Skip to main content

Azure CLI List blobs pagination

Azure CLI: list blobs with pagination

See az storage blob list for more information.

The context

I have an Azure storage account that has several containers and some containers have much blobs in

All the blobs have been individually archived in "Archive" access tier

I want to list all the blobs in the storage account and change the access tier of the blobs to "Hot" tier

However, the command "az storage blob list" only returns 5000 blobs at most, and I have 65k blobs

I need then to use the pagination feature of the command to list all the blobs

The problem

When I use the command "az storage blob list":

  • It returns 5000 blobs at most
  • It returns a continuation token, but in standard error output
  • I need to store the standard output (the blob list) and the standard error output (the continuation token) in different places
  • Because I need to give the continuation token to the command to list the next 5000 blobs in a loop until the continuation token is not outputed anymore (that means we reached the end of the listing)
  • There is no way to just get the continuation token without the blob list

I know how to store the standard output somewhere, how to store the standard error output somewhere else, but here I need to store both in different places.

The solution

Here is the solution I found (click here to see the code):

  • Store the standard output in a file
  • Store the standard error output in a variable
  • Use the variable to give the continuation token to the command to list the next 5000 blobs
  • Loop until the continuation token is not outputed anymore

Popular posts from this blog

npm run build base-href

Using NPM to specify base-href When building an Angular application, people usually use "ng" and pass arguments to that invocation. Typically, when wanting to hard code "base-href" in "index.html", one will issue: ng build --base-href='https://ngx.rktmb.org/foo' I used to build my angular apps through Bamboo or Jenkins and they have a "npm" plugin. I got the habit to build the application with "npm run build" before deploying it. But the development team once asked me to set the "--base-href='https://ngx.rktmb.org/foo'" parameter. npm run build --base-href='https://ngx.rktmb.org/foo did not set the base href in indext.html After looking for a while, I found https://github.com/angular/angular-cli/issues/13560 where it says: You need to use −− to pass arguments to npm scripts. This did the job! The command to issue is then: npm run build -- --base-href='https://ngx.rktmb.org/foo&
Read more

Jenkins invalid privatekey

Publish over SSH, Message "invalid privatekey:" With quite recent (June-July 2020) installations of Jenkins and OpenSSH, I have the following error message when using the "Deploy overs SSH" Jenkins plug-in and publishing artifacts to the target overs SSH: jenkins.plugins.publish_over.BapPublisherException: Failed to add SSH key. Message [invalid privatekey: [B@d8d395a] This problem seems to be referenced here: https://issues.jenkins-ci.org/browse/JENKINS-57495 Just regenerate a key with the right parameters To solve it: ssh-keygen -t rsa -b 4096 Or ssh-keygen -t rsa -b 4096 -m PEM
Read more

AzureCLI Custom Python

Installing Azure CLI on Archlinux When trying to install Azure CLI on Archlinux, I follow the documentation, in the "script" tab , and it leads to the following errors: [mihamina@arch-00 ~]$ curl -L https://aka.ms/InstallAzureCli | bash [...] Running install script. -- Verifying Python version. -- Python version 3.11.3 okay. [...] -- Executing: ['/usr/bin/python3', 'virtualenv.py', '--python', '/usr/bin/python3', '/home/mihamina/lib/azure-cli'] /tmp/tmpn0w4l6w9/virtualenv-16.7.11/virtualenv.py:24: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives import distutils.spawn /tmp/tmpn0w4l6w9/virtualenv-16.7.11/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead import distutils.sysconfig Already using interpreter /u
Read more