Aveek Das

Getting started with the Azure Queue Storage using Python

September 14, 2021 by

In this article, we are going to talk about the Azure Queue Storage service using python as a programming language. As it goes by the name, Azure Queue Storage is a simple queue and messaging service in Azure that can be used to build queues and process messages in an asynchronous fashion. Queues help to decouple multiple systems and provide the middle layer in a distributed architecture. The Queue Storage service in Azure allows users to connect to and store messages up to 64 KB in size from anywhere in the world. It is exposed by using HTTP or HTTPS endpoints and can contain millions of messages that need to be processed.

In this article, we are going to use python to programmatically create and delete Azure Queue Storage in a step-by-step manner. We will also have a look at the four important scenarios that include the following.

  • Adding a message to the queue
  • Peeking a message from the queue
  • Getting a message from the queue
  • Deleting messages from the queue

Pre-Requisites

In order to proceed with this article, I recommend that you must have the following pre-requisites.

  • Python 3 – Since this article is focussed mainly on accessing the Azure Storage Queue elements programmatically, you must have Python 3 installed on your machine
  • A valid Azure Account – Also, before proceeding forward, please make sure that you have access to a valid Azure Account where you have sufficient permission to create and delete Queue Storage

Azure Queue Storage components

Let us first start by understanding the important components of the Queue Storage service in Azure.

Queue Storage Service Components

Figure 1 – Azure Queue Storage Service Components

As you can see in the figure above, there are the following components to the Queue Storage account in Azure.

  • Storage Account – In order to build any queue in Azure, it is necessary to have a storage account created. In that storage account, we can create multiple queues as per the requirements
  • Queue – This is the actual queue that is going to contain the messages. We can connect to this queue and send messages. There can be multiple queues created under a storage account
  • Message – Once the queues are in place, we can start sending messages to the queue. This can be done either programmatically or by using the Azure Portal

In the figure above, we have a storage account with the name “my-storage” and two queues under it, namely “orders” and “notification“. These queues are defined to logically handle how messages should be handled and what services do they offer.

Additionally, in order to connect to a queue in Azure, we need to know the URL for that specific queue. It is usually of the following format.

http://<STORAGE_ACCOUNT>.queue.core.windows.net/<QUEUE_NAME>

If we have the “orders” queue in the “my-storage” account, then the URL would be as follows:

Creating a Storage Account in Azure

In order to get started, we should create a storage account in Azure under which we can create our queues. The easiest way to do this is to use the Azure portal. Alternatively, you can also use the Azure CLI or PowerShell to create the Storage Account in Azure. Navigate to the Azure Portal and search for Storage Accounts. Click on Create and then Create a storage account blade will appear on the right.

Creating the Storage Account

Figure 2 – Creating the Storage Account

Once you have provided the basic details, you can click on Review + Create to create the storage account with the default settings. However, you are free to choose other settings by navigating through the other tabs during the configuration of the Storage Account. The deployment might take some time but once it is done, you can see it on the portal and confirm that there are no queues in the storage account so far.

Storage Account created in Azure

Figure 3 – Storage Account created in Azure

Once the Storage Account is created, we need to obtain the Access Keys in order to connect to the Storage account programmatically using python. Navigate to Access Keys under Security + Networking and click on Show Keys. Copy the Connection string and store it for later use.

Fetching the Access Keys for the Storage Account

Figure 4 – Fetching the Access Keys for the Storage Account

Installing the Azure Storage SDK for Python

The next step after creating the storage account in Azure is that we would need to install the python SDK for Azure Storage on our machine. This can be done by downloading and installing the package from the regular Python Package Manager. Head over to your terminal and write the following command:

Downloading and Installing Azure Queue Storage SDK from the PyPi

Figure 5 – Downloading and Installing Azure Queue Storage SDK from the PyPi

Creating the program in python to connect to the Azure Queue Storage

Now that we have all the necessary requirements set up, we are good to create our program in Python. For the purpose of this article, I am going to use a Jupyter Notebook in Visual Studio Code to explain the steps as follows.

Setting up the connecting string

Assign the connection string to a variable to be used later.

Connection String

Figure 6 – Connection String

Import the necessary libraries to the program

We need to import the Azure Queue Storage SDK to our program in order to leverage it to connect to the queue storage in Azure.

Importing the SDK to the program

Figure 7 – Importing the SDK to the program

Create the Queue Client and the Queue

Provide a name for the queue and create the queue client using the SDK and the connecting string. Using the queue client, you can now create the queue.

Creating the queue in Azure Queue Storage

Figure 8 – Creating the queue in Azure Queue Storage

As soon as the queue is created, you will see the response in JSON format. Also, you can verify the queue from the portal as well.

Queue has been created in the portal

Figure 9 – Queue has been created in the portal

Sending messages to the queue

Once the queue is created, you can start sending messages to the queue. This can be simply done by using the send_message() method of the queue client.

Sending a message to the queue in Azure Storage

Figure 10 – Sending a message to the queue in Azure Storage

Once the message has been sent, you will receive the JSON response that states the result of the operation.

Peeking messages in the queue

Peeking messages in the queue helps us to view the messages in the queue without removing them. This is helpful when you need to gauge what the messages are before processing them from the queue. It can be done with the help of the peek_messages() function.

Peeking messages from the Azure Storage Queue

Figure 11 – Peeking messages from the Azure Storage Queue

Processing and Removing messages from the queue

Now that we know how to peek at messages from the queue, let us also learn how to read and process messages from the queue. Also, it is important to remove the message from the queue after it has been processed so that it is not processed again.

Processing and removing messages from the queue

Figure 12 – Processing and removing messages from the queue

By default, when the receive_messages() function is called, the message becomes invisible to other programs reading from the same queue to prevent unnecessary processing. The invisibility time is set to 30 seconds after which it becomes visible again.

Deleting the queue from the Storage Account

You can delete the queue from the Storage Account by using the delete_queue() function.

Deleting the queue

Figure 13 – Deleting the queue

The entire code for the article can be found here.

Conclusion

In this article, we have understood in detail Azure Queue Storage and how to create and delete queues programmatically using python. We have also learned how to add messages to the queue, peek and get messages and finally deleting messages off the queue. These are the four different services that users can use to interact with the Azure Storage Queue service. Instead of python, you are also free to choose any other runtime of your choice.

Aveek Das
Azure, Python

About Aveek Das

Aveek is an experienced Data and Analytics Engineer, currently working in Dublin, Ireland. His main areas of technical interest include SQL Server, SSIS/ETL, SSAS, Python, Big Data tools like Apache Spark, Kafka, and cloud technologies such as AWS/Amazon and Azure. He is a prolific author, with over 100 articles published on various technical blogs, including his own blog, and a frequent contributor to different technical forums. In his leisure time, he enjoys amateur photography mostly street imagery and still life. Some glimpses of his work can be found on Instagram. You can also find him on LinkedIn View all posts by Aveek Das

168 Views