Python – Asynchronous Task using RabbitMQ
Last Updated :
26 May, 2020
Have you ever faced an issue where you need to perform a task in background that will take up a lot of time to complete? Have you ever wanted to perform a task after a certain interval of time? If your answer was “YES” for any of the above questions, then Python has got you covered. We will be demonstrating how to perform Asynchronous tasks using RabbitMQ.
What exactly Asynchronous means ?
Asynchronous means controlling the timing of operations to be performed by the use of signals sent when the previous operation is completed rather than at regular intervals.
What RabbitMQ is ?
RabbitMQ is a message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP) and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT), and other protocols. If that sounded complicated to you then don’t worry, we have got you covered. So, In simple terms, it provides a Queue which performs tasks in the background by running a server of its own.
With all that cleared up, let’s get started with the installation of RabbitMQ and other necessary tools. So, we will be using a python package called Celery for connecting with RabbitMQ. Celery provides an easy way of connecting and sending tasks to the Queue (RabbitMQ). More technically speaking, Celery is a Python Task-Queue system that handle distribution of tasks on workers across threads or network nodes.
You should have python installed on your system. Then you need to install Celery on your system. To do so, simply type the following
pip install celery==4.4.2
Next, install RabbitMQ on your machine. To do so, head over to their official page and download the installer depending on your operating system. You might need o download ErLang along with RabbitMQ. Once installed type the following in your terminal
rabbitmq-server restart
You should get an output like this
Do not close this terminal. Now your broker should be running. To check that, go to http://localhost:15672/ in your browser and enter the username and password. The default username is guest and the default password is also guest. You should see something like this
Now, let’s get started with the fun part a.k.a. coding.
We will be downloading a YouTube video to our system. We will be using pytube module for downloading it. To install pytube type the following
pip install pytube3
Downloading YouTube video can take up a lot of time. We will now see, how our python file gets executed instantly and the video keeps on downloading in background.
First, we will create a python file called task_queue.py
from celery import Celery
import sys
from pytube import YouTube
BASEDIR ="D:\\"
app = Celery('downloader',
@app.task
def download(url, filename):
try:
yt = YouTube(url)
except:
print("Connection Error")
yt.streams\
.filter(progressive = True, file_extension ='mp4')\
.order_by('resolution')[-1]\
.download(output_path = BASEDIR, filename = filename)
print('Task Completed !')
|
We need another python file to run this code (you can do it in the same file but it’s a good practice to call the function from another file). Create another file with the name runtask.py
import sys
from task_queue import download
link = sys.argv[1]
filename = sys.argv[2]
download.delay(link, filename)
|
Now, start Celery. To do so, open up a terminal in your working directory and type the following
celery -A task_queue worker --pool=solo -E
If you get any error in this, then either your RabbitMQ server is not running or you are not in your working directory. You should get an output like this
Now we have Celery running. now open up another terminal in the same directory and run the following command
$ python runtask.py https://www.youtube.com/watch?v=vG2PNdI8axo Geeksforgeeks
After you complete the above command, wait for few seconds to allow the download to finish in background. Then go to your D: drive and there you would see a file named Geeksforgeeks.mp4. So, that’s it. Your video is downloaded.
Asynchronous tasks are very essential in real-life cases. You can not have your software or website stuck at a loading page while your code performs tasks. Tasks which take longer time to execute are performed in background without halting the main thread.
Similar Reads
Asynchronous HTTP Requests with Python
Performing multiple HTTP requests is needed while working with applications related to data preprocessing and web development. In the case of dealing with a large number of requests, using synchronous requests cannot be so efficient because each request must wait for the previous request to get comp
4 min read
Python Taskgroups with asyncIO
In this article, we will see the use of Task Groups with asyncIO in Python. What is a Task group? Task groups are introduced in Python 3.11 along with Exception Groups and as the name suggests it groups similar or different tasks together. It is already possible using the gather() method of asyncio
6 min read
Integrating RabbitMQ with Python
In this article, we will guide you through setting up and using RabbitMQ with Python. RabbitMQ is a powerful message broker that allows applications to communicate with each other via messages. This practical guide will show you how to connect to RabbitMQ, publish messages to a queue, and consume me
5 min read
Simple Chat Room using Python
This article demonstrates - How to set up a simple Chat Room server and allow multiple clients to connect to it using a client-side script. The code uses the concept of sockets and threading. Socket programming Sockets can be thought of as endpoints in a communication channel that is bi-directional
8 min read
Asyncio Vs Threading In Python
In Python, both Asyncio and Threading are used to achieve concurrent execution. However, they have different mechanisms and use cases. This article provides an in-depth comparison between Asyncio and Threading, explaining their concepts, key differences, and practical applications. Table of Content
6 min read
Flask Asynchronous Programming Using async.io
Flask is inherently synchronous, meaning each request is processed one at a time. However, modern web applications often require handling multiple tasks simultaneously, such as: Making external API callsProcessing large datasets Managing real-time communicationAsynchronous programming allows us to e
5 min read
Dominos Chatbot using Python
Chatbots are gaining popularity as a means for businesses to interact with their customers. Domino's Pizza is one such company that has used a chatbot to improve its customer service. In this article, we'll look at how to use Python to create a chatbot for Domino's Pizza. Tools and Technologies Used
11 min read
Python Falcon - Waitress
In this article, we will explore Python Falcon-Waitress, a powerful combination for building web APIs with Falcon, and we'll illustrate its usage with practical examples. Falcon-Waitress offers seamless integration of the Falcon framework with the Waitress production-quality WSGI server, enabling yo
7 min read
How to use Thread in Tkinter Python
Prerequisite: Python GUI – tkintermultithreading Python offers multiple options for developing GUI (Graphical User Interface). Out of all the GUI methods, tkinter is the most commonly used method. It is a standard Python interface to the Tk GUI toolkit shipped with Python. Python with tkinter is the
2 min read
Send message to Telegram user using Python
Have you ever wondered how people do automation on Telegram? You may know that Telegram has a big user base and so it is one of the preferred social media to read people. What good thing about Telegram is that it provides a bunch of API's methods, unlike Whatsapp which restricts such things. So in t
3 min read