top of page

How to Improve Efficiency with Asynchronicity Using Celery and FastAPI?


Cartoon-style illustration of gear wheels resembling a computer CPU, with 'FastAPI' and 'Celery' at the center, connected by metal pipes.

When it comes to developing backend systems for our projects, we often default to synchronous architectures. Synchronous backends, such as simple REST APIs, are easy to implement, and Python offers a plethora of frameworks that make this process a breeze. However, these architectures have their limitations and can present challenges when dealing with long-running tasks or delivering a responsive user experience. This blog post will explore asynchronous backends and how to implement them using Celery and FastAPI.


What are the Limitations of Having Synchronous Backends?

Synchronous backends have their advantages, but they also come with notable downsides:

  • Synchronous backends struggle to manage long-running tasks without additional logic, often tacked on as an afterthought.

  • Long-running tasks can block the main thread in popular languages like Python and JavaScript, leading to rejected incoming requests and a degraded user experience.

  • Reporting progress of a long-running process to the caller can be challenging with synchronous architectures.

  • When using services like AWS API Gateway, you can exceed the maximum timeout, leading to failed requests as the server terminates the connection.

How are Asynchronous APIs used to improve Backend Efficiency?

Asynchronous APIs provide an elegant solution to these problems. The primary goal of an asynchronous architecture is to deliver quick responses to users while efficiently handling time-consuming tasks. However, like any architectural choice, asynchronous APIs have their own set of challenges:

  • Increased Complexity: Asynchronous backends require more moving parts to create a reliable and scalable system.

  • Debugging Complexity: Debugging asynchronous code can be more challenging and may require additional configuration.

  • Deployment Complexity: Deploying asynchronous architectures can be trickier due to the increased number of architectural components.

Meet Celery

Celery is a powerful tool for building asynchronous backends. It is an open-source asynchronous task queue that relies on distributed message passing. While it supports task scheduling, its primary focus is on real-time operations. Celery provides several essential components for building asynchronous backends:


1. Message Broker

A core part of the asynchronous architecture, the message broker handles async messaging and message queuing. While you could use various solutions like plain text files or databases, dedicated message brokers offer better reliability and speed.


2. Results Backend

To store the results of long-running tasks, you need a results backend. Since you may need to access these results at a later time, a dedicated storage solution is essential.


3. Worker Node

Worker nodes are responsible for executing tasks and providing results. They act as the backbone of your asynchronous system, handling the actual workload.


4. Web Node

Web nodes serve as the frontend to your backend system. They expose an API for requesting tasks and retrieving results once they are ready.


Flowchart depicting a process with distinct paths originating from two different sources, 'sync' and 'async', illustrating their different processes of response to the same thing.
The flowchart illustrates the differences between 'synchronous' and 'asynchronous' architectures, showcasing the faster response of the 'asynchronous' path.

How to optimize web nodes using FastAPI and asynchronous techniques?

FastAPI is a high-performance Python framework that enables the creation of asynchronous APIs. According to its documentation, FastAPI's performance rivals that of Node.js and Go, making it an excellent choice for building reliable and fast web nodes for your asynchronous backend.


Main Workflow Differences: Synchronous vs. Asynchronous

Let's highlight the main workflow differences between synchronous and asynchronous APIs to understand the advantages of the latter better:


Synchronous Workflow:

  • A client sends a request to the server.

  • The server processes the request synchronously, potentially blocking the main thread for long-running tasks.

  • The server sends a response back to the client when the task is complete.

  • During this time, the allocated thread working on that particular request can't handle other incoming requests.

Asynchronous Workflow:

  • A client sends a request to the server.

  • The server acknowledges the request and starts processing it asynchronously.

  • The server immediately responds to the client, indicating that the task is in progress.

  • The server continues processing the task in the background.

  • Once the task is complete, the server notifies the client or the client can poll for the result.

  • During this time, the server can efficiently handle other incoming requests.


Conclusion

In the world of backend architecture, it's essential to consider the specific needs of your project. While synchronous backends are simple to implement, they can fall short when handling long-running tasks and delivering a responsive user experience. Asynchronous architectures, powered by tools like Celery and FastAPI, offer a compelling solution to these challenges.

By embracing asynchronous APIs, you can ensure that your users enjoy a snappy experience while your backend efficiently manages complex and time-consuming tasks. While there may be added complexity in terms of setup and debugging, the benefits in terms of performance and scalability make the effort worthwhile. As you embark on your backend development journey, consider asynchronous architectures as a powerful tool to be used in your future projects.



Check out this repo to see an asynchronous backend example using Celery and FastAPI: https://github.com/Wason1797/FastAPI-Celery-example


1 Comment


Esteban Molina
Esteban Molina
Oct 26, 2023

It really looks like a very good solution for long-running tasks in the backend, great post!

Like
bottom of page