Skip to content

Comparison to Celery

There are some things to be aware of when using Mognet, if you're coming from Celery.

Result Persistence

As of the time of writing this document (June 2021), in Celery, if you submit a task Signature for execution, its corresponding AsyncResult will only exist after the task started, and attempting to retrieve a non-existing AsyncResult will instead result in a default instance being returned with PENDING status.

Mognet, on the other hand, will create a Result instance on the Result Backend at the time the Request gets sent to the Task Broker. This ensures that, unless the message gets lost in the Task Broker, a PENDING state really does mean that.

No multi-worker concurrency

Mognet does not use a multi-worker architecture, be it via Threads, Processes, Greenlets, Eventlets, etc. Instead, it leverages only asyncio to run itself and the tasks you define.

By default, it is configured to only run one task at once. You can override this by setting minimum_concurrency on AppConfig to a value greater than 1.

Bear in mind that the higher the value, the more resources you will need on your worker. This value will vary greatly depending on the nature of your tasks. For example, if your tasks are primarily CPU-bound, it is best to launch two separate mognet ... run processes instead.

To run multiple workers in production, we recommend:

  • Using a tool like Supervisor, http://supervisord.org/, to launch multiple processes (and much more)
  • Clustering software like Kubernetes, which also gives you support for scaling horizontally across multiple machines (depending on your cluster)

No early acknowledgements

For durability purposes, task messages received from the Broker are only acknowledged after the task finishes (either successfully or with failure).

This means that, should the mognet ... run process crash (e.g., a SIGKILL), the task can still be picked up by a different worker and retried. Note that this isn't done forever; if a task is retried too many times (default is 3, as set in max_retries on AppConfig), the task is marked as FAILURE and discarded.

This, however, has the drawback where long-running tasks can run into issues with RabbitMQ-based task brokers, see RabbitMQ Disconnects. One alternative to this is using Stateful Tasks.