Long running tasks is cancelled - Celery5.2.6

I have a project hosted in Digital Ocean in a Basic Droplet with 2 GB Ram. In my local machine, the long-running task runs between 8-10 minutes and is still successful.However in Digital Ocean droplets, often times the celery will not succeed in the long-running task.

Current celery - celery 5.2.6 I have two configurations in supervisor

  1. Running the celery worker celery -A myproject worker -l info
  2. Running the celery beat celery -A myproject beat -l info

This is the message from celeryd.log

CPendingDeprecationWarning: In Celery 5.1 we introduced an optional breaking change which on connection, loss cancels all currently executed tasks with late acknowledgment enabled. These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2022-07-07 04:25:36,998: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379//: Error 111 connecting to localhost:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)

[2022-07-07 04:25:39,066: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379//: Error 111 connecting to localhost:6379. Connection refused..
Trying again in 4.00 seconds... (2/100)

For a temporary solution, what I did is to restart the server, and re-run new tasks, but this will not guarantee that the long-running task will be successful, the problem with this is the previously failed task will not restart.

My goal is,

  1. Prevent long-running tasks from being canceled
  2. If the long-running task is already canceled and cancellation can't be avoided, I need it to rerun and continue instead of starting a new task.

Is this possible? Any ideas on how?



Comments

Popular posts from this blog

Spring Elasticsearch Operations

Network Error and Timeout on Authorize.net JS

Object oriented programming concepts (OOPs)