It's modified from celerybeat-mongo (https://github.com/zakird/celerybeat-mongo)
See Changelog in CHANGES.md
This is a Celery Beat Scheduler (http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html) that stores both the schedules themselves and their status information in a backend Redis database.
- Full-featured celery-beat scheduler
- Dynamically add/remove/modify tasks
- Support multiple instance by Active-Standby model ( not tested)
Set your python path with the source or you can just install in editable mode using
pip install -e .
and specify the scheduler when running Celery Beat, e.g.
$ celery beat -S celerybeatredis.schedulers.RedisScheduler
Settings for the scheduler are defined in your celery configuration file similar to how other aspects of Celery are configured
CELERY_REDIS_SCHEDULER_URL = "redis://localhost:6379/1"
CELERY_REDIS_SCHEDULER_KEY_PREFIX = 'tasks:meta:'
You must make sure these two values are configured. CELERY_REDIS_SCHEDULER_URL
is used to store tasks. CELERY_REDIS_SCHEDULER_KEY_PREFIX
is used to generate
keys in redis. The keys will be of the form
tasks:meta:task-name-here
tasks:meta:test-fib-every-3s
There is also an optional setting
CELERY_REDIS_SCHEDULER_LOCK_TTL = 30
This value determines how long the redis scheduler will hold on to it's lock, which prevents multiple beat instances from running at the same time. However, in some cases -- such as a hard crash -- celery beat will not be able to clean up after itself and release the lock. Therefore, the lock is given a configurable time-to-live, after which it will expire and be released, if it is not renewed by the beat instance that acquired it. If said beat instance does die, another instance will be able to pick up the baton, as it were, and run instead.
After installed and configure the needs by above, you can make a try with test, cd to test directory, start a worker by:
$ celery worker -A tasks -l info --autoscale=10,3
then start the beat by:
$ celery beat -S celerybeatredis.schedulers.RedisScheduler
celerybeat-redis will load the entry from celeryconfig.py
There was two ways to add a period task:
Celery provide a CELERYBEAT_SCHEDULE
entry in config file, when
celerybeat-redis starts with such a config, it will load tasks to redis, create
them as a celerybeat-redis task.
It's perfect for quick test
Schedules can be manipulated in the Redis database from any redis-cli. There exist two types of schedules, interval and crontab.
The example from Celery User Guide::Periodic Tasks.
CELERYBEAT_SCHEDULE = {
'interval-test-schedule': {
'task': 'tasks.add',
'schedule': timedelta(seconds=30),
'args': (param1, param2)
}
}
Becomes the following::
{
"name" : "interval test schedule",
"task" : "task.add",
"enabled" : true,
"schedule" : {
"every" : 30,
"period" : "seconds",
},
"args" : [
"param1",
"param2"
]
}
The following command should add an interval task in redis. Please note that the key has to have prefix 'tasks:meta:'
set tasks:meta:multiply-every-10-minutes "{\"name\":\"multiply-every-10-minutes\",\"task\":\"tasks.multiply\",\"enabled\":true,\"schedule\": { \"period\": \"minutes\", \"every\": 10 },\"args\":[3,2] }"
The following fields are required: name, task, crontab || interval,
enabled when defining new tasks.
total_run_count
and last_run_at
are maintained by the
scheduler and should not be externally manipulated.
The example from Celery User Guide::Periodic Tasks. (see: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules)
Similarly a crontab schedule like this -
CELERYBEAT_SCHEDULE = {
# Executes every Monday morning at 7:30 A.M
'add-every-monday-morning': {
'task': 'tasks.add',
'schedule': crontab(hour=7, minute=30, day_of_week=1),
'args': (16, 16),
},
}
Becomes:
{
"name" : "add-every-monday-morning",
"task" : "tasks.add",
"enabled" : true,
"schedule" : {
"minute" : "30",
"hour" : "7",
"day_of_week" : "1",
"day_of_month" : "*",
"month_of_year" : "*"
},
"args" : [
16,
16
]
}
The following command will add cron tasks like the one above -
set tasks:meta:multiply-every-20-minutes "{\"name\":\"multiply-every-20-minutes\",\"task\":\"tasks.multiply\",\"enabled\":true,\"schedule\": { \"minute\": \"*/20\", \"hour\":\"*\", \"day_of_week\":\"*\", \"day_of_month\":\"*\", \"month_of_year\":\"*\" },\"args\":[13,13] }"
You can add one time tasks on the fly as usual from the code like the following -
>>> from tasks import multiply
>>> result = multiply.delay(12312,123)
>>> result.state
'PENDING'
>>> result.state
'SUCCESS'
>>> result.get()
1514376
Original celery beat doesn't support multiple node deployment, multiple beat will send multiple tasks and make worker duplicate execution, celerybeat-redis use a redis lock to deal with it. Only one node running at a time, other nodes keep tick with minimal task interval, if this node down, when other node ticking, it will acquire the lock and continue to run.
WARNING: this is an experiment feature, need more test, not production ready at this time.