Because of the busy of work, I doesn't got much time on this time, and one major reason is now I am programming with Java... So I won't push any new release from now on, if someone is interest with it, just fork and maintenance your own version, Sorry for the inconvenient.
It's modified from celerybeat-mongo (https://github.com/zakird/celerybeat-mongo)
See Changelog in CHANGES.md
This is a Celery Beat Scheduler (http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html) that stores both the schedules themselves and their status information in a backend Redis database.
- Full-featured celery-beat scheduler
- Dynamically add/remove/modify tasks
- Support multiple instance by Active-Standby model
It can be installed by installing the celerybeat-redis Python egg:
# pip install celerybeat-redis
And specifying the scheduler when running Celery Beat, e.g.
$ celery beat -S celerybeatredis.schedulers.RedisScheduler
Settings for the scheduler are defined in your celery configuration file similar to how other aspects of Celery are configured
CELERY_REDIS_SCHEDULER_URL = "redis://localhost:6379/1"
CELERY_REDIS_SCHEDULER_KEY_PREFIX = 'tasks:meta:'
You must make sure these two values are configured. CELERY_REDIS_SCHEDULER_URL
is used to store tasks. CELERY_REDIS_SCHEDULER_KEY_PREFIX
is used to generate
keys in redis. The keys will be of the form
tasks:meta:task-name-here
tasks:meta:test-fib-every-3s
There is also an optional setting
CELERY_REDIS_SCHEDULER_LOCK_TTL = 30
This value determines how long the redis scheduler will hold on to it's lock, which prevents multiple beat instances from running at the same time. However, in some cases -- such as a hard crash -- celery beat will not be able to clean up after itself and release the lock. Therefore, the lock is given a configurable time-to-live, after which it will expire and be released, if it is not renewed by the beat instance that acquired it. If said beat instance does die, another instance will be able to pick up the baton, as it were, and run instead.
After installed and configure the needs by above, you can make a try with test, cd to test directory, start a worker by:
$ celery worker -A tasks -l info
then start the beat by:
$ celery beat -S celerybeatredis.schedulers.RedisScheduler
celerybeat-redis will load the entry from celeryconfig.py
There was two ways to add a period task:
Celery provide a CELERYBEAT_SCHEDULE
entry in config file, when
celerybeat-redis starts with such a config, it will load tasks to redis, create
them as a celerybeat-redis task.
It's perfect for quick test
You can create task by insert specify data to redis, according to following described:
Schedules can be manipulated in the Redis database through direct database manipulation. There exist two types of schedules, interval and crontab.
{
"name" : "interval test schedule",
"task" : "task-name-goes-here",
"enabled" : true,
"interval" : {
"every" : 5,
"period" : "minutes"
},
"args" : [
"param1",
"param2"
],
"kwargs" : {
"max_targets" : 100
},
"total_run_count" : 5,
"last_run_at" : {
"__type__": "datetime",
"year": 2014,
"month": 8,
"day": 30,
"hour": 8,
"minute": 10,
"second": 6,
"microsecond": 667
}
}
The example from Celery User Guide::Periodic Tasks.
CELERYBEAT_SCHEDULE = {
'interval-test-schedule': {
'task': 'tasks.add',
'schedule': timedelta(seconds=30),
'args': (param1, param2)
}
}
Becomes the following::
{
"name" : "interval test schedule",
"task" : "task.add",
"enabled" : true,
"interval" : {
"every" : 30,
"period" : "seconds",
},
"args" : [
"param1",
"param2"
],
"kwargs" : {
"max_targets" : 100
},
"total_run_count": 5,
"last_run_at" : {
"__type__": "datetime",
"year": 2014,
"month": 8,
"day": 30,
"hour": 8,
"minute": 10,
"second": 6,
"microsecond": 667
}
}
The following fields are required: name, task, crontab || interval,
enabled when defining new tasks.
total_run_count
and last_run_at
are maintained by the
scheduler and should not be externally manipulated.
The example from Celery User Guide::Periodic Tasks. (see: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#crontab-schedules)
CELERYBEAT_SCHEDULE = {
# Executes every Monday morning at 7:30 A.M
'add-every-monday-morning': {
'task': 'tasks.add',
'schedule': crontab(hour=7, minute=30, day_of_week=1),
'args': (16, 16),
},
}
Becomes:
{
"name" : "add-every-monday-morning",
"task" : "tasks.add",
"enabled" : true,
"crontab" : {
"minute" : "30",
"hour" : "7",
"day_of_week" : "1",
"day_of_month" : "*",
"month_of_year" : "*"
},
"args" : [
"16",
"16"
],
"kwargs" : {},
"total_run_count" : 1,
"last_run_at" : {
"__type__": "datetime",
"year": 2014,
"month": 8,
"day": 30,
"hour": 8,
"minute": 10,
"second": 6,
"microsecond": 667
}
}
Original celery beat doesn't support multiple node deployment, multiple beat will send multiple tasks and make worker duplicate execution, celerybeat-redis use a redis lock to deal with it. Only one node running at a time, other nodes keep tick with minimal task interval, if this node down, when other node ticking, it will acquire the lock and continue to run.
WARNING: this is an experiment feature, need more test, not production ready at this time.