celery

  1. # 启动 celery
  2. $ celery -A celery_tasks  worker --loglevel=info
  4. # 后台启动
  5. nohup celery -A celery_tasks worker > /var/log/celery_worker.log 2>&1 &
  6. nohup celery -A celery_tasks beat > /var/log/celery_beat.log 2>&1 &
  8. # 启动 celery , flower 任务面板
  9. $ celery -A celery_tasks  flower --address=0.0.0.0 --port=5555
  11. 测试环境可以使用一个进程：
  12. celery -A celery_tasks worker -B

启动详情

  1. $ celery -A celery_tasks  worker --loglevel=info
  2. /home/hwmg/beancluster.cfg
  3. /home/hwmg
  4. /home/hwmg
  5. ['hwfirewalld', 'sh/hw_firewalld.sh', '172.28.78.201']
  6. ['h3cRouter', 'sh/h3cRouter.sh', '172.28.78.202']
  7. ['shanshi', 'sh/shanshi.sh', '172.28.78.203']
  8. ['CiscoFirewalld', 'sh/cisco_firewalld.sh', '172.28.78.206']
  9. ['CiscoRouter', 'sh/CiscoTranfer.sh', '172.28.78.205']
  10. ['iKuaiOS', 'sh/ikuai.sh', '172.28.78.204']
  11. ['JuniperFirewalld', 'sh/Juniper.sh', '172.28.78.207']
  12. ['MikrotikRouter', 'sh/Mikrotik.sh', '172.28.78.208']
  13. /home/hwmg
  14. /usr/local/lib/python3.6/site-packages/celery/platforms.py:835: SecurityWarning: You're running the worker with superuser privileges: this is
  15. absolutely not recommended!
  17. Please specify a different user using the --uid option.
  19. User information: uid=0 euid=0 gid=0 egid=0
  21.   uid=uid, euid=euid, gid=gid, egid=egid,
  23.  -------------- celery@antiy246 v5.1.2 (sun-harmonics)
  24. --- ***** -----
  25. -- ******* ---- Linux-3.10.0-862.el7.x86_64-x86_64-with-centos-7.5.1804-Core 2022-01-14 14:49:35
  26. - *** --- * ---
  27. - ** ---------- [config]
  28. - ** ---------- .> app:         celery:0x7fa51d466a90
  29. - ** ---------- .> transport:   redis://10.42.0.107:6379/9
  30. - ** ---------- .> results:     redis://10.42.0.107:6379/9
  31. - *** --- * --- .> concurrency: 16 (prefork)
  32. -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
  33. --- ***** -----
  34.  -------------- [queues]
  35.                 .> celery           exchange=celery(direct) key=celery
  38. [tasks]
  39.   . celery_tasks.cronbjob.counter.Monitor
  40.   . celery_tasks.cronbjob.detect.detectAliveIP
  41.   . celery_tasks.cronbjob.detect.portDetect
  42.   . celery_tasks.cronbjob.replay.replayMP4
  43.   . celery_tasks.go_tasks.CreateQemu
  44.   . celery_tasks.go_tasks.DeleteQemu
  45.   . celery_tasks.go_tasks.RestoreQemu
  46.   . celery_tasks.go_tasks.StartQemu
  47.   . celery_tasks.go_tasks.StopQemu
  48.   . celery_tasks.go_tasks.add
  49.   . celery_tasks.go_tasks.sum
  50.   . celery_tasks.py_tasks.add
  51.   . celery_tasks.py_tasks.assets.batchCreateAssets
  52.   . celery_tasks.py_tasks.assets.createAssets
  53.   . celery_tasks.py_tasks.assets.releaseAssets
  54.   . celery_tasks.py_tasks.iptables.createIPtables
  55.   . celery_tasks.py_tasks.restore.restoreManual
  56.   . celery_tasks.py_tasks.servicesCURD.createInstance
  57.   . celery_tasks.py_tasks.servicesCURD.deleteInstances
  58.   . celery_tasks.py_tasks.servicesCURD.restartInstances
  59.   . celery_tasks.py_tasks.servicesCURD.startStopInstances
  60.   . celery_tasks.py_tasks.sftp.addSFTPConf
  61.   . celery_tasks.py_tasks.sftp.delSFTPConf
  62.   . celery_tasks.py_tasks.sum
  64. [2022-01-14 14:49:37,779: INFO/MainProcess] Connected to redis://10.42.0.107:6379/9
  65. [2022-01-14 14:49:37,802: INFO/MainProcess] mingle: searching for neighbors
  66. [2022-01-14 14:49:38,842: INFO/MainProcess] mingle: all alone
  67. [2022-01-14 14:49:38,905: INFO/MainProcess] celery@antiy246 ready.

APScheduler

advanceded python scheduler 实现了 Quartz 的所有功能的一个python定时任务框架。提供基于日期、固定时间间隔以及crontab类型的任务，并且可以持久化任务。

特点

  • 类型 linux cron 的调度（可选的开始/结束时间 ）
  • 基于时间间隔的执行调度（周期性调度，可选的开始/结束时间）
  • 一次性执行任务（在设定的日期/时间运行一次任务）

基本概念与执行逻辑

image.png

示例

  1. from apscheduler.schedulers.blocking import BlockingScheduler
  2. from datetime import datetime
  4. # 输出时间
  5. def job():
  6.     print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
  8.     # BlockingScheduler
  9. sched = BlockingScheduler()
  10. sched.add_job(my_job, 'interval', seconds=5, id='my_job_id')
  11. sched.start()

flask

基于 flask 集成时任务队列时，一个严重的问题是（运行在 uwsgi 上的多进程 work 问题）
https://blog.miguelgrinberg.com/post/run-your-flask-regularly-scheduled-jobs-with-cron

参考

https://www.biaodianfu.com/python-schedule.html