Scheduler

单机scheduler实现，包含客户端和服务器

Travis-CI
GoDoc
codecov
Report card

https://github.com/songxinjianqwe/scheduler

初衷

1、毕设计划实现一个简化版的Docker容器
2、目前需要熟悉Go语言
3、学习Docker的架构

  • Docker分为Docker Daemon（HTTP Server）和Docker CLI（命令行工具），而我自己实现的scheduler也是这种架构

4、后续可以将命令执行环境从宿主机移至容器中，以增强隔离性。计划依赖containerd来实现，进一步熟悉容器技术。

架构

CS架构，Client为CLI工具，Server为HTTP Server，均使用Go语言编写。

安装

客户端CLI

  1. go get "github.com/songxinjianqwe/scheduler"
  2. cd $GOPATH/src/github.com/songxinjianqwe/scheduler/cli
  3. go install
  4. ./cli

服务器

  1. cd $GOPATH/src/github.com/songxinjianqwe/scheduler/daemon
  2. go install
  3. ./daemon

    启动服务器后会在命令行中打印出REST API：
  • 返回所有任务，对应客户端list命令

    ROUTE: /api/tasks

    Path regexp: ^/api/tasks[/]?$

    Queries templates:

    Queries regexps:

    Methods: GET
  • 返回该id对应的任务，对应客户端get命令

    ROUTE: /api/tasks/{id}

    Path regexp: /]+)[/]?$

    Queries templates:

    Queries regexps:

    Methods: GET
  • 提交任务，对应客户端submit命令

    ROUTE: /api/tasks

    Path regexp: ^/api/tasks[/]?$

    Queries templates:

    Queries regexps:

    Methods: POST
  • 停止任务，对应客户端stop命令

    ROUTE: /api/tasks/{id}

    Path regexp: /]+)[/]?$

    Queries templates:

    Queries regexps:

    Methods: PUT
  • 删除任务，对应客户端delete命令

    ROUTE: /api/tasks/{id}

    Path regexp: /]+)[/]?$

    Queries templates:

    Queries regexps:

    Methods: DELETE

功能

查询任务列表

命令

./cli list

示例image.png

提交任务

命令

./cli submit task_id -type=task_type -time=delay_or_interval -script="shell script"

  • task_id：必填，任务ID，要求是全局唯一
  • -type：必填，任务类型；可以是delay，表示延迟任务，也可以是cron，表示定时任务
  • -time：必填，延时时间；格式类似于10s,1m,20h等，可以参考Go的time.Duration字符串格式
  • -script：必填，shell脚本，如果脚本中含有空格，则需要用引号包裹

示例

  • ./cli submit print_ls_per_10s -type=cron -time=10s -script="ls"
    • 表示提交一个任务ID为print_ls_per_10s的定时任务，每隔10秒，会在服务器上执行一次ls命令
  • ./cli submit do_calc_after_1min -type=delay -time=1m -script="sleep 10s;echo $((1+2))"
    • 表示提交一个任务ID为do_calc_after_1min的延时任务，在1分钟后，会睡眠10秒，然后打印出1+2的结果

读取/监听单个任务

命令

./cli get task_id [-watch=true]

  • task_id：任务ID，要求是全局唯一
  • -watch：选填，是否需要监听；默认为false，如果设置为true，那么get会不断返回该任务的最新状态

示例

  • 。./cli get do_calc_after_1mins
    image.png
  • ./cli get do_calc_after_1mins -watch=true
    image.png

停止任务

命令

./cli stop task_id
停止任务：

  • 如果是延迟任务，则：
    • 可以在任务开始前停止，则任务不会被执行，终态为Stopped
    • 如果任务已经开始执行，则无法停止，且报错，终态为Executed
    • 如果任务已经执行完毕，则无法停止，且报错，终态为Executed
  • 如果是定时任务，则：
    • 可以在任务开始前停止，则任务一次都不会被执行，终态为Stopped
    • 如果在某一次任务开始执行后停止，则任务本次执行不会被中止，且会保存本次执行结果，终态为Stopped
    • 如果在某一个任务任务执行后，下一次任务执行前停止，则下次执行会被跳过，终态为Stopped

示例

image.pngimage.png

删除任务

命令

./cli delete task_id

示例

image.png

难点

并发

写写并发：submit任务之后的execute与stop不会在同一个goroutine中执行，由此会带来并发问题。
读写并发：List读取操作并不要求返回快照，否则代价太大，需要加全局锁，阻塞所有写操作（然后拷贝一份）。使用go test -race会检测到一系列的读写race，基本都是List()的读操作与对某些task的状态的写操作造成的，但这并不代表程序状态错误。
sync.Map#Range能做到的应该是遍历目前现存的所有元素，但不会保证每个元素的值都是在同一时刻的快照。而我们确实也不需要保证读到的是快照。

首次检测race
  1. ==================
  2. WARNING: DATA RACE
  3. Read at 0x00c0001b62b8 by goroutine 15:
  4.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  5.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:86 +0x89
  6.   sync.(*Map).Range()
  7.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  8.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  9.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  10.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  11.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  12.   net/http.HandlerFunc.ServeHTTP()
  13.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  14.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  15.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  16.   net/http.serverHandler.ServeHTTP()
  17.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  18.   net/http.(*conn).serve()
  19.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  20. Previous write at 0x00c0001b62b8 by goroutine 23:
  21.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  22.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:174 +0x27b
  23.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  24.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  25.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  26.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  27. Goroutine 15 (running) created at:
  28.   net/http.(*Server).Serve()
  29.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  30.   net/http.(*Server).ListenAndServe()
  31.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  32.   net/http.ListenAndServe()
  33.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  34.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  35.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  36. Goroutine 23 (finished) created at:
  37.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  38.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  39.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  40.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  41.   net/http.HandlerFunc.ServeHTTP()
  42.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  43.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  44.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  45.   net/http.serverHandler.ServeHTTP()
  46.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  47.   net/http.(*conn).serve()
  48.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  49. ==================
  50. ==================
  51. WARNING: DATA RACE
  52. Read at 0x00c0001b62d0 by goroutine 15:
  53.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  54.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:86 +0x89
  55.   sync.(*Map).Range()
  56.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  57.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  58.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  59.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  60.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  61.   net/http.HandlerFunc.ServeHTTP()
  62.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  63.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  64.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  65.   net/http.serverHandler.ServeHTTP()
  66.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  67.   net/http.(*conn).serve()
  68.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  69. Previous write at 0x00c0001b62d0 by goroutine 23:
  70.   github.com/songxinjianqwe/scheduler/common.(*Task).updateStatus()
  71.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:190 +0x42
  72.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  73.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:177 +0x2f2
  74.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  75.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  76.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  77.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  78. Goroutine 15 (running) created at:
  79.   net/http.(*Server).Serve()
  80.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  81.   net/http.(*Server).ListenAndServe()
  82.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  83.   net/http.ListenAndServe()
  84.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  85.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  86.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  87. Goroutine 23 (finished) created at:
  88.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  89.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  90.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  91.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  92.   net/http.HandlerFunc.ServeHTTP()
  93.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  94.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  95.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  96.   net/http.serverHandler.ServeHTTP()
  97.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  98.   net/http.(*conn).serve()
  99.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  100. ==================
  101. ==================
  102. WARNING: DATA RACE
  103. Read at 0x00c0001b62d8 by goroutine 15:
  104.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  105.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:86 +0x89
  106.   sync.(*Map).Range()
  107.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  108.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  109.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  110.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  111.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  112.   net/http.HandlerFunc.ServeHTTP()
  113.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  114.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  115.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  116.   net/http.serverHandler.ServeHTTP()
  117.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  118.   net/http.(*conn).serve()
  119.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  120. Previous write at 0x00c0001b62d8 by goroutine 23:
  121.   github.com/songxinjianqwe/scheduler/common.(*Task).updateStatus()
  122.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:191 +0x8d
  123.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  124.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:177 +0x2f2
  125.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  126.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  127.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  128.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  129. Goroutine 15 (running) created at:
  130.   net/http.(*Server).Serve()
  131.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  132.   net/http.(*Server).ListenAndServe()
  133.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  134.   net/http.ListenAndServe()
  135.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  136.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  137.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  138. Goroutine 23 (finished) created at:
  139.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  140.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  141.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  142.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  143.   net/http.HandlerFunc.ServeHTTP()
  144.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  145.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  146.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  147.   net/http.serverHandler.ServeHTTP()
  148.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  149.   net/http.(*conn).serve()
  150.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  151. ==================
  152. ==================
  153. WARNING: DATA RACE
  154. Read at 0x00c0001b62f0 by goroutine 15:
  155.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  156.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:86 +0x89
  157.   sync.(*Map).Range()
  158.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  159.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  160.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  161.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  162.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  163.   net/http.HandlerFunc.ServeHTTP()
  164.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  165.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  166.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  167.   net/http.serverHandler.ServeHTTP()
  168.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  169.   net/http.(*conn).serve()
  170.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  171. Previous write at 0x00c0001b62f0 by goroutine 23:
  172.   github.com/songxinjianqwe/scheduler/common.(*Task).increaseVersionAndSignalListeners()
  173.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:196 +0x63
  174.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  175.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:185 +0x303
  176.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  177.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  178.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  179.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  180. Goroutine 15 (running) created at:
  181.   net/http.(*Server).Serve()
  182.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  183.   net/http.(*Server).ListenAndServe()
  184.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  185.   net/http.ListenAndServe()
  186.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  187.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  188.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  189. Goroutine 23 (finished) created at:
  190.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  191.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  192.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  193.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  194.   net/http.HandlerFunc.ServeHTTP()
  195.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  196.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  197.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  198.   net/http.serverHandler.ServeHTTP()
  199.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  200.   net/http.(*conn).serve()
  201.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  202. ==================
  203. ==================
  204. WARNING: DATA RACE
  205. Read at 0x00c0001a0780 by goroutine 15:
  206.   reflect.typedmemmove()
  207.       /usr/local/Cellar/go/1.11.5/libexec/src/runtime/mbarrier.go:177 +0x0
  208.   reflect.packEface()
  209.       /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:119 +0x103
  210.   reflect.valueInterface()
  211.       /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:1008 +0x16f
  212.   reflect.Value.Interface()
  213.       /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:978 +0x51
  214.   encoding/json.marshalerEncoder()
  215.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:448 +0x99
  216.   encoding/json.(*structEncoder).encode()
  217.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:647 +0x307
  218.   encoding/json.(*structEncoder).encode-fm()
  219.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:661 +0x7b
  220.   encoding/json.(*arrayEncoder).encode()
  221.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:769 +0x12a
  222.   encoding/json.(*arrayEncoder).encode-fm()
  223.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:776 +0x7b
  224.   encoding/json.(*sliceEncoder).encode()
  225.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:743 +0xf1
  226.   encoding/json.(*sliceEncoder).encode-fm()
  227.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:755 +0x7b
  228.   encoding/json.(*structEncoder).encode()
  229.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:647 +0x307
  230.   encoding/json.(*structEncoder).encode-fm()
  231.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:661 +0x7b
  232.   encoding/json.(*arrayEncoder).encode()
  233.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:769 +0x12a
  234.   encoding/json.(*arrayEncoder).encode-fm()
  235.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:776 +0x7b
  236.   encoding/json.(*sliceEncoder).encode()
  237.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:743 +0xf1
  238.   encoding/json.(*sliceEncoder).encode-fm()
  239.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:755 +0x7b
  240.   encoding/json.(*encodeState).reflectValue()
  241.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:333 +0x93
  242.   encoding/json.(*encodeState).marshal()
  243.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:305 +0xad
  244.   encoding/json.Marshal()
  245.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:160 +0x73
  246.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  247.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:30 +0xfd
  248.   net/http.HandlerFunc.ServeHTTP()
  249.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  250.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  251.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  252.   net/http.serverHandler.ServeHTTP()
  253.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  254.   net/http.(*conn).serve()
  255.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  256. Previous write at 0x00c0001a0780 by goroutine 23:
  257.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  258.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:174 +0x22e
  259.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  260.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  261.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  262.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  263. Goroutine 15 (running) created at:
  264.   net/http.(*Server).Serve()
  265.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  266.   net/http.(*Server).ListenAndServe()
  267.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  268.   net/http.ListenAndServe()
  269.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  270.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  271.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  272. Goroutine 23 (finished) created at:
  273.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  274.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  275.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  276.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  277.   net/http.HandlerFunc.ServeHTTP()
  278.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  279.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  280.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  281.   net/http.serverHandler.ServeHTTP()
  282.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  283.   net/http.(*conn).serve()
  284.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  285. ==================
  286. ==================
  287. WARNING: DATA RACE
  288. Read at 0x00c0001a0798 by goroutine 15:
  289.   reflect.Value.String()
  290.       /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:1711 +0x5c
  291.   encoding/json.stringEncoder()
  292.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:610 +0xda
  293.   encoding/json.(*structEncoder).encode()
  294.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:647 +0x307
  295.   encoding/json.(*structEncoder).encode-fm()
  296.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:661 +0x7b
  297.   encoding/json.(*arrayEncoder).encode()
  298.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:769 +0x12a
  299.   encoding/json.(*arrayEncoder).encode-fm()
  300.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:776 +0x7b
  301.   encoding/json.(*sliceEncoder).encode()
  302.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:743 +0xf1
  303.   encoding/json.(*sliceEncoder).encode-fm()
  304.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:755 +0x7b
  305.   encoding/json.(*structEncoder).encode()
  306.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:647 +0x307
  307.   encoding/json.(*structEncoder).encode-fm()
  308.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:661 +0x7b
  309.   encoding/json.(*arrayEncoder).encode()
  310.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:769 +0x12a
  311.   encoding/json.(*arrayEncoder).encode-fm()
  312.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:776 +0x7b
  313.   encoding/json.(*sliceEncoder).encode()
  314.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:743 +0xf1
  315.   encoding/json.(*sliceEncoder).encode-fm()
  316.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:755 +0x7b
  317.   encoding/json.(*encodeState).reflectValue()
  318.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:333 +0x93
  319.   encoding/json.(*encodeState).marshal()
  320.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:305 +0xad
  321.   encoding/json.Marshal()
  322.       /usr/local/Cellar/go/1.11.5/libexec/src/encoding/json/encode.go:160 +0x73
  323.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  324.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:30 +0xfd
  325.   net/http.HandlerFunc.ServeHTTP()
  326.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  327.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  328.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  329.   net/http.serverHandler.ServeHTTP()
  330.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  331.   net/http.(*conn).serve()
  332.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  333. Previous write at 0x00c0001a0798 by goroutine 23:
  334.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  335.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:174 +0x22e
  336.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  337.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  338.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  339.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  340. Goroutine 15 (running) created at:
  341.   net/http.(*Server).Serve()
  342.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  343.   net/http.(*Server).ListenAndServe()
  344.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  345.   net/http.ListenAndServe()
  346.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  347.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  348.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  349. Goroutine 23 (finished) created at:
  350.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  351.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  352.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  353.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  354.   net/http.HandlerFunc.ServeHTTP()
  355.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  356.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  357.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  358.   net/http.serverHandler.ServeHTTP()
  359.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  360.   net/http.(*conn).serve()
  361.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  362. ==================
  363. --- FAIL: TestSubmit (3.01s)
  364.     testing.go:771: race detected during execution of test
  365. time="2019-02-11T09:42:54+08:00" level=info msg="Receive a task:common.Task{Id:\"test_delay_7ebfd222-c94c-4c71-b923-88122c260b54\", TaskType:\"delay\", Time:2000000000, Script:\"echo 1\", Results:[]common.TaskResult(nil), Status:0, LastStatusUpdated:time.Time{wall:0xc6dbb70, ext:63685446174, loc:(*time.Location)(0x17d8900)}, Version:0, timer:(*time.Timer)(nil), ticker:(*time.Ticker)(nil), lock:(*sync.Mutex)(nil), watchCond:(*sync.Cond)(nil)}"
  366. time="2019-02-11T09:42:56+08:00" level=info msg="start executing task[test_delay_7ebfd222-c94c-4c71-b923-88122c260b54]"
  367. time="2019-02-11T09:42:56+08:00" level=info msg="Task stdout: 1\n"
  368. time="2019-02-11T09:42:57+08:00" level=info msg="start stopping task[test_delay_7ebfd222-c94c-4c71-b923-88122c260b54]"
  369. FAIL
  370. coverage: 60.3% of statements
  371. Found 6 data race(s)

仔细观察，发现其中很大一部分是先写（比如task的状态更新），再在List()中json序列化时读，出现读写冲突。
但是在List()时遍历理论上是拷贝了一份的！
为了我编写了一个示例：

  1. type Person struct {
  2.    Name string
  3.    Age int
  4.    Cars []Car
  5. }
  6. type Car struct {
  7.    Name string
  8. }
  9. func createPerson(Name string, Age int, carNames []string) *Person {
  10.    person := Person{}
  11.    person.Name = Name
  12.    person.Age = Age
  13.    var cars []Car
  14.    for _, carName := range carNames {
  15.       cars = append(cars, createCar(carName))
  16.    }
  17.    person.Cars = cars
  18.    return &person
  19. }
  20. func createCar(name string) Car {
  21.    car := Car{}
  22.    car.Name = name
  23.    return car
  24. }
  25. var allPersons = []*Person{
  26.    createPerson("p1", 1, []string{"c1","c2"}),
  27.    createPerson("p2", 2, []string{"c1","c2"}),
  28. }
  29. func getPersonList() []Person {
  30.    var personList []Person
  31.    for _, p := range allPersons {
  32.       personList = append(personList, *p)
  33.    }
  34.    return personList
  35. }
  36. func TestCopy(t *testing.T) {
  37.    list := getPersonList()
  38.    for _, p := range list {
  39.       fmt.Printf("%#v\n", p)
  40.    }
  41.    allPersons[0].Name="p3"
  42.    allPersons[0].Cars[0].Name = "c3"
  43.    for _, p := range list {
  44.       fmt.Printf("%#v\n", p)
  45.    }
  46. }

输出结果：

  1. main.Person{Name:"p1", Age:1, Cars:[]main.Car{main.Car{Name:"c1"}, main.Car{Name:"c2"}}}
  2. main.Person{Name:"p2", Age:2, Cars:[]main.Car{main.Car{Name:"c1"}, main.Car{Name:"c2"}}}
  3. main.Person{Name:"p1", Age:1, Cars:[]main.Car{main.Car{Name:"c3"}, main.Car{Name:"c2"}}}
  4. main.Person{Name:"p2", Age:2, Cars:[]main.Car{main.Car{Name:"c1"}, main.Car{Name:"c2"}}}

注意，这里修改person的Name，并没有影响到拷贝后的personList；但是修改car的状态，会影响！
原因是[]Car是一个切片类型，切片类型是引用类型，而struct结构体拷贝是浅拷贝，所以没有拷贝[]Car！

再次检测race

这是List()原来的实现，我

  1. // List返回的是原来的一份拷贝
  2. func (this *StandAloneEngine) List() ([]common.Task, error) {
  3.    var tasks []common.Task
  4.    this.tasks.Range(func(key, value interface{}) bool {
  5.       tasks = append(tasks, *value.(*common.Task))
  6.       return true
  7.    })
  8.    if tasks == nil {
  9.       tasks = make([]common.Task, 0)
  10.    }
  11.    return tasks, nil
  12. }

然后我修改了第5行：
tasks = append(tasks, value.(*common.Task).Clone())

  1. func (this *Task) Clone() Task {
  2.    aCopy := *this
  3.    aCopy.lock = nil
  4.    aCopy.ticker = nil
  5.    aCopy.timer = nil
  6.    aCopy.watchCond = nil
  7.    aCopy.Results = make([]TaskResult, len(this.Results))
  8.    copy(aCopy.Results, this.Results)
  9.    return aCopy
  10. }

再次检测race，又发现了问题：

  1. ==================
  2. WARNING: DATA RACE
  3. Read at 0x00c0000c0b78 by goroutine 15:
  4.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  5.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:201 +0xe2
  6.   sync.(*Map).Range()
  7.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  8.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  9.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  10.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  11.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  12.   net/http.HandlerFunc.ServeHTTP()
  13.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  14.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  15.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  16.   net/http.serverHandler.ServeHTTP()
  17.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  18.   net/http.(*conn).serve()
  19.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  20. Previous write at 0x00c0000c0b78 by goroutine 22:
  21.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  22.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:174 +0x27b
  23.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  24.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  25.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  26.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  27. Goroutine 15 (running) created at:
  28.   net/http.(*Server).Serve()
  29.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  30.   net/http.(*Server).ListenAndServe()
  31.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  32.   net/http.ListenAndServe()
  33.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  34.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  35.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  36. Goroutine 22 (finished) created at:
  37.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  38.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  39.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  40.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  41.   net/http.HandlerFunc.ServeHTTP()
  42.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  43.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  44.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  45.   net/http.serverHandler.ServeHTTP()
  46.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  47.   net/http.(*conn).serve()
  48.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  49. ==================
  50. ==================
  51. WARNING: DATA RACE
  52. Read at 0x00c0000c0b90 by goroutine 15:
  53.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  54.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:201 +0xe2
  55.   sync.(*Map).Range()
  56.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  57.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  58.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  59.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  60.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  61.   net/http.HandlerFunc.ServeHTTP()
  62.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  63.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  64.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  65.   net/http.serverHandler.ServeHTTP()
  66.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  67.   net/http.(*conn).serve()
  68.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  69. Previous write at 0x00c0000c0b90 by goroutine 22:
  70.   github.com/songxinjianqwe/scheduler/common.(*Task).updateStatus()
  71.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:190 +0x42
  72.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  73.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:177 +0x2f2
  74.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  75.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  76.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  77.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  78. Goroutine 15 (running) created at:
  79.   net/http.(*Server).Serve()
  80.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  81.   net/http.(*Server).ListenAndServe()
  82.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  83.   net/http.ListenAndServe()
  84.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  85.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  86.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  87. Goroutine 22 (finished) created at:
  88.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  89.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  90.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  91.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  92.   net/http.HandlerFunc.ServeHTTP()
  93.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  94.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  95.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  96.   net/http.serverHandler.ServeHTTP()
  97.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  98.   net/http.(*conn).serve()
  99.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  100. ==================
  101. ==================
  102. WARNING: DATA RACE
  103. Read at 0x00c0000c0b98 by goroutine 15:
  104.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  105.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:201 +0xe2
  106.   sync.(*Map).Range()
  107.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  108.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  109.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  110.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  111.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  112.   net/http.HandlerFunc.ServeHTTP()
  113.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  114.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  115.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  116.   net/http.serverHandler.ServeHTTP()
  117.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  118.   net/http.(*conn).serve()
  119.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  120. Previous write at 0x00c0000c0b98 by goroutine 22:
  121.   github.com/songxinjianqwe/scheduler/common.(*Task).updateStatus()
  122.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:191 +0x8d
  123.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  124.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:177 +0x2f2
  125.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  126.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  127.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  128.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  129. Goroutine 15 (running) created at:
  130.   net/http.(*Server).Serve()
  131.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  132.   net/http.(*Server).ListenAndServe()
  133.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  134.   net/http.ListenAndServe()
  135.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  136.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  137.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  138. Goroutine 22 (finished) created at:
  139.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  140.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  141.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  142.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  143.   net/http.HandlerFunc.ServeHTTP()
  144.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  145.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  146.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  147.   net/http.serverHandler.ServeHTTP()
  148.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  149.   net/http.(*conn).serve()
  150.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  151. ==================
  152. ==================
  153. WARNING: DATA RACE
  154. Read at 0x00c0000c0bb0 by goroutine 15:
  155.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  156.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:201 +0xe2
  157.   sync.(*Map).Range()
  158.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  159.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  160.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  161.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  162.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  163.   net/http.HandlerFunc.ServeHTTP()
  164.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  165.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  166.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  167.   net/http.serverHandler.ServeHTTP()
  168.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  169.   net/http.(*conn).serve()
  170.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  171. Previous write at 0x00c0000c0bb0 by goroutine 22:
  172.   github.com/songxinjianqwe/scheduler/common.(*Task).increaseVersionAndSignalListeners()
  173.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:196 +0x63
  174.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  175.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:185 +0x303
  176.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  177.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  178.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  179.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  180. Goroutine 15 (running) created at:
  181.   net/http.(*Server).Serve()
  182.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  183.   net/http.(*Server).ListenAndServe()
  184.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  185.   net/http.ListenAndServe()
  186.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  187.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  188.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  189. Goroutine 22 (finished) created at:
  190.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  191.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  192.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  193.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  194.   net/http.HandlerFunc.ServeHTTP()
  195.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  196.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  197.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  198.   net/http.serverHandler.ServeHTTP()
  199.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  200.   net/http.(*conn).serve()
  201.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  202. ==================
  203. ==================
  204. WARNING: DATA RACE
  205. Read at 0x00c0000892c0 by goroutine 15:
  206.   runtime.slicecopy()
  207.       /usr/local/Cellar/go/1.11.5/libexec/src/runtime/slice.go:221 +0x0
  208.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List.func1()
  209.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:207 +0x211
  210.   sync.(*Map).Range()
  211.       /usr/local/Cellar/go/1.11.5/libexec/src/sync/map.go:337 +0x13c
  212.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).List()
  213.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:85 +0x77
  214.   github.com/songxinjianqwe/scheduler/daemon/handler.GetAllTasksHandler()
  215.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:25 +0xa1
  216.   net/http.HandlerFunc.ServeHTTP()
  217.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  218.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  219.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  220.   net/http.serverHandler.ServeHTTP()
  221.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  222.   net/http.(*conn).serve()
  223.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  224. Previous write at 0x00c0000892c0 by goroutine 22:
  225.   github.com/songxinjianqwe/scheduler/common.(*Task).appendResultAndUpdateStatusAtomically()
  226.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:174 +0x22e
  227.   github.com/songxinjianqwe/scheduler/common.(*Task).Execute()
  228.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/common/task.go:97 +0x2a6
  229.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit.func1()
  230.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:57 +0x62
  231. Goroutine 15 (running) created at:
  232.   net/http.(*Server).Serve()
  233.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2851 +0x4c5
  234.   net/http.(*Server).ListenAndServe()
  235.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2764 +0xe8
  236.   net/http.ListenAndServe()
  237.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:3004 +0xef
  238.   github.com/songxinjianqwe/scheduler/daemon/server.Run()
  239.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/server/server.go:49 +0x522
  240. Goroutine 22 (finished) created at:
  241.   github.com/songxinjianqwe/scheduler/daemon/engine/standalone.(*StandAloneEngine).Submit()
  242.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/engine/standalone/standalone_engine_impl.go:55 +0x3c0
  243.   github.com/songxinjianqwe/scheduler/daemon/handler.SubmitTask()
  244.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/daemon/handler/handler.go:85 +0x35f
  245.   net/http.HandlerFunc.ServeHTTP()
  246.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1964 +0x51
  247.   github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
  248.       /Users/jasper/go/src/github.com/songxinjianqwe/scheduler/vendor/github.com/gorilla/mux/mux.go:212 +0x12e
  249.   net/http.serverHandler.ServeHTTP()
  250.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:2741 +0xc4
  251.   net/http.(*conn).serve()
  252.       /usr/local/Cellar/go/1.11.5/libexec/src/net/http/server.go:1847 +0x80a
  253. ==================
  254. --- FAIL: TestSubmit (3.01s)
  255.     testing.go:771: race detected during execution of test
  256. time="2019-02-11T16:34:33+08:00" level=info msg="Receive a task:common.Task{Id:\"test_delay_266aa0c3-fc83-4fa7-abb6-1ed69d692ae3\", TaskType:\"delay\", Time:2000000000, Script:\"echo 1\", Results:[]common.TaskResult(nil), Status:0, LastStatusUpdated:time.Time{wall:0xf9dbde0, ext:63685470873, loc:(*time.Location)(0x17d7640)}, Version:0, timer:(*time.Timer)(nil), ticker:(*time.Ticker)(nil), lock:(*sync.RWMutex)(nil), watchCond:(*sync.Cond)(nil)}"
  257. time="2019-02-11T16:34:35+08:00" level=info msg="start executing task[test_delay_266aa0c3-fc83-4fa7-abb6-1ed69d692ae3]"
  258. time="2019-02-11T16:34:35+08:00" level=info msg="Task stdout: 1\n"
  259. time="2019-02-11T16:34:36+08:00" level=info msg="start stopping task[test_delay_266aa0c3-fc83-4fa7-abb6-1ed69d692ae3]"
  260. FAIL
  261. Found 5 data race(s)
  262. FAIL   github.com/songxinjianqwe/scheduler/cli/client 9.045s
  263. ?      github.com/songxinjianqwe/scheduler/cli/command    [no test files]
  264. ?      github.com/songxinjianqwe/scheduler/common [no test files]
  265. ?      github.com/songxinjianqwe/scheduler/common/run [no test files]
  266. ?      github.com/songxinjianqwe/scheduler/daemon [no test files]
  267. ?      github.com/songxinjianqwe/scheduler/daemon/engine  [no test files]
  268. ok     github.com/songxinjianqwe/scheduler/daemon/engine/standalone   43.060s
  269. ?      github.com/songxinjianqwe/scheduler/daemon/handler [no test files]
  270. ?      github.com/songxinjianqwe/scheduler/daemon/server  [no test files]

其中有sliceCopy的读与task的appendResult之间的读写冲突，于是我在Clone外面加了一层读锁，以此保证读到的results是在一次完整的写操作之后、下次写操作之前的一个正确的快照。

  1. func (this *Task) Clone() Task {
  2.    this.lock.RLock()
  3.    defer this.lock.RUnlock()
  4.    aCopy := *this
  5.    aCopy.lock = nil
  6.    aCopy.ticker = nil
  7.    aCopy.timer = nil
  8.    aCopy.watchCond = nil
  9.    aCopy.Results = make([]TaskResult, len(this.Results))
  10.    copy(aCopy.Results, this.Results)
  11.    return aCopy
  12. }

此时再次检测race就没有问题了！

long polling（watch）

watch是自己实现了一个HTTP长轮询

待改进

  • 支持多种script，如js、python、groovy
  • 服务端对任务结果数保留一定数量，避免OOM
  • 将shell命令放在沙箱环境中运行