Keywords: Golang, go, concurrency, JSON

{GpX} Go Concurrent Processes Executer

Concurrency is not paralelism

---

What is gcpex ?

Messing around *nix, a need arises frequently for me, concurrent execution of multiple, non related proccesses.

Each one must be launched with their own parameters and directed to their own custom log Files.

Umm ... Just use a bash Script you idiot ...

Well, that was my first approach and worked nicely ... but the code was kind of ugly and cumbersome .. not to mention It's relative poor performance.

During my Go learning journey I read this article that came to my mind naturally when trying to solve this task.

Based on that knowledge I built my own tool gcpex.

👉 Note: Only stdlib packages and just one source file used.

Demo

Tell me about the installation

Obviously you need go installed in your machine.

Then just:

go get -v github.com/klashxx/gcpex

And the executable will be compiled and placed in your $GOPATH/bin directory.

$ which gcpex
~/Documents/dev/go/bin/gcpex

Easy enough 😎

OK, now ... how does this thing works?

The syntax is neat:

$ gcpex
  -in string
        cmd JSON file repo. [mandatory]
  -out string
        Respond JSON file.
  -routines int
        max parallel execution routines (default 5)

---

1. -in: Mandatory requisite, a JSON File to Configure our bunch of executions.

The format is pretty self explanatory:

[
 {
  "cmd": "a_command",
  "args": ["arg1", "arg2"],
  "log": "/stdout/log/path/a_command.log",
  "err": "/stderr/log/path/a_command.err",
  "overwrite": true
 },
 {
  "cmd": "another_command",
  "env": ["PATH=/my/custom/path"],
  "log": "/non_existent/commands.out"
 }
 ]

Schema definition:

• cmd: Executable {mandatory}
• args: List of arguments to parse to the executable {optional}
• log: Path to the log File attached to cmd stdout. {optional} (missed if not specified)
• err: Path to the err File attached to cmd stderr. {optional} (missed if not specified)
• env: List of environment variables to use for launch the process, if env is null it uses the current environment
• overwrite: Must be switched to true (bool value) to overwrite a previous log and/or err file. {optional} (default = false)

2. -out: an optional JSON file where the response will be written.

Format:

[
 {
  "Cmd": "a_command",
  "Path": "/path/to/command",
  "Env": null,
  "Args": [
    "arg1",
    "arg2"
  ],
  "Success": true,
  "Pid": 11111,
  "Duration": 15,
  "Errors": [],
  "Log": "/stdout/log/path/a_command.log",
  "Err": "/stderr/log/path/a_command.err",
  "Overwrite": true
},
{
  "Cmd": "another_command",
  "Path": "/my/custom/path",
  "Env": [
    "PATH=/my/custom/path"
  ],
  "Args": [],
  "Success": false,
  "Pid": 0,
  "Duration": 0,
  "Errors": [
    "/non_existent/commands.out: file base dir does not exists"
  ],
  "Log": "/non_existent/commands.out",
  "Err": "",
  "Overwrite": false
}
]

Schema definition:

• Cmd: Full path to the cmd executed
• Path: Dir path to executable.
• Env: List of environment variables used to launch the process.
• Args: List of arguments parsed to the executable.
• Success: A bool value, will be true when cmd exit code is 0.
• Pid: Process Identification Number during the execution. Zero when process fails.
• Duration: Number of seconds exec took to complete.
• Errors: List of errors presented during the execution.
• Log: Path to file used to store stdout.
• Err: Path to file used to store stderr.
• Overwrite: A bool flag, allows Log and Err overwriting (when true).

3. -routines: number of routines to digester the commands stored in our JSON -in file.

---

Examples

Having this commands_01.json file:

[
  {
  "cmd": "echo",
  "args": ["5"]
 },
 {
  "cmd": "ls",
  "args": ["-j"],
  "log": "/tmp/ls.out",
  "err": "/tmp/ls.err"
 },
 {
  "cmd": "sleep",
  "args": ["5"]
 },
 {
  "cmd": "sleep",
  "args": ["5"]
 },
  {
  "cmd": "dummy02",
  "args": ["5"]
 },
  {
  "cmd": "cat",
  "args": ["commands.json"],
  "log": "/tmp/commands.out"
 },
  {
  "cmd": "cat",
  "args": ["commands.json"],
  "log": "/non_existent/commands.out"
 }
]

Using two routines to digester and storing the result in reponse.json:

$ gcpex -in commands_01.json -routines 2 -out response.json
2017/02/03 00:12:46 Start -> Cmd: echo          Args: 5               Pid:  8845
2017/02/03 00:12:46 Start -> Cmd: ls            Args: -j              Pid:  8846
2017/02/03 00:12:46 End   -> Cmd: echo          Args: 5               Pid:  8845 Success: true  Elapsed: 0000
2017/02/03 00:12:46 ERROR -> Cmd: ls            Args: -j              Err: exit status 1
2017/02/03 00:12:46 Start -> Cmd: sleep         Args: 5               Pid:  8847
2017/02/03 00:12:46 Start -> Cmd: sleep         Args: 5               Pid:  8848
2017/02/03 00:12:51 End   -> Cmd: sleep         Args: 5               Pid:  8848 Success: true  Elapsed: 0005
2017/02/03 00:12:51 End   -> Cmd: sleep         Args: 5               Pid:  8847 Success: true  Elapsed: 0005
2017/02/03 00:12:51 ERROR -> Cmd: dummy02       Args: 5               Err: exec: "dummy02": executable file not found in $PATH
2017/02/03 00:12:51 ERROR -> Cmd: cat           Args: commands.json   Err: /non_existent/commands.out: file base dir does not exists
2017/02/03 00:12:51 Start -> Cmd: echo          Args: Lorem ipsum dolor sit amet Pid:  8849
2017/02/03 00:12:51 End   -> Cmd: echo          Args: Lorem ipsum dolor sit amet Pid:  8849 Success: true  Elapsed: 0000
2017/02/03 00:12:51 Final -> Elapsed (seconds): 0005                  Executions (tot/ok/ko): 007 / 004 / 003
$ echo $?
1

Log of ls command:

$ cat /tmp/ls.out
$ cat /tmp/ls.err
ls: illegal option -- j
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

Log of echo excution:

$ cat /tmp/commands.out
Lorem ipsum dolor sit amet ,consectetur adipiscing elit,  sed do eiusmod tempor incididunt ut labore et dolore magna aliqua

Content of the result file response.json:

[
{
  "Cmd": "echo",
  "Path": "/bin/echo",
  "Env": null,
  "Args": [
    "5"
  ],
  "Success": true,
  "Pid": 8845,
  "Duration": 0,
  "Errors": null,
  "Log": "",
  "Err": "",
  "Overwrite": false
},
{
  "Cmd": "ls",
  "Path": "/bin/ls",
  "Env": null,
  "Args": [
    "-j"
  ],
  "Success": false,
  "Pid": 8846,
  "Duration": 0,
  "Errors": [
    "exit status 1"
  ],
  "Log": "/tmp/ls.out",
  "Err": "/tmp/ls.err",
  "Overwrite": false
},
{
  "Cmd": "sleep",
  "Path": "/bin/sleep",
  "Env": null,
  "Args": [
    "5"
  ],
  "Success": true,
  "Pid": 8848,
  "Duration": 5,
  "Errors": null,
  "Log": "",
  "Err": "",
  "Overwrite": false
},
{
  "Cmd": "sleep",
  "Path": "/bin/sleep",
  "Env": null,
  "Args": [
    "5"
  ],
  "Success": true,
  "Pid": 8847,
  "Duration": 5,
  "Errors": null,
  "Log": "",
  "Err": "",
  "Overwrite": false
},
{
  "Cmd": "dummy02",
  "Path": "",
  "Env": null,
  "Args": [
    "5"
  ],
  "Success": false,
  "Pid": 0,
  "Duration": 0,
  "Errors": [
    "exec: \"dummy02\": executable file not found in $PATH"
  ],
  "Log": "",
  "Err": "",
  "Overwrite": false
},
{
  "Cmd": "cat",
  "Path": "/bin/cat",
  "Env": null,
  "Args": [
    "commands.json"
  ],
  "Success": false,
  "Pid": 0,
  "Duration": 0,
  "Errors": [
    "/non_existent/commands.out: file base dir does not exists"
  ],
  "Log": "/non_existent/commands.out",
  "Err": "",
  "Overwrite": false
},
{
  "Cmd": "echo",
  "Path": "/bin/echo",
  "Env": null,
  "Args": [
    "Lorem ipsum dolor sit amet"
  ],
  "Success": true,
  "Pid": 8849,
  "Duration": 0,
  "Errors": null,
  "Log": "/tmp/commands.out",
  "Err": "",
  "Overwrite": false
}
]

---

Nice? Let's try Another one ...

Suppose a commands_02.json file with 30 sleep 5 processes:

[
 {
  "cmd": "sleep",
  "args": ["5"]
 },
 {
  "cmd": "sleep",
  "args": ["5"]
 },
 ...
 ]

Add so on ....

🏁 Fact: A sequential process would take 150 seconds to complete.

Let's to use Ten simultaneous routines to do our job:

$ gcpex -in commands_02.json -routines 10
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7961
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7960
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7962
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7966
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7967
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7963
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7968
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7969
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7965
2017/02/03 00:03:53 Start -> Cmd: sleep         Args: 5     Pid:  7964
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7960 Success: true  Elapsed: 0005
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7961 Success: true  Elapsed: 0004
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7970
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7971
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7964 Success: true  Elapsed: 0005
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7962 Success: true  Elapsed: 0005
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7963 Success: true  Elapsed: 0005
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7972
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7973
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7965 Success: true  Elapsed: 0005
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7966 Success: true  Elapsed: 0005
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7974
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7967 Success: true  Elapsed: 0005
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7975
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7976
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7977
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7968 Success: true  Elapsed: 0005
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7978
2017/02/03 00:03:58 End   -> Cmd: sleep         Args: 5     Pid:  7969 Success: true  Elapsed: 0005
2017/02/03 00:03:58 Start -> Cmd: sleep         Args: 5     Pid:  7979
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7970 Success: true  Elapsed: 0005
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7971 Success: true  Elapsed: 0005
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7980
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7981
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7972 Success: true  Elapsed: 0005
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7973 Success: true  Elapsed: 0005
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7974 Success: true  Elapsed: 0005
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7982
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7983
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7975 Success: true  Elapsed: 0005
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7976 Success: true  Elapsed: 0005
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7984
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7978 Success: true  Elapsed: 0005
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7977 Success: true  Elapsed: 0005
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7985
2017/02/03 00:04:03 End   -> Cmd: sleep         Args: 5     Pid:  7979 Success: true  Elapsed: 0005
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7986
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7987
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7988
2017/02/03 00:04:03 Start -> Cmd: sleep         Args: 5     Pid:  7989
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7980 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7981 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7982 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7983 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7986 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7985 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7984 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7987 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7988 Success: true  Elapsed: 0005
2017/02/03 00:04:08 End   -> Cmd: sleep         Args: 5     Pid:  7989 Success: true  Elapsed: 0005
2017/02/03 00:04:08 Final -> Elapsed (seconds): 0015        Executions (tot/ok/ko): 030 / 030 / 000

As expected the total execution time is 15 seconds.

Now ... We're going to use the 🏇 Calvary.

Thirty routines in action:

$ gcpex -in commands_02.json -routines 30
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8102
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8104
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8105
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8106
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8103
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8109
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8110
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8111
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8107
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8108
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8112
2017/02/03 00:05:39 Start -> Cmd: sleep         Args: 5     Pid:  8113
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8114
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8115
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8116
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8117
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8118
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8119
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8120
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8121
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8122
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8123
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8124
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8125
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8126
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8127
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8128
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8129
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8130
2017/02/03 00:05:40 Start -> Cmd: sleep         Args: 5     Pid:  8131
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8102 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8105 Success: true  Elapsed: 0004
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8103 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8104 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8106 Success: true  Elapsed: 0004
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8109 Success: true  Elapsed: 0004
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8107 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8108 Success: true  Elapsed: 0004
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8110 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8112 Success: true  Elapsed: 0005
2017/02/03 00:05:44 End   -> Cmd: sleep         Args: 5     Pid:  8111 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8113 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8114 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8115 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8116 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8117 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8118 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8119 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8120 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8121 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8122 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8123 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8124 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8125 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8126 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8128 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8127 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8129 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8130 Success: true  Elapsed: 0005
2017/02/03 00:05:45 End   -> Cmd: sleep         Args: 5     Pid:  8131 Success: true  Elapsed: 0005
2017/02/03 00:05:45 Final -> Elapsed (seconds): 0005        Executions (tot/ok/ko): 030 / 030 / 000

Again... the result makes sense, the program took five seconds to process it all.

Licensing

gcpex is licensed under the MIT license.

Contact me

You can find me out here

Made with ❤️ in Almería, Spain.