Merge pull request #23 from Issif/v2.0.0

## 2.0.0 - 2019-05-23
#### New 
- New output : **Elasticsearch** ([issue #14](https://github.com/Issif/falcosidekick/issues/14))
- **New configuration method : we can now use a config file in YAML and/or env vars** (see *README*) ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
- New endpoint : `/debug/vars` gives access to Golang + Custom metrics (see *README*) ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
#### Enhancement
- Add a lot of unit tests for code coverage ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
- Some log outputs have been reformated
- 💥 Some env variables have been renamed again to match fields in YAML config files (*see README*)
#### Fix
- Panic are now catched to avoid crashes

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 2.0.0 - 2019-05-23
+#### New 
+- New output : **Elasticsearch** ([issue #14](https://github.com/Issif/falcosidekick/issues/14))
+- **New configuration method : we can now use a config file in YAML and/or env vars** (see *README*) ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
+- New endpoint : `/debug/vars` gives access to Golang + Custom metrics (see *README*) ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
+#### Enhancement
+- Add a lot of unit tests for code coverage ([issue #17](https://github.com/Issif/falcosidekick/issues/17))
+- Some log outputs have been reformated
+- :boom: Some env variables have been renamed again to match fields in YAML config files (*see README*)
+#### Fix
+- Panic are now catched to avoid crashes
+
 ## 1.1.0 - 2019-05-10
 #### Enhancement
 -  **all outputs use new generic methods (`NewClient()` + `Post()`), new output integration will be easier**

README.md

@@ -1,30 +1,37 @@
+# Falcosidekick
+
 ![falcosidekick](https://github.com/Issif/falcosidekick/raw/master/imgs/falcosidekick.png)
 
 ![release](https://flat.badgen.net/github/release/Issif/falcosidekick/latest?color=green) ![last commit](https://flat.badgen.net/github/last-commit/Issif/falcosidekick) ![licence](https://flat.badgen.net/badge/license/MIT/blue) ![docker pulls](https://flat.badgen.net/docker/pulls/issif/falcosidekick?icon=docker)
 
-# Falcosidekick
-A simple daemon to help you with falco's outputs (https://sysdig.com/opensource/falco/). It takes a falco's event and forwards it to different outputs. 
+## Description
 
-# Outputs
+A simple daemon to help you with falco's outputs (https://sysdig.com/opensource/falco/). It takes a falco's event and forwards it to different outputs.
+
+## Outputs
 
 Currently available outputs are :
-* Slack
-* Datadog
-* AlertManager
 
-# Usage
+* **Slack**
+* **Datadog**
+* **AlertManager**
+* **Elasticsearch**
+
+## Usage
 
 Run the daemon as any other daemon in your architecture (systemd, k8s daemonset, swarm service, ...)
 
-## With docker
-```
+### With docker
+
+```bash
 docker run -d -p 2801:2801 -e SLACK_WEBHOOK_URL=XXXX -e DATADOG_API_KEY=XXXX issif/falcosidekick
 ```
 
-## Falco's config
+### Falco's config
 
 Add this (adapted to your environment) in your *falco.yaml* :
-```
+
+```bash
 json_output: true
 json_include_output_property: true
 program_output:
@@ -33,55 +40,132 @@ program_output:
   program: "curl -d @- localhost:2801/"
 ```
 
-## Env variables 
+### Configuration
+
+Configuration is made by *file (yaml)* and *env vars*, both can be used but *env vars* override values from *file*.
+
+#### YAML File
 
-Configuration of the daemon is made by Env vars :
+See **config_example.yaml** :
+
+```yaml
+#listenport: 2801 #port to listen for daemon (default: 2801)
+debug: false #if true all outputs will print in stdout the payload they send (default: false)
+
+slack:
+  webhookurl: "" # Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
+  #footer: "" #Slack footer
+  #icon: "" #Slack icon (avatar)
+  outputformat: "text" # all (default), text, fields
+
+datadog:
+  #apikey: ""  #Datadog API Key, if not empty, Datadog output is enabled
+
+alertmanager:
+  # hostport: "" # http://{domain or ip}:{port}, if not empty, Alertmanager output is enabled
+
+elasticsearch:
+  # hostport: "" # http://{domain or ip}:{port}, if not empty, Elasticsearch output is enabled
+  # index: "falco" # index (default: falco)
+  # type: "event" 
+```
+
+Usage : 
+
+```bash
+usage: falcosidekick [<flags>]
+
+Flags:
+      --help                     Show context-sensitive help (also try --help-long and --help-man).
+  -c, --config-file=CONFIG-FILE  config file
+```
 
-* **LISTEN_PORT** : port to listen for daemon (default: 2801)
-* **SLACK_WEBHOOK_URL** : Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not `empty`, Slack output is *enabled*
+#### Env vars
+
+Configuration of the daemon can be made also by *env vars*, these values override these from *yaml file*.
+
+The *env vars* "match" field names in *yaml file with this structure (**take care of lower/uppercases**) : `yaml: a.b --> envvar: A_B` :
+
+* **LISTENPORT** : port to listen for daemon (default: 2801)
+* **DEBUG** : if *true* all outputs will print in stdout the payload they send (default: false)
+* **SLACK_WEBHOOKURL** : Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not `empty`, Slack output is *enabled*
 * **SLACK_FOOTER** : Slack footer
 * **SLACK_ICON** : Slack icon (avatar)
-* **SLACK_OUTPUT_FORMAT** : `all` (default), `text` (only text is displayed in Slack), `fields` (only fields are displayed in Slack)
-* **DATADOG_API_KEY** : Datadog API Key, if not `empty`, Datadog output is *enabled*
-* **ALERTMANAGER_HOST_PORT** : AlertManager host:port, if not `empty`, AlertManager is *enabled*
-* **DEBUG** : if *true* all outputs will print in stdout the payload they send
+* **SLACK_OUTPUTFORMAT** : `all` (default), `text` (only text is displayed in Slack), `fields` (only fields are displayed in Slack)
+* **DATADOG_APIKEY** : Datadog API Key, if not `empty`, Datadog output is *enabled*
+* **ALERTMANAGER_HOSTPORT** : AlertManager http://host:port, if not `empty`, AlertManager is *enabled*
+* **ELASTICSEARCH_HOSTPORT** : Elasticsearch http://host:port, if not `empty`, Elasticsearch is *enabled*
+* **ELASTICSEARCH_INDEX** : Elasticsearch index (default: falco)
+* **ELASTICSEARCH_TYPE** : Elasticsearch document type (default: event)
 
-# Handlers
+## Handlers
 
 Different URI (handlers) are available :
 
 * `/` : main and default handler, your falco config must be configured to use it
 * `/ping` : you will get a  `pong` as answer, useful to test if falcosidekick is running and its port is opened (for healthcheck purpose for example)
 * `/test` : (for debug only) send a test event to all enabled outputs.
+* `/debug/vars` : get statistics from daemon (in JSON format), it uses classic `expvar` package and some custom values are added 
 
-# Logs
+## Logs
 
 All logs are sent to `stdout`.
 
+```bash
+2019/05/10 14:32:06 [INFO] : Enabled Outputs : Slack Datadog
 ```
-2019/05/10 14:32:06 [INFO] : Enable Outputs : Slack Datadog
-2019/05/10 14:32:06 [INFO] : Disable Outputs : AlertManager
-```
 
-# Examples
+## Metrics
+
+The daemon exposes the common *Golang* metrics and some custom values in JSON format. It's useful for monitoring purpose.
+
+![expvar json](https://github.com/Issif/falcosidekick/raw/master/imgs/expvar_json.png)
+![expvarmon](https://github.com/Issif/falcosidekick/raw/master/imgs/expvarmon.png)
+
+## Examples
 
 Run you daemon and try (from falco's documentation) :
-```
-curl "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2017-10-09T23:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"}}'
+
+```bash
+curl "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"}}'
 ```
 
 You should get :
 
-**Slack** :
+### Slack
 
+(SLACK_OUTPUT_FORMAT="all")
 ![slack example](https://github.com/Issif/falcosidekick/raw/master/imgs/slack.png)
+(SLACK_OUTPUT_FORMAT="fields")
 ![slack no fields example](https://github.com/Issif/falcosidekick/raw/master/imgs/slack_no_fields.png)
 
-**Datadog** :
+### Datadog
 
 *(Tip: filter on `sources: falco`)*
 ![datadog example](https://github.com/Issif/falcosidekick/raw/master/imgs/datadog.png)
 
-**AlertManager** :
+### AlertManager
+
+![alertmanager example](https://github.com/Issif/falcosidekick/raw/master/imgs/alertmanager.png)
+
+### Elasticsearch (with Kibana)
+
+![kibana example](https://github.com/Issif/falcosidekick/raw/master/imgs/kibana.png)
+
+## Development
+
+### Build
+
+```bash
+go build
+```
+
+### Test & Coverage
+
+```bash
+go test ./outputs -count=1 -cover -v
+```
+
+## Author
 
-![alertmanager example](https://github.com/Issif/falcosidekick/raw/master/imgs/alertmanager.png)
+Thomas Labarussias (https://github.com/Issif)

config.go

@@ -0,0 +1,55 @@
+package main
+
+import (
+	"log"
+	"path"
+	"path/filepath"
+	"strings"
+
+	"github.com/Issif/falcosidekick/types"
+
+	"github.com/spf13/viper"
+	kingpin "gopkg.in/alecthomas/kingpin.v2"
+)
+
+func getConfig() *types.Configuration {
+	c := &types.Configuration{}
+
+	configFile := kingpin.Flag("config-file", "config file").Short('c').ExistingFile()
+	kingpin.Parse()
+
+	v := viper.New()
+	v.SetDefault("ListenPort", 2801)
+	v.SetDefault("Debug", false)
+	v.SetDefault("Slack.WebhookURL", "")
+	v.SetDefault("Slack.Footer", "https://github.com/Issif/falcosidekick")
+	v.SetDefault("Slack.Icon", "https://raw.githubusercontent.com/Issif/falcosidekick/master/imgs/falcosidekick.png")
+	v.SetDefault("SlackOutput.OutputFormat", "all")
+	v.SetDefault("Datadog.APIKey", "")
+	v.SetDefault("Alertmanager.HostPort", "")
+	v.SetDefault("Elasticsearch.HostPort", "")
+	v.SetDefault("Elasticsearch.Index", "falco")
+	v.SetDefault("Elasticsearch.Type", "event")
+
+	v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
+	v.AutomaticEnv()
+	if *configFile != "" {
+		d, f := path.Split(*configFile)
+		if d == "" {
+			d = "."
+		}
+		v.SetConfigName(f[0 : len(f)-len(filepath.Ext(f))])
+		v.AddConfigPath(d)
+		err := v.ReadInConfig()
+		if err != nil {
+			log.Printf("Error when reading config file: %v\n", err)
+		}
+	}
+	v.Unmarshal(c)
+
+	if c.ListenPort == 0 || c.ListenPort > 65536 {
+		log.Fatalf("[ERROR] : Bad port number\n")
+	}
+
+	return c
+}

config_example.yaml

@@ -0,0 +1,19 @@
+# listenport: 2801 #port to listen for daemon (default: 2801)
+debug: false #if true all outputs will print in stdout the payload they send (default: false)
+
+slack:
+  webhookurl: "" # Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
+  #footer: "" #Slack footer
+  #icon: "" #Slack icon (avatar)
+  outputformat: "text" # all (default), text, fields
+
+datadog:
+  #apikey: ""  #Datadog API Key, if not empty, Datadog output is enabled
+
+alertmanager:
+  # hostport: "" # http://{domain or ip}:{port}, if not empty, Alertmanager output is enabled
+
+elasticsearch:
+  # hostport: "" # http://{domain or ip}:{port}, if not empty, Elasticsearch output is enabled
+  # index: "falco" # index (default: falco)
+  # type: "event" 

go.mod

@@ -1,3 +1,10 @@
 module github.com/Issif/falcosidekick
 
 go 1.12
+
+require (
+	github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc // indirect
+	github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf // indirect
+	github.com/spf13/viper v1.3.2
+	gopkg.in/alecthomas/kingpin.v2 v2.2.6
+)

go.sum

@@ -0,0 +1,43 @@
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
+github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
+github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
+github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
+github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
+github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
+github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
+github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
+github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
+github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
+github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
+github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
+github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
+github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/viper v1.3.2 h1:VUFqw5KcqRf7i70GOzW7N+Q7+gxVBkSSqiXB12+JQ4M=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
+golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a h1:1n5lsVfiQW3yfsRGu98756EH1YthsFqr/5mxHduZW2A=
+golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=