Advanced error monitoring using kibana logs.
Dashboard:
Error details:
- node.js 8+
- mysql or mariadb 5+ (pull requests for Postgres welcome)
- kibana
- some free space to store your logs
- you should be able to run socket based interaction
git clone https://github.com/jehy/shodan.git
cp config/default.json config/runtime.json
- Set up
runtime.json
config (see below) - Run
npm run migrate
to set up DB - Run
npm i
npm run server
andnpm run updater
(those are 2 different processes!)- Enjoy!
-
kibana
"cookie": "_oauth2_proxy=XXXXX"
your own kibana cookie"version": "5.5.2",
your kibana version"url": "https://logs.google.com",
your kibana url"firstSearchFor": 24,
first search should begin fetching entries from 24 hours ago"searchFor": 1,
new entries are fetched 1 last hour"crawlDelay": 1
delay by 1 minute to avoid ignoring entries which were added to the past"preference": 1521709471107,
some preference id from kibana requests, dunno wtf"indexes": ["some-app"],
index names to search for"indexFilterOut": ["some-app-extra"],
filter for indices"fetchNum": 500,
max entries to fetch"updateInterval": 30,
how often we should fetch updates, in seconds"storeLogsFor": 7
for how many days do you want to store logs
-
"userAgent": "Mozilla/5.0 (X11; Linux x86_64)",
useragent string to use for kibana search -
"db"
standard database options"client": "mysql",
"connection":
"host": "127.0.0.1",
"user": "your_database_user",
"password": "your_database_password",
"database": "myapp_test",
"charset": "utf8"
"migrations":
"tableName": "knex_migrations"
-
errorNotifyBot
bot configmonitoringChannelId
- channel where bot takes guys who are on duty.outputChannelId
- channel to write error messages to.interval
- error check interval.diffMinutesBetweenSendSlack
- min interval for posting same messages.minStageErrors
- minimal stage errors number to react to.minProdErrors
- minimal production errors number to react to.minStageErrorsNew
- minimal NEW stage errors number to react to.minProdErrorsNew
-minimal NEW prod errors number to react to.minStageErrorsGrown
- minimal stage errors growth to react to.minProdErrorsGrown
- minimal prod errors growth to react to.maxSlackMessages
- max number of messages per slack postblackListMsgNames
, ex["fillDistances", "getGMToffset_0"]
- array of error message names to ignore