Monitor PostgreSQL Data Table and logging data to elasticsearch system with composable index template.
This repository based on the Docker ELK implementation by deviantony(https://github.com/deviantony/docker-elk)
Run the latest version(8.8.1) of the Elastic stack with Docker and Docker Compose.
It gives you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and the visualization power of Kibana.
Based on the official Docker images from Elastic:
- Elasticsearch
- Logstash
- Kibana
See more information -> https://github.com/deviantony/docker-elk
- You can create custom elastic user & role in
setup/entrypoint.sh
,.env
andsetup/roles/
. - Make User
# entrypoint.sh
declare -A users_passwords
users_passwords=(
[logstash_internal]="${LOGSTASH_INTERNAL_PASSWORD:-}"
[kibana_system]="${KIBANA_SYSTEM_PASSWORD:-}"
[monitoring_internal]="${MONITORING_INTERNAL_PASSWORD:-}"
# add new user
)
- Make Role
// setup/roles/custom_role.json example
{
"cluster": ["manage_index_templates", "monitor", "manage_ilm"],
"indices": [
{
"names": ["logs-generic-default", "logstash-*", "ecs-logstash-*"],
"privileges": ["write", "create", "create_index", "manage", "manage_ilm"]
},
{
"names": ["logstash", "ecs-logstash"],
"privileges": ["write", "manage"]
}
]
}
# entrypoint.sh
declare -A users_roles
users_roles=(
[logstash_internal]='logstash_writer'
[monitoring_internal]='remote_monitoring_collector'
# [Username]='{role name}'
)
declare -A roles_files
roles_files=(
[logstash_writer]='logstash_writer.json'
# [role name] = 'role file path'
)
-
You have to create your own Index Template which is matched with your Data Table.
-
Analyzer / Tokenizer / Filter and Mapping information should be included.
-
You can find composable index template examples under
logstash/templates
.If your elastic version > 7.8.x, you have to use composable format index template. If not, legacy index format is okay. For more information, please see documentation(https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)
- Change
jdbc_connection_string
,jdbc_user
,jdbc_password
. - If you wanna use
sql_last_run
value to prevent duplication of data, you can uselast_run_metadata_path
. It tracks last run data depending ontracking_column
value in Data Table and save value inlast_run_metadata_path
. - Of course, you can make your own logstash pipeline under
logstash/pipeline/
.
$ docker-compose up setup
$ docker-compose up
$ docker-compose down -v