![elastic_stack_dashboard](https://private-user-images.githubusercontent.com/19597087/342579494-7a2e9686-0b14-4867-bd4d-a67dbab4b289.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUwNDkxODAsIm5iZiI6MTcyNTA0ODg4MCwicGF0aCI6Ii8xOTU5NzA4Ny8zNDI1Nzk0OTQtN2EyZTk2ODYtMGIxNC00ODY3LWJkNGQtYTY3ZGJhYjRiMjg5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMwVDIwMTQ0MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTMxMjFiMDc3MjljOTk0YjY0NDFhOTRjOGE4MDQwNThkNDYxZGQ2YWU5NGM2YjVlNDRhMDEzNDJlODIyMWQ0MzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.WIZLfAM-4Hk_hEToJ936S_MRpZNYLYle0-htGGxF3pQ)
This repository showcases how to centralize and manage logs from microservice applications using the Elastic Stack (Logstash, Elasticsearch, and Kibana) and Loguru for logging within a FastAPI app.
This project is designed for a school management system comprising multiple microservices:
- Students App
- Library Management App
- Faculty App
Each of these microservices can operate independently but are currently integrated using a common API gateway. We use Loguru for logging and the Elastic Stack for centralized log management.
The FastAPI app serves as the core of the school management system, integrating the various microservices.
- Logstash: Collects and processes logs from the FastAPI app.
- Elasticsearch: Stores and indexes logs for efficient searching and analysis.
- Kibana: Provides a web interface for visualizing and exploring logs.
All applications, including the Elastic Stack, are dockerized for easy deployment and management.
To run this project, you need to have Docker installed on your machine.
git clone https://github.com/george-mountain/Centralized-Log-Management.git
cd centralized-log-management
Create a .env
file in the root directory of the project and configure it as shown in the env-sample
file provided in the repository.
Your .env
file should look like this:
ELASTIC_PASSWORD=YOURELASTICPASSWORDHERE #e.g elastic1234
ELASTICSEARCH_USERNAME=YOURELASTICUSERNAMEHERE #e.g elastic
ELASTICSEARCH_PASSWORD=YOURELASTICSEARCHPASSWORDHERE #e.g elastic1234
Use Docker Compose to start all services:
docker-compose up --build
This command will start the FastAPI app, Logstash, Elasticsearch, and Kibana.
The docker-compose.yml
file is configured to:
- Start the Elasticsearch container with security enabled.
- Start the Logstash container to collect logs from the FastAPI app.
- Start the Kibana container to visualize logs.
- Start the FastAPI app for the school management system.
The Logstash configuration file logstash.conf
uses environment variables for credentials, ensuring sensitive information is not hard-coded.
Environment variables for credentials are stored in the .env
file, ensuring they are securely managed and can be easily changed without modifying the codebase.
The FastAPI app will be available at http://localhost:8000
.
The fastapi endpoints are:
http://localhost:8000/docs#/ --- This is the central application endpoint documentation
http://localhost:8000/school-management/student/docs#/ --- This is the student app endpoints documentation
http://localhost:8000/school-management/library/docs#/ --- This is the library app endpoints documentation
http://localhost:8000/school-management/faculty/docs#/ --- This is the faculty app endpoints documentation.
Kibana will be available at http://localhost:5601
. Use the credentials configured in your .env
file to log in.
Note: When you access the above url, you will be promted to login. Use the username and password which you configured
on the .env file for the elastic stack.
Elasticsearch will be available at http://localhost:9200
. Use the credentials configured in your .env
file to access it.
For any questions or issues, please open an issue in the repository