An application that facilitates monitoring Mutual Fund movement trends. API data is provided by Indian Mutual Fund API. The focus of this repository is managing and scheduling Airflow DAG's which run ETL data pipelins through Spark jobs in a containerized environment. The processed data files are stored in Amazon S3 which are used by a Flask webserver to plot and monitor tends using Plotly.
The following core functionalities are implemeted:
- Creating a Docker containerized enviroment which runs Airflow (webserver, worker, scheduler) and Spark cluster (master and worker nodes)
- Built custom docker images for Airflow and Spark to facilitate interaction of the containers.
- PySpark scripts to run ETL operations via spark jobs on requested mutual funds data.
- Airflow DAG script to schedule and manage PySpark scripts.
- Saving the processed data into an AWS S3 bucket as
.parquet
files for efficient access. - Testing the ETL script with unittests.
- Flask webserver which allows the user to thematically group Mutual funds, initiate data extraction, and plot the data on a dashboard.
For example, Mutual Funds grouped to the theme "technology"
are shown in the Figure
For each mutual fund in the group, a new DAG is initiated in Airflow. The DAG requires the SchemeCode
as an input which is provided through the airflow-client API.
-
mkdir -p ./dags ./logs ./plugins
# Before starting the containers, setup AWS credentials in docker-compose.yml AWS_ACCESS_KEY_ID: <> AWS_SECRET_ACCESS_KEY: <> AWS_DEFAULT_REGION: <>
-
Run
docker-compose up --build
in the project directory -
Access the Airflow web UI at
http://localhost:8080
-
Use the provided Airflow credentials to log in (
airflow:airflow
) -
Explore and manage your DAGs (Directed Acyclic Graphs) in the Airflow web UI
-
Monitor Spark jobs using the Spark web UI at
http://localhost:8000
-
Run Flask webserver
pip install -r requirements.txt cd dashboard python app.py
This Docker Compose file orchestrates a multi-container environment for running Apache Airflow with Spark. The project involves several services, including Apache Airflow components, a PostgreSQL database, a Redis message broker, and Apache Spark components.
-
PostgreSQL Container (
postgres
):- Image:
postgres:13
- Responsible for hosting the PostgreSQL database used by Apache Airflow.
- Credentials:
- Username:
airflow
- Password:
airflow
- Database:
airflow
- Username:
- Healthcheck: Verifies the availability of the PostgreSQL service.
- Image:
-
Redis Container (
redis
):- Image:
redis:latest
- Acts as a message broker for Celery, which is used for distributed task execution in Apache Airflow.
- Healthcheck: Pings the Redis server to ensure its availability.
- Image:
-
Spark Master Container (
spark-master
):- Image: Built from a custom Dockerfile located in the
./spark_image
directory. - Serves as the master node for Apache Spark.
- Exposes ports
8000
for the Spark web UI and7077
for the Spark master service. - Healthcheck: Checks the availability of the Spark web UI.
- Image: Built from a custom Dockerfile located in the
-
Spark Worker Container (
spark-worker
):- Image: Built from the same custom Dockerfile as the Spark Master.
- Acts as a worker node for Apache Spark, connected to the Spark Master.
- Healthcheck: Verifies the availability of the Spark worker.
-
Airflow Webserver Container (
airflow-webserver
):- Image: Built from a custom Dockerfile located in the
./custom_airflow_image
directory. - Runs the Apache Airflow webserver.
- Exposes port
8080
for accessing the Airflow web UI. - Healthcheck: Checks the availability of the Airflow web UI.
- Image: Built from a custom Dockerfile located in the
-
Airflow Scheduler Container (
airflow-scheduler
):- Image: Same as the Airflow Webserver.
- Manages the scheduling of tasks in Apache Airflow.
-
Airflow Worker Container (
airflow-worker
):- Image: Same as the Airflow Webserver.
- Executes tasks distributed by the Airflow scheduler using Celery.
-
Airflow Triggerer Container (
airflow-triggerer
):- Image: Same as the Airflow Webserver.
- Handles triggering of tasks in Apache Airflow.
-
Airflow Initialization Container (
airflow-init
):- Image: Same as the Airflow Webserver.
- Initializes the Apache Airflow environment, including database setup and user creation.
- Executes the
airflow version
command.
-
Airflow CLI Container (
airflow-cli
):- Image: Same as the Airflow Webserver.
- Provides a command-line interface for Apache Airflow tasks.
-
Flower Container (
flower
):- Image: Same as the Airflow Webserver.
- Runs Flower, a real-time web-based monitoring tool for Celery clusters.
- Exposes port
5555
for accessing the Flower web UI. - Healthcheck: Verifies the availability of the Flower web UI.
- Common Environment Variables (
&airflow-common-env
):- Sets up common environment variables shared across Airflow containers.
- Defines executor, database connection, Celery broker, and other configurations.
- Includes AWS access key, secret key, and default region.
- PostgreSQL Database Volume (
postgres-db-volume
):- Persists the PostgreSQL database data to ensure data durability.