DSND Term 2 Project: Disaster Response Pipeline
Build a Natural Language Processing (NLP) model that classifies messages during disaster event (e.g. an earthquake or hurricane). The model is capable of classifying messages into several categories so that the messages can be directed to the appropriate aid agencies to act.
.
├── README.md
├── app
│ ├── run.py
│ └── templates
│ ├── go.html
│ └── master.html
├── data
│ ├── DisasterResponse.db
│ ├── disaster_categories.csv
│ ├── disaster_messages.csv
│ └── process_data.py
├── models
│ ├── disaster_response_prediction.pkl
│ └── train_classifier.py
└── notebook
├── ETL Pipeline Preparation.ipynb
└── ML Pipeline Preparation.ipynb
-
ETL Pipeline Data cleaning pipeline contained in data/process_data.py:
- Loads the messages and categories datasets
- Merges the two datasets
- Cleans the data
- Stores it in a SQLite database
-
ML Pipeline Machine learning pipeline contained in model/train_classifier.py:
- Loads data from the SQLite database
- Splits the dataset into training and test sets
- Builds a text processing and machine learning pipeline
- Trains and tunes a model using GridSearchCV
- Outputs results on the test set
- Exports the final model as a pickle file
-
Flask Web App is the web application to classify messages into respective categories. The web app also displays visualizations of the data.
- Python 3.6+
- Machine Learning Libraries: NumPy, SciPy, Pandas, Sciki-Learn
- Natural Language Process Libraries: NLTK
- SQLlite Database Libraqries: SQLalchemy
- Model Loading and Saving Library: Joblib
- Web App and Data Visualization: Flask, Plotly
-
Execute the following commands in the project's directory to set up the database, train model and save the model.
-
Run ETL pipeline to clean data and store the processed data in the database.
python data/process_data.py data/disaster_messages.csv data/disaster_categories.csv data/ DisasterResponse.db
-
Run the ML pipeline that loads data from DB, trains classifier and saves the classifier as a pickle file.
python models/train_classifier.py data/DisasterResponse.db models/disaster_response_prediction.pkl
-
Run the following command in the app's directory to run your web app.
python run.py
-
-
Go to http://0.0.0.0:3001/
In the notebooks folder you can find two jupyter notebook that demonstrated how the model is built step by step:
- ETL Preparation Notebook: ETL pipeline implementation
- ML Pipeline Preparation Notebook: Machine Learning Pipeline developed with NLTK and Scikit-Learn
Credit to Figure Eight for the message data.