Built a Scalable Data Science Pipeline in Spark using PySpark for predicting if a PC has Malware or not. Data consistes of information about the hardware and Anti Virus Softwares installed in the Machine.
This dataset is available on kaggle: https://www.kaggle.com/c/microsoft-malware-prediction
1. Gradient Boosting Decision Tree Classifier
2. Random Forest Classifier
3. Logistic Regression
a. Data Cleaning
b. Feature Engineering
c. Feature Selection
d. Data Encoding
a. Identification of correlated features
b. Data Exploring
c. For Model Interpretation
a. Build Default models as baseline models
b. Experiment with hyperameters (to identify extreme limits of these hyperparameters for overfitting)
c. Tune the models by Grid Search using Cross Validation (Based on the resonable parameter grid identified by experiments)
d. Evaluate all the models in each step using AUC (best metric for binary classification of balanced classes)
a. Obtain Feature Importances from each model
b. Identify to top 10 important features
c. Analyse those features to give recommendations