Example directory structure:
.
├── code
├── data
│ ├── cifar10
│ │ ├── test
│ │ └── train
│ └── mnist
│ ├── test
│ └── train
├── weights
│ ├── cifar10
│ └── mnist
├── results
│ ├── plots
│ └── raw
└── README.md
- MNIST (
mnist
): Download and extract from https://github.com/myleott/mnist_png/blob/master/mnist_png.tar.gz - CIFAR-10 (
cifar10
): Download and extract from https://www.kaggle.com/swaroopkml/cifar10-pngs-in-folders - Fashion-MNIST (
fmnist
): Download and extract from https://github.com/DeepLenin/fashion-mnist_png/raw/master/data.zip - Run
python3 data2csv.py <dataset_name>
,<dataset_name>
can bemnist/cifar10
- This will create a file
data.csv
in thedata/<dataset_name>
directory - Run
python3 mean_and_std.py <dataset_name>
to compute the channel-wise mean and std of the data and add these values toconfig.py
- Run
python3 split_data.py -d <dataset_name> -s <subset>
- This creates a new dataset in the
data/
directory with the name<dataset_name>_<ext>
with only the data from the specified subset - For example,
python3 split_data -d mnist -s "3,5,6,9"
creates a new datasetmnist_3569
with only the specified classes
- Add model definition to
models
- Change the model used in
run_classifier.py
- Run
python3 run_classifier.py <dataset_name>
- This saves the weights of the trained model in
weights/<dataset_name>
directory
- To add/modify a OOD score, add it as a function in
scores.py
- Change pretrained model path in
test_ood.py
- Run
python3 test_ood.py -i <id_dataset_name> -o <ood_dataset_name> -s <score_name> -n <model_name> -m <model_metric> -e <model_epochs>
- This saves scores for ID and OOD data as
.npy
files inresults/raw
- Run
python3 finetune_ood.py -i <id_dataset_name> -f <finetune_dataset_name> -n <model_name> -m <model_metric> -e <model_epochs>
- This behaves like
run_classifier.py
and saves the model and the test predictions in the same way
- To add/modify a finetuning loss, add it as a
nn.Module
inlosses.py
- Run
python3 analyze_scores.py -i <id_dataset_name> -o <ood_dataset_name> -s <score_name>
- This saves the density plots for the scores in
results/plots
- Create a Google Drive folder having the same root structure and upload the data
- Add a colab notebook to this folder, mount drive
cd
to created folder and run the.py
scripts from notebook command line- Sync to and from Drive when needed
(This work was done as a course project for CS726: Advanced Machine Learning, Spring 2022, IIT Bombay)