Submitted to ICML 2023.
To reproduce the results, you will need to install the packages listed in requirements.txt
and have python>=3.7
installed.
Please run pip install -r requirements.txt
to install them.
This section will explain how to reproduce results.
Simply run the code:
python -m src.${method}.train \
--model_name ${model}_${dataset} \
--seed ${seed}
Where the variables you can choose from:
-
method
:GiniSelector
, which will train the model with the cross-entropy (CE
) loss.SelectiveNet
.ConfidNet
.DeepGamblers
.SelfAdaptiveTraining
.
-
model
:vgg16
densenet121
resnet34
-
dataset
:cifar-10
cifar-100
svhn
-
seed
: an integer
The training will save the best weights as well as logs, logits, and scores in the checkpoints/${method}/${model}_${dataset}/${seed}
folder.
Abiding by the principles of open science, we provide the checkpoints for most methods and models on the release page. PS.: The checkpoints for SelectiveNet
are not available due to the size of the files (~30GB in total).
Run the following code to download and unzip a checkpoint from the release:
mkdir checkpoints/
cd checkpoints/
wget -c https://github.com/giniselector/giniselector/releases/download/checkpoints/${method}-${model}_${dataset}.zip
unzip ./${method}-${model}_${dataset}.zip
Or go to the release page and download the checkpoints manually.
Then you can save the logits and scores from a specific method and model by running:
python -m src.${method}.test \
--model_name ${model}_${dataset} \
--seed ${seed}
To evaluate the benchmark, please run:
python -m src.analysis.save_results \
--model_name ${model}_${dataset}
The results will be exported to .csv
in the results/
repository, and the evaluation metrics are defined in src/utils/eval.py
Run the code:
python -m src.analysis.plots \
--model_name ${model}_${dataset}
The images will be saved to the images/
folder.
The post-hoc Gini
score can be plugged into any deep classifier trained with the cross-entropy loss. The function implemented in python that takes the logits as input is defined below.
import torch
def gini(logits: torch.Tensor):
g = torch.sum(torch.softmax(logits, 1) ** 2, 1)
return 1 - g
Please, place the following lines in your .env
file if you want to modify any of the default folders.
#.env
export DATA_DIR=""
export IMAGENET_ROOT=""
export CHECKPOINTS_DIR="checkpoints/"
Where:
DATA_DIR
is the directory where the datasets will be downloaded.IMAGENET_ROOT
is the directory where the ImageNet dataset is located.CHECKPOINTS_DIR
is the directory where the pre-trained models will be placed.