Layer-Peeled Model

This is the code repository for the PNAS paper Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training. If you use this code for your work, please cite

@article{fang2021exploring,
  title={Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training},
  author={Fang, Cong and He, Hangfeng and Long, Qi and Su, Weijie J},
  journal={Proceedings of the National Academy of Sciences},
  volume={118},
  number={43},
  pages={e2103091118},
  year={2021},
  publisher={National Acad Sciences}
}

Installing Dependencies

Use virtual environment tools (e.g miniconda) to install packages and run experiments
python==3.6.7
pip install -r requirements.txt

Code Organization

The code is organized as follows:

• optimization.py (solving the relaxed convex optimization program of the nonconvex Layer-Peeled Model)
• train_models.py (training deep learning models for minority collapse)
• minority_collapse.py (analyzing minority collapse in deep neural networks)
• train_models_oversampling.py (training deep learning models for the oversampling algorithm)
• minority_collapse_oversampling.py (analyzing minority collapse in the oversampling algorithm)

Change the Dir Path

Change the /path/to/working/dir to your working dir.

Reproducing deep learning experiments

To reproduce the experiments for neural collapse

python train_models.py data_option=$data_option model_option=$model_option t1=10 R=1
python minority_collapse.py data_option=$data_option model_option=$model_option t1=10 R=1
python train_models.py data_option=cifar10 model_option=VGG13 t1=10 R=1 (an example)
python minority_collapse.py data_option=cifar10 model_option=VGG13 t1=10 R=1 (an example)

To reproduce the experiments for minority collapse (Figure 4)

python train_models.py data_option=$data_option model_option=$model_option t1=$t1 R=$R
python minority_collapse.py data_option=$data_option model_option=$model_option t1=$t1 R=$R
python train_models.py data_option=cifar10 model_option=VGG13 t1=5 R=10 (an example)
python minority_collapse.py data_option=cifar10 model_option=VGG13 t1=5 R=10 (an example)

To reproduce the experiments for the oversampling algorithm (Figure 6)

python train_models_oversampling.py data_option=$data_option model_option=$model_option t1=$t1 R=1000 weight_ratio=$weight_ratio
python minority_collapse_oversampling.py data_option=$data_option model_option=$model_option t1=$t1 R=1000 weight_ratio=$weight_ratio
python train_models_oversampling.py data_option=cifar10 model_option=VGG13 t1=5 R=1000 weight_ratio=10 (an example)
python minority_collapse_oversampling.py data_option=cifar10 model_option=VGG13 t1=5 R=1000 weight_ratio=10 (an example)

Reproducing experiments for Layer-Peeled Model

In this part, you need to install cvxpy==1.1.7 successfully (in a new environment), which might be incompatible with dependencies in deep learning experiments.
To reproduce the experiments for the relaxed convex optimization program of the Layer-Peeled Model (Figure 3)

python optimization.py