Problem Statement

Although TinyGAN [1] successfully reduces the number of parameters of BigGAN [2], while keeping a competitive performance, the model still requires a respectable amount of computational resources both for training and generating images. The purpose of this project is to investigate whether or not it is possible to generate quality images with an even shallower network than TinyGAN, by leveraging their proposed distillation framework. Due to our limited computational resources, we opted to use the CIFAR-10 [3] dataset, which although has a small spatial size (32x32), it is still complex enough to require a large model to generate quality images. In particular, the current state of the art model in Class-conditional Image Generation on CIFAR-10 is StyleGAN2 [4], which contains more than 20 million trainable parameters. In the following sections we will describe the distillation technique used, along with the selected teacher and student networks.

References

[1] Chang, Ting-Yun, and Chi-Jen Lu. "Tinygan: Distilling biggan for conditional image generation." Proceedings of the Asian Conference on Computer Vision. 2020.

[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database. IEEE Computer Vision and Pattern Recognition (CVPR), 2009.

[3] Krizhevsky, Alex, and Geoffrey Hinton. "Learning multiple layers of features from tiny images." (2009): 7.

[4] Kang, Minguk, et al. "Rebooting acgan: Auxiliary classifier gans with stable training." Advances in Neural Information Processing Systems 34 (2021): 23505-23518.