Knowledge distillation related

[andynnnnn]

Hello, it's great to see your achievements. I've been working with large CV models similar to Yolo-world and Grounding Dino, training them on my own dataset to obtain pre-trained files. I'd like to know how I can derive pre-trained files with relatively smaller model parameters while still retaining the Zero-Shot detection capability of the models.

[yoshitomo-matsubara]

Hi @andynnnnn

Could you clarify what you mean by

how I can derive pre-trained files with relatively smaller model parameters while still retaining the Zero-Shot detection capability of the models.

?
Are you talking about quantization or pruning?

[andynnnnn]

Hello,

What I mean is that I intend to use knowledge distillation to train a student model based on zero-shot detection models such as Grounding Dino and YOLO-WORLD. My approach is to replace the original backbone, swin_T_224_1k, with a backbone that has significantly fewer parameters. I am wondering which distillation algorithm I should employ for this purpose. Do you have any recommendations? Thank you.

[yoshitomo-matsubara]

I've already answered similar questions in other discussions. Search similar discussions in Discussions page
e.g., #428