extract_features.md

Extract CNN and Faster R-CNN Feature on Your Own

This document provides a walk-through of how we pre-compute ResNet50 features and Faster R-CNN objects features

CNN-pooling features

See video_dialogue_model/extract_features/run_resnet.py

Faster R-CNN objects features

1. Install vqa-maskrcnn-benchmark repository and download the model and config.

   cd data
   wget https://dl.fbaipublicfiles.com/vilbert-multi-task/detectron_model.pth
   wget https://dl.fbaipublicfiles.com/vilbert-multi-task/detectron_config.yaml
   

2. Extract features for images

   See video_dialogue_model/extract_features/run_rcnn.py

   For every x.jpg image, we will get a x.jpg.npy file, which contains all infos generated by Faster R-CNN.

3. Gather all npy file to build objects.mmap files

   See video_dialogue_model/extract_features/build_rcnn_mmap.py