This is an official Github repository for the MobiCom paper "NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices". This project is built upon Google libvpx, Android Exoplayer, and Qualcomm SNPE and consists of C/C++/Java/Python.
[Project homepage] [Paper] [Video]
If you use our work for research, please cite it.
@inproceedings{yeo2020nemo,
title={NEMO: enabling neural-enhanced video streaming on commodity mobile devices},
author={Yeo, Hyunho and Chong, Chan Ju and Jung, Youngmok and Ye, Juncheol and Han, Dongsu},
booktitle={Proceedings of the 26th Annual International Conference on Mobile Computing and Networking},
pages={1--14},
year={2020}
}
Lastly, NEMO is currently protected under the patent and is retricted to be used for the commercial usage.
BY-NC-SA
– Attribution-NonCommercial-ShareAlike
./nemo
├── video # Python: Video downloader/encoder
├── dnn # Python: DNN trainer/converter
├── cache_profile # Python: Anchor point selector
├── player # Java, C/C++: Android video player built upon Exoplayer and the SR-integrated codec
./third_party
├── libvpx # C/C++: SR-integrated codec
- OS: Ubuntu 16.04 or higher versions
- HW: NVIDIA GPU
- Docker: https://docs.docker.com/install/
- NVIDIA docker: https://github.com/NVIDIA/nvidia-docker
- Qualcomm SNPE SDK (v1.40.0): https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/tools (We cannot provide this due to the Qualcom license policy.)
We provide a step-by-step guide with a single video (which content is product review).
All the folloiwing commands must be executed inside the docker.
- Clone the NEMO docker repository
git clone https://github.com/chaos5958/nemo-docker.git
- Build the docker image
cd ${HOME}/nemo-docker
./build.sh
- Run & Attach to the docker
cd ${HOME}/nemo-docker
./run.sh
- Clone the NEMO main repository
git clone --recurse-submodules https://github.com/kaist-ina/nemo.git ${NEMO_CODE_ROOT}
- Download/Setup the Qualcomm SNPE SDK as follow:
./nemo
├── third_party
├── snpe
├── benchmarks
├── bin
├── include
├── lib
├── models
├── share
...
- Download a Youtube video
$NEMO_CODE_ROOT/nemo/tool/script/download_video.sh -c product_review
- Encode the video
$NEMO_CODE_ROOT/nemo/tool/script/encode_video.sh -c product_review
Details are described in this file.
- Train a DNN
$NEMO_CODE_ROOT/nemo/dnn/script/train_video.sh -g 0 -c product_review -q high -i 240 -o 1080
- (Optional) Convert the TF model to the Qualcomm SNPE dlc
$NEMO_CODE_ROOT/nemo/dnn/script/convert_tf_to_snpe.sh -g 0 -c product_review -q high -i 240 -o 1080
- (Optional) Test the dlc on Qualcomm devices
$NEMO_CODE_ROOT/nemo/dnn/script/test_snpe.sh -g 0 -c product_review -q high -r 240 -s 4 -d [device id]
Details are described in this file.
- Setup: Build the SR-integrated codec (x86_64)
$NEMO_CODE_ROOT/nemo/cache_profile/script/setup.sh
- Generate the cache profile using the codec
$NEMO_CODE_ROOT/nemo/cache_profile/script/select_anchor_points.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo
- (Optional) Analyze frame dependencies & frame types
$NEMO_CODE_ROOT/nemo/cache_profile/script/analyze_video.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo
Details are described in this file.
- Setup: Build the SR-integrated codec (arm64-v8)
$NEMO_CODE_ROOT/nemo/test/script/setup_local.sh
- Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/test/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
- Measure the latency
$NEMO_CODE_ROOT/nemo/test/script/measure_latency.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id]
- Measure the quality
$NEMO_CODE_ROOT/nemo/test/script/measure_quality.sh -g 0 -c product_review -q high -i 240 -o 1080 -a nemo_0.5
Details are described in this file.
- Setup: Copy data to mobile devices
$NEMO_CODE_ROOT/nemo/player/script/setup_device.sh -c product_review -q high -r 240 -a nemo_0.5 -d [device id] -a nemo_0.5_16
- Run the NEMO player
- Enable USB debugging and USB install at mobiles.
- Open
$NEMO_CODE_ROOT/nemo/player
using Android Studio. - Set ndk path in local.properties as
ndk.dir=[ndk folder]/android-ndk-14b
; I usedandroid-ndk-14b
. - Build & run it!
App information
* product review: You can choose content.
* 1: it is deprecated.
* high: You can choose DNN quality level.
* Decode-Cache: You can choose Decode, Decode-SR (=per frame super-resolution), Decode-cache (=NEMO).
* None: You can choose total playback time.
* 0.5: You can choose the cache profile, 0.5 represents the quality threshold compared to per-frame super-resolutionl.