MVL Stereo Processor
~~~~~~~~~~~~~~~~~~~~~~~~~~
(C) 2014-2016 Rok Mandeljc <[email protected]>
Machine Vision Laboratory
University of Ljubljana, Faculty of Electrical Engineering


1. Introduction
~~~~~~~~~~~~~~~
Machine Vision Laboratory (MVL) Stereo Processor is a command-line
stereo video processing utility, based on the stereo pipeline elements
from MVL Stereo Toolbox.

It supports processing of individual frames or sequences of frames,
and combined left/right video streams, outputting extracted frames,
rectified images, computed disparity maps and reprojected points
for each frame.


2. Installation
~~~~~~~~~~~~~~~

MVL Stereo Video Processor has following dependencies:
- C++ compiler and CMake
- libmvl_stereo_pipeline library from MVL Stereo Toolbox >= 2.0.0
- Qt >= 5.6
- OpenCV >= 3.0.0

To build the program, use cmake; in unpacked source directory, issue:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local
make
make install

The above sequence configures, builds and installs the program into
/usr/local/bin.

If MVL Stereo Toolbox is installed in a path not searched by CMake
by default (e.g., /usr/local prefix), you need to supply the path to
CMake config file via CMake switch:
-DCMAKE_PREFIX_PATH=/usr/local/lib64/cmake
or directly via:
-Dlibmvl_stereo_pipeline_DIR=/usr/local/lib64/cmake


3. Use
~~~~~~

The program supports several operation modes, depending on provided
arguments. Below we show some illustrative examples; for full list of
command-line options, consult the output of program run with --help option.

3.1 Input data types
~~~~~~~~~~~~~~~~~~~~

The program supports three types of input data, all of which are
specified as an input-file string. The program attempts to auto-detect
the input data type based on the file name suffix; if you wish to force
a certain type, use the --input-type switch.

Input types:
- image sequence: given as a filename format string, in which placeholders
  %{f} and %{s} will be replaced with frame number and camera side label
  ('L' and 'R'), respectively. The frame number in the filename can be
  zero-padded, and width can be specified in the token itself. For
  exampe, placeholder token %{f|04d} will be substituted with 0000, 0001,
  and so on.
- video file: in this case, each frame is assumed to contain left and
  right image in side-by-side configuration (i.e., the frame is split
  horizontally in half)
- VRMS video: private video format used by our project. Enabled only if
  corresponding library is available.


3.2 Frames
~~~~~~~~~~

To enable export of original frames (for example, from a video file),
use the --output-frames switch, followed by the output name format.
Left and right images are stored in separate files, whose full names
are determined via %{f} (frame number) and %{s} (side; 'L' or 'R')
placeholders.

Images can be stored in any format supported by OpenCV's imwrite()
function.

NOTES (apply to all subsequent output formats as well):
- the program will try to create any parent directories in the output
  image path.
- each --output-* switch can be specified multiple times to enable
  simultaneous export to multiple formats

In the following example, left and right frames are extracted from
a video file, and stored twice; once as JPEGs and once as PNGs. The
JPEG image files are named:
/tmp/frames-jpg/0L.jpg, /tmp/frames-jpg/0R.jpg,
/tmp/frames-jpg/1L.jpg, /tmp/frames-jpg/1R.jpg, and so on.
The the PNG images, on the other hand, are named:
/tmp/frames-png/0000L.png, /tmp/frames-png/0000R.png,
/tmp/frames-png/0001L.png, /tmp/frames-png/0001R.png, and so on.

mvl-stereo-processor \
    /tmp/input-video.avi \
    --output-frames="/tmp/frames-jpg/%{f}%{s}.jpg" \
    --output-frames="/tmp/frames-png/%{f|04d}%{s}.png"


3.2 Rectified images
~~~~~~~~~~~~~~~~~~~~

In order to obtain rectified images, stereo calibration file, exported
from MVL Stereo Toolbox, needs to be provided via --stereo-calibration
option.

The format for rectified images is specified by --output-rectified. Left
and right images are stored in separate files, whose full names
are determined via %{f} (frame number) and %{s} (side; 'L' or 'R')
placeholders.

In the following example, a video file is used as an input, producing
a series of rectified images named /tmp/rectified/0L.jpg,
/tmp/rectified/0R.jpg, /tmp/rectified/1L.jpg, /tmp/rectified/1R.jpg, etc.

mvl-stereo-processor \
    /tmp/input-video.avi \
    --stereo-calibration=/tmp/stereo-calibration.yaml \
    --output-rectified="/tmp/rectified/%{f}%{s}.jpg"


3.3 Frame range
~~~~~~~~~~~~~~~

It is also possible to specify a range of frames to be processed, using
--frame-range option. The argument consists of either two or three
numbers, separated by colon (:), i.e., start:end or start:step:end.
The 'start' or 'end' values can be ommited, implying that processing
should start at the beginning, and end at the end of the sequence.

The --frame-range option can be specified multiple times to enable
processing of multiple frame ranges.

The following example processes frames from 0 to 100, frames from
500 to 1000 with step 10, and frames from 5000 to the end with step
100.

mvl-stereo-processor \
    "/tmp/frames-png/%{f|04d}%{s}.png \
    --stereo-calibration=/tmp/stereo-calibration.yaml \
    --output-rectified="/tmp/rectified/%{f}%{s}.jpg" \
    --frame-range :99 \
    --frame-range 500:10:1000 \
    --frame-range 5000:100:


3.4 Disparity
~~~~~~~~~~~~~

In order to obtain disparity maps, stereo method configuration, exported
from MVL Stereo Toolbox, must be provided via --stereo-method option.
In order to compute stereo disparity, rectified images are needed, however,
stereo calibration can be omitted if the input frames are already rectified.

The disparity maps are stored in a format governed by --output-disparity
option. If the resulting file name has a valid image format suffix, disparity
is stored as a color-coded image. If the suffix is .yml, .yaml or .xml, then
OpenCV's cv::FileStorage is used to serialize the resulting matrix. If
.bin suffix is used, the custom binary format implemented by MVL Stereo Toolbox
is used instead.

Note that the stereo method plugin to use is determined from the provided
stereo method configuration file.

mvl-stereoo-processor \
    /tmp/input-video.avi \
    --stereo-calibration=/tmp/stereo-calibration.yaml \
    --stereo-method-config=/tmp/stereo-method-bm.yaml \
    --output-disparity="/tmp/disparity/%{f|02d}.bin"


3.5 Reprojected points
~~~~~~~~~~~~~~~~~~~~~~

In order to obtain reprojected points, both stereo calibration
(--stereo-calibration) and stereo method configuration (--stereo-method)
must be provided.

The output format is governed by --output-points option; the
filename's suffix can be either .yml, .yaml or .xml (for OpenCV's
cv::FileStorage serialization), .bin (for binary format implemented by
MVL Stereo Toolbox), or .pcd (for PCL point cloud).


In the following example, all outputs are obtained by the same time:

mvl-stereo-processor \
    /tmp/input-video.avi \
    --stereo-calibration=/tmp/stereo-calibration.yaml \
    --stereo-method=/tmp/stereo-method-bm.yaml \
    --output-frames="/tmp/frames/%{f|04d}%{s}.jpg" \
    --output-rectified="/tmp/rectified/%{f|04d}%{s}.jpg" \
    --output-disparity="/tmp/disparity/%{f|04d}.bin" \
    --output-points="/tmp/points/%{f|04d}.bin" \
    --output-points="/tmp/point-cloud/%{f|04d}.pcd"