A codebase with examples of visual-inertial odometry, mesh-based obstacle avoidance, underwater image enhancement, and stereo depth estimation.
I spent about 15 months working on a startup called Blue Meadow. Our original idea was to develop an autonomous robot for monitoring and performing tasks on offshore aquaculture farms (primarily seaweed and oysters). This would help farmers detect disease, optimize growing conditions, and reduce labor costs through fewer visits to the farm site.
Most ocean robots are extremely expensive due to their reliance on acoustic navigation (e.g sidescan sonar). One of our main goals was to reduce hardware cost by adopting a visual-inertial approach to navigation, which relies on a cheap IMU and camera. To help the vehicle constrain its absolute position in the world, it also receives range measurements from one or more acoustic beacons attached to the farm.
Eventually, we moved away from the idea of a mobile task-automating robot, and started developing and simpler static sensor package. Since I spent a significant amount of time developing a vision-based navigation system for our original idea, I thought I'd make the code public in case it's useful or interesting to others.
NOTE: This codebase is not in a very user-friendly state right now, but I'm working on making it easier for others to use. If there are particular modules you're interested, let me know and I can prioritize those.
- Underwater navigation a simulated ocean environment
- Hybrid smoother/filter state estimator demo
- Stereo object meshing algorithm (simulated)
- Stereo object meshing algorithm (real world; ZED camera)
The main software modules are located in src/vehicle
:
core
: widely used math and data typesdataset
: classes for working with a few underwater stereo datasetsfeature_tracking
: classes for sparse feature detection, optical flow, and stereo matchingimaging
: underwater image enhancement algorithmslcm_util
: utils to convert between internal C++ types and LCM typesmesher
: applies Delaunay triangulation to tracked stereo features in order to approximate local obstaclesparams
: a home-grown system for loading params into C++ classespatchmatch_gpu
: faster CUDA implementation of the Patchmatch stereo algorithmrrt
: ignore; not fully implemented or testedstereo
: classic OpenCV block matching and a Patchmatch implementationvio
: a full stereo visual odometry pipeline, using GTSAM as a backendvision_core
: widely used computer vision types
Most of these modules have correspond tests in the test
directory.
If you're taking a quick glance at this codebase, the modules I'm most proud of are vio
, mesher
, and patchmatch_gpu
.
We use a homegrown system for loading YAML configuration files into C++ classes, allowing parameters to change without recompiling. It also avoids having to write massive contructors for classes, or set lots of class members manually.
The YAML configuration files in the config
folder. For a guide on how our param system is designed, see src/vehicle/params/README.md
.
We use the LCM library for communicating across processes and languages. This allows us to define a message type once, and generate bindings in C++
, Python
, and our C#
Unity simulator.
See lcmtypes
for message type definitions.
- Make repository public
- Add better demos and pictures of outputs
- Stop using
catkin
; switch tocmake
and make build more lightweight - Improve setup/build/demo documentation
- Add documentation to each module
- Remove abandoned modules
GTSAM is the Georgia-Tech Smoothing and Mapping library. It's used widely in the robotics community to represent and solve problems as factor graphs.
I'm working off of a fork of GTSAM here. It has a couple factors that aren't fixed/merged into the GTSAM develop branch yet.
To install this dependency:
- Clone the fork of GTSAM
git checkout develop
just to be safemkdir build && cd build && cmake .. && make
- Can run tests with
make check
sudo make install
to put the custom library into/usr/local
- This repo should include from and link against the installed fork version
I had to install Java before building and installing LCM from source to get the lcm-spy
tool.
sudo apt install openjdk-8-jre
sudo apt install openjdk-8-jdk
- Clone our fork of LCM
- This has a fix for
lcm-spy
(latest version1.4.0
fails on Ubuntu 18.04) mkdir build && cd build && cmake .. && sudo make install
Use the usual cmake
process:
mkdir build && cd build
cmake ..
make -j8
- https://github.com/ethz-asl/eigen_catkin/wiki/Eigen-Memory-Issues#mixed-use-of-StdVector
- I've run into a boatload of issues when using Eigen types and structs containing Eigen types with
std::vector
,std::make_shared
and other memory-allocating things. - DO NOT include `eigen3/Eigen/StdVector> and try to use that workaround. It only caused lower-level bugs to show up.
- DO use
EIGEN_MAKE_ALIGNED_OPERATOR_NEW
in any struct that has an Eigen type member std::bad_alloc
exceptions seem to implicate an Eigen type allocation issue