flybody
is an anatomically-detailed body model of the fruit fly Drosophila melanogaster for MuJoCo physics simulator and reinforcement learning applications.
The fly model was developed in a collaborative effort by Google DeepMind and HHMI Janelia Research Campus.
We envision our model as a platform for fruit fly biophysics simulations and for modeling neural control of sensorimotor behavior in an embodied context; see our accompanying publication.
The fruit fly body model lives in this directory. To visualize it, you can drag-and-drop fruitfly.xml
or floor.xml
to MuJoCo's simulate
viewer.
Beginning interacting with the model via Python is as simple as:
from dm_control import mujoco
physics = mujoco.Physics.from_xml_path('flybody/fruitfly/assets/fruitfly.xml') # Load model.
physics.step() # Step simulation.
The quickest way to get started with flybody
is to take a look at a tutorial notebook or .
Also, this notebook shows examples of the flight, walking, and vision-guided flight RL task environments.
To train the fly, try the distributed RL training script, which uses Ray to parallelize the DMPO agent training.
Follow these steps to install flybody
:
-
Clone this repo and create a new conda environment:
git clone https://github.com/TuragaLab/flybody.git cd flybody conda env create -f flybody.yml conda activate flybody
flybody
can be installed in one of the three modes described next. Also, for installation in editable (developer) mode, use the commands as shown. For installation in regular, not editable, mode, drop the-e
flag. -
Core installation: minimal installation for experimenting with the fly model in MuJoCo or prototyping task environments. ML dependencies such as Tensorflow and Acme are not included and policy rollouts and training are not automatically supported.
pip install -e .
-
ML extension (optional): same as core installation, plus ML dependencies (Tensorflow, Acme) to allow running policy networks, e.g. for inference or for training using third-party agents not included in this library.
pip install -e .[tf]
-
Ray training extension (optional): same as core installation and ML extension, plus Ray to also enable distributed policy training in the fly task environments.
pip install -e .[ray]
- Create a new conda environment:
Proceed with installation in one of the three modes (described above):
conda create --name flybody python=3.10 pip ipython cudatoolkit cudnn=8.2.1=cuda11.3_0 conda activate flybody
- Core installation:
pip install git+https://github.com/TuragaLab/flybody.git
- ML extension (optional):
pip install "flybody[tf] @ git+https://github.com/TuragaLab/flybody.git"
- Ray training extension (optional):
pip install "flybody[ray] @ git+https://github.com/TuragaLab/flybody.git"
-
You may need to set MuJoCo rendering environment varibles, e.g.:
export MUJOCO_GL=egl export MUJOCO_EGL_DEVICE_ID=0
-
Also, for the ML and Ray extensions,
LD_LIBRARY_PATH
may require an update, e.g.:export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/path/to/miniconda3/envs/flybody/lib
-
You may want to run
pytest
to test the main components of theflybody
installation.
See our accompanying publication. Thank you for your interest in our fly model:)
@article{flybody,
title = {Whole-body simulation of realistic fruit fly locomotion with
deep reinforcement learning},
author = {Roman Vaxenburg and Igor Siwanowicz and Josh Merel and Alice A Robie and
Carmen Morrow and Guido Novati and Zinovia Stefanidi and Gwyneth M Card and
Michael B Reiser and Matthew M Botvinick and Kristin M Branson and
Yuval Tassa and Srinivas C Turaga},
journal = {bioRxiv},
doi = {https://doi.org/10.1101/2024.03.11.584515},
url = {https://www.biorxiv.org/content/10.1101/2024.03.11.584515},
year = {2024},
}