README.md

Jupyter experiment viewer

Introduction

"The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more." from jupyter.org

Swan uses Jupyter Notebook to filter, process and visualize results from experiments.

You need to have Docker installed to run Jupyter notebooks easily. We have provided a Docker image that will make running notebooks as easy as possible. You just need to run on your vagrant instance:

docker run -p 127.0.0.1:8888:8888 intelsdi/swan-jupyter

If you're running Cassandra using the cassandra.service file provided in this repository, please use the following command to run Jupyter, so that it will have direct access to Cassandra:

docker run --network="container:cassandra-swan" intelsdi/swan-jupyter 

Additionally Cassandra's container already exposes the necessary port for Jupyter (8888), so there is no need to expose it manually.

Explore the Example Jupyter Notebook

To access Jupyter notebook on your host machine easily add following lines to your ~/.ssh.config file:

Host local-vagrant
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /path/to/swan/vagrant/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL
  ForwardAgent yes
  ForwardX11 yes
  LocalForward 8888 127.0.0.1:8888

Now, by entering ssh local-vagrant you are forwarding jupyter notebook web interface port to your localhost while connecting to your remote machine. Then, from within the Jupyter web interface, open a template notebook by clicking on example.ipynb.

This is very simple notebook that will generate only sensitivity profile for the experiment. The first step is to set the following variables:

• EXPERIMENT_ID is the identifier of the experiment which will be examined

After filling the variables, navigate to the green box using the keyboard arrows so that it points to the first variable and press [Shift] [Enter] to evaluate it. Evaluation actually means executing the code in the box. Evaluate further and observe the output. Experiment object's construction will look like:

# An experiment can now be loaded from the database by its ID.
from swan import Experiment, SensitivityProfile
profile1 = SensitivityProfile(EXPERIMENT_ID, slo=500)

It may take a while since it will retrieve data from Cassandra and store it in the variable profile1.

The next step is to render the sensitivity profile from the loaded samples and draw sensitivity chart. The former will be generated after evaluating:

profile1.latency()

resulting with:

To learn more about Sensitivity Profile read the Sensitivity Experiment README.