This tool can create many images with articulated cuboids in different scenes. These images are labeled and can be reused and manipulated at specific features as for example the articulatioon or the apperence which can be changed individualy without changing other aspects of the image. The framework and the rendering are achieved through the game engine Unity 3D which communicates with the python scripts via a TCP Socket connnection. Unity receives parameters from the script to build up a specific scene and sends the rendered image back to python.
This repository was created in a project at the research group Computer Vision at the Heidelberg Collaboratory for Image processing.
This read me file contains parts of a a detailed report and the used python code is commented in this Documentation. If you are intrested in this project have a look at these links.
-
Download this repository or
-
Go to the directory you want to clone this project to and copy the following code into you terminal.
```git clone https://github.com/R-Haecker/python_unity_images.git}```
-
Try out the examples given in the next section and modify them as you desire to learn to work with this project.
- You can have a look into the Documentation of the code and the meaning of all functions as well as their parameters here.
You need all of the following packages installed:
- Both Files:
- PIL, numpy, json, logging, os, time
- dataset.py:
- matplotlib, tkinter, math, copy,
- client.py:
- socket, sys, io, subprocess, inspect,
The parameters used in this project are all specified in spherical coordinates and the angles are set in degrees. The coordinated system follows the standard convention as described here. The only difference is that the y axis is named z in Uinty and the z axis is named y.
This annotated example shows important parameters as for instance the parameter phi which specifies the rotation of all objects around the vertical axis. The scale parameters lambda specify the height of every cuboid and the angle theta denotes the angle between two cuboids.
The following code represents a simple example how to use this project. This code creates and plots eight randomly generated images.
Here is the figure of the created images:
import dataset
data = dataset.dataset_cuboids(dataset_name = "simple_example")
dictionaries = []
for i in range(8):
dictionaries.append(data.get_example(save_para = True, save_image = True))
data.exit()
data.plot_images(dictionaries, save_fig = True)
- In the first line the file dataset.py is imported.
- The first thing you want to do is to initialize an object of the class
dataset_cuboids()
.- This starts Unity and connects to it with code from
client.py
. - make sure that the string
dataset_name
does not contain any blank spaces.
- This starts Unity and connects to it with code from
- In this example we create random images with the function
get_example()
.- This function returns an dictionnary with the keys:
index
,parameters
andimage
. - The argumets
save_para
andsave_image
areTrue
. This means that the parameters of the created scene are saved inside a unique folder for your dataset. This folder can be found indata/dataset/
, it is named with a time stamp and the name of your dataset specified indataset_name
. The parameters and and images are saved separately. You can have a look at the saved data of the example above. - This is done eight times and every returned dictionary is save in a list.
- This function returns an dictionnary with the keys:
- If we are done with requesting images you should always close and exit the Unity application and the connection with
exit()
. - At last we want to have a look at the created images with the function
plot_images()
.save_fig = True
means that the resulting figure is saved atdata/figures
.
That is it we have successfully created and saved images with a ground truth.
import dataset
data = dataset.dataset_cuboids(dataset_name = "advanced_example", unique_data_folder = False)
data.reset_index()
dictionaries = []
for i in range(10):
dictionaries.append(data.get_example(save_para=True, save_image=True))
data.exit()
data.plot_images(dictionaries, images_per_row=5, save_fig=True)
data2 = dataset.dataset_cuboids(dataset_name = "advanced_example", unique_data_folder = False)
data2.set_config(total_cuboids=[2,3],same_theta=False, DirectionalLightTheta=[80,90], totalPointLights=None, totalSpotLights=None)
dictionaries2 = []
for i in range(10):
dictionaries2.append(data2.get_example(save_para=True, save_image=True))
data2.exit()
data2.plot_images(dictionaries2, images_per_row=5, save_fig=True, show_index=False)
-
The first block of code is pretty similar to the simple example
-
The initialization differs by one additional argument.
unique_data_folder = True
means that if you later save images or parameters your personal dataset folder will not include a time stamp. This enables another instance asdata2
with the samedataset_name
to use the same dataset folder to store more and different images and parameters.- The folder in
data/dataset/
will now just be named:advanced_example
.
-
Since the simple example was executed befor this example the index is currently at nine. It is stored externally in
data/python/index.txt
. -
With the function
reset_index()
we now set it back to zero. Keep in mind that if you choose to setunique_data_folder = True
and reset the index you can overwrite your old data. -
In the function
plot_images()
we specified the shape of the figure.- Since we created 10 images and
images_per_row = 5
, we now plot 2 rows of each 5 images next to each other.
- Since we created 10 images and
-
Now we create another
dataset_cuboids
instance and save into the same dataset folder. This block of code can also be executed in a different python file. -
We will also use the function
get_example()
, but first we want to personalize our boundaries and settings for the randomly generated images to create different images. -
The intervals, which define in what range a parameter can be generated are saved in the config of the instance of the class
dataset_cubiods()
. -
Every instance has a default config initialized wich can be changed with the function
set_config()
. You should habe a look into the Documentation at the functionset_config
and to learn what the specific parameters mean, you should go to the functionwrite_json_crane()
in theclient_communicator_to_unity
class.- In our example we choose with the argument
total_cuboids = [2,3]
that only two or three cuboids are created on the main "branch". - with
same_theta = False
we specified that all angels between cuboids should be different. - with
totalPointLights=None
there will not be any Pointlights. - with
totalSpotLights=None
there will not be any Spotlights as well. - with
DirectionalLightTheta=[80,90]
we specify that the only light the "Sun" will have the polar angle theta between 80 and 90 degrees which means there will be dawn. Note that the common spherical coordinates are implemented as seen here.
- In our example we choose with the argument
-
This time when using the function
plot_images()
we setshow_index = False
to not show the index displayed on top of the images.
We now created a dataset with two different configs in the same dataset folder.
Here is the figure of images with the default config:
Here is the figure of images with the modified config: