Robust way to explore GAN model latent space on web using t-SNE statistical method
GAN explorer example
This is example on how you can create latent space web explorer for any GAN model using js and python
Example is based on three.js library, t-SNE statistical method and vanilla js
t-SNE allows us to position images near other similar-looking images in the 2D coordinate space and three.js library to build a WebGL-powered visualization that can display tens of thousands of images in an interactive 3D environment
Inside examples folder there is celeba_gan_e30.h5
that has been trained on celeba dataset in case someone wants to try and build custom GAN explorer but doesn't have machines to train model
- clone github project and set
githubPages
tofalse
insidemain.js
npm install
command inside project root foldernode app.js
serve app using node
-
Clone the project and run calculate_latent_size.py and set variables inside file that will fit your model this is important for atlases and how your latent space is visualized with three.js
python calculate_latent_size.py
-
Using your GAN model generate latent points and save each as image i.e
# example.py def generate_images(examples, n): for i in range(n): pyplot.imsave(f'./faces/generated_face_{i}.png', examples[i, :, :]) model = load_model('celeba_gan_e30.h5') n = 35301 latent_points = generate_latent_points(100, n) X = model.predict(latent_points) X = (X + 1) / 2.0 generate_images(X, n)
-
Use classify_images.py to create vectorized representation of each image
python classify_images.py 'faces/*'
-
Run create_tsne_projection.py to get t-SNE coordinates
python create_tsne_projection.py
inside file make sure to change latent_size and other variables same goes for other files below -
Split latent space images into equal parts for atlas
python split_latent_space.py
-
Generate atlases and serve the app
python create_atlases.py
by now you should haveimage_tsne_projections.json
andimages_..jpg
those are your atlases next is to create new folder inside/src/assets/
and copy generated files inside, create new configconfig.js
inside/src/
folder that will fit your model and add insidemain.js
-index.html
option to have your config selectable inside web page
serve app using nodenode app.js
and visitlocalhost:3000
to explore your GAN latent space
python3
node.js
imagemagick
- Port to React.js
- Convert classify_images.py to tf2 and have script only create vectorized representations of each image
- Add 3D visualization
- Allow objects to be interactive for interpolation / zoom / vector arithmetic ... etc
- Dynamic atlas size
- Indication of load progress
- Guide through the visualization
machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/
medium.com/mlearning-ai/latent-space-representation-a-hands-on-tutorial-on-autoencoders-in-tensorflow
douglasduhaime.com/posts/visualizing-tsne-maps-with-three-js
github.com/YaleDHLab/pix-plot
artsexperiments.withgoogle.com/tsnemap
towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d