DCGAN model for Flux v0.10 #207

matsueushi · 2020-02-29T21:06:58Z

This is a DCGAN implementation for Flux v0.10. I know there are already pending pull requests for DCGAN (#47, #111), but they are incompatible with Zygote.

A linear layer is used as the last layer of the discriminator and losses are calculated using logitbinarycrossentropy. This is because the combination of sigmoid and binarycrossentropy may cause numerical issues (FluxML/Flux.jl#914).

It ouputs generated digits for a fixed noise every 1000 iterations. I believe it is helpful to trace its training process :)

0 steps

3000 steps

6000 steps

final result (=9380 steps)

DCGAN model for Flux v0.10

CarloLucibello · 2020-03-01T09:35:37Z

vision/mnist/dcgan.jl

+end
+
+cd(@__DIR__)
+train()


newline here

sorry, I meant a newline at the very end of the file, so that we don't have github complaining with a red arrow

Oh, sorry. I'll fix that

CarloLucibello · 2020-03-01T09:44:46Z

very nice. Is this the architecture from the original paper (or any other)?. If so, can you add a reference in a comment?

CarloLucibello · 2020-03-01T09:47:31Z

you should add the model to the README

CarloLucibello · 2020-03-01T09:51:54Z

Maybe this is the right moment to start reorganizing this repo. We could have a folder structure like this domain/model_dataset/, and each example should come with a Project.toml and a Manifest.toml.
So, this should go under vision/dcgan_mnist/.
Is this too cumbersome? @matsueushi what do you think?

CarloLucibello · 2020-03-01T09:55:31Z

vision/mnist/dcgan.jl

+                @info("Train step $(train_steps), Discriminator loss = $(loss_dscr), Generator loss = $(loss_gen)")
+                # Save generated fake image
+                output_image = create_output_image(gen, fixed_noise)
+                save(@sprintf("dcgan_steps_%06d.png", train_steps), output_image)


you could also add to this PR those figures you produced, it's nice to have a reference

CarloLucibello · 2020-03-01T10:05:42Z

Could you also use https://github.com/JuliaML/MLDatasets.jl instead of Flux.Data.MNIST? It's more future proof since we are going to excise Julia.Data soon

matsueushi · 2020-03-01T17:41:46Z

@CarloLucibello , thank you for reviewing my PR.

very nice. Is this the architecture from the original paper (or any other)?. If so, can you add a reference in a comment?

Basically the architecture follows the DCGAN tutorial for tensorflow (https://www.tensorflow.org/tutorials/generative/dcgan). I would change the hyperparams if necessary.

Maybe this is the right moment to start reorganizing this repo. We could have a folder structure like this domain/model_dataset/, and each example should come with a Project.toml and a Manifest.toml.
So, this should go under vision/dcgan_mnist/.
Is this too cumbersome? @matsueushi what do you think?

I'm on board. It will be easier to maintain models and update deps.

matsueushi · 2020-03-01T17:44:20Z

I will replace Flux.Data.MNIST by MLDatasets

CarloLucibello · 2020-03-01T17:55:27Z

vision/dcgan_mnist/dcgan.jl

+using Printf
+
+const BATCH_SIZE = 128
+const NOISE_DIM = 100


maybe LATENT_DIM sounds better here

CarloLucibello · 2020-03-01T18:07:03Z

I very much like the overall style, this would be a nice template for the other models.
My only perplexity is on the use of global variables: I have a REPL based workflow and typically have all parameters as keyword arguments to the main function, then pass them around to the various methods, usually grouping them together in a Params struct I create with https://github.com/mauro3/Parameters.jl. Of course, this is just my personal taste, what you did
here is perfectly fine, just wondering what would be the "julian" way to write deep learning scripts that most people would consider as convenient templates for their code

matsueushi · 2020-03-01T20:31:02Z

I used global variables just because they are used in other vision models. So if it is time to change, I would switch them to Params.

matsueushi · 2020-03-01T22:50:58Z

Params is already used by Zygote (https://github.com/FluxML/Zygote.jl/blob/6f17f8b5c40e48e6d7732afb91dba9a1ddac145b/src/compiler/interface.jl#L53-L57) so I used HyperParams for struct

CarloLucibello · 2020-03-01T23:05:23Z

vision/dcgan_mnist/dcgan.jl

+
+function train()
+    # Model Parameters
+    hparams = HyperParams()


here we can do:

function train(; kws...) hparams = HyperParams(; kws...) ....

CarloLucibello · 2020-03-01T23:16:16Z

vision/dcgan_mnist/dcgan.jl

+    # Load MNIST dataset
+    images, _ = MLDatasets.MNIST.traindata(Float32)
+    # Normalize to [-1, 1] and convert it to WHCN
+    image_tensor = permutedims(reshape(@.(2f0 * images - 1f0), 28, 28, 1, :), (2, 1, 3, 4))


ouch, the permutedims is a bit annoying but I guess we cannot do much about it without breaking changes in MLDatasets

good catch anyways

CarloLucibello · 2020-03-01T23:19:39Z

could you commit the pngs as well? then I think we are good to go

vision/dcgan_mnist/dcgan.jl

Co-Authored-By: Carlo Lucibello <[email protected]>

matsueushi · 2020-03-01T23:24:42Z

could you commit the pngs as well? then I think we are good to go

Sure, I will rerun a whole training process again to make sure everything works fine and add ouput images

matsueushi · 2020-03-01T23:56:58Z

0 steps

3000 steps

6000 steps

9380 steps

CarloLucibello · 2020-03-02T00:14:58Z

vision/dcgan_mnist/dcgan.jl

+generator_loss(fake_output) = mean(logitbinarycrossentropy.(fake_output, 1f0))
+
+function train_discriminator!(gen, dscr, batch, opt_dscr, hparams)
+    noise = randn(Float32, hparams.latent_dim, hparams.batch_size) |> gpu


this is not ideal, we should avoid data transfer and use CuArrays.randn if training on gpu. I don't know what would be an elegant way to do that, maybe we can revisit in the future

we might write something like

randn!(similar(batch, (hparams.latent_dim, hparams.batch_size)))

https://docs.julialang.org/en/v1/stdlib/Random/#Random.randn!, https://github.com/JuliaGPU/CuArrays.jl/blob/da2389f0db46d611a5b625c2ce6bba79aa20163f/src/rand/random.jl#L88-L95

CarloLucibello · 2020-03-02T00:26:00Z

vision/dcgan_mnist/dcgan.jl

+        for batch in data
+            # Update discriminator and generator
+            loss_dscr = train_discriminator!(gen, dscr, batch, opt_dscr, hparams)
+            loss_gen = train_generator!(gen, dscr, batch, opt_gen, hparams)


here part of the computation done in train_discriminator! could be reused in train_generator!, i.e. in principle one need only a single forward of the generator.

This is also done here https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html

I don't know how to do it properly with zygote, needs some thinking. If you have a simple solution in mind you could add this optimization, otherwise we can revisit in the future

Unfortunately I don't know how to do it, too.

CarloLucibello · 2020-03-02T00:43:10Z

better move the images to a subfolder

CarloLucibello · 2020-03-02T00:49:32Z

last tiny detail, consider renaming batch to x

matsueushi · 2020-03-02T03:19:29Z

Thanks, I applied the changes

CarloLucibello · 2020-03-02T06:20:05Z

terrific works, thanks!

matsueushi · 2020-03-02T12:01:21Z

@CarloLucibello, thank you for your awesome detailed review!

matsueushi added 2 commits February 29, 2020 17:46

Add DCGAN model

0cd7a7e

DCGAN model for Flux v0.10

Readme, clean up

5561af6

CarloLucibello reviewed Mar 1, 2020

View reviewed changes

matsueushi added 2 commits March 1, 2020 16:38

New line, reference to sample outputs

c75aafb

Add project files

4f0fd33

CarloLucibello reviewed Mar 1, 2020

View reviewed changes

CarloLucibello mentioned this pull request Mar 1, 2020

This commit fixes #191 #205

Open

matsueushi added 2 commits March 1, 2020 22:05

Load MNIST using MLDatasets

7a53854

Use Parameters.jl to hold hyperparameters

f48c7dd

CarloLucibello reviewed Mar 1, 2020

View reviewed changes

Allow keyword arguments

81eb020

CarloLucibello reviewed Mar 1, 2020

View reviewed changes

vision/dcgan_mnist/dcgan.jl Outdated Show resolved Hide resolved

Update vision/dcgan_mnist/dcgan.jl

038d869

Co-Authored-By: Carlo Lucibello <[email protected]>

Add output images

0702d4b

CarloLucibello reviewed Mar 2, 2020

View reviewed changes

Move images to subfolder, avoid data transfer

67d7d09

CarloLucibello merged commit 5c289c9 into FluxML:master Mar 2, 2020

CarloLucibello mentioned this pull request Mar 2, 2020

Optimize DCGAN implementation #208

Open

matsueushi deleted the mnist-dcgan branch March 2, 2020 12:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DCGAN model for Flux v0.10 #207

DCGAN model for Flux v0.10 #207

matsueushi commented Feb 29, 2020

CarloLucibello Mar 1, 2020

CarloLucibello Mar 1, 2020

matsueushi Mar 1, 2020

CarloLucibello commented Mar 1, 2020

CarloLucibello commented Mar 1, 2020

CarloLucibello commented Mar 1, 2020 •

edited

Loading

CarloLucibello Mar 1, 2020

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020

matsueushi commented Mar 1, 2020

CarloLucibello Mar 1, 2020

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020 •

edited

Loading

matsueushi commented Mar 1, 2020

CarloLucibello Mar 1, 2020

matsueushi Mar 1, 2020

CarloLucibello Mar 1, 2020

CarloLucibello Mar 1, 2020

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020

matsueushi commented Mar 1, 2020

CarloLucibello Mar 2, 2020

matsueushi Mar 2, 2020

CarloLucibello Mar 2, 2020

CarloLucibello Mar 2, 2020

matsueushi Mar 2, 2020

CarloLucibello commented Mar 2, 2020

CarloLucibello commented Mar 2, 2020

matsueushi commented Mar 2, 2020

CarloLucibello commented Mar 2, 2020

matsueushi commented Mar 2, 2020

DCGAN model for Flux v0.10 #207

DCGAN model for Flux v0.10 #207

Conversation

matsueushi commented Feb 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarloLucibello commented Mar 1, 2020

CarloLucibello commented Mar 1, 2020

CarloLucibello commented Mar 1, 2020 • edited Loading

Choose a reason for hiding this comment

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020

matsueushi commented Mar 1, 2020

Choose a reason for hiding this comment

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020 • edited Loading

matsueushi commented Mar 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarloLucibello commented Mar 1, 2020

matsueushi commented Mar 1, 2020

matsueushi commented Mar 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarloLucibello commented Mar 2, 2020

CarloLucibello commented Mar 2, 2020

matsueushi commented Mar 2, 2020

CarloLucibello commented Mar 2, 2020

matsueushi commented Mar 2, 2020

CarloLucibello commented Mar 1, 2020 •

edited

Loading

matsueushi commented Mar 1, 2020 •

edited

Loading