208 — PixelNN: Example-based Image Synthesis

Many of the existing generative models appear are difficult to interpret or fine-tune, and because the associated loss-functions cannot account for all possible variations in the output image, it is possible to generate a wide variety of very different outputs with little or no control.

Much like StackGAN, this generative model stacks two networks on top of each other, first using a CNN to generate a very smoothed (blurry) representation of an image, and then using a pixelwise nearest-neighbor (PixelNN) to map this smoothed image to a high resolution output.

This is an interesting case in which a generative model is not using a GAN; GANs are so common for this use-case that it’s unexpected to see alternative methods in use. To use a nearest-neighbor in lieu of deep-learning is so uncouth, it’s hip.

Aside from its greater image-to-image predictability, this approach has another great strength: The images that it generates are highly consistent between runs. This means that this model can be used in places where consistency is important; for example, in enabling reproducible analysis on dynamic data.