3. Conditional Generative Adversarial Network (cGAN)

So far, we have seen the impressive generation capabilities of GANs. However, a key limitation is the lack of control over what the GAN generates, for example, generating images of a specific class or outputs from other modalities like text.

(cGANs) address this limitation by introducing an additional conditioning variable y, which can be a class label, text embedding, or any other feature vector derived from a different modality. In this framework, both the generator and the discriminator are conditioned on y:

• Generator: The latent noise vector z is concatenated with y, enabling the model to learn the conditional probability distribution \( p(x|y) \).
• Discriminator: Both the generated/real sample x and the conditioning variable y are provided as input, allowing the model to evaluate whether x is a realistic sample given y.

The objective function for a conditional GAN becomes:

cGANs have been successfully used to generate both unimodal outputs (e.g., generating images from class labels) and multimodal outputs (e.g., generating descriptive tags from images). Here, the author tried to demonstrate automated tagging of images, with multi-label predictions, using conditional adversarial nets to generate a (possibly multi-modal) distribution of tag-vectors conditional on image features. They used the skip-model trained on the dictionary of size 247465 and the convolution networks to generate tags and images generation respectively.

4. Cycle Generative Adversarial Network (CycleGAN)

CycleGAN enables image-to-image translation between two visual domains without requiring paired training examples. When trained on two unpaired image collections, CycleGAN learns to capture the domain-specific characteristics of each and translates between them. The model assumes that there is some underlying shared structure between domains — for example, both being different renderings or styles of the same scene — and seeks to learn a mapping between them.

CycleGAN has become widely popular for tasks like style transfer, object transfiguration, season conversion, and photo enhancement. Its unique capability to perform translations without paired data makes it powerful for real-world applications where aligned datasets are unavailable.

A key insight from CycleGAN is that adversarial loss alone is insufficient to ensure meaningful translations. The model might map multiple inputs to the same output or introduce artifacts. To solve this, CycleGAN introduces the concept of cycle consistency, which enforces that translating from one domain to another and back again should yield the original image ideally. This is analogous to translating a sentence from English to French and then back to English — the result should closely resemble the original sentence.

Adversarial Loss

The adversarial loss ensures that the generated images from one domain are indistinguishable from real images in the target domain.

\[ \mathcal{L}_{GAN}(G, D_Y, X, Y) = \mathbb{E}_{y \sim p_{data}(y)}[\log D_Y(y)] + \mathbb{E}_{x \sim p_{data}(x)}[\log(1 - D_Y(G(x)))] \]

Similarly, for the reverse direction:

\[ \mathcal{L}_{GAN}(F, D_X, Y, X) = \mathbb{E}_{x \sim p_{data}(x)}[\log D_X(x)] + \mathbb{E}_{y \sim p_{data}(y)}[\log(1 - D_X(F(y)))] \]

Cycle Consistency Loss

To preserve the content between translations, CycleGAN uses a cycle consistency loss. This ensures that: \[ F(G(x)) \approx x \quad \text{and} \quad G(F(y)) \approx y \]

The cycle consistency loss is defined as:

\[ \mathcal{L}_{cyc}(G, F) = \mathbb{E}_{x \sim p_{data}(x)}[\|F(G(x)) - x\|_1] + \mathbb{E}_{y \sim p_{data}(y)}[\|G(F(y)) - y\|_1] \]

Notice that our model can be viewed as training two “autoencoders” [20]: we learn one autoencoder F o G : X -> X jointly with another G o F : Y -> Y. However, these autoencoders each have special internal structures: they map an image to itself via an intermediate representation that is a translation of the image into another domain. Such a setup can also be seen as a special case of “adversarial autoencoders”, which use an adversarial loss to train the bottleneck layer of an autoencoder to match an arbitrary tarm get distribution. In our case, the target distribution for the X-> Xautoencoderisthat of the domain Y.

Full Objective

The full objective combines both adversarial and cycle consistency losses:

\[ \mathcal{L}(G, F, D_X, D_Y) = \mathcal{L}_{GAN}(G, D_Y, X, Y) + \mathcal{L}_{GAN}(F, D_X, Y, X) + \lambda \cdot \mathcal{L}_{cyc}(G, F) \]

The optimization problem becomes:

\[ G^*, F^* = \arg\min_{G, F} \max_{D_X, D_Y} \mathcal{L}(G, F, D_X, D_Y) \]

Stabilizing Training with Least Squares Loss

To improve training stability and gradient flow, CycleGAN replaces the binary cross-entropy loss with a least squares loss. This change modifies the generator and discriminator objectives as follows:

• Generator loss: \( \mathbb{E}_{x \sim p_{data}(x)}[(D(G(x)) - 1)^2] \)
• Discriminator loss: \( \mathbb{E}_{y \sim p_{data}(y)}[(D(y) - 1)^2] + \mathbb{E}_{x \sim p_{data}(x)}[(D(G(x)))^2] \)

This formulation leads to smoother gradients, helping the model learn more effectively, and avoids issues like vanishing gradients often seen in vanilla GAN training.

In summary, CycleGAN elegantly solves the unpaired image-to-image translation problem by combining adversarial training with cycle consistency, enabling powerful and flexible cross-domain generation without requiring paired data.