Super Resolution GAN (SRGAN)

Super-Resolution Generative Adversarial Networks (SRGAN) represents an approach to image upscaling that addresses one of the major challenges in computer vision, which is how to recover fine-grained details when enlarging low-resolution images. SRGAN uses adversarial training to generate high-resolution images that preserve textures and patterns often lost in traditional upsampling methods.

Understanding the Problem

Traditional image super-resolution methods, such as bilinear interpolation have drawbacks. They can enlarge image dimensions but often produce overly smooth outputs lacking the fine details of true high-resolution images. This happens because traditional techniques depend on simple mathematical interpolation rather than understanding image structures and patterns.

• They fail to capture textures and sharp edges accurately.
• The smoothing effect reduces the perceived quality of the upscaled images.
• The objective is not only to minimize pixel-wise differences but also to generate images that appear realistic to human viewers.

Architecture Overview

SRGAN follows the classic GAN framework with two competing neural networks: a generator that creates super-resolution images from low-resolution inputs and a discriminator that attempts to distinguish between real high-resolution images and generated super-resolution images. This setup drives the generator to produce increasingly realistic results.

Generator Architecture

The generator employs a residual network (ResNet) architecture instead of traditional deep convolutional networks. This choice is important because residual networks use skip connections that allow gradients to flow more effectively during training, enabling the construction of much deeper networks without the vanishing gradient problem.

The generator consists of 16 residual blocks, each containing two convolutional layers with 3×3 kernels and 64 feature maps. Each convolutional layer is followed by batch normalization and Parametric ReLU (PReLU) activation. Unlike standard ReLU or LeakyReLU, PReLU adapts and learns the slope parameter for negative values, providing better performance with minimal computational overhead.

The upsampling process uses two trained sub-pixel convolution layers that efficiently increase the spatial resolution. Sub-pixel convolution rearranges elements from the channel dimension to spatial dimensions, effectively performing learned upsampling rather than simple interpolation.

Discriminator Architecture

The discriminator follows a structure, using eight convolutional layers with 3×3 kernels. The number of feature maps doubles from 64 to 512 as the spatial resolution decreases through strided convolutions. The architecture concludes with two dense layers and a sigmoid activation function to output a probability indicating whether the input image is real or generated.

Loss Function Design

SRGAN introduces a sophisticated loss function called perceptual loss, which combines content loss and adversarial loss. This combination is essential for achieving both pixel-level accuracy and quality.

Content Loss

Traditional super-resolution methods typically use Mean Squared Error (MSE) as the content loss, which measures pixel-wise differences between generated and target images. However, MSE tends to produce overly smooth images because it averages over all possible high-resolution images that could relate to a given low-resolution input.

l^{SR}_{VGG/i,j} = \frac{1}{W_{i,j} H_{i,j}} \sum_{x=1}^{W_{i,j}} \sum_{y=1}^{H_{i,j}} \left( \left( \phi_{i,j}(I^{HR})_{x,y} - \phi_{i,j}(G_{\theta_G}(I^{LR}))_{x,y} \right)^2 \right)

• l^{SR}_{VGG/i,j}​: Perceptual (VGG) loss at layer (i,j).
• W_{i,j}, H_{i,j}​: Width and height of the VGG feature map, used for normalization.
• \phi_{i,j}​: Feature map extracted from layer (i,j) of the pre-trained VGG network.
• I^{HR}: Ground-truth high-resolution image.
• I^{LR}: Low-resolution input image.
• G_{\theta_G}(I^{LR}): Super-resolved output image generated by the generator GGG.
• (x,y): Spatial position in the feature map.

SRGAN proposes using VGG loss instead, which computes the difference between feature representations extracted from a pre-trained VGG-19 network. This approach focuses on perceptually important features rather than raw pixel values. The VGG loss can be computed at different network depths:

• VGG2,2: Features from the second convolution layer before the second max-pooling (low-level features)
• VGG5,4: Features from the fourth convolution layer before the fifth max-pooling (high-level features)

Adversarial Loss

The adversarial loss encourages the generator to produce images that the discriminator cannot distinguish from real high-resolution images. This loss component is crucial for generating sharp, realistic textures that make the upscaled images visually appealing.

l^{SR}_{Gen} = \sum_{n=1}^{N} -\log D_{\theta_D}(G_{\theta_G}(I^{LR}))

• l^{SR}_{Gen}: Adversarial (generator) loss for super-resolution.
• N: Total number of training samples.
• G_{\theta_G}(I^{LR}): Super-resolved image generated by the generator GGG using low-resolution input I^{LR}.
• D_{\theta_D}(\cdot): Discriminator’s probability that the input image is real.
• -\log D_{\theta_D}(G_{\theta_G}(I^{LR})): Penalizes the generator if the discriminator easily detects the fake image.

Total Loss - Perceptual loss

l^{SR} = l^{SR}_X + 10^{-3} l^{SR}_{Gen}

• l^{SR}: Overall super-resolution loss.
• l^{SR}_X: Content loss (often based on VGG perceptual loss).
• l^{SR}_{Gen}​: Adversarial loss from the generator.

Training Process and Results

During training, high-resolution images are first downsampled to create low-resolution inputs. This adversarial process, involving a generator and a discriminator, progressively improves the realism of the generated images.

• The generator focuses on producing high-resolution images from low-resolution inputs.
• The discriminator evaluates the authenticity of the images, pushing the generator to improve.
• SRGAN delivers superior results in both objective metrics and Mean Opinion Score (MOS).

Limitations and Considerations

SRGAN has several important limitations to consider:

• Training Stability: SRGAN can suffer from training instability, mode collapse or convergence issues. Careful hyperparameter tuning and training monitoring are essential.
• Computational Requirements: The model is computationally intensive, requiring significant GPU memory and training time. Real-time applications may need model compression or specialized hardware.
• Dataset Dependency: Performance heavily depends on the training dataset. The model may not generalize well to image types significantly different from the training data.
• Perceptual vs. Pixel Accuracy Trade-off: While SRGAN produces visually appealing results, it may not achieve the highest pixel-wise accuracy compared to methods optimized purely for MSE.

Practical Applications

SRGAN is widely used in domains such as medical imaging, satellite imagery enhancement and mobile photography. It is especially useful when visual quality takes importance over pixel-perfect accuracy, as in consumer applications where the focus is on improving perceived image quality for viewers.

• Its success has led to several improved variants, including Enhanced SRGAN (ESRGAN) and Real-ESRGAN.
• These advancements continue to set new standards in single-image super-resolution.
• Image upscaling is becoming more practical and accessible across various applications.