Scribd
Upload
36 views

Deep Image Prior

Deep image prior is a convolutional neural network approach that uses the structure of an untrained generator as an implicit prior for solving inverse problems like denoising, super-resolution and inpainting. The network is randomly initialized and optimized to minimize a task-dependent loss function without using external training data. This allows the network to learn image statistics from the target image itself. The approach has been applied successfully to problems like denoising, super-resolution, inpainting and flash-no flash image reconstruction. Several open-source implementations of deep image prior are available.

Uploaded by

joseph458
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
36 views

Deep Image Prior

Deep image prior is a convolutional neural network approach that uses the structure of an untrained generator as an implicit prior for solving inverse problems like denoising, super-resolution and inpainting. The network is randomly initialized and optimized to minimize a task-dependent loss function without using external training data. This allows the network to learn image statistics from the target image itself. The approach has been applied successfully to problems like denoising, super-resolution, inpainting and flash-no flash image reconstruction. Several open-source implementations of deep image prior are available.

Uploaded by

joseph458
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Deep image prior

Deep image prior is a type of convolutional neural network used to enhance a given image with no prior
training data other than the image itself. A neural network is randomly initialized and used as prior to solve
inverse problems such as noise reduction, super-resolution, and inpainting. Image statistics are captured by
the structure of a convolutional image generator rather than by any previously learned capabilities.

Method

Background

Inverse problems such as noise reduction, super-resolution, and inpainting can be formulated as the
optimization task , where is an image, a corrupted representation of
that image, is a task-dependent data term, and R(x) is the regularizer. This forms an energy
minimization problem.

Deep neural networks learn a generator/decoder which maps a random code vector to an
image .

The image corruption method used to generate is selected for the specific application.

Specifics

In this approach, the prior is replaced with the implicit prior captured by the neural network (where
for images that can be produced by a deep neural networks and otherwise). This
yields the equation for the minimizer and the result of the optimization
process .

The minimizer (typically a gradient descent) starts from a randomly initialized parameters and descends
into a local best result to yield the restoration function.

Overfitting

A parameter θ may be used to recover any image, including its noise. However, the network is reluctant to
pick up noise because it contains high impedance while useful signal offers low impedance. This results in
the θ parameter approaching a good-looking local optimum so long as the number of iterations in the
optimization process remains low enough not to overfit data.

Deep Neural Network Model

Typically, the deep neural network model for deep image prior uses a U-Net like model without the skip
connections that connect the encoder blocks with the decoder blocks. The authors in their paper mention
that "Our findings here (and in other similar comparisons) seem to suggest that having deeper architecture is
beneficial, and that having skip-connections that work so well for recognition tasks (such as semantic
segmentation) is highly detrimental."[1]

Applications

Denoising

The principle of denoising is to recover an image from a noisy observation , where . The
[2]
distribution is sometimes known (e.g.: profiling sensor and photon noise ) and may optionally be
incorporated into the model, though this process works well in blind denoising.

The quadratic energy function is used as the data term, plugging it into the
equation for yields the optimization problem .

Super-resolution

Super-resolution is used to generate a higher resolution version of image x. The data term is set to
where d(·) is a downsampling operator such as Lanczos that decimates the
image by a factor t.

Inpainting

Inpainting is used to reconstruct a missing area in an image . These missing pixels are defined as the
binary mask . The data term is defined as (where is
the Hadamard product).

The intuition behind this is that the loss is computed only on the known pixels in the image, and the
network is going to learn enough about the image to fill in unknown parts of the image even though the
computed loss doesn't include those pixels. This strategy is used to remove image watermarks by treating
the watermark as missing pixels in the image.

Flash–no-flash reconstruction

This approach may be extended to multiple images. A straightforward example mentioned by the author is
the reconstruction of an image to obtain natural light and clarity from a flash–no-flash pair. Video
reconstruction is possible but it requires optimizations to take into account the spatial differences.

Implementations
A reference implementation rewritten in Python 3.6 with the PyTorch 0.4.0 library was
released by the author under the Apache 2.0 license: deep-image-prior (https://github.com/D
mitryUlyanov/deep-image-prior) [3]
A TensorFlow-based implementation written in Python 2 and released under the CC-SA 3.0
license: deep-image-prior-tensorflow (https://github.com/beala/deep-image-prior-tensorflow)
A Keras-based implementation written in Python 2 and released under the GPLv3:
machine_learning_denoising (https://github.com/rodgzilla/machine_learning_denoising)

References
1. https://sites.skoltech.ru/app/data/uploads/sites/25/2018/04/deep_image_prior.pdf
2. jo (2012-12-11). "profiling sensor and photon noise .. and how to get rid of it" (http://www.dar
ktable.org/2012/12/profiling-sensor-and-photon-noise/). darktable.
3. "DmitryUlyanov/Deep-image-prior" (https://github.com/DmitryUlyanov/deep-image-prior).
GitHub. 3 June 2021.

Ulyanov, Dmitry; Vedaldi, Andrea; Lempitsky, Victor (30 November 2017). "Deep Image
Prior". arXiv:1711.10925v2 (https://arxiv.org/abs/1711.10925v2) [cs.CV (https://arxiv.org/arch
ive/cs.CV)].

Retrieved from "https://en.wikipedia.org/w/index.php?title=Deep_image_prior&oldid=1152976978"

You might also like