IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO.

3, JULY 2023 903

Underwater Image Enhancement Based on

Zero-Reference Deep Network
Yifan Huang , Fei Yuan , Fengqi Xiao , Jianxiang Lu, and En Cheng

Abstract—Due to underwater light absorption and scattering, in turbid water [1]. In general, light with the red wavelength
underwater images usually suffer from severe color attenuation attenuates fastest in water. In contrast, light with the blue-green
and contrast reduction. Most mainstream underwater image pro- wavelength suffers from the slowest attenuation, causing most
cessing methods based on deep learning require a large amount
of underwater paired training data, leading to a complex network underwater images to appear in blue-green tones [2].
structure, longer training time, and higher computational cost. To In natural lighting scenes, the light reflected from the scene
address this problem, a novel Zero-Reference Deep Network for reaches the camera after being dispersed and absorbed. As
Underwater Image Enhancement (Zero-UIE) is proposed in this shown in Fig. 1, the light reaching camera is divided into three
paper, which transforms the enhancement of an underwater image components: direct light, forward scattered light, and backward
into a specific parameter map estimation by using a deep network.
The underwater curve model based on the classical haze image scattered light [3], [4]. The haze effect is mostly caused by sus-
formation principle is specially designed to remove underwater pended particles such as sand, minerals, and plankton in lakes,
color dispersion and cast. A lightweight deep network is designed rivers, and oceans. When light is reflected from objects toward
to estimate the dynamic adjustment parameters of the underwater the camera, a portion of the light encounters these suspended
curve model, and then adjust the dynamic range of the given particles, resulting in the absorption and scattering effect. It
image pixels according to the model. A set of non-reference loss
functions are designed according to the characteristics of under- usually expands to multiple scattering in an environment without
water images, which can implicitly drive the network learning. blackbody emission, which further disperses the light beam into
In addition, adaptive color compensation can be optionally used homogeneous background light [5].
as the pre-processing step to further improve the robustness and To obtain underwater images with higher visual quality, using
visual performance. The significant contribution of the proposed artificial light sources for active illumination can solve the above
method is zero reference, i.e., it does not require any paired or
unpaired reference data for training. Extensive experiments on var- problems to a certain extent. However, the intensity of the artifi-
ious benchmarks demonstrate that the proposed method is superior cial light source centering at the strongest point would gradually
to state-of-the-art methods subjectively and objectively, which is attenuate along the radial direction, resulting in the uneven
competitive and applicable to diverse underwater conditions. Most background, false contours, and shadows in the acquired images.
importantly, it is an innovative exploration of zero reference for In addition, the artificial light source is limited in turbid water,
underwater image enhancement.
making the subsequent tasks increasingly difficult. Moreover,
Index Terms—Deep network, image enhancement, underwater the actual application cost also needs to be considered. For un-
image, zero reference. derwater tasks, especially the underwater robot tasks, real-time
image enhancement shows broad application prospects [6].
I. INTRODUCTION In addition to artificial light sources, several methods based
on image enhancement, image restoration, and data-driven have
INCE the suspended particles in the water lead to light
S absorption and scattering, the underwater tasks based on
optical vision face enormous challenges. Underwater images
been well explored [7]. Image enhancement methods such as
histogram equalization (HE) [8] and contrast limited adaptive
histogram equalization (CLAHE) [9] aim to adjust pixel values
acquired by the optical vision system suffer from severe degra- to enhance specific qualities of the image, e.g., color, contrast,
dation, e.g., color distortion and contrast reduction. The visible and brightness. Image restoration methods (UDCP) [10] regard
distance in clear water is about 20 m, while it is within 5 m improving image quality as the inverse imaging problem. The
image can be restored by utilizing the physical imaging model
Manuscript received 27 February 2022; revised 27 June 2022, 13 September while exploring image priors as constraints. In recent years,
2022, and 2 November 2022; accepted 7 January 2023. Date of publication 3 due to the powerful modeling capabilities and the character-
April 2023; date of current version 14 July 2023. This work was supported in
part by the National Natural Science Foundation of China under Grant 62071401 istics of learning rich representations from massive training
and Grant 62271425, and in part by the Great Science and Technology Projects data, data-driven methods [11], [12] have achieved outstand-
of Xiamen under Grant 3502Z20231008. (Corresponding author: Fei Yuan.) ing performance in image processing. However, most of these
Associate Editor: R. Cong.
The authors are with the Key Laboratory of Underwater Acoustic Commu- schemes rely on paired data for supervised training, while a few
nication and Marine Information Technology, Ministry of Education, Xiamen unsupervised learning schemes do not require paired data, but
University, Xiamen 361005, China (e-mail: [email protected]; yuan- still need unpaired reference data. Unfortunately, collecting
[email protected]; [email protected]; [email protected].
cn; [email protected]). paired data would introduce high cost, and the image pairs
Digital Object Identifier 10.1109/JOE.2023.3245686 generated by simulation algorithms are different from the real

1558-1691 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 904 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 1. General underwater task scene under natural light.

data, leading to poor generalization capabilities of the network. is then adjusted pixel by pixel based on the proposed underwater
Therefore, the zero-reference learning method is more suitable curve model using the estimated parameter map to remove the
for underwater scenes with small samples and lack of reference underwater color dispersion and cast. Since the image values
data. However, most of the existing zero-reference learning output by UIE-curve may overflow in the channels with high
methods are applied to low-light image enhancement, and few intensity, and the dynamic pixel range in some channels will be
approaches are suitable for underwater image enhancement. The compressed, the normalization layer is added to the network.
low-light enhancement in the air focuses on the illumination The layer transforms the dynamic range of the image to [0,1],
adjustment of images, which cannot be directly applied to un- which increases the dynamic pixel range of channels with com-
derwater scenes due to the severe color cast and haze blurring pressed intensities, and can further stretch the image contrast,
of underwater images. while the seriously attenuated channels can also be re-enhanced.
Therefore, the article presents a novel zero-reference deep In particular, we can learn the adjustable parameters of the
network-based method for underwater image enhancement underwater curve model based on a deep convolutional neural
(Zero-UIE). Inspired by low light curve adjustment, an un- network. The proposed network is lightweight, and the network
derwater curve model is specially designed to transform the training does not require massive amounts of data. Subsequently,
underwater image enhancement task into an estimation problem since the light is severely absorbed and attenuated underwater,
of specific curve parameters, instead of directly performing the the proposed adaptive color compensation can optionally be
end-to-end mapping. The end-to-end task is much more com- used as preprocessing on the channels with severe attenuation,
plex than parameter map estimation. For complex end-to-end thereby compensating the attenuated information in advance.
networks, when only relying on a small number of samples Experiments show that the optional module can further improve
without reference data, the training effect often fails to meet visual performance.
expectations. To better achieve lightweight and zero reference, The significant advantage of the proposed method is exploring
the network needs to be designed as simple as possible for pa- the deep learning-based zero-reference strategy for underwater
rameter map estimation tasks. In this article, the zero-reference image enhancement, i.e., it does not require any paired or un-
learning strategy is explored to estimate the parameter map paired reference data during the training process. To achieve zero
of the underwater curve model, eliminating the requirements reference, a set of nonreference loss functions are specially de-
for paired and unpaired reference data, which can deal with signed according to underwater image characteristics, including
various underwater degradation scenes such as color distortion underwater color constancy loss and underwater color relative
and contrast reduction. dispersion loss. Simultaneously, illumination smoothness loss
The proposed network takes the underwater image as in- is introduced to maintain the monotonic relationship between
put and outputs the estimated parameter feature map of the adjacent pixels.
underwater curve model. The dynamic range of the input image
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 905

The rest of this article is organized as follows. In Section II, prior (ULAP) and an adjusted reversed saturation map (ARSM)
the related work on underwater image processing is introduced. were applied to compensate and modify the transmission map.
In Section III, the underwater imaging model is described. The Next, the attenuation ratio differences between R and G/B
details of the proposed strategy are presented in Section IV. The channels were calculated, thereby estimating the transmission
additional optional module (adaptive color compensation) for map of G/B channels. Finally, white balance was introduced as
the network is illustrated in Section V. In Section VI, experimen- postprocessing. Zhou et al. [24] developed a dehazing method
tal results and discussions are elaborated. Finally, Section VII using the Akkaynak–Treibitz underwater imaging model, which
concludes this article. relies on the scene depth map and color correction method to
eliminate color distortion. First, an underwater image depth
II. RELATED WORK estimation method was designed to create the depth map, and
The existing underwater image processing technologies can then estimated the channel based on the modified model to
be roughly divided into three categories. The enhancement- remove backscattering. In addition, a color correction approach
based and restoration-based methods are manually designed, was proposed to adjust the global color distribution of the image
while the data-driven strategies can automatically learn the automatically. Limited by scene prior or depth information, these
nonlinear mapping function from training data. image restoration methods are greatly restricted in practical
Image enhancement technology aims to improve spe- applications.
cific image quality by directly adjusting pixel values, with- In recent years, data-driven methods have been fully explored
out considering the physical process of underwater image for underwater image processing, whose key is acquiring train-
degradation. Ghani [13] integrated homomorphic filtering, ing sets and the generalization ability of the convolution model.
recursive-overlapped CLAHS, and dual-image wavelet fusion to For image processing, data-driven methods mainly consist of
improve the visibility of deep underwater images. Wong CNN-based and GAN-based methods. Li et al. [25] proposed an
et al. [14] proposed an integrated approach using adaptive gray underwater image enhancement convolutional neural network
world (AGW) and differential gray-levels histogram equaliza- (UWCNN) based on underwater scene prior, which directly
tion for color images (DHECI) to eliminate color cast and im- reconstructed the clear underwater image without estimating
prove image contrast. Inspired by the Retinex framework, Zhang model parameters. Lu et al. [26] proposed an underwater image
et al. [15] extended the multiscale Retinex to enhance underwater color restoration network (UICRN) to obtain real image color
images by combining bilateral and trilateral filters on the three by estimating the main parameters of the underwater imaging
channels of the image in CIELAB color space, according to the model. An encoder neural network was used to extract features
characteristics of each channel. Ancuti et al. [16], [17] proposed from the input underwater image. Then three independent de-
a fusion-based method and achieved impressive performance for coders were used to estimate the direct light transmission map,
underwater image and video enhancement. A multiscale fusion backscattered light transmission map, and background light.
strategy was used to fuse the two images into the final result, Guo et al. [27] proposed a multiscale dense generative adver-
which were directly derived from white-balanced and contrast- sarial network (GAN) for underwater image enhancement. The
enhanced versions. Zhang et al. [18] proposed a method based multiscale dense residual block in the generator can boost the
on subinterval linear transformation to remove the color distor- performance and render details. Spectral normalization was used
tion, and then enhanced image contrast through the bi-interval to stabilize the training of the discriminator. At the same time,
histogram based on optimal equalization threshold strategy and the nonsaturating GAN loss function was designed to constrain
S-shaped function. Image enhancement-based methods can be training. Li et al. [28] proposed an unsupervised generative
simply implemented with low complexity. However, since the adversarial network (WaterGAN), which took in-air images
underwater environment is extremely complex and changeable, and depth pairs as input, and generated synthetic underwater
these methods are not completely suitable for underwater scenes images. Then a color correction network was proposed with
with multiple degradations. the original unlabeled underwater images as input and output
Image restoration technology constructs a mathematical restored underwater images. Wang et al. [29] proposed an unsu-
model to invert the image degradation process and obtain the un- pervised generative adversarial network (UWGAN), based on
degraded image in an ideal state. The most widely used underwa- an improved underwater imaging model to generate realistic
ter imaging model is the Jaffe–McGlamery underwater optical underwater images from in-air images and depth map pairs.
model [3], [4]. To estimate model parameters, many researchers Then U-Net was trained on the synthetic underwater data set
focus on exploring effective image priors to constrain the inverse for color restoration and dehazing. Islam et al. [30] presented a
problem [19], [20]. Wang et al. [21] proposed a method based fast underwater image enhancement method for improved visual
on nonlocal prior, and used an adaptive attenuation curve to perception (FUnIEGAN). A model based on the conditional
simulate the light attenuation process, realizing the restoration generative adversarial network was proposed for real-time un-
of underwater optical images. Peng et al. [22] proposed a depth derwater image enhancement. In addition, seven cameras were
estimation method for underwater scenes based on image blur- used to capture images under various visibility conditions and
riness and light absorption. The image formation model (IFM) contributed to the EUVP data set, which contains a set of paired
was introduced to restore underwater images. Song et al. [23] and unpaired underwater images.
roughly estimated the transmission map of the R channel based Most of the above data-driven schemes are based on super-
on the proposed underwater dark channel prior (NUDCP). Then vised training and paired data generated by simulation algo-
a scene depth map based on the underwater light attenuation rithms. Although a few GAN schemes based on unsupervised
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 906 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 2. Attenuation characteristics of underwater lights with different wavelengths. (a) Attenuation coefficient of lights with different wavelengths. (b) Water
depth can be reached by lights with different wavelengths.

learning do not require paired data, unpaired data is strictly re- frequency, attenuating faster than blue–green light, which causes
quired. Deep learning has made significant progress in modeling most underwater images to appear in blue–green tones.
complex linear systems, but it requires a large amount of training In addition to wavelength, the residual energy ratio is also
data, which is greatly restricted in the deep-sea environment. related to the proportion of salt in the water. According to
Moreover, the supervised scheme is impractical in many cases, the amount of suspended particles and the proportion of salt,
because the collection of paired data incurs high costs, and the seawaters fall into three types: general seawater, turbid tropical-
image pairs generated by simulation algorithms are different subtropical water, and midlatitude water [32]. When the light
from the real data, which may lead to poor generalization capa- beam passes through general seawater per meter, the Rer value
bilities of the network. reaches 82% for red light (700 μm), 95% for green light
Therefore, the article presents a novel underwater image en- (520 μm), and 97.5% for blue light (440 μm). The Rer values
hancement method based on the Zero-UIE. Aiming at the prob- of different underwater environments can be adjusted based
lem of underwater image degradation, a zero-reference learning on the value of general seawater. The Rer values of various
strategy is explored to eliminate the requirements for paired wavelengths are empirically expressed as [33]
and unpaired data. The network is trained by well-designed ⎧
underwater nonreference loss functions, which can be used to ⎪
⎨0.8 ∼ 0.85, if λ = 650 ∼ 750 μm
implicitly evaluate the image quality as well. Rer(λ) = 0.93 ∼ 0.97, if λ = 490 ∼ 550 μm (2)
⎪
⎩
0.95 ∼ 0.99, if λ = 400 ∼ 490 μm.
III. UNDERWATER IMAGING MODEL
Besides the attenuation effect, the light in water still suffers
It is challenging to capture clear images in underwater en-
from additional scattered noise, which causes image dispersion.
vironments, due to the haze effect caused by scattering and
Ignoring the influence of forwarding scattering and camera
color cast caused by varying light attenuation with different
factors, the scattering model can be simplified as [5]
wavelengths. Color scattering and cast introduce blurred details
 
and reduced contrast in underwater images. The degree of im- L (λ, d(x)) = Bλ · 1 − (Rer(λ))d(x) , λ ∈ {R, G, B}
age degradation depends on the size and amount of suspended
(3)
particles in the water, and the absorption and scattering of light
where L(λ, d(x)) represents the scattered light, and Bλ is the
increase with the scene depth. The attenuation model can be
uniform background light. The final underwater degradation
expressed as [5], [31]
model can be obtained by combining the attenuation and scatter-
EO (λ, d(x)) = EI (λ, 0) · (Rer(λ))d(x) , λ ∈ {R, G, B}. ing models. The underwater image Iλ (x) captured by the camera
(1) can be expressed as the weighted sum of the directly attenuated
The above formula represents the attenuated light from the light and the scattered background light [34]
scene to the observation point. The energy of light beam be-
fore and after passing through the medium are EI (λ, 0) and Iλ (x) = EI (λ, 0) · (Rer(λ))d(x)
EO (λ, d(x)) with scene depth d(x). λ is the light wavelength.  
+ Bλ · 1 − (Rer(λ))d(x) , λ ∈ {R, G, B}. (4)
x is a point in the image. Light will attenuate when passing
through the medium. The residual energy ratio Rer represents
Consider the following assumption:
the ratio of residual energy to initial energy per unit distance.
The Rer of different light wavelengths are different in the water. EO (λ, d(x))
tλ (x) = = 10−β(λ)d(x) = (Rer(λ))d(x) (5)
As shown in Fig. 2, red light has longer wavelength and lower EI (λ, 0)

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 907

where tλ (x) represents the residual energy ratio after the light training data. Considering the problem of insufficient image
beam passes through the medium. β(λ) represents the attenua- samples and the difficulty of acquiring paired/unpaired images
tion coefficient, which characterizes the ability of the medium to in certain underwater scenes, this article presents an underwater
scatter light in all directions. Since tλ (x) depends on wavelength image enhancement framework based on a Zero-UIE. Compared
λ and scene depth d(x) between x and camera, it brings color with other deep learning methods, the training process of our
dispersion and cast to the image. Assuming EI (λ, 0) = Jλ (x), approach does not require any reference image.
which represents the reflected light that is directly transmitted, The presented Zero-UIE framework is shown in Fig. 3. A
the model can be expressed as simple nonlinear model curve mapping is utilized to achieve
image enhancement. Instead of end-to-end mapping, the task is
Iλ (x) = Jλ (x) · tλ (x) + Bλ · (1 − tλ (x)) , λ ∈ {R, G, B}. modeled as an estimation problem of the specific image curve
(6) model. The underwater image is input to the underwater image
The above model can be used to eliminate haze and restore enhancement network (UIE-Net), which estimates the parameter
color. The restored underwater image Jλ (x) can be modeled feature map of the underwater curve model. And the dynamic
as [5] range of the input image can be adjusted pixel by pixel according
Iλ (x) − Bλ to the estimated parameter map for color restoration and contrast
Jλ (x) = + Bλ , λ ∈ {R, G, B}. (7)
tλ (x) enhancement. Subsequently, the normalization layer is used to
normalize the dynamic pixel range of the UIE-curve output,
Therefore, the key to restoring underwater image is to estimate which can further enhance the brightness of the compressed
Bλ and tλ (x). The typical method of estimating Bλ is based on image channels. The critical components of Zero-UIE will be in-
the dark channel theory. In particular, the dark channel refers to troduced in detail, including the underwater image enhancement
the phenomenon that the brightness of at least one pixel is close curve model (UIE-curve), UIE-Net, and underwater nonrefer-
to zero in the neighborhood Ω(x) surrounding any given point x ence loss functions. The significant advantage of the proposed
in an outdoor haze-free image. The dark channel Jdark (x) can method is exploring a deep learning-oriented zero-reference
be defined as strategy for underwater image enhancement, which does not

require any paired or unpaired reference data during training. In
Jdark (x) = min min Jλ (y)
λ y∈Ω(x) addition, color compensation can be optionally used as prepro-
cessing on the input underwater image to further improve the
Jdark (x) → 0, λ ∈ {R, G, B}. (8) visual performance. The proposed algorithm exhibits pleasing
Perform the minimum operation in the local patch Ω(x) on visual performance in brightness, color, contrast, and naturalness
the hazy image Iλ (x) by combining deep networks and classical techniques.

min Iλ (y) = min (Jλ (y)) tλ (x)

y∈Ω(x) y∈Ω(x) A. Underwater Curve Model
+ Bλ (1 − tλ (x)) , λ ∈ {R, G, B}. (9) Inspired by curve adjustment in photo editing software and
the Zero-DCE method proposed by Guo et al. [35], this article
The dark channel is obtained by taking the smallest value
attempts to design a curve model suitable for underwater im-
among RGB channels to form a grayscale image, and then per-
ages based on the haze formation model. We follow the curve
forming guided filtering. Subsequently, the first 0.1% brightness
adjustment mode to automatically map the underwater degraded
of the pixels are selected from the dark channel map. Finally,
image to its enhanced version, where the curve parameter feature
corresponding to these selected positions, the point with the
map estimated by the network only depends on the input image.
maximum brightness is extracted in the raw image as the value
The differentiable underwater curve model must be carefully
of uniform background light Bλ . It is worth noting that the
designed to adjust model parameters through the deep network,
estimated background light of the underwater image is a 3-D
while maintaining the range of image values and the contrast
value, including the information of RGB components. It can be
of adjacent pixels. As mentioned in Section III, the classical
seen from the above analysis that after Bλ is obtained, the core
underwater imaging model can be modeled as (6), and the
problem is to estimate tλ (x). Subsequently, the focus is to build
underwater image restoration process can be expressed as (7).
a zero-reference network estimating the parameter map related
Due to the absence of reference information, it is impossible to
to tλ (x) for underwater image enhancement.
directly use the zero-reference network to estimate the transmis-
sion map, which cannot be well constrained. Therefore, inspired
IV. PROPOSED METHOD
by the curve adjustment model in photo editing software, the
It usually costs a great deal to collect enough paired data in underwater haze formation model is deformed into curve form
underwater scenes, whereas the underwater images generated
by simulation are different from the real ones. Therefore, the Jλ (x) = (Iλ (x)tλ (x) − Iλ (x)tλ (x) + Iλ (x)
supervised underwater image enhancement method based on
paired data sets is limited, due to the relatively poor gener- − Bλ (1 − tλ (x))) /tλ (x)
alization ability, additional artifacts, and color cast. Although = Iλ (x) + (Iλ (x) (1 − tλ (x))) /tλ (x)
unsupervised underwater image enhancement does not require
paired data sets, it still requires carefully selected unpaired − (Bλ (1 − tλ (x))) /tλ (x) (10)
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 908 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 3. Framework of Zero-UIE.

Fig. 4. UIE-curves with different adjustment parameters. The red box shows the background light area, and the right shows the corresponding RGB values.
(a) Original image. (b) UIE-curve of R channel. (c) UIE-curve of G channel. (d) UIE-curve of B channel.

where Iλ (x) is the input underwater degraded image, Jλ (x) input Iλ (x). Tλ (x) is the trainable curve parameter map with
indicates the clear image to be restored, Bλ is the background the same size as the given image, which is used to adjust the
light, and tλ (x) represents the transmission map. Define curve amplitude. It can be seen that the relevant terms of the
Tλ (x)Iλ (x) = (tλ (x) − 1)/tλ (x), then transmission map are transformed into the product form of a new
parameter map Tλ (x) and the input image. Since tλ (x) ∈ (0, 1],
Jλ (x) = Iλ (x) − Iλ (x)Tλ (x)Iλ (x) + Bλ Tλ (x)Iλ (x). (11)
Tλ (x) ∈ (−∞, 0].
Compared with the low-illuminance enhancement curve The UIE-curves with different adjustment parameters α (α ∈
LE(x) in the literature [35], the background light Bλ is addi- Tλ ) are shown in Fig. 4. The red box in Fig. 4(a) shows
tionally added to the above formula. Therefore, the underwater the selected background light area and its component values.
curve model (UIE-curve) based on the haze formation principle Fig. 4(b)–(d) shows the UIE-curves corresponding to R, G, and
can be defined as B channels, respectively. Although there are different inflection
points in these curves, the distribution laws are consistent while
U IE(x, λ) = Iλ (x) + Tλ (x)Iλ (x) (Bλ − Iλ (x)) (12)
meeting the requirements of monotonic and differentiable. It
where x represents the pixel coordinates, λ is the light wave- is worth noting that, unlike LE-curve [35], the upper bound
length, and U IE(x, λ) is the enhanced version of the given of the UIE-curve may be greater than 1 before normalization.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 909

In a physical sense, consistent with the imaging model 7,

when the input I(x) is at point 1 (less than the background
light Bλ ), the brightness value of the pixel is lower than the
background light. After the curve adjustment, the dynamic range
of this part of the pixels are compressed and the brightness is
reduced. This is because, under the influence of the haze effect,
the air light formed by the scattering effect of ambient light can
cause various problems, such as decreased saturation, gradual
disappearance of scene colors, and increased brightness values.
Therefore, the area where point 1 is located in the UIE-curve
eliminates the haze effect of the underwater image by reducing
the dynamic range and brightness value of the image pixels.
When the input I(x) is at point 3 (greater than the background
light Bλ ), the brightness value of the pixel is higher than the
background light, which proves that the point is regarded as
Fig. 5. Example of UIE-curve. the brighter target area rather than the background light. After
the curve adjustment, the dynamic range of this part of the pixels
are expanded and the intensity value increases. This is because
In fact, when Bλ < 1, the output value of the UIE-curve may the point has been recognized as the target area, and the area
indeed overflow (e.g., when Iλ (x) > Bλ , the value of U IE(x, λ) where point 3 is located in the UIE-curve enhances the contrast
may be greater than 1). As an extreme example, when Bλ < 1 of the target by expanding the dynamic range and brightness
and Iλ (x) = 1, then U IE(x, λ) ≥ 1. If and only if Tλ (x) = 0, value of the image pixels.
we can obtain U IE(x, λ) = 1. It is noted that the UIE-curve According to the above analysis, the image values output
has a one-to-one correspondence with the boundaries of the by the UIE-curve may have overflow problems in some high-
underwater imaging model 7. intensity channels (such as blue–green channels), and the pixel
The RGB color model uses the RGB pattern to assign the dynamic range of some channels will be compressed, resulting
intensity value in the range [0, 255] to the RGB components in the image being dark. The histogram normalization layer is
of each pixel in the image (the normalized value [0,1] in this added to the network to transform the value of the image channel
article), and the sensor also adopts this standard. In the case of to [0,1]. This can increase the dynamic range and brightness of
output overflow (> 255), the sensor will truncate by default and the pixels with compressed intensities, and reenhance the seri-
sample the pixel value of 255. ously attenuated channel, further improving the image contrast.
For the UIE-curve, the output pixel value may be greater than Since global mapping leads to insufficient or excessive en-
1, but to meet the goals [35]: 1) the curve should be monotonic hancement in local regions, all operations are pixelwise. We
to maintain the contrast of adjacent pixels; 2) in the process can obtain the best fitting curve for each pixel and adjust its
of gradient backpropagation, the form of the curve should be dynamic range separately. And the curve is applied to the RGB
as simple and differentiable as possible, the UIE-curve does channels, which can achieve image enhancement by adjust-
not truncate and normalize before backpropagation. That is, the ing the dynamic range of the input image. In Section IV-B,
entire operation is to keep the original data value. When we a deep network is proposed to estimate the parameter map
store the image, we will first perform a normalization operation Tλ (x), and then adjust the underwater image according to the
to solve the overflow problem, i.e., scale the output result of the UIE-curve.
UIE-curve to the range [0,1]. After normalization, the enhanced
image still satisfies the goal: 3) the value of each pixel in the
enhanced image should be within the normalized range [0,1]. B. Underwater Image Enhancement Network
Take the UIE-curve in Fig. 5 as an example, where I(x) repre- An underwater image enhancement network (UIE-Net) is
sents the input image pixel, U IE(x) represents the output pixel carefully designed to learn the mapping between input images
value after curve adjustment, and α (α ∈ Tλ ) is the adjustable and their best-fitting curve parameter map. The input of UIE-Net
parameter, with the range of (−∞, 0]. The I(x) corresponding to is the underwater degraded image Iλ (x), and it outputs a set of
point 2 is the value of the background light Bλ , and the UIE-curve pixelwise curve parameter map Tλ (x). The UIE-Net consists
for each pixel in the image will meet at this point. of the plain convolutional neural network of ten convolutional
According to [36], if the pixel with the largest value in the layers. Each intermediate convolutional layer is composed of 32
image is used as the background light, it may lead to errors. In convolution kernels size of 3 × 3, with stride = 1, and the ReLU
a real-world image, the brightest pixels may appear in the white activation function is used after each convolutional layer. Some
target areas, not in the background light areas. Therefore, we skip connections are used to introduce the features from shal-
use the dark channel to estimate the background light, which low convolutional layers to obtain rich low-level information.
may not be the brightest point in the entire image, resulting in The batch normalization layer that breaks the relationship of
point 2. neighboring pixels is not included. For T(x) ∈ (−∞, 0], the last

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 910 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 6. Network structure of UIE-Net.

TABLE I
RESOURCE OCCUPANCY OF UIE-NET

convolutional layer is followed by the ReLU activation function, loss Lcol is defined as [38]
and inversion operation is additionally executed. Finally, the
3-layer parameter map is output, corresponding to RGB three Lcol = (μG − μR )2 + (μB − μR )2 + (μG − μB )2
channels. Fig. 6 shows the network structure of UIE-Net. = (μp − μq )2 , ε = {(R, G), (R, B), (G, B)}
The entire network only consists of 104 259 trainable param- ∀(p,q)∈ε
eters. The size of the input and output image is 256 × 256 × 3.
(13)
More detailed network resource information is shown in Table I.
where μp denotes the average intensity value of p channel of the
enhanced image, and (p, q) represents a pair of channels. The
smaller the underwater color constant loss, the closer the average
C. Underwater Nonreference Loss Functions
values of the RGB components, and the closer the output image
To achieve zero-reference learning in UIE-Net, a set of dif- is to the real world.
ferentiable nonreference loss functions are designed to train the 2) Underwater Color Relative Dispersion Loss: Not only
network, aiming at the characteristics of underwater images. the distance of the average intensity reflects the image color
These loss functions can also be implicitly used to evaluate deviation, but also the distance of standard deviation reflects
the image enhancement quality: the underwater color constant the difference of RGB channels. The image standard deviation
loss is used to restore image color, the underwater color relative represents the dispersion degree between the pixel and average
dispersion loss is adopted to normalize the channel differences, value. Therefore, the standard deviation of RGB channels are
and the illumination smoothness loss is utilized to maintain the calculated as σR , σG , σB , and the underwater color relative
monotonic relationship between adjacent pixels. dispersion loss is defined as [38]
1) Underwater Color Constant Loss: Underwater image suf-
fers from serious color cast, so we first focus on solving the color Lcrd = (σG − σR )2 + (σB − σR )2 + (σG − σB )2
restoration problem. Following the color constancy hypothesis = (σp − σq )2 , ε = {(R, G), (R, B), (G, B)}.
based on the gray world [37], the average value of each color
∀(p,q)∈ε
channel in the real-world image tends to be approximately equal
(14)
over the entire image. Based on the assumption that the color
channels of the gray image should be similar, underwater color The smaller the underwater color relative dispersion loss, the
constant loss is designed to assist in correcting the image color. closer the pixel dispersion degrees of RGB components, and the
To decrease the color cast and saliency, the distribution dis- closer the output image is to the real world. In this case, the
tances of RGB channels need to be reduced as much as possible. relative change of the three channels can reach a relatively small
We mainly focus on the image average. The average intensity state.
values of RGB channels are calculated as μR , μG , μB , which 3) Illumination Smoothness Loss: Instead of the end-to-end
reflect the average brightness. The underwater color constant model, the proposed network is used to estimate the curve

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 911

parameter map. Therefore, it is necessary to make constraints on analysis of RGB color components on the collected 127 under-
the parameter map, of which the smoothness can significantly water images is performed, which covers various underwater
affect the image quality. To maintain the monotonic relationship degradation scenes, as shown in Fig. 7.
between adjacent pixels, illumination smoothness loss Ltv is Underwater images with different degradation types exhibit
implemented to constraint the parameter map [35] apparent differences in color appearance, so the statistical anal-
ysis of RGB color components is the main focus. Fig. 8(a) shows
Ltv = (|x Tc | + |y Tc |)2 , ξ = {R, G, B} (15) the average value of the RGB components of each image, and
c∈ξ Fig. 8(b) shows the total average value of the 127 images. The
horizontal axis represents the image number, and the vertical
where x and y represent the gradient operators in the hori- axis represents the intensity value. It can be seen that the red
zontal and vertical directions, respectively. The smaller the Ltv , components of most underwater images are significantly lower
the higher the smoothness of the parameter map. than the blue–green components, which means that the red signal
4) Total Loss: The total loss function can be expressed as suffers from severe attenuation. In contrast, the blue–green
components show different distribution patterns.
Ltotal = Wcol Lcol + Wcrd Lcrd + Wtv Ltv (16)
Furthermore, the statistical results inspired us to propose
where Wcol , Wcrd , and Wtv represent the weights of the losses. an adaptive color compensation strategy. The color compensa-
tion [39] is optionally available as the preprocessing step before
the network, which can compensate the channels suffering from
V. OPTIONAL MODULE severe attenuation and make the image input to the network
Due to the powerful modeling capabilities and rich represen- closer to “gray,” thereby improving the color performance.
tations learned from training data, deep learning has achieved Specifically, adaptive color compensation is implemented
excellent performance in underwater image enhancement. Al- according to the color distribution of the tested image. The
though significant progress has been made, underwater light maximum color channel is used to compensate for the mini-
absorption and attenuation are still enormous challenges. On mum color channel. And optionally, the middle color channel is
the one hand, the complex underwater imaging environment appropriately compensated as well.
makes it impractical to find a universal method that relies only Assuming an underwater degraded image I, the average
on manual design, usually using a simplified model. On the values of the RGB components are calculated as IR , IG , IB .
other hand, the parameters of the deep network will be fixed Then calculate the maximum, middle and minimum values in
after training. Data-driven methods, therefore, lack sufficient (IR , IG , IB ), expressed as Imax , Imid , Imin . Subsequently, the
flexibility to deal with the changeable underwater environment. maximum channel is used to compensate for the minimum
And the deep network cannot be adjusted according to the input channel, because the minimum channel has been severely at-
data different from the training set. Furthermore, the existing tenuated. The first-level compensation strategy for each pixel
data-driven methods require a large number of parameters to can be expressed as [39]
learn complex mapping functions, which limits their potential c1
Imin (i, j) = Imin (i, j) + α (Imax (i, j)
values in practical applications [6]. Especially for the zero
reference tasks, the network cannot be guided to train toward − Imin (i, j)) (1 − Imin (i, j)) Imax (i, j) (17)
the expected direction without reasonable constraints. c1
where Imin (i, j) represents the first compensated channel, and
Therefore, to effectively improve the quality of underwater
α is a constant, which is set to 1 empirically. The compensation
images, a new framework is proposed by combining the ad-
value in the second term is proportional to the difference of mean
vantages of the deep network model and the manual-design
values, and (Imax (i, j) − Imin (i, j)) reflects the attenuation im-
method. In particular, considering deep network cannot cover
balance of these two channels.
all underwater image types, the optional module named adaptive
After completing the first-level compensation, a simple and
color compensation is designed as preprocess to make up for the
effective judgment procedure is undergone to determine the next
network.
color compensation strategy. If Imid < k × (Imax + Imin ), then
The gray world hypothesis [37] holds that the average values
the maximum channel is used to compensate the middle channel
of RGB components tend to have the same gray value for images
(empirically set k = 1/2). In this case, the middle channel also
with significant color changes. In a physical sense, it is assumed
suffers from relatively severe attenuation. Similarly, the second-
that the average reflection value of light in the natural scene is
level compensation strategy can be expressed as
fixed, which is close to gray, so that the influence of ambient
light can be eliminated. Therefore, the additional color compen- c2
Imid (i, j) = Imid (i, j) + α (Imax (i, j)
sation [17] is added as the preprocessing step before inputting
− Imid (i, j)) (1 − Imid (i, j)) Imax (i, j) (18)
to the network, which can preequalize the RGB channels of the
image and improve the visual performance. where c2
Imid (i, j)
represents the second compensated channel.
Most underwater images exhibit problems such as blue– Finally, the two compensated color channels are combined
green color cast, low illumination, turbidity, etc. For various to form the output color-compensated image. Adaptive color
underwater degradation types, the color distribution is different. compensation can be added as the potential preprocessing step
To achieve color restoration more scientifically, the statistical for network input. To ensure fairness with other algorithms,

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 912 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 7. Some examples of underwater images.

Fig. 8. Statistical characteristics analysis on color distribution of underwater images.

preprocessing is not considered when testing on the benchmarks. and 0.02 standard deviation. Bias is initialized as a constant.
In particular, the effectiveness of preprocessing to improve the Adam optimizer with default parameters and fixed learning
visual performance of the network will be additionally demon- rate 1e−4 is adopted. The experiment environment includes
strated. To distinguish whether color compensation is used or NVIDIA GeForce RTX 3090Ti GPU, 256 GB RAM, and Intel
not, the proposed simplified version without color compensation (R) i9-10900 k CPU.
is defined as Zero-UIE, and the improved version with color Several underwater image processing algorithms are used
compensation is defined as Zero-UIE+ . The former is used as comparisons, including two traditional methods, three su-
for testing on various data sets in Sections VI-A to VI-E, and pervised schemes, and a similar unsupervised method: the
the latter is only used as an experimental discussion to further underwater depth estimation and image restoration method
improve the visual performance in Section VI-F. (UDCP) by Drews et al. [10], the underwater image restora-
tion method based on image blurriness and light absorption
(IBLA) by Peng et al. [22], an underwater image enhancement
VI. EXPERIMENT RESULTS AND DISCUSSION network (WaterNet) by Li et al. [40], the enhancing underwater
To improve the generalization performance of the network, imagery using Generative Adversarial Networks (UGAN) by
underwater images with different degradation types are incor- Fabbri et al. [41], an underwater image enhancement method
porated into the training set. Specifically, 400 images randomly via medium transmission-guided multicolor space embedding
selected from the UIEB data set [40] and 2600 underwater (Ucolor) by Li et al. [42], and an unsupervised underwater image
images from laboratory data are used for training (the rest for restoration method (USUIR) by Fu et al. [43]. In particular,
testing). The number of iterations is set to 50. The experiment to ensure fairness with other algorithms, preprocessing is not
is implemented with the PyTorch framework, and the training considered when testing on the benchmarks in Sections VI-A
images are resized to 256 × 256 × 3. The weights of each to VI-E (the proposed version is displayed as Zero-UIE). Abla-
layer are initialized as the Gaussian function with zero mean tion experiments are discussed in Section VI-D. Section VI-E

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 913

Fig. 9. Example of standard color card and tested color card. (a) Standard color card. (b) Tested color card.

Fig. 10. Comparison of color card image experiments. (a) Original image. (b) UDCP [10]. (c) IBLA [22]. (d) WaterNet [40]. (e) UGAN [41]. (f) Ucolor [42].
(g) USUIR [43]. (h) Proposed.

elaborates the time performance. As a further improvement of South China Sea. The algorithm performance can be evaluated
the visual performance, the proposed adaptive color compen- by calculating the full-reference color indicators of the standard
sation can be optionally used as preprocessing (the improved color card image and tested image, which are shown in Fig. 9.
version is displayed as Zero-UIE+ ), which is presented in Sec- The restoration results of the color card experiment are pre-
tion VI-F. sented in Fig. 10. It can be seen that UDCP and IBLA cannot
effectively remove the color cast. Although UGAN and USUIR
can remove part of the color cast, blue-green distortion remains.
A. Evaluation on Color Card Images WaterNet based on traditional technology fusion and Ucolor can
To quantitatively evaluate the color restoration performance correct the color attenuation to a certain extent, but the overall
of the algorithm, underwater color card image experiments images tend to be gray and dark, whose brightness is attenuated.
have been carried out in the South China Sea and swimming Even though the proposed method has few background color
pool, contributing to a set of color card image test data sets. casts remaining, it can restore the color card area well and
Specifically, color card images at 5–15 m are collected in the produce the optimal image contrast.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 914 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

TABLE II
QUANTITATIVE EVALUATION ON COLOR CARD IMAGES, USING ΔC ∗ AND ΔE ∗ METRICS

Simultaneously, two indicators are designed to evaluate the objectively, which can restore more realistic underwater scene
color and brightness of the color card image, including chro- colors and higher contrast.
maticity difference ΔC ∗ and color difference ΔE ∗ [39]. As-
suming the two corresponding color blocks C1 and C2 are B. Evaluation on UIEB Data Sets
selected from the standard color card and tested color card,
To quantitatively evaluate the comprehensive performance
respectively, the three components of C1 and C2 in the CIELab
of the proposed method, the rest 490 underwater images are
color space [44] are represented as L∗1 , a∗1 , b∗1 and L∗2 , a∗2 ,
selected from the UIEB data set [40]. Several nonreference
b∗2 . The pixel values of the uniform area at the center of the
image quality evaluation indicators are selected for performance
24 color patches of the color card are first selected. Subsequently,
evaluation, including commonly used underwater color image
the brightness error ΔL∗ between the tested color card and the
quality evaluation index (UCIQE) [45] and nonreference under-
standard color card, the chromaticity error Δa∗ from red to
water image quality index (UIQM) [46]. The larger the value of
green, and the chromaticity error Δb∗ from yellow to blue are
the indicators, the better the performance of the algorithm.
calculated
It can be seen from Fig. 11 that UDCP cannot effectively
remove the color cast. IBLA produces brightness distortion
ΔL∗ = |L∗1 − L∗2 | , Δa∗ = |a∗1 − a∗2 | , Δb∗ = |b∗1 − b∗2 | .
on specific images. Although UGAN, Ucolor, and USUIR can
(19)
remove haze and blur, some color cast remains. WaterNet can
The chromaticity difference ΔC ∗ and the color difference
achieve color restoration on most images, but fails on several
ΔE ∗ are defined as
certain images, and the results show grayish tones. The proposed
⎧ method can achieve the best color restoration and contrast
⎨ΔC ∗ = (Δa∗ )2 + (Δb∗ )2
enhancement, which perform the most significant target and
(20)
⎩ΔE ∗ = (ΔL∗ )2 + (Δa∗ )2 + (Δb∗ )2 . pleasing brightness.
Table III shows the evaluation results of UIQM and UCIQE
indicators on the single image, as well as the average values on
The smaller the values of the indicator ΔC ∗ and ΔE ∗ , the
the UIEB data set. The “avg.” represents the average value of the
closer the tested color card is to the standard color card, the
several above images, and the “testset” represents the average
better the algorithm performance.
value of the data set. The proposed algorithm can achieve optimal
Table II shows the color card evaluation indicators ΔC ∗ and
or suboptimal results on most images. Although the similar
ΔE ∗ of each tested image and their average values. The red
unsupervised scheme USUIR is able to show acceptable results
font indicates the best value, and the blue font indicates the
on several images, the proposed method can obtain the optimal
suboptimal value. It can be seen that the proposed method can
values on the overall data set.
achieve optimal or suboptimal results on most images, while per-
forming best on the average set. Two supervised-based schemes,
UGAN and Ucolor, also achieve pleasing results on some im- C. Evaluation on RUIE Data Sets
ages. Compared with the similar unsupervised scheme USUIR, To verify the performance of the proposed method on multiple
the proposed method achieves the best results subjectively and benchmarks, 726 underwater images are selected from the RUIE
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 915

Fig. 11. Comparison on UIEB data sets. (a) Original image. (b) UDCP [10]. (c) IBLA [22]. (d) WaterNet [40]. (e) UGAN [41]. (f) Ucolor [42]. (g) USUIR [43].
(h) Proposed.

data set [47], while using the nonreference metrics UIQM and And the proposed method can reach the optimal level both on
UCIQE to evaluate the enhancement performance as well. the average of the displayed images and the overall data set.
As shown in Fig. 12, UDCP cannot improve image bright- It is worth noting that, compared with other deep learning
ness and color. IBLA still performs blur and color cast. Wa- methods, the proposed approach is trained without any reference
terNet and Ucolor have a certain ability to correct color, but image. In conclusion, extensive experiments on benchmarks
lack in enhancing image contrast. Although UGAN can en- prove that the approach is superior to state-of-the-art methods
hance image contrast, it cannot completely remove the color subjectively and objectively, which demonstrates the potential
distortion. The similar unsupervised scheme USUIR still has of zero reference in underwater applications.
residual color casts. The proposed method can achieve the
best color restoration, contrast improvement, and brightness
enhancement. D. Ablation Study
Table IV shows the indicator values of each image and the Several ablation studies are conducted to prove the
average results of the data set. The “avg.” represents the average effectiveness of each component of Zero-UIE, focusing on the
value of the several above images, and the “testset” represents the loss functions and normalization layer.
average value of the data set. It can be seen that IBLA and USUIR 1) Ablation Study on Loss Function: The results trained by
show suboptimal results on UIQM and UCIQE, respectively. various combinations of loss functions are presented in Fig. 13,
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 916 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

TABLE III
QUANTITATIVE EVALUATION ON UIEB DATA SETS, USING UIQM AND UCIQE METRICS

where “w/o” means “without.” To directly compare the impact noting that after UIE-curve, the image pixel dynamic range
of loss functions on network training, only the network output will be compressed, maybe resulting in low brightness. Fur-
results are displayed without the color compensation and nor- thermore, UIE-curve adding the normalization layer (displayed
malization. The results without the illumination smoothness loss as Zero-UIE) can further stretch the dynamic range and image
Ltv hinder the correlations between adjacent regions, resulting contrast, contributing to a more natural appearance and better
in apparent artifacts and unsmooth areas. The color cast cannot visual effect.
be completely removed when the underwater color constant As shown in Fig. 14, (1)–(4), although UIE-curve behaves in
loss Lcol is discarded. Additional undesired color casts such as some color casts, it can achieve a pleasing dehazing effect. After
over-enhanced red area will be introduced into the images when adding normalization, Zero-UIE can better enhance image color
the underwater color relative dispersion Lcrd is not considered. and contrast. But there are some exceptions. Fig. 14, (5)–(8)
Therefore, these several loss functions complement each other, shows that normalization does not contribute to improving image
and the full results (displayed as Total) can achieve the best color quality, while the UIE-curve can remove color cast and haze
restoration and haze removal. independently, which proves the role of the proposed UIE-curve.
2) Ablation Study on Normalization: Fig. 14 shows the de- To visualize the role of the network more intuitively, we
tails of normalization. If the normalization is directly performed compute the difference images of the enhanced results at each
on the raw images, it cannot achieve effective color correction step with the original input images of Fig. 14. Fig. 15(a)–(c)
and contrast enhancement. This is because the pixels of the represents the difference images with the input images and the
raw images already have a large dynamic range, and stretching enhanced versions (UIE-curve, normalization, and Zero-UIE).
them directly has little effect, resulting in the residual color Image difference is to subtract the corresponding pixel values
cast and haze blur. The images output by the Zero-UIE without of two images to weaken the similar parts and highlight the
normalization (displayed as UIE-curve) can achieve a certain changed parts of the images, which tends to reflect the enhanced
improvement in color. More importantly, UIE-curve can remove intensities. As seen in Fig. 15, directly normalizing the raw
haze well and enhance the image contrast greatly. It is worth images contributes almost nothing. UIE-curve can significantly

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 917

Fig. 12. Comparison on RUIE data sets. (a) Original image. (b) UDCP [10]. (c) IBLA [22]. (d) WaterNet [40]. (e) UGAN [41]. (f) Ucolor [42]. (g) USUIR [43].
(h) Proposed.

improve the image contrast and details, while Zero-UIE can be used to measure the texture details. In the autocorrelation
further enhance the visual quality, improving the red components function graph, the larger the surface change trend, the more
and removing color casts. complex the image texture, which corresponds to higher image
Fig. 16 illustrates the corresponding RGB histogram of each contrast. It can be seen that UIE-curve contributes the most to
enhanced version, where (a) shows the source image. Normaliz- the image texture, and the normalized raw image has almost no
ing the source images directly contributes little to the histogram. improvement in texture details. Furthermore, the full Zero-UIE
Zero-UIE can further reduce the distribution distance of the RGB can produce higher contrast and finer texture details based on
components than UIE-curve, thereby removing color casts and UIE-curve.
enhancing image contrast. Several nonreference image quality evaluation indicators are
Fig. 17 shows the autocorrelation function calculation results selected to test the performance, including commonly used
for each enhanced version. The image roughness can reflect the underwater color image quality evaluation index (UCIQE) [45]
texture information, and the autocorrelation function can often and nonreference underwater image quality index (UIQM) [46].

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 918 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 13. Ablation study of the loss functions (illumination smoothness loss Ltv , underwater color constant loss Lcol , and underwater color relative dispersion
loss Lcrd ). (a) Input. (b) Total. (c) w/o Ltv . (d) w/o Lcol . (e) w/o Lcrd .

Fig. 14. Ablation study of normalization. (a) Input. (b) UIE-curve (w/o normalization). (c) Normalization for raw images. (d) Zero-UIE (with normalization).

Fig. 15. Difference images of the enhanced results and the original inputs in Fig. 14. (a) UIE-curve (w/o normalization). (b) Normalization for raw images.
(c) Zero-UIE (with normalization).

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 919

TABLE IV
QUANTITATIVE EVALUATION ON RUIE DATA SETS, USING UIQM AND UCIQE METRICS

Fig. 16. RGB components of the subimages in Fig. 14. (a) Input raw image. (b) Input. (c) UIE-curve (w/o normalization). (d) Normalization for raw images. (e)
Zero-UIE (with normalization).

Fig. 18 shows the normalized evaluation results of each en- Fig. 19 shows the growth rate of the enhanced version on
hanced version on UIQM, UCIQE, and each subindicator of the metrics relative to the raw input image quality. It can be
UIQM. UIE-curve contributes much more to the quality im- seen that UIE-curve greatly contributes to the image sharpness
provement of the raw images than normalization, while the full (higher UIQM value). Although the normalization layer per-
Zero-UIE shows the optimal performance on each indicator. forms well in brightness, there are still haze blur and color

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 920 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 17. Autocorrelation function visualization. (a) Input raw image. (b) Raw autocorrelation function. (c) UIE-curve (w/o normalization). (d) Normalization for
raw images. (e) Zero-UIE (with normalization).

Fig. 18. Normalized quantitative evaluation of each enhanced version, using UICM, UISM, UIConM, UIQM, and UCIQE metrics. From left to right: Input,
UIE-curve, normalization for raw images, and Zero-UIE. (a) UIEB data set. (b) RUIE data set.

Fig. 19. Growth rate of the enhanced version relative to the raw input image, using UIQM and UCIQE metrics. (a) UIEB data set. (b) RUIE data set.

casts, resulting in the lowest UIQM value. The full Zero- E. Testing Runtime
UIE achieves the optimal results after adding the normaliza-
tion layer to UIE-curve. This demonstrates the potential of To demonstrate the efficiency of the proposed model, this
the proposed zero-reference framework for underwater image article compares the average testing runtime of different meth-
enhancement. ods. The tested images are selected from the UIEB data set size

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 921

TABLE V
COMPARISON ON TESTING RUNTIME (RT) (IN SECONDS)

Fig. 20. Color compensation details. The bottom is the corresponding RGB histogram. (a) Input. (b) First-level color compensation. (c) Second-level color
compensation. (d) Zero-UIE (w/o compensation). (e) Zero-UIE+ (with compensation).

of 256 × 256. And the runtime is measured on a PC with an severely attenuated components to a certain degree and reduce
NVIDIA RTX 3090Ti GPU and Intel (R) i9-10900 k CPU. The the distribution distance of the three components. The results of
average running time is shown in Table V, where RT represents Zero-UIE+ can obtain more balanced RGB channels, producing
the required runtime per image. The image quality evaluation more natural colors than Zero-UIE.
metrics UIQM and UCIQE are also shown for reference. The The ablation study of the color compensation is presented in
time efficiency of the proposed Zero-UIE is slightly better than Fig. 21. Using color compensation alone does not provide the
USUIR. Several other methods have relatively longer runtime ability to correct color casts. If the color compensation is dis-
and require complex inference on each image. In addition, the carded, the target may be over-enhanced in some cases [shown in
proposed method achieves the optimal metrics evaluation results Fig. 21 (3)–(7)], resulting in red artifacts and additional noise.
with the least time consumption. In these cases, the color cast cannot be removed entirely. For
The proposed strategy consists of a lightweight network for exceptions, as shown in Fig. 21 (1)–(2), Zero-UIE (without
practical application, which can be mounted on devices with color compensation) can produce natural colors, while the color
limited computing resources, such as underwater vehicles, to compensation is redundant. From subjective vision, Zero-UIE+
obtain underwater optical images with higher visual quality. (with color compensation) can restore realistic scene colors
while avoiding additional noise in some cases, but resulting in
F. Further Improvement of Visual Performance slightly reduced image contrast.
Even obtaining optimal performance on benchmarks, in some In practical applications, it can flexibly choose whether to add
severely degraded underwater scenes, Zero-UIE may have resid- preprocessing operation according to the specific requirements.
ual noise and color casts. As a further improvement of the visual In fact, the simplified version (Zero-UIE) is already suitable
performance, the proposed adaptive color compensation (in for most application scenarios. Experiments on multiple bench-
Section V) can be optionally used as preprocessing to eliminate marks in the previous sections demonstrate that the simplified
the above problems, which can improve the robustness of the version (Zero-UIE) has excellent potential for underwater image
model. Fig. 20 shows the color compensation details, where enhancement. If the visual quality needs further improvement,
Fig. 20(d) shows the output results of Zero-UIE without color the adaptive color compensation can additionally be selected as
compensation (displayed as Zero-UIE), and Fig. 20(e) shows the preprocessing (Zero-UIE+ ) to remove the residual color casts
output results of Zero-UIE with color compensation (displayed at the expense of some time. This provides an essential idea for
as Zero-UIE+ ). In this case, color compensation can stretch the improving model performance in future work.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 922 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Fig. 21. Further improvement of visual performance by color compensation. (a) Input. (b) Color compensation for raw images. (c) Zero-UIE (w/o compensation).
(d) Zero-UIE+ (with compensation).

VII. CONCLUSION REFERENCES

This article presents a novel zero-reference deep network for [1] C. Desai, R. A. Tabib, S. S. Reddy, U. Patil, and U. Mudenagudi, “Ruig:
Realistic underwater image generation towards restoration,” in Proc.
underwater image enhancement (Zero-UIE), eliminating the re- IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 2181–2189.
quirements for paired or unpaired data. The image enhancement [2] Z. Liang, Y. Wang, X. Ding, Z. Mi, and X. Fu, “Single underwater
is transformed into the parameter estimation of the underwater image enhancement by attenuation map guided color correction and detail
preserved dehazing,” Neurocomputing, vol. 425, pp. 160–172, 2021.
curve model. And a set of differentiable underwater nonrefer- [3] B. L. McGlamery, “A computer model for underwater camera systems,”
ence loss functions are designed to drive the network training. Proc. SPIE, vol. 208, pp. 221–231, 1980.
Adaptive color compensation can be optionally selected to im- [4] J. S. Jaffe, “Computer modeling and the design of optimal underwater
imaging systems,” IEEE J. Ocean. Eng., vol. 15, no. 2, pp. 101–111,
prove image color for higher visual quality. It is worth noting that Apr. 1990.
compared to other deep learning methods, the proposed method [5] J. Y. Chiang, Y.-C. Chen, and Y.-F. Chen, “Underwater image enhance-
does not require any reference image in the training process. ment: Using wavelength compensation and image dehazing (WCID),” in
Proc. Int. Conf. Adv. Concepts Intell. Vis. Syst., 2011, pp. 372–383.
The Zero-UIE exhibits pleasing visual performance in bright- [6] X. Fu and X. Cao, “Underwater image enhancement with global–local
ness, color, contrast, and naturalness even under zero-reference networks and compressed-histogram equalization,” Signal Process.: Im-
training. Extensive experiments on multiple benchmarks demon- age Commun., vol. 86, 2020, Art. no. 115892.
[7] P. Zhuang, “Retinex underwater image enhancement with multiorder
strate that the proposed method outperforms state-of-the-art
gradient priors,” in Proc. IEEE Int. Conf. Image Process., 2021, pp. 1709–
methods both on qualitative and quantitative evaluations. It 1713.
has better robustness, larger image dynamic adjustment range, [8] W. Frei, “Image enhancement by histogram hyperbolization,” Comput.
Graph. Image Process., vol. 6, no. 3, pp. 286–294, 1977.
and lower computational burden. Moreover, the lightweight
[9] K. Zuiderveld, “Contrast limited adaptive histogram equalization,” in
network makes it perform well under the condition of lacking Graphic Gems IV, San Diego, CA, USA: Academic, 1994, pp. 474–485.
the sample pairs and small training samples. Due to the above [10] P. L. J. Drews, E. R. Nascimento, S. S. C. Botelho, and M. F. M.
Campos, “Underwater depth estimation and image restoration based on
advantages, it has great value in practical industrial applications, single images,” IEEE Comput. Graph. Appl., vol. 36, no. 2, pp. 24–35,
e.g., the real-time image processing equipped on underwater Mar./Apr. 2016.
vehicles. In future work, the physical parameters will be com- [11] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using
deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell.,
bined to improve the rationality of the zero-reference network vol. 38, no. 2, pp. 295–307, Feb. 2016.
in underwater image processing tasks. And the loss functions [12] J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast
will be improved to further enhance the ability of haze re- enhancer from multi-exposure images,” IEEE Trans. Image Process.,
vol. 27, no. 4, pp. 2049–2062, Apr. 2018.
moval and color restoration for extremely degraded underwater [13] A. S. A. Ghani, “Image contrast enhancement using an integration of
scenes. recursive-overlapped contrast limited adaptive histogram specification and
dual-image wavelet fusion for the high visibility of deep underwater
image,” Ocean Eng., vol. 162, pp. 224–238, 2018.
ACKNOWLEDGMENT [14] S.-L. Wong, R. Paramesran, I. Yoshida, and A. Taguchi, “An integrated
method to remove color cast and contrast enhancement for underwater
The authors would like to thank the Associate Editor and the image,” IEICE Trans. Fund. Electron., Commun. Comput. Sci., vol. 102,
reviewers for their helpful feedback that improved this paper. no. 11, pp. 1524–1532, 2019.
Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 HUANG et al.: UNDERWATER IMAGE ENHANCEMENT BASED ON ZERO-REFERENCE DEEP NETWORK 923

[15] S. Zhang, T. Wang, J. Dong, and H. Yu, “Underwater image enhancement [41] C. Fabbri, Md J. Islam, and J. Sattar, “Enhancing underwater imagery
via extended multi-scale retinex,” Neurocomputing, vol. 245, pp. 1–9, using generative adversarial networks,” in Proc. IEEE Int. Conf. Robot.
2017. Autom., 2018, pp. 7159–7165.
[16] C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater [42] C. Li, S. Anwar, J. Hou, R. Cong, C. Guo, and W. Ren, “Underwater image
images and videos by fusion,” in Proc. IEEE Conf. Comput. Vis. Pattern enhancement via medium transmission-guided multi-color space embed-
Recognit., 2012, pp. 81–88. ding,” IEEE Trans. Image Process., vol. 30, pp. 4985–5000, May 2021.
[17] C. O. Ancuti, C. Ancuti, C. De Vleeschouwer, and P. Bekaert, “Color [43] Z. Fu et al., “Unsupervised underwater image restoration: From a ho-
balance and fusion for underwater image enhancement,” IEEE Trans. mology perspective,” Proc. AAAI Conf. Artif. Intell., vol. 36, no. 1,
image Process., vol. 27, no. 1, pp. 379–393, Jan. 2018. pp. 643–651, Jun. 2022.
[18] W. Zhang, L. Dong, T. Zhang, and W. Xu, “Enhancing underwater image [44] K. McLaren, “Xiii-the development of the CIE 1976 (l* a* b*) uniform
via color correction and bi-interval contrast enhancement,” Signal Pro- colour space and colour-difference formula,” J. Soc. Dyers Colourists,
cess.: Image Commun., vol. 90, 2021, Art. no. 116030. vol. 92, no. 9, pp. 338–341, 1976.
[19] C. O. Ancuti, C. Ancuti, C. De Vleeschouwer, and R. Garcia, “Locally [45] M. Yang and A. Sowmya, “An underwater color image quality evaluation
adaptive color correction for underwater image dehazing and matching,” metric,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 6062–6071,
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2017, Dec. 2015.
pp. 1–9. [46] K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired un-
[20] C. Li, J. Guo, C. Guo, R. Cong, and J. Gong, “A hybrid method for derwater image quality measures,” IEEE J. Ocean. Eng., vol. 41, no. 3,
underwater image correction,” Pattern Recognit. Lett., vol. 94, pp. 62–67, pp. 541–551, Jul. 2016.
2017. [47] R. Liu, X. Fan, M. Zhu, M. Hou, and Z. Luo, “Real-world underwater
[21] Y. Wang, H. Liu, and L.-P. Chau, “Single underwater image restoration enhancement: Challenges, benchmarks, and solutions under natural light,”
using adaptive attenuation-curve prior,” IEEE Trans. Circuits Syst. I: IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 12, pp. 4861–4875,
Regular Papers, vol. 65, no. 3, pp. 992–1002, Mar. 2018. Dec. 2020.
[22] Y.-T. Peng and P. C. Cosman, “Underwater image restoration based on
image blurriness and light absorption,” IEEE Trans. image Process.,
vol. 26, no. 4, pp. 1579–1594, Apr. 2017.
[23] W. Song, Y. Wang, D. Huang, A. Liotta, and C. Perra, “Enhancement Yifan Huang received the master’s degree in com-
of underwater images with statistical model of background light and munication and information systems from the De-
optimization of transmission map,” IEEE Trans. Broadcast., vol. 66, no. 1, partment of Information and Communication Engi-
pp. 153–169, Mar. 2020. neering, Xiamen University, Xiamen, China, in 2020.
[24] J. Zhou, T. Yang, W. Ren, D. Zhang, and W. Zhang, “Underwater image She is currently working toward the Ph.D. degree
restoration via depth map and illumination estimation based on a single in communication and information systems from the
image,” Opt. Exp., vol. 29, no. 19, pp. 29864–29886, 2021. Department of Information and Communication En-
[25] C. Li, S. Anwar, and F. Porikli, “Underwater scene prior inspired deep gineering with the Key Lab of Underwater Acoustic
underwater image and video enhancement,” Pattern Recognit., vol. 98, Communication and Marine Information Technol-
2020, Art. no. 107038. ogy, Ministry of Education, Xiamen University.
[26] J. Lu, F. Yuan, W. Yang, and E. Cheng, “An imaging information estimation Her professional direction is communication and
network for underwater image color restoration,” IEEE J. Ocean. Eng., information systems. Her research interests fall in the underwater image pro-
vol. 46, no. 4, pp. 1228–1239, Oct. 2021. cessing and video/image quality measurement.
[27] Y. Guo, H. Li, and P. Zhuang, “Underwater image enhancement using a
multiscale dense generative adversarial network,” IEEE J. Ocean. Eng.,
vol. 45, no. 3, pp. 862–870, Jul. 2020.
[28] J. Li, K. A. Skinner, R. M. Eustice, and M. Johnson-Roberson, “Watergan:
Unsupervised generative network to enable real-time color correction of
monocular underwater images,” IEEE Robot. Autom. Lett., vol. 3, no. 1, Fei Yuan received the Ph.D. degree in communica-
pp. 387–394, Feb. 2017. tion and information systems from the Communi-
[29] N. Wang, Y. Zhou, F. Han, H. Zhu, and J. Yao, “Uwgan: Underwater cation Engineering Department, Xiamen University,
gan for real-world underwater color restoration and dehazing,” 2019, Xiamen, China, in 2008.
arXiv:1912.10269. He is currently a Professor with the Department
[30] M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement of Information and Communication Engineering,
for improved visual perception,” IEEE Robot. Autom. Lett., vol. 5, no. 2, School of Information, Xiamen University, and a key
pp. 3227–3234, Apr. 2020. member of the Key Lab of Underwater Acoustic
[31] S. B. Sami, A. Muqeet, and H. Tariq, “A novel image dehazing and Communication and Marine Information Technol-
assessment method,” in Proc. IEEE 8th Int. Conf. Inf. Commun. Technol., ogy, Ministry of Education, Xiamen University. His
2019, pp. 72–77. current research focuses on the general area of under-
[32] N. G. Jervov, Optical Oceanography. New York, NY, USA: Elsevier, 1968. water multimedia communication, communication and sensing integration, and
[33] S. Q. Duntley, “Light in the sea,” JOSA, vol. 53, no. 2, pp. 214–233, 1963. underwater intelligent Internet of Things applications.
[34] J. Houghton, The Physics of Atmospheres. Cambridge, U.K.: Cambridge
Univ. Press, 2002.
[35] C. Guo et al., “Zero-reference deep curve estimation for low-light image
enhancement,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
2020, pp. 1780–1789. Fengqi Xiao received the undergraduate degree in
[36] K. He, J. Sun, and X. Tang, “Single image haze removal using dark electronic information science and technology from
channel prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 12, the Electronic Information Science and Technology
pp. 2341–2353, Dec. 2011. Department, Ocean University of China, Qingdao,
[37] G. Buchsbaum, “A spatial processor model for object colour perception,” China, in 2016. He is currently working toward the
J. Franklin Inst., vol. 310, no. 1, pp. 1–26, 1980. Ph.D. degree in communication and information sys-
[38] A. Galdran, D. Pardo, A. Picón, and A. Alvarez-Gila, “Automatic red- tems from the Department of Information and Com-
channel underwater image restoration,” J. Vis. Commun. Image Represen- munication Engineering with the Key Lab of Under-
tation, vol. 26, pp. 132–145, 2015. water Acoustic Communication and Marine Infor-
[39] Y. Huang, M. Liu, and F. Yuan, “Color correction and restoration based mation Technology, Ministry of Education, Xiamen
on multi-scale recursive network for underwater optical image,” Signal University, Xiamen, China.
Process.: Image Commun., vol. 93, 2021, Art. no. 116174. His professional direction is communication and information systems. His
[40] C. Li et al., “An underwater image enhancement benchmark dataset and research interests fall in the underwater image processing and video/image
beyond,” IEEE Trans. Image Process., vol. 29, pp. 4376–4389, Nov. 2019. quality measurement.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.
 924 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 48, NO. 3, JULY 2023

Jianxiang Lu received the undergraduate degree in En Cheng received the Ph.D. degree in communi-
electronic information engineering from the Electron- cation and information systems from the Communi-
ics Department, and the masters degree in communi- cation Engineering Department, Xiamen University,
cation and information systems from the Department Xiamen, China, in 2006.
of Information and Communication Engineering, Xi- He is currently a Professor with the Communi-
amen University, Xiamen, China, in 2021. cation Engineering Department, Xiamen University
His main research interests include computer vi- and the Director with the Key Lab of Underwater
sion and machine learning. Acoustic Communication and Marine Information
Technology, Ministry of Education, Xiamen Univer-
sity. His research interests fall in the general area of
underwater acoustic communication and networking,
spanning from the communication networks, underwater acoustic communica-
tion, multimedia signal processing and communication, video/image quality
measurement, and embedded system design.

Authorized licensed use limited to: Huazhong University of Science and Technology. Downloaded on March 27,2024 at 06:34:08 UTC from IEEE Xplore. Restrictions apply.