Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in

Scientific Computing

Salah A. Faroughia,∗, Nikhil M. Pawara , Célio Fernandesa,b , Subasish Dasc , Nima K. Kalantarid , Seyed
Kourosh Mahjoura
a Geo-Intelligence Laboratory, Ingram School of Engineering, Texas State University, San Marcos, Texas, 78666, USA
b Associate Laboratory of Intelligent Systems, Institute for Polymers and Composites, Polymer Engineering Department, School
of Engineering, University of Minho, Campus of Azurém, 4800-058 Guimarães, Portugal
c Artificial Intelligence in Transportation Lab, Ingram School of Engineering, Texas State University, San Marcos, Texas,

78666, USA
d Computer Science and Engineering Department, Texas A&M University, College Station, Texas, 77843, USA

Abstract
Recent breakthroughs in computing power have made it feasible to use machine learning and deep learning
arXiv:2211.07377v1 [cs.LG] 14 Nov 2022

to advance scientific computing in many fields, such as fluid mechanics, solid mechanics, materials science, etc.
Neural networks, in particular, play a central role in this hybridization. Due to their intrinsic architecture,
conventional neural networks cannot be successfully trained and scoped when data is sparse; a scenario that is
true in many scientific fields. Nonetheless, neural networks offer a strong foundation to digest physical-driven
or knowledge-based constraints during training. Generally speaking, there are three distinct neural network
frameworks to enforce underlying physics: (i) physics-guided neural networks (PgNN), (ii) physics-informed
neural networks (PiNN) and (iii) physics-encoded neural networks (PeNN). These approaches offer unique
advantages to accelerate the modeling of complex multiscale multi-physics phenomena. They also come with
unique drawbacks and suffer from unresolved limitations (e.g., stability, convergence, and generalization) that
call for further research. This study aims to present an in-depth review of the three neural network frame-
works (i.e., PgNN, PiNN, and PeNN) used in scientific computing research. The state-of-the-art architectures
and their applications are reviewed; limitations are discussed; and future research opportunities in terms of
improving algorithms, considering causalities, expanding applications, and coupling scientific and deep learn-
ing solvers are presented. This critical review provides a solid starting point for researchers and engineers to
comprehend how to integrate different layers of physics into neural networks.
Keywords: Physics-guided Neural Networks, Physics-informed Neural Networks, Physics-encoded Neural
Networks, Solid Mechanics, Fluid Mechanics, Machine Learning, Deep Learning, Scientific Computing

1. Introduction

Machine learning (ML) and deep learning (DL) are becoming the key technologies to advance scientific
research and computing in a variety of fields, such as fluid mechanics [1], solid mechanics [2], material design [3],
etc. The emergence of multiteraflop machines with thousands of processors for scientific computing combined
with advanced sensory-based experimentation has heralded an explosive growth of structured and unstructured
heterogeneous data in science and engineering fields. ML and DL approaches were initially introduced to
scientific computing to solve the lack of efficient data modeling procedures, preventing scientists from rapidly
interacting with heterogeneous complex data [4]. These approaches show transformative potential because
they enable the exploration of vast design spaces, the identification of multidimensional connections, and
the management of ill-posed issues [5, 6, 7]. However, conventional ML and DL methods are unable to
extract interpretative information and expertise from complex multidimensional data. They may be effective
in mapping observational or computational data, but their predictions may be physically irrational or dubious,
resulting in poor generalization [8, 9, 10]. For this reason, scientists initially considered these methodologies like
a magic black box devoid of a solid mathematical foundation and incapable of interpretation. Notwithstanding,
learning techniques constitute a new paradigm for accurately solving scientific and practical problems orders
of magnitude faster than the conventional solvers.
Deep learning (i.e., neural networks mimicking the human brain) and scientific computing share common
historical and intellectual links that are normally unrealized, e.g., differentiability [8]. Figure 1 shows a
schematic representation of the history of development for a plethora of scientific computing and DL approaches
(only seminal works are included). In the last decade, breakthroughs in DL and computing power have made it
possible to employ DL in a broad variety of scientific computing, especially in fluid mechanics [1, 10, 11], solid

∗ Corresponding author: [email protected]

Preprint submitted to Elsevier November 15, 2022

Figure 1: A schematic representation of the history of development, including only seminal works, for scientific
computing and DL approaches. For scientific computing, the following are listed: Finite Difference Method
(FDM) [23], Molecular Dynamics (MD) [24], Finite Element Method (FDM) [25], Large Eddy Simulation
(LES) [26], Discrete Element Method (DEM) [27], Finite Volume Method (FVM) [28], Immersed Boundary
Method (IBM) [29], Smoothed Particle Hydrodynamics (SPH) [30], Lattice Boltzmann Method (LBM) [31],
and Discontinuous Galerkin (DG) [32]. For deep learning, the following are listed: Deep Neural Network (DNN)
[33], Recurrent Neural Network (RNN) [34], Physics-guided Neural Network (PgNN) [35], Convolutional Neural
Network (CNN) [36], Generative Adversarial Network (GAN) [37], Physics-informed Neural Network (PiNN)
[38], Fourier Neural Operator (FNO) [39], and Physics-encoded Neural Network (PeNN) [40].

mechanics [2, 12, 13] and materials science [14, 15, 16], albeit at the cost of accuracy and loss of generality
[17]. These data-driven methods are routinely applied to fulfill one of the following goals: (i) accelerate
direct numerical simulations using surrogate modeling [18], (ii) accelerate adjoint sensitivity analysis [8], (iii)
accelerate probabilistic programming [19], and (iv) accelerate inverse problems [20]. For example, in the first
goal, the physical parameters of the system (e.g., dimensions, mass, momentum, temperature, etc.) are used
as inputs to predict the next state of the system or its effects (i.e., outputs), and in the last goal, the outputs of
a system (e.g., a material with targeted properties) are used as inputs to infer the intrinsic physical attributes
that meet the requirements (i.e., model’s outputs). To accomplish these goals, lightweight DL models can be
constructed to partially or fully replace a bottleneck step in the scientific computing processes [17, 21, 22].
Due to the intrinsic architecture of conventional DL methods, their learning is limited to the scope of the
datasets with which the training is conducted (e.g., specific boundary conditions, material types, spatiotem-
poral discretization, etc.), and inference cannot be successfully scoped under any unseen conditions (e.g., new
geometries, new material types, new boundary conditions, etc.). Because the majority of the scientific fields
are not (big) data-oriented domains and cannot provide comprehensive datasets that cover all possible condi-
tions, these models trained based on sparse datasets are accelerated but not predictive [22]. Thus, it is logical
to leverage the wealth of prior knowledge, the underlying physics, and domain expertise to further constrain
these models while training on available, sparse data points. Neural networks (NN) are better suited to digest
physical-driven or knowledge-based constraints during training. Generally speaking, there are three distinct
neural network frameworks to enforce underlying physics: (i) physics-guided neural networks (PgNN), (ii)
physics-informed neural networks (PiNN), and (iii) physics-encoded neural networks (PeNN).
PgNN constructs the model as a black-box to learn a surrogate mapping between the formatted inputs
and the outputs. This process requires a rich and sufficient dataset to train a reliable model. The physics-
guided models map a set of inputs x to a related set of outputs y using an appropriate function F with
unknown parameters w such that y = F (x; w). By specifying a particular structure for F, a data-driven
approach generally attempts fine-tuning the parameters w so that the overall error between true values, ŷ,
and those from model predictions, y, is minimized [7]. In PgNNs, x and y are generated using experimentation
and computation in a controlled setting and curated through extensive processes to ensure compliance with
physics principles and fundamental rules [22]. However, for complex physical systems, the data is likely sparse
due to the high cost of data acquisition [41]. The vast majority of state-of-the-art PgNNs lack robustness
and fail to provide any guarantees of generalization (i.e., interpolation [38, 42] and extrapolation [43]). To
remediate this issue, PiNNs have been introduced to perform supervised learning tasks while abiding given
laws of physics in the form of general non-linear differential equations [44, 10, 45, 46, 6].

2
 The PiNN-based models respect the physical laws by incorporating a weakly imposed loss function con-
sisting of the residuals of physics equations and boundary constraints. They leverage automatic differentiation
[47] to differentiate the neural network outputs with respect to their inputs (i.e., spatiotemporal coordinates
and model parameters). By minimizing the loss function, the network can closely approximate the solution
[48]. As a result, PiNNs lay the groundwork for a new modeling and computation paradigm that enriches DL
with the long-standing achievements in mathematical physics [44]. PiNNs still have a number of unresolved
issues and limitations relating to theoretical considerations (e.g., convergence and stability [49, 6, 50]) and
implementation issues (e.g., neural networks design, boundary conditions management, and optimization as-
pects) [40, 10]. In addition, in cases where the explicit form of differential equations governing the complex
dynamics is not fully known a priori, PiNNs encounter serious limitations [51]. For these cases, another family
of DL approaches known as physics-encoded neural networks (PeNN) has been proposed [40].
The PeNN-based models leverage physical-driven or knowledge-based constraints directly in their archi-
tecture to resolve issues with data sparsity and the lack of generalization encountered by both PgNNs and
PiNNs models [40]. In PeNNs, the known physics of non-linear dynamics of multiscale systems is forcibly
encoded into the neural networks’ architecture to facilitate learning in a data-driven manner [40]. The enforc-
ing mechanisms of the underlying physics in PeNNs are thus fundamentally different from PiNNs [52, 39, 53];
although they can be combined to achieve the desired non-linearity of the model [43]. It is demonstrated that
the resulting neural networks using the PeNN paradigm possess remarkable robustness against data sparsity
and model generalizability compared with PgNNs and PiNNs [40].
This review paper highlights the general architecture, advantages, and drawbacks of various DL methods
with relevant applications to computational fluid and solid mechanics. The remainder of this work is structured
as follows: In Section 2, the potential of PgNNs to accelerate scientific computing is discussed. Section 3
provides an overview of PiNNs and discusses their potential to advance PgNNs. Section 4 reviews several
leading PeNN architectures to address critical limitations in PgNNs and PiNNs. Finally, in Section 5, an
outlook for future research directions is provided.

2. Physics-guided Neural Network, PgNN

PgNN is a supervised DL model that statistically learns the known physics of a desired phenomenon
by extracting features or attributes from raw training data [54]. PgNNs consist of one or a combination of
Multilayer Perceptron (MLP, alternatively called artificial neural networks, ANN, or deep neural networks,
DNN, in different studies relevant to this review) [54], CNN [54], RNN [54], GAN [55], and graph neural
networks (GRNN) [56]. A schematic representation of a sample PgNN architecture is illustrated in Fig. 2.
Any physical problem includes a set of independent features or input features as x = [X1 , X2 , X3 , ..., Xn ] and
a set of dependent variables or desired outputs as y = [Y1 , Y2 , Y3 , ..., Yn ]. The data describing this physical
phenomenon can be generated by experimentation (e.g., sensor-based observation, etc.), closure laws (e.g.,
Fourier’s law, Darcy’s Law, drag force, etc.), or the solution of governing ordinary differential equations
(ODE) and/or partial differential equations (PDE), e.g., Burger’s equation, Navier-Stokes equations, etc. The
dependent variables and independent features thus comply with physics principles and the trained neural
network is guided inherently by physics while training.
In PgNNs, the neurons in each layer are connected to the neurons in the next layer through a set of
weights. The output of each node is obtained by applying an activation function (e.g., Rectified Linear Unit
(ReLU), Tanh, Sigmoid, Linear, etc.) to the weighted sum of the outputs of the neurons in the previous layer
plus an additional bias [57]. Starting from the input, the output of the neurons in each layer is obtained using
this process in a sequential manner. This process is typically called forward propagation. Subsequently, in
order to evaluate the accuracy of the prediction, a loss function (or alternatively, a cost function) is defined
and calculated. Commonly used loss functions for regression are L1 [58] and mean-squared-error (MSE) [58].
The next step in training involves error backpropagation, which calculates the partial derivatives/gradients
of the cost function with respect to weights and biases (i.e., θ as shown in Fig. 2). Finally, an optimization
technique, such as gradient descent [59], stochastic gradient descent [59], or mini-batch gradient descent [59], is
used to minimize the loss function and simultaneously compute and update θ parameters using the calculated
gradients from the backpropagation procedure. The process is iterated until the desired level of accuracy is
obtained for a PgNN.
In recent years, PgNN has been extensively used to accelerate computational fluid dynamics (CFD) [60],
computational solid mechanics [61], and multi-functional material designs [62]. It has been employed in all
computationally expensive and time-consuming components of scientific computing, such as (i) pre-processing
[63, 60, 64], e.g., mesh generation; (ii) discretization and modeling [65, 66, 67], e.g., Finite Difference (FDM),
Finite Volume (FVM), Finite Element (FEM), Discrete Element Method (DEM), Molecular Dynamics (MD),
etc.; and (iii) post-processing, e.g., output assimilation and visualization [68, 69, 70]. These studies are arranged
(i) to train shallow networks on small datasets to replace a bottleneck (i.e., a computationally expensive step)
in conventional forward numerical modeling, e.g., drag coefficient calculation in concentrated complex fluid
flow modeling [22, 71, 72, 73, 74]; or (ii) to train relatively deep networks on larger datasets generated for
a particular problem, e.g., targeted sequence design within the coarse-grained polymer genome [75]. These

3
Figure 2: A schematic architecture of PgNNs. Panel (a) shows the typical generation of training datasets
using known closure laws, direct numerical simulation of PDEs and ODEs, or experimentation to comply with
physical principles. Panel (b) shows the architecture of a PgNN model consisting of a simple feed forward neural
network (which can be replaced with any other network type). The loss function made of L1 , L2 regularization,
MSE, or other user-defined error functions is minimized iteratively in the training phase. θ is the learnable
parameter corresponding to weights/biases in the neural network that can be learned simultaneously while
minimizing the loss function.

networks acknowledge the physical principles upon which the training data is generated and accelerate the
simulation process [70, 22].
Although the training of PgNNs appears to be straightforward, generating the data by tackling the
underlying physics for complex physical problems can consume a significant amount of computational power
and time [6, 13]. However, once trained, a PgNN can substantially accelerate the computation speed for the
phenomena of interests. It is worth noting that while a PgNN model may have the best accuracy on the
training set based on numerous attempts, it is more likely to memorize the trends, noise, and detail in the
training set rather than intuitively comprehend the pattern in the dataset. PgNN overfitting can be mitigated
in different ways [76, 77, 78] to enhance the predictability of the model within the scope of the training data;
however, PgNN still loses its prediction ability when inferred/tested outside the scope of the training datasets.
In the following subsections, we review the existing literature and highlight some of the most recent studies
that applied PgNNs to accelerate different steps in scientific computing.

2.1. Pre-Processing
The pre-processing is often the most work intensive component in scientific computing, irrespective of the
model (e.g., FEM, FDM, FVM, etc.). The main steps in this component are the disassembly of the domain
into small, but finite, parts (i.e., mesh generation, evaluation, and optimization), and the mesh properties
upscaling and/or downscaling to use a spatiotemporally coarse mesh while implicitly solving for unresolved
fine-scale physics. These two steps are time-consuming and require expert-level knowledge, and hence, are
potential candidates to be replaced by accelerated PgNN-based models.

2.1.1. Mesh Generation

Mesh generation is a critical step for numerical simulations. Zhang et al. [63] suggested the automatic
generation of an unstructured mesh based on the prediction of the required local mesh density throughout the
domain. For that purpose, an ANN was trained to guide a standard mesh generation algorithm. They also
proposed to extend the study to other architectures, such as CNN or GRNN for future studies with larger
datasets and/or higher-dimensional problems. Huang et al. [60] adopted a DL approach to identify optimal
mesh densities. They generated optimized meshes using classical CFD tools (e.g., Simcenter STAR-CCM+
[79]) and proposed training a CNN to predict optimal mesh densities for arbitrary geometries. The addition
of an adaptive mesh refinement version accelerated the overall process without compromising accuracy and
resolution. The authors proposed learning optimal meshes (generated by corresponding solvers with adjoint
functionality) using ANN, which may be utilized as a starting point in other simulation tools irrespective of the
specific numerical approach [60]. Wu et al. [64] also proposed a mesh optimization method by combining the
moving mesh method with DL in order to solve the mesh optimization problem. With the experiments carried
out, they developed a neural network with extremely high precision to optimize the mesh while maintaining

4
the specified number of nodes and topology of the initially given mesh. They also showed that the moving
mesh algorithm is independent of the CFD calculation using their method [64].
In mesh generation, a critical issue has been the evaluation of mesh quality due to a lack of general and
effective criteria. Chen et al. [80] presented a benchmark dataset (i.e., the NACA-Market reference dataset) to
facilitate the quality assessment of a mesh. They proposed a method using a deep CNN, dubbed as GridNet,
to perform the automatic evaluation of the mesh quality. This method receives the mesh as input and conducts
the evaluation. The mesh quality evaluation using a deep CNN model trained on the NACA-Market dataset
proved to be viable with an accuracy of 92.5 percent [80].

2.1.2. Cross-scaling Techniques

It is always desirable to numerically solve a multi-physics problem on a spatiotemporally coarser mesh
to minimize computational cost. For this reason, different upscaling [81, 82], downscaling [83], and cross-
scaling [84] methods have been introduced to determine accurate numerical solutions to non-linear problems
spanning a wide range of length-scales and time-scales. One viable choice is to use a coarse mesh that reliably
depicts long-wavelength dynamics and accounts for unresolved small-scale physics. Deriving the mathematical
model (e.g., boundary conditions) for coarse representations, on the other hand, is relatively hard. Bar-Sinai
et al. [82] proposed a PgNN model for learning optimum PDE approximations based on actual solutions to
known underlying equations. The ANN outputs spatial derivatives, which are then optimized in order to best
satisfy the equations on a low-resolution grid. When compared to typical discretization methods (e.g., finite
difference), the recommended ANN method was considerably more accurate while integrating the collection
of non-linear equations at a resolution that was 4x to 8x coarser [82]. The main challenge in this approach,
however, is to systematically derive these kinds of solution-adaptive discrete operators. Maddu et al. [81]
developed a PgNN, dubbed as STENCIL-NET, for learning resolution-specific local discretization of non-linear
PDEs. By combining spatially and temporally adaptive parametric pooling on regular Cartesian grids with
knowledge about discrete time integration, STENCIL-NET can accomplish numerically stable discretization of
the operators for any arbitrary non-linear PDE. The STENCIL-NET model can also be used to determine PDE
solutions over a wider spatiotemporal scale than the training dataset. In their paper, the authors employed
STENCIL-NET for long-term forecasting of chaotic PDE solutions on coarse spatiotemporal grids to test
their hypothesis. When the STENCIL-NET model was compared to base-line numerical techniques (e.g., fully
vectorized WENO [85]), the predictions on coarser grids were faster by up to 25 to 150X on GPUs and 2x to
14x on CPUs, while providing identical accuracy [81].
A non-exhaustive list of recent studies that leveraged PgNNs to accelerate the pre-processing part of
scientific computing is reported in Table 1. These studies collectively concluded that PgNN can be successfully
integrated to achieve a considerable speed-up factor in mesh generation, mesh evaluation, and cross-scaling,
which are vital to many complex problems explored using scientific computing techniques. The next subsection
discusses the potential of PgNN to be incorporated into modeling components, thus achieving a higher speed-up
factor or higher modeling accuracy.

2.2. Modeling and Post-processing

2.2.1. PgNN for Fluid Mechanics
PgNN has gained considerable attention from the fluid mechanics community. The study by Lee and
Chen [89] on estimating fluid properties using ANN was among the first studies that applied PgNN to fluid
mechanics. Since then, the application of PgNNs in fluid mechanics has been extended to a wide range of
applications, e.g., laminar and turbulent flows, non-Newtonian fluid flows, aerodynamics, etc., especially to
speed up the traditional computational fluid dynamics (CFD) solvers.
For incompressible laminar flow simulations, the numerical procedure to solve Navier–Stokes equations is
considered as the main bottleneck. To alleviate this issue, PgNNs have been used as a part of the resolution
process. For example, Yang et al. [90] proposed a novel data-driven projection method using an ANN to avoid
iterative computation of the projection step in grid-based fluid simulations. The efficiency of the proposed
data-driven projection method was shown to be significant, especially in large-scale fluid flow simulations.
Tompson et al. [91] used a CNN for predicting the numerical solutions to the inviscid Euler equations for
fluid flows. An unsupervised training that incorporates multi-frame information was proposed to improve
the long-term stability. The CNN model produced very stable divergence free velocity fields with improved
accuracy when compared to the ones obtained by the commonly used Jacobi method [92]. Chen et al. [93] later
developed a U-net-based architecture, a particular case of a CNN model, for the prediction of velocity and
pressure field maps around arbitrary 2D shapes in laminar flows. The CNN model is trained with a dataset
composed of random shapes constructed using Bézier curves and then by solving Navier-Stokes equations
using a CFD solver. The predictive efficiency of the CNN model was also assessed on unseen shapes, using ad
hoc error functions, specifically, the MSE levels for these predictions were found to be in the same order of
magnitude as those obtained on the test subset, i.e., between 1.0 × 10−5 and 5.0 × 10−5 for both pressure and
velocity, respectively.
Moving from laminar to turbulent flow regimes, PgNNs have been extensively used for the formulation
of turbulence closure models [94]. Ling et al. [95] used a feed-forward MLP and a specialized neural network

5
Table 1: A non-exhaustive list of recent studies that leveraged PgNNs to accelerate the pre-processing part
in scientific computing.

Area of application NN Type Objective Reference

Mesh Generation ANN Generating unstructured mesh [63]

CNN Predicting meshes with optimal density and ac- [60]

celerating meshing process without compromising
performance or resolution

ANN Generating high quality tetrahedral meshes [86]

ANN Developing a mesh generator tool to produce high [87]

quality FEM meshes

ANN Generating finite element mesh with less complex- [88]

ities

Mesh Evaluation CNN Conducting automatic mesh evaluation and qual- [80]
ity assessment

Mesh Optimisation ANN Optimizing mesh while retaining the same number [64]
of nodes and topology as the initially given mesh

Cross-scaling ANN Utilizing data-driven discretization to estimate [82]

spatial derivatives that are tuned to best fulfill
the equations on a low-resolution grid.

ANN Providing solution-adaptive discrete operators to [81]

(STENCIL- predict PDE solutions on bigger spatial domains
NET)
and for longer time frames than it was trained

to predict Reynolds-averaged Navier–Stokes (RANS) and Large Eddy Simulation (LES) turbulence problems.
Their specialized neural network embeds Galilean invariance [96] using a higher-order multiplicative layer. The
performance of this model was compared with that of MLP and ground truth simulations. They concluded
that the specialized neural network can predict the anisotropy tensor on an invariant tensor basis, resulting
in significantly more accurate predictions than MLP. Maulik et al. [97] presented a closure framework for
subgrid modeling of Kraichnan turbulence [98]. To determine the dynamic closure strength, the proposed
framework used an implicit map with inputs as grid-resolved variables and eddy viscosities. Training an ANN
with extremely subsampled data obtained from high-fidelity direct numerical simulations (DNSs) yields the
optimal map. The ANN model was found to be successful in imbuing the decaying turbulence problem with
dynamic kinetic energy dissipation, allowing accurate capture of coherent structures and inertial range fidelity.
Later, Kim and Lee [99] used simple linear regression, SLinear, multiple linear regression, MLinear, and a
CNN to predict the turbulent heat transfer (i.e., the wall-normal heat flux, qw ) using other wall information,
including the streamwise wall-shear stress, spanwise wall-shear stress or streamwise vorticity, and pressure
fluctuations, obtained by DNSs of a channel flow (see Fig. 3(a)). The constructed network was trained using
adaptive moment estimation (ADAM) [100, 101], and the grid searching method [102, 103] was performed to
optimize the depth and width of the CNN. Their finding showed that the PgNN model is less sensitive to the
input resolution, indicating its potential as a good heat flux model in turbulent flow simulation. Yousif et al.
[104] also proposed an efficient method for generating turbulent inflow conditions based on a PgNN formed
by a combination of a multiscale convolutional auto-encoder with a subpixel convolution layer (MSCSP-AE)
[105, 106] and a long short-term memory (LSTM) [107, 108] model. The proposed model was found to have
the capability to deal with the spatial mapping of turbulent flow fields.
PgNNs have also been applied in the field of aerodynamics. Kou and Zhang [109] presented a review paper
on typical data-driven methods, including system identification, feature extraction, and data fusion, that have
been employed to model unsteady aerodynamics. The efficacy of those data-driven methods is described by
several benchmark cases in aeroelasticity. Wang et al. [110] described the application of ANN to the modeling
of the swirling flow field in a combustor (see Fig. 3(b)). Swirling flow field data from particle image velocimetry
(PIV) was used to train an ANN model. The trained PgNN model was successfully tested to predict the swirling
flow field under unknown inlet conditions. Chowdhary et al. [111] studied the efficacy of combining ANN models
with projection-based (PB) model reduction techniques [112, 113] to develop an ANN-surrogate model for
computationally expensive, high-fidelity physics models, specifically, for complex hypersonic turbulent flows.
The surrogate model was used to perform Bayesian estimation of freestream conditions and parameters of the

6
SST (shear stress transport) turbulence model. The surrogate model was then embedded in the high-fidelity
(Reynolds-averaged Navier–Stokes) flow simulator, using shock-tunnel data. Siddiqui et al. [114] developed a
non-linear data-driven model, encompassing Time Delay Neural Networks (TDNN), for a pitching wing. The
pitch angle was considered as the input to the model, while the lift coefficient was considered as the output.
The results showed that the trained models were able to capture the non-linear aerodynamic forces more
accurately than linear and semi-empirical models, especially at higher offset angles. Wang et al. [115] also
proposed a multi-fidelity reduced-order model based on multi-task learning ANNs to efficiently predict the
unsteady aerodynamic performance of an iced airfoil. The results indicated that the proposed model achieves
higher accuracy and better generalization capability compared with single-fidelity and single-task modeling
approaches.

Figure 3: Panel (a) shows a comparison of the high-order moments of the heat flux data obtained from
DNS of turbulent heat transfer and predictions obtained by SLinear, MLinear and CNN (i.e., PgNN) models
developed by Kim and Lee [99]. Notice that qw,rms is the root-mean-squared-error (RMSE) of qw , k denotes
the index of the weights in the network and the angle bracket denotes the average over all test points. Panel
(b) shows the comparison of a PgNN model and PIV technique for the prediction of the swirling flow field
in a combustor Wang et al. [110]. The changes in maximum and minimum vorticity, ω (1/s), in a swirling
flow field are shown for several pressure drops, ∆p (MPa). Panel (c) shows the performance of the PgNN
model developed by Faroughi et al. [22] against the blind dataset generated to predict the drag coefficient of
a spherical particle translating in a Giesekus fluid at Reynolds number Re = 75, retardation ratio ζ = 0.8
and mobility parameter α = 0.4. Panel (d) shows a comparison of the L22 error (E) and simulation run time
of WENO-NN, weighted ENO-Jiang Shu (WENO5-JS) scheme convergent at fifth order, and weighted ENO
(WENO1) scheme convergent at first order, to simulate shock wave interactions Stevens and Colonius [116].

The simulation of complex fluid flows, specifically using fluids that exhibit viscoelastic nature and non-
linear rheological behaviors, is another topic where PgNNs have been applied [117, 118]. The dynamics of
these fluids are generally governed by non-linear constitutive equations that lead to stiff numerical problems
[119, 120]. Faroughi et al. [22] developed a PgNN model to predict the drag coefficient of a spherical particle
translating in viscoelastic fluids (see Fig. 3(c)). The PgNN considered a stacking technique (i.e., ensembling
Random Forrest [121], Extreme Gradient Boosting [122] and ANN models) to digest inputs (Reynolds number,
Weissenberg number, viscosity ratio,and mobility factor considering both Oldroyd-B and Giesekus fluids) and
outputs drag predictions based on the individual learner’s predictions and an ANN meta-regressor. The
accuracy of the model was successfully checked against blind datasets generated by DNSs. Lennon et al. [123]
also developed a tensor basis neural network (TBNN) allowing rheologists to construct learnable constitutive

7
models that incorporate essential physical information while remaining agnostic to details regarding particular
experimental protocols or flow kinematics. The TBNN model incorporates a universal approximator within
a materially objective tensorial constitutive framework, which, by construction, respects physical constraints,
such as frame-invariance and tensor symmetry, required by continuum mechanics. Due to the embedded
TBNN, the developed rheological universal differential equation quickly learns simple yet accurate and highly
general models for describing the provided training data, allowing a rapid discovery of constitutive equations.
Lastly, PgNNs have also been extensively used to improve both the accuracy and speed of CFD solvers.
Stevens and Colonius [116] developed a DL model (the weighted essentially non-oscillatory neural network,
WENO-NN) to enhance a finite-volume method used to discretize PDEs with discontinuous solutions, such
as the turbulence–shock wave interactions (see Fig. 3(d)). Kochkov et al. [18] used hybrid discretizations,
combining CNNs and subcomponents of a numerical solver, to interpolate differential operators onto a coarse
mesh with high accuracy. The training of the model was performed within a standard numerical method for
solving the underlying PDEs as a differentiable program, and the method allows for end-to-end gradient-based
optimization of the entire algorithm. The method learns accurate local operators for convective fluxes and
residual terms and matches the accuracy of an advanced numerical solver running at 8 to 10× finer resolution,
while performing the computation 40 to 80× faster. Cai et al. [124] implemented a least-squares ReLU neural
network (LSNN) for solving the linear advection-reaction problem with a discontinuous solution. They showed
that the proposed method outperformed mesh-based numerical methods in terms of the number of DOFs
(degrees of freedom). Haber et al. [125] suggested an auto-encoder CNN to reduce the resolution cost of a scalar
transport equation coupled to the Navier–Stokes equations. Lara and Ferrer [126] proposed to accelerate high
order discontinuous Galerkin methods using neural networks. The methodology and bounds were examined
for a variety of meshes, polynomial orders, and viscosity values for the 1D viscous Burgers’ equation. List
et al. [127] employed CNN to train turbulence models to improve under-resolved low-resolution solutions to the
incompressible Navier–Stokes equations at simulation time. The developed method consistently outperforms
simulations with a two-fold higher resolution in both spatial and temporal dimensions. For mixing layer cases,
the hybrid model on average resembles the performance of three-fold reference simulations, which corresponds
to a speed-up of 7.0x for the temporal layer and 3.7x for the spatial mixing layer.
Table 2 reports a non-exhaustive list of recent studies that leveraged PgNN to model fluid flow problems.
These studies collectively concluded that PgNNs can be successfully integrated with CFD solvers or used as
standalone surrogate models to develop accurate and yet faster modeling components for scientific computing
in fluid mechanics. In the next section, the potential application of PgNNs in computational solid mechanics
is discussed.

2.2.2. PgNN for Solid Mechanics

Physics-guided neural networks (PgNNs) have also been extensively adopted by the solid mechanics com-
munity. The study by Andersen et al. [35] on welding modeling using ANN was among the first studies that
applied PgNN to solid mechanics. Since then, the application of PgNN has been extended to a wide range
of problems, e.g., structural analysis, topology optimization, inverse materials design and modeling, health
condition assessment, etc., especially to speed up the traditional forward and inverse modeling methods in
computational mechanics.
In the area of structural analysis, Tadesse et al. [132] proposed ANN for predicting mid-span deflections
of a composite bridge with flexible shear connectors. The ANN was tested on six different bridges yielding a
maximum root-mean-squared-error (RMSE) of 3.79%, which can be negligible in practice. The authors also
developed ANN-based close-form solutions to be used for rapid prediction of deflection in everyday design.
Güneyisi et al. [133] employed ANN to develop a new formulation for flexural over-strength factor for steel
beams. They considered 141 experimental data samples with different cross-sectional typologies to train the
model. The results showed a comparable training and testing accuracy of 99 percent, indicating that the ANN
model provided a reliable tool to estimate beams’ over-strength. Hung et al. [134] leveraged ANN to predict
the ultimate load factor of a non-linear inelastic steel truss. They considered a planar 39-bar steel truss to
demonstrate the efficiency of the proposed ANN. They used the cross-sections of members as the input and
load-factor as the output. The ANN-based model yielded a high degree of accuracy, with an average loss
of less than 0.02, in predicting the ultimate load-factor of the non-linear inelastic steel truss. Chen et al.
[135] also used ANN to solve a three-dimensional (3D) inverse problem of a collision between an elastoplastic
hemispherical metal shell and a rigid impactor. The goal was to predict the position, velocity, and duration
of the collision based on the shell’s permanent plastic deformation. For static and dynamic loading, the ANN
model predicted the location, velocity, and collision duration with high accuracy. Hosseinpour et al. [136] used
PgNN for buckling capacity assessment of castellated steel beams subjected to lateral-distortional buckling.
As shown in Fig. 4(a), the ANN-based model provided higher accuracy than well-known design codes, such as
AS4100 [137], AISC [138], and EC3 [139] for modeling and predicting the ultimate moment capacities.
Topology optimization of materials and meta-materials is yet another domain where PgNNs have been
employed [140, 141]. Topology optimization is a technique that identifies the optimal material placement
inside a prescribed domain to achieve the optimal structural performance [142]. For example, Abueidda et al.
[143] developed a CNN model that performs real-time topology optimization of linear and non-linear elastic

8
 Table 2: A non-exhaustive list of studies that leveraged PgNNs to model fluid flow problems.

Area of application NN Type Objective Reference

Laminar Flows CNN Calculating numerical solutions to the inviscid Euler [91]
equations

CNN Predicting the velocity and pressure fields around ar- [93]
bitrary 2D shapes

Turbulent Flows ANN Developing a model for the Reynolds stress anistropy [95]
tensor using high-fidelity simulation data

ANN Modelling of LESs of a turbulent plane jet flow con- [128]

figuration

CNN Designing and training artificial neural networks based [129]

on local convolution filters for LES

ANN Developing subgrid modelling of Kraichnan turbu- [97]

lence

CNN Estimating turbulent heat transfer based on other wall [99]

information acquired from channel flow DNSs

CNN-LSTM Generating turbulent inflow conditions with accurate [104]

statistics and spectra

Aerodynamics CNN-MLP Predicting incompressible laminar steady flow field [130]

over airfoils

ANN Developing a high-dimensional PgNN model for high [131]

Reynolds number turbulent flows around airfoils

ANN Modeling the swirling flow field in a combustor [110]

PCA-ANN Creating surrogate models of computationally expen- [111]

sive, high-fidelity physics models for complex hyper-
sonic turbulent flows

ANN Predicting unsteady aerodynamic performance of iced [115]

airfoil

Viscoelastic Flows ANN Predicting drag coefficient of a spherical particle trans- [22]
lating in viscoelastic fluids

ANN Constructing learnable constitutive models using a [123]

universal approximator within a materially objective
tensorial constitutive framework

Enhance CFD Solvers ANN Developing an improved finite-volume method for sim- [116]
ulating PDEs with discontinuous solutions

CNN Interpolating differential operators onto a coarse mesh [18]

with high accuracy

LSNN Solving the linear advection-reaction problem with [124]

discontinuous solution

CNN Modeling the scalar transport equation to reduce the [125]

resolution cost of forced cooling of a hot workpiece in
a confined environment

CNN Accelerating high order discontinuous Galerkin meth- [126]

ods

CNN Developing turbulence model to improve under- [127]

resolved low-resolution solutions to the incompressible
9 equations at simulation time
Navier–Stokes
materials under large and small deformations. The trained model can predict the optimal designs with great
accuracy without the need for an iterative process scheme and with very low inference computation time. Yu
et al. [144] suggested an integrated two-stage technique made up of a CNN-based encoder and decoder (as
the first stage) and a conditional GAN (as the second stage) that allows for the determination of a near-
optimal topological design. This integration resulted in a model that determines a near-optimal structure
in terms of pixel values and compliance with considerably reduced computational time. Banga et al. [145]
also proposed a 3D encoder-decoder CNN to speed up 3D topology optimization and determine the optimal
computational strategy for its deployment. Their findings showed that the proposed model can reduce the
overall computation time by 40% while achieving accuracy in the range of 96%. Li et al. [146] then presented
a GAN-based non-iterative near-optimal topology optimizer for conductive heat transfer structures trained
on black-and-white density distributions. A GAN for low resolution topology was combined with a super
resolution generative adversarial network, SRGAN, [147, 148] for a high resolution topology solution in a
two-stage hierarchical prediction-refinement pipeline. When compared to conventional topology optimization
techniques, they showed this strategy has clear advantages in terms of computational cost and efficiency.

Figure 4: Panel (a) shows a comparison between ANN and other international codes’ accuracy (e.g., R-squared
and RMSE) to predict the ultimate moment capacities of castellated beams subjected to lateral-distortional
buckling (adapted from Hosseinpour et al. [136]). Panel (b) shows a two-stage PgNN architecture for predicting
the design parameters of meta-materials for spindoid topologies. The first ANN (i.e., inverse PgNN) takes
the query stiffness as input and outputs the design parameters. The second ANN (i.e., forward PgNN) takes
the predicted design parameters and reconstructs the stiffness to verify inverse network accuracy. R-squared
values for prediction of stiffness component C1111 and design parameter ρ are shown in the subsets (adapted
from Kumar et al. [149]). Panel (c) shows a comparison between conditional GAN and ground truth made by
elastography to predict elastic modulus from strain data (adapted from Ni and Gao [150]).

PgNN has also been applied for inverse design and modeling in solid mechanics. Messner [151] employed a
CNN to develop surrogate models that estimate the effective mechanical properties of periodic composites. As
an example, the CNN-based model was applied to solve the inverse design problem of finding structures with
optimal mechanical properties. The surrogate models were in good agreement with well-established topology
optimization methods, such as solid isotropic material with penalization (SIMP) [152], and were sufficiently
accurate to recover optimal solutions for topology optimization. Lininger et al. [153] also used CNN to solve
an inverse design problem for meta-materials made of thin film stacks. The authors demonstrated the CNN’s
remarkable ability to explore the large global design space (up to 1012 parameter combinations) and resolve
all relationships between meta-material structure and associated ellipsometric and reflectance/transmittance
spectra [154, 153]. Kumar et al. [149] proposed a two-stage ANN model, as shown in Fig. 4(b), for inverse
design of meta-materials. The model generates uniform and functionally graded cellular mechanical meta-
materials with tailored anisotropic stiffness and density for spinodoid topologies. The ANN model used in
this study is a combination of two-stage ANN, first ANN (i.e., inverse PgNN) takes query stiffness as input
and outputs design parameters, e.g., Θ. The second ANN (i.e., forward PgNN) takes the predicted design

10
parameters as input and reconstructs the stiffness to verify the first ANN results. The prediction accuracy for
stiffness and design parameter was validated against ground truth data for both networks; sample comparisons
and their corresponding R-squared values are shown in Fig. (4(b). Ni and Gao [150] proposed a combination
of representative sampling spaces and conditional GAN, cGAN [155, 156], to address the inverse problem of
modulus identification in the field of elasticity. They showed that the proposed approach can be deployed with
high accuracy, as shown in Fig. 4(c), while avoiding the use of costly iterative solvers used in conventional
methods, such as the adjoint weighted approach [157]. This model is especially suitable for real-time elastog-
raphy and high-throughput non-destructive testing techniques used in geological exploration, quality control,
composite material evaluation, etc.
The PgNN models have also been used to overcome some of the computational limitations of multiscale
simulations in solid mechanics. This is achieved by (i) bypassing the costly lower-scale calculations and thereby
speeding the macro-scale simulations [61], or (ii) replacing a step or the complete simulation with surrogate
models [61]. For example, Liang et al. [158] developed an ANN model that takes finite element based aorta
geometry as input and output the aortic wall stress distribution directly, bypassing FEM calculation. The
difference between the stress calculated by FEM and the one estimated by the PgNN model is practically
negligible, while the PgNN model produces output in just a fraction of the FEM computational time. Mozaffar
et al. [159] successfully employed RNN-based surrogate models for material modeling by learning the reversible,
irreversible, and history-dependent phenomena that occur when studying material plasticity. Mianroodi et al.
[2] used a CNN-based solver to predict the local stresses in heterogeneous solids with highly non-linear material
response and mechanical contrast features. When compared to common solvers like FEM, the CNN-based
solver offered an acceleration factor of 8300x for elasto-plastic materials. Im et al. [5] proposed a PgNN
framework to construct a surrogate model for a high-dimensional elasto-plastic FEM model by integrating an
LSTM network with the proper orthogonal decomposition (POD) method [160, 161]. The suggested POD-
LSTM surrogate model allows rapid, precise, and reliable predictions of elasto-plastic structures based on the
provided training dataset exclusively. For the first time, Long et al. [162] used a CNN to estimate the stress
intensity factor of planar cracks. Compared to FEM, the key benefit of the proposed light-weight CNN-based
crack evaluation methodology is that it can be installed on an unmanned machine to automatically monitor
the severity of a crack in real-time.
Table 3 reports a non-exhaustive list of recent studies that leveraged PgNNs in solid mechanics and
materials design problems. These studies collectively concluded that PgNNs can be successfully integrated
with conventional solvers (e.g., FEM solvers) or used as standalone surrogate models to develop accurate and
yet faster modeling components for scientific computing in solid mechanics. Albeit, PgNNs come with their
own limitations and shortcomings that might compromise solutions under different conditions, as discussed in
the next section.

2.3. PgNN Limitations

Even though, PgNN-based models show great potential to accelerate the modeling of non-linear phenom-
ena described by input-output interdependencies, they suffer from several critical limitations and shortcomings.
Some of these limitations become more pronounced when the training datasets are sparse.

• The main PgNNs’ limitation stems from the fact that their training process is solely based on statistics
[54]. Even though the training datasets are inherently constrained by physics (e.g., developed by direct
numerical simulation, closure laws, and de-noised experimentation), PgNN generates models based on
correlations in statistical variations. The outputs (predictions), thus, are naturally physics-agnostic
[38, 171] and may violate the underlying physics [6].
• Another important PgNNs’ limitation stems from the fact that training datasets are usually sparse,
especially in the scientific fields discussed in this paper. When the training data is sparse and does not
cover the entire range of underlying physiochemical attributes, the PgNN-based models fail in blind-
testing on conditions outside the scope of training [43], i.e., they do not offer extrapolation capabilities
in terms of spatiotemporal variables and/or other physical attributes.
• PgNN’s predictions might be severely compromised, even for inputs within the scope of sparse training
datasets [22]. The lack of interpolation capabilities is more pronounced in complex and non-linear
problems where the range of the physiochemical attributes is extremely wide (e.g., the range of Reynolds
numbers from creeping flow to turbulent flow).

• PgNNs may not fully satisfy the initial conditions and boundary conditions using which the training
datasets are generated [38]. The boundary conditions and computational domain vary from one problem
to another, making the data generation and training process prohibitively costly. In addition, a significant
portion of scientific computing research involves inverse problems in which unknown physiochemical
attributes of interest are estimated from measurements or calculations that are only indirectly related to
these attributes [172, 173, 10, 13]. For instance, in groundwater flow modeling, we leverage measurements
of the pressure of a fluid immersed in an aquifer to estimate the aquifer’s geometry and/or material

11
 Table 3: A non-exhaustive list of studies that leveraged PgNNs to model solid mechanics problems.

Area of application NN Type Objective Reference

Accelerating Simulations ANN Predicting the aortic wall stress distribution using [158]
FEM aorta geometry

RNN Developing surrogate models for material model- [159]

ing by learning reversible, irreversible, and history-
dependent phenomena

CNN Predicting local stresses in heterogeneous solids [2]

with highly non-linear material response and me-
chanical contrast features

CNN Estimating stress intensity factor of planar cracks [162]

Topology Optimization CNN Optimizing topology of linear and non-linear elastic [143]
materials under large and small deformations

CNN-GAN Determining near-optimal topological design [144]

CNN Accelerating 3D topology optimization [145]

GAN- Generating near-optimal topologies for conductive [146]

SRGAN heat transfer structures

Inverse Modeling CNN Estimating effective mechanical properties for peri- [151]
odic composites

CNN Solving an inverse design problem for meta- [153]

materials made of thin film stacks

cGAN Addressing inverse problem of modulus identifica- [150]

tion in elasticity

CVAE Designing nano-patterned power splitters for pho- [150]

tonic integrated circuits

Structural Elements ANN Predicting non-linear buckling load of an imperfect [163]

reticulated shell

ANN Optimizing dynamic behavior of thin-walled lami- [164]

nated cylindrical shells

ANN Determining and identifying loading conditions for [135]

shell structures

Structural Analysis CNN Forecasting stress fields in 2D linear elastic can- [165]
tilevered structures subjected to extrenal static
loads

ANN Estimaating the thickness and length of reinforced [166]

walls based on previous architectural projects

Condition Assessment Auto- Learning mapping between vibration characteristics [167]

encoder-NN and structural damage

CNN Providing a real-time crack assessment method [168]

RNN Nonparametric identification of large civil struc- [169]

tures subjected to dynamic loadings

CNN Damage Identification of truss structures using [170]

noisy incomplete modal data

12
 characteristics [174]. Such requirements further complicate the process of developing a simple neural
network that is predictive under any conditions.
• PgNNs-based models are not resolution-invariant by construction [39], hence they cannot be trained on
a lower resolution and be directly inferred on a higher resolution. This shortcoming is due to the fact
that PgNN is only designed to learn the solution of a physical phenomena for a single instance (i.e.
inputs-outputs).
• Through the training process, PgNN-based networks learn the input-output interdependencies across the
entire dataset. Such a process could potentially consider slight variations in the functional dependencies
between different input and output pairs as noise, and produce an average solution. Consequently,
while these models are optimal with respect to the entire data, they may produce suboptimal results on
individual cases.
• PgNN models may struggle to learn the underlying process when the training dataset is diverse, i.e.,
when the interdependencies between different input and output pairs are drastically different. Although
this issue can be mitigated by increasing the model size, more data is required to train such a network,
making the training costly and, in some cases, impractical.

One way to resolve some of the PgNNs’ limitations is to generate more training data. However, this is
not always a feasible solution due to the high cost of data acquisition. Alternatively, PgNNs can be further
constrained by governing physical laws without any prior assumptions, reducing the need for large datasets.
The latter is a plausible solution because, in most cases, the physical phenomenon can be fully and partially
described using explicit ODEs, PDEs, and/or closure laws. This approach led to the development of a physics-
informed neural network, which is described and reviewed in the next section.

3. Physics-informed Neural Network, PiNN

In scientific computing, physical phenomena are often described using a strong mathematical form con-
sisting of governing differential equations as well as initial and boundary conditions. At each point inside a
domain, the strong form specifies the constraints that a solution must meet. The governing equations are
usually linear or non-linear PDEs and/or ODEs. Some of the PDEs are notoriously challenging to solve, e.g.,
the Navier-Stokes equations to explain a wide range of fluid flows [10], Föppl–von Kármán equations to de-
scribe large deflections in solids [175, 175], etc. Other important PDE examples are heat equations [176], wave
equation [177], Burgers’ equation [178], Laplace’s equation [179], Poisson’s equation [180], amongst others.
This wealth of well-tested knowledge can be logically leveraged to further constrain PgNNs while training on
available data points, if any [38]. To this end, mesh-free physics-informed neural networks (PiNNs) have been
developed [38, 44], quickly extended [181, 182], and extensively deployed in a variety of scientific and applied
fields [183, 184, 185, 186, 187, 188].
A schematic representation of a sample PiNN architecture is illustrated in Fig. 5. In PiNNs, the underlying
physics is incorporated outside the neural network architecture to constrain the model while training, thereby
ensuring outputs follow known physical laws. The most common method to emulate this process is through
a weakly imposed penalty loss that penalizes the network when not following the physical constraints. As
shown in Fig. 5, a neural network with spatiotemporal features (i.e., x and t) as input parameters and the
PDE solution elements as output parameters (i.e., u) can be used to emulate any PDEs.
The network’s outputs are then fed into the next layer, which is an automated differentiation layer. In
this instance, multiple partial derivatives are generated by differentiating the outputs with regard to the input
parameters (x and t). With the goal of optimizing the PDE solution, these partial derivatives are used to
generate the required terms in the loss function. The loss function in PiNN is a combination of the loss owing
to labelled data (LData ), governing PDEs (LP DE ), applied initial conditions (LIC ) and applied boundary
conditions (LBC ) [10]. The LBC ensures that the PiNN’s solution meets the specified boundary constraints,
whereas LData assures that the PiNN follows the trend in the training dataset (i.e., historical data, if any).
Furthermore, the structure of the PDE is enforced in PiNN through the LP DE , which specifies the collocation
points where the solution to the PDE holds [38]. The weights for the loss due to the initial conditions,
boundary conditions, data, and PDE can be specified as wi , wb , wd , and wp , respectively. The next step is to
check, for a given iteration, if the loss is within the accepted tolerance, . If not, the learnable parameters of
the network (θ) and unknown PDE parameters (λ) are updated through error backpropagation. For a given
number of iterations, the entire cycle is repeated until the PiNN model produces learnable parameters with loss
functions less than . Note that the training of PiNNs is more complicated compared to PgNNs, as PiNNs are
composed of sophisticated non-convex and multi-objective loss functions that may result in instability during
optimization [38, 6, 10].
Dissanayake and Phan-Thien [189] were the first to investigate the incorporation of prior knowledge into
a neural network. Subsequently, Owhadi [190] introduced the concept of physics-informed learning models
as a result of the ever-increasing computing power, which enables the use of increasingly complex networks
with more learnable parameters and layers. The PiNN, as a new computing paradigm for both forward and
13
Figure 5: A schematic architecture of Physics-informed Neural Networks (PiNNs). The network digests
spatiotemporal coordinates (x ,t) as inputs to approximate the multi-physics solution û. The last layer gener-
ates the derivatives of the predicted solution u with respect to inputs, which are calculated using automatic
differentiation (AD). These derivatives are used to formulate the residuals of the governing equations in the
loss function, which is composed of multiple terms weighted by different coefficients. θ and λ are the learnable
parameters for weights/biases and unknown PDE parameters, respectively, that can be learned simultaneously
while minimizing the loss function.

inverse modeling, was introduced by Raissi et al. in a series of papers [38, 191, 44]. Raissi et al. [38] deployed
two PiNN models, a continuous and a discrete time models, on examples consisting of different boundary
conditions, critical non-linearities, and complex valued solutions such as Burgers’, Schrodinger’s and Allen-
Cahn’s equations. The results for Burgers’ equation demonstrated that, given a sufficient number of collocation
points (i.e., as the basis for the continuous model), an accurate and data-efficient learning procedure can be
obtained [38].
In continuous PiNN models, when dealing with higher dimensional problems, the number of collocation
points increases exponentially, making learning processing difficult and computationally expensive [38, 6].
Raissi et al. [38] presented a discrete time model based on the Runge-Kutta technique [192] to address the
computational cost issue. This model simply takes a spatial feature as input, and over time steps, PiNN
converges to the underlying physics. For all the examples explored by Raissi et al. [38], continuous and discrete
PiNN models were able to satisfactorily build physics-informed surrogate models. Nabian et al. [193] proposed
an alternate method for managing collocation points. They investigated the effect of sampling collocation
points according to distribution and discovered that it was proportional to the loss function. This concept
requires no additional hyperparameters and is simpler to deploy in existing PiNN models. In their study, they
claimed that a sampling approach for collocation points enhanced the PiNN model’s behavior during training.
The results were validated by deploying the hypothesis on PDEs for solving problems related to elasticity,
diffusion, and plane stress physics.
In order to use PiNN to handle inverse problems, the loss function of the deep neural network must satisfy
both the measured and unknown values at a collection of collocation sites distributed throughout the problem
domain. Raissi et al. [44] also showcased the potential of both continuous and discrete time PiNN models to
solve benchmark inverse problems such as the propagation of non-linear shallow-water waves (Korteweg–De
Vries equation) [194] and incompressible fluid flows (Navier-Stokes equations) [195].
Compared to PgNNs, the PiNN models provide more accurate predictions for forward and inverse model-
ing, particularly in scenarios with high non-linearities, limited data, or noisy data [196]. As a result, it has been
implemented in several fundamental scientific and applied fields. Aside from forward and inverse problems,
the PiNN can also be used to develop partial differential equations for unknown phenomena if training data
representing the phenomenon’s underlying physics is available [44]. Raissi et al. [44] leveraged both continuous
time and discrete time PiNN models for generating universal PDEs depending on the type and structure of
the available data. In the remainder of this section, we review the recent literature on PiNN’s applications in
the fluid and solid mechanics fields.

3.1. PiNN for Fluid Mechanics

The application of PiNNs to problems involving fluid flow is an active, ongoing field of study [197,
198]. Raissi et al. [191], in a seminal work, developed a PiNN, so-called hidden fluid mechanics (HFM),

14
to encode physical laws governing fluid motions, i.e., Navier-Stokes equations. They employed underlying
conservation laws to derive hidden quantities of interest such as velocity and pressure fields from spatiotemporal
visualizations of a passive scalar concentration, e.g., dye, transported in arbitrarily complex domains. Their
algorithm to solve the data assimilation problem is agnostic to the boundary and initial conditions as well
as to the geometry. Their model successfully predicted 2D and 3D pressure and velocity fields in benchmark
problems inspired by real-world applications. Figure 6, adapted from Raissi et al. [191], compares the PiNN
prediction with the ground truth for the classical problem of a 2D flow past a cylinder. The model can be
used to extract valuable quantitative information such as wall shear stresses, and lift and drag forces for which
direct measurements are difficult to obtain.

Figure 6: A comparison between ground truth simulation results and PiNN predictions for a 2D flow past
a circular cylinder. Comparisons are shown for the concentration of passive scalar, c(t, x, y), and resulting
velocity fields, u,v, and pressure field, p (adapted from Raissi et al. [191]).

Zhang et al. [199] also developed a PiNN framework for the incompressible fluid flow past a cylinder
governed by Navier-Stokes equations. PiNN learns the relationship between simulation output (i.e., velocity
and pressure) and the underlying geometry, boundary, initial conditions, and inherently fluid properties. They
demonstrated that the generalization performance is enhanced across both the temporal domain and design
space by including Fourier features [200], such as frequency and phase offset parameters. Cheng and Zhang
[201] developed Res-PiNN (i.e., Resnet blocks along with PiNN) for simulating cavity flow and flow past a
cylinder governed by Burgers’ and Navier-Stokes equations. Their results showed that Res-PiNN had better
predictive ability than conventional PgNN and PiNN algorithms. Lou et al. [202] also demonstrated the
potential of PiNN for solving inverse multiscale flow problems. They used PiNN for inverse modeling in both
the continuum and rare-field regimes represented by the Boltzmann-Bhatnagar-Gross-Krook (BGK) collision
model. The results showed that PiNN-BGK is a unified method (i.e., it can be used for forward and inverse
modeling), easy to implement, and effective in solving ill-posed inverse problems [202].
Wessels et al. [203] employed PiNN to develop an updated Lagrangian method for the solution of incom-
pressible free surface flow subject to the inviscid Euler equations, the so-called Neural Particle Method (NPM).
The method does not require any specific algorithmic treatment, which is usually necessary to accurately re-
solve the incompressibility constraint. In their work, it was demonstrated that NPM is able to accurately
compute a pressure field that satisfies the incompressibility condition while avoiding topological constraints on
the discretization process [203]. In addition, PiNN has also been employed to model complex non-Newtonian
fluid flows involving non-linear constitutive PDEs able to characterize the fluid’s rheological behavior [204].
Haghighat et al. [205] trained a PiNN model to solve the dimensionless form of the governing equations
of coupled multiphase flow and deformation in porous media. Almajid and Abu-Al-Saud [206] compared the
predictions of PiNN with those of PgNN, i.e., a conventional artificial neural network, for solving the gas
drainage problem of water-filled porous media. The study showed that PgNN performs well under certain
15
conditions (i.e., when the observed data consists of early and late time saturation profiles), while the PiNN
model performs robustly even when the observed data contains only an early time saturation profile (where
extrapolations are needed). Depina et al. [207] applied PiNN to model unsaturated groundwater flow problems
governed by the Richards PDE and van Genuchten constitutive model [208]. They demonstrated that PiNNs
can efficiently estimate the van Genuchten model parameters and solve the inverse problem with a relatively
accurate approximation of the solution to the Richards equation.
Some of the other variants of PiNN models employed in fluid mechanics are: nn-PiNN, where PiNN
is employed to solve constitutive models in conjunction with conservation of mass and momentum for non-
Newtonian fluids [204]; ViscoelasticNet, where PiNN is used for stress discovery and viscoelastic flow models
selection [209], such as Oldroyd-B [119], Giesekus and Linear PTT [210]; RhINN which is a rheology-informed
neural networks employed to solve constitutive equations for a Thixotropic-Elasto-Visco-Plastic complex fluid
for a series of flow protocols [183]; CAN-PiNN, which is a coupled-automatic-numerical differential framework
that combines the benefits of numerical differentiation (ND) and automatic differentiation (AD) for robust
and efficient training of PiNN [211]; ModalPiNN, which is a combination of PiNN with enforced truncated
Fourier decomposition [212] for periodic flow reconstruction [213]; GAPiNN, which is a geometry aware PiNN
consisted of variational auto encoder, PiNN and boundary constraining network for real-world applications
with irregular geometries without parameterization [214]; Spline-PiNN, which is a combination of PiNN and
Hermite spline kernels based CNN employed to train a PiNN without any pre-computed training data and
provide fast, continuous solutions that generalize to unseen domains [215]; cPiNN, which is a conservative
physics-informed neural network consisting of several PiNNs communicating through the sub-domain interfaces
flux continuity for solving conservation laws [181]; SA-PiNN, which is a self-adaptive PiNN to address the
adaptive procedures needed to force PiNN to fit accurately the stubborn spots in the solution of stiff PDEs
[49]; and XPiNN, which is an extended PiNN to enhance the representation and parallelization capacity of
PiNN and generalization to any type of PDEs with respect to cPINN [182].
Table 4 reports a non-exhaustive list of recent studies that leveraged PiNN to model fluid flow problems.
Furthermore, Table 5 reports a non-exhaustive list of recent studies that developed other variants of PiNN
architectures to improve the overall prediction accuracy and computational cost in fluid flow problems.

3.2. PiNN for Solid Mechanics

The application of PiNN in computational solid mechanics is also an active, ongoing field of study. The
study by Haghighat et al. [226] on modeling linear elasticity using PiNN was among the first papers that
introduced PiNN in the solid mechanics community. Since then, the framework has been extended to other
solid-mechanics problems (e.g., linear and non-linear elastoplasticity, etc.).
Shukla et al. [227] used PiNN for surrogate modeling of the micro-structural properties of poly-crystalline
nickel. In their study, in addition to employing the PiNN model, they applied an adaptive activation function
to accelerate the convergence of numerical modeling. The resulting PiNN-based surrogate model demonstrated
a viable strategy for non-destructive material evaluation. Henkes et al. [228] modeled non-linear stress and
displacement fields induced by inhomogeneities in materials with sharp phase transitions using PiNN. To
overcome the PiNN’s convergence issues in this problem, they used adaptive training approaches and domain
decomposition [203]. According to their results, the domain decomposition approach is capable of properly
resolving non-linear stress, displacement, and energy in heterogeneous microstructures derived from real-world
µCT-scans images [228]. Goswami et al. [229] used PiNN in the V-DeepONet framework to determine failure
pathways, failure zones, and damage along failure in brittle fractures for quasi-brittle materials. They trained
the model to map the initial configuration of the defect to the relevant fields of interest (e.g., damage and
displacements). They showed that their model can rapidly predict the solution to any initial crack configuration
and loading steps. In fact, the proposed model can be used for design optimization, reliability analysis, and
uncertainty quantification in brittle fracture mechanics.
Zhang and Gu [230] trained a PiNN model with a loss function based on the minimal energy criteria
to investigate digital materials. The model tested on 1D tension, 1D bending, and 2D tensile problems
demonstrated equivalent performance when compared to supervised DL methods (i.e., PgNNs). By adding a
hinge loss for the Jacobian matrix, the PiNN method was able to properly approximate the logarithmic strain
and rectify any erroneous deformation gradient.
Rao et al. [231] proposed a PiNN architecture with mixed-variable (displacement and stress component)
outputs to handle elastodynamic problems without labeled data. The method was found to boost the network’s
accuracy and trainability in contrast to the pure displacement-based PiNN model. Figure 7 compares the
ground truth stress fields generated by the FEM with the ones estimated by mixed-variable PiNN for an
elastodynamic problem [231]. It can be observed that stress components can be accurately estimated by
mixed-variable PiNN. Rao et al. [231] also proposed a composite scheme of PiNN to enforce the initial and
boundary conditions in a hard manner as opposed to the conventional (vanilla) PiNN with soft initial and
boundary condition enforcement. This model was tested on a series of dynamics problems (e.g., the defected
plate under cyclic uni-axial tension and the elastic wave propagation), and resulted in the mitigation of
inaccuracies near the boundaries encountered by PiNN.

16
 Table 4: A non-exhaustive list of recent studies that leveraged PiNN to model fluid flow problems.

Area of Application Objectives Reference

Incompressible Flows Accelerating the modeling of Navier-Stokes equations to infer the [44]
solution for various 2D and 3D flow problems

Learning the relationship between output and underlying geometry [199]

as well as boundary conditions

Simulating ill-posed (e.g., lacking boundary conditions) or inverse [216]

laminar and turbulent flow problems

Turbulent Flows Solving vortex-induced and wake-induced vibration of a cylinder [217]

at high Reynolds number

Simulating turbulent incompressible flows without using any spe- [218]

cific model or making turbulence assumptions

Reconstructing Reynolds stress disparities described by Reynolds- [219]

averaged Navier-Stokes equations

Geofluid Flows Solving well-based groundwater flow equations without utilizing [220]
labeled data

Predicting high-fidelity multi-physics data from low-fidelity fluid [221]

flow and transport phenomena in porous media

Estimating Darcy’s law-governed hydraulic conductivity for both [222]

saturated and unsaturated flows

Non-Newtonian Flows Solving systems of coupled PDEs adopted for non-Newtonian fluid [204]
flow modeling

Simulating linear viscoelastic flow models such as Oldroyd-B, [209]

Giesekus and Linear PTT

Simulating direct and inverse solutions of rheological constitutive [183]

models for complex fluids

Biomedical Flows Enabling the seamless synthesis of non-invasive in-vivo measure- [223]
ment techniques and computational flow dynamics models derived
from first physical principles

Enhancing the quantification of near-wall blood flow and wall shear [224]
stress arterial in diseased arterial flows

Supersonic Flows Solving inverse supersonic flow problems involving expansion and [198]
compression waves

Surface Water Flows Solving ill-posed strongly non-linear and weakly-dispersive surface [225]
water waves governed by Serre-Green-Naghdi equations using only
data of the free surface elevation and depth of the water.

17
Table 5: A non-exhaustive list of different variants of PiNN architectures used in modeling fluid flow problems.

PiNN Structure Objective Reference

CAN-PiNN Providing PiNN’s robustness and efficient training by integrating ND- [211]
and AD-based approach to solve incompressible Navier–Stokes equations

ModalPiNN Providing a simpler representation of PiNN for oscillating phenomena [213]

to improve performance with respect to sparsity, noise and lack of syn-
chronisation in the data

GA-PiNN Enhancing PiNN to develop a parameter-free, irregular geometry-based [214]

surrogate model for fluid flow modeling

Spline-PiNN Improving the generalisation of PiNN by combining it with Hermite [215]

spline CNN to solve incompressible Navier-Stokes equations

cPiNN Enhancing PiNN to solve high dimensional non-linear conservation laws [181]
requiring high computational and memory requirements

SA-PiNN Improving the PiNN’s convergence and accuracy problem for stiff PDEs [49]
using self-adaptive weights in the training

XPiNN Improving PiNN and cPiNN in terms of generalization, representation, [182]

parllelization capacity, and computational cost

Fang and Zhan [232] proposed a PiNN model to design the electromagnetic meta-materials used in various
practical applications such as cloaking, rotators, concentrators, etc. They studied PiNN’s inference issues for
Maxwell’s equation [233] with a high wave number in the frequency domain by improving the activation
function to overcome the high wave number problems. The proposed PiNN recovers not only the continuous
functions but also piecewise functions, which is a new contribution to the application of PiNN in practical
problems.
Zhang et al. [234] employed PiNN to identify nonhomogenous materials in elastic imaging for application
in soft tissues. Two PiNNs were used, one for the approximate solution of the forward problem and another
for approximating the field of the unknown material parameters. The results showed that the unknown
distribution of mechanical properties can be accurately recovered using PiNN. Abueidda et al. [235] employed
PiNN to simulate 3D hyperelasticity problems. They proposed an Enhanced-PiNN architecture consisting of
the residuals of the strong form and the potential energy [236], producing several loss terms contributing to the
definition of the total loss function to be minimized. The enhanced PiNN outperformed both the conventional
(vanilla) PiNN and deep energy methods, especially when there were areas of high solution gradients.
Haghighat et al. [13] tested a different variant of PiNN to handle inverse problems and surrogate modeling
in solid mechanics. Instead of employing a single neural network, they implemented a PiNN with multiple
neural networks in their study. They deployed the framework on linear elastostatic and non-linear elasto-
plasticity problems, and showed that the improved PiNN model provides a more reliable representation of
the physical parameters. In addition, they investigated the domain of transfer learning in PiNN and found
that the training phase converges more rapidly when transfer learning is used. Yuan et al. [237] proposed an
auxiliary PiNN model (dubbed as A-PiNN) to solve inverse problems of non-linear integro-differential equa-
tions (IDEs). A-PiNNs circumvent the limitation of integral discretization by establishing auxiliary output
variables in the governing equation to represent the integral(s) and by substituting the integral operator with
automated differentiation of the auxiliary output. Therefore, A-PiNN, with its multi-output neural network,
is constructed such that it determines both primary and auxiliary outputs to approximate both the variables
and integrals in the governing equations. The A-PiNNs were used to address the inverse issue of non-linear
IDEs, including the Volterra equation [238]. As demonstrated by their findings, the unknown parameters can
be determined satisfactorily even with noisy data.
Some of the other variants of PiNN used in solid mechanics are: PhySRNet, which is a PiNN-based super-
resolution framework for reconstructing high resolution output fields from low resolution counterparts without
requiring high-resolution labelled data [239]; PDDO-PiNN, which is a combination of peridynamic differential
operator (PDDO) [240] and PiNN to overcome degrading performance of PiNN under sharp gradients [241];
PiELM, which is a combination of PiNN and extreme learning machine (ELM) [242] employed to solve direct
problems in linear elasticity [243]; DPiNN, which is a distributed PiNN utilizing a piecewise-neural network
representation for the underlying field, instead of the piece-polynomial representation commonly used in FEM
[50]; and PiNN-FEM, which is a mixed formulation based on PiNN and FE for computational mechanics in
heterogeneous domain [244].
Table 6 reports a non-exhaustive list of recent studies that leveraged PiNN in computational solid mechan-

18
Figure 7: A comparison between mixed-variable PiNN’s prediction and ground truth generated by FEM to
predict the stress fields in a defected plate under uni-axial load (adapted from Rao et al. [231]).

ics. Furthermore, Table 7 reports a non-exhaustive list of recent studies that developed other variants of PiNN
architectures to improve overall prediction accuracy and computational cost in solid mechanics modeling.

3.3. PiNN Limitations

PiNNs show a great potential to be used in modeling dynamical systems described by ODEs and/or PDEs,
however, they come with several limitations and shortcomings that must be considered:

• Vanilla PiNNs use deep networks consisting of a series of fully connected layers and a variant of gradient
descent optimization. The learning process and hyperparameter tuning are conducted manually and are
sample size- and problem-dependent. The training, thus, may face gradient vanishing problems and can
be prohibitively slow for practical three-dimensional problems [257]. In addition, vanilla PiNNs impose
limitations on low-dimensional spatiotemporal parameterization due to the usage of fully connected layers
[40].

• There is no theoretical proof of convergence for PiNNs, even when applied to smooth problems. PiNNs
contain several terms in the loss function with relative weighting that greatly affects the final solution.
There are, currently, no guidelines for selecting weights optimally [50]. Different terms in the loss
function may compete with each other during training and this competition may reduce the stability of
the training process. Therefore convergence to the global minimum cannot be assured [6]. PiNNs also
suffer during training when confronted by an ill-posed optimization problem due to their dependence on
soft physical constraints [40].
• PiNNs suffer from low-frequency induced bias and frequently fail to solve non-linear PDEs for problems
governed by high-frequency or multiscale structures [258]. In fact, PiNNs may experience difficulty
propagating information from initial conditions or boundary conditions to unseen parts of the domain
or to future times, especially in large computational domains (e.g., unsteady turbulent flow) [43].
• PiNNs are solution learning algorithms, i.e., they learn the solutions to a given PDE for a single instance.
For any given new instance of the functional parameters or coefficients, PiNNs require training a new
neural network. This is because, by construction, PiNNs cannot learn the physical operation of a given
phenomenon, and that limits their generalization (e.g., spatiotemporal extrapolation) [259]. The PiNN
approach, thus, suffers from the same computational issue as classical solvers, especially in 3D problems
(e.g., FEM, FVM, etc.), as the optimization problem needs to be solved for every new instance of PDE
parameters, boundary conditions, and initial conditions.
• PiNNs encounter difficulties while learning the solutions to inverse problems in heterogeneous media, e.g.,
a composite slab composed of several materials [257]. In such cases, the parameters of the underlying
PDE (e.g., conductivity or permeability coefficients) change across the domain, yet a PiNN outputs
unique parameter values over the whole domain due to its inherent design.

Despite the shortcomings, PiNNs offer a strong promise for complex domains that are hard to mesh and
practical problems where data acquisition is expensive. To circumvent some of the limitations of PiNN, several
techniques have been proposed. For instance, to address the first limitation listed above, discrete learning
techniques using convolutional filters, such as HybridNet [260], dense convolutional encoder-decoder network
[261], auto-regressive encoder-decoder model [262], TF-Net [263], DiscretizationNet [264], and PhyGeoNet
[265], just to name a few, have been employed that exceed vanilla PiNN in terms of computational efficiency. As
another example, to address the last limitation listed above, Dwivedi et al. [257] proposed a Distributed PiNN
(DPiNN) that has potential advantages over existing PiNNs to solve the inverse problems in heterogeneous

19
Table 6: A non-exhaustive list of recent studies that leveraged PiNN in computational solid mechanics.

Area of Application Objectives Reference

Elasticity Solving forward and inverse problems in linear elastostatic and [13]
non-linear elasticity problems

Simulating forward and discovery problems for linear elasticity [226]

Resolving the non-homogeneous material identification problem [234]

in elasticity imaging

Estimating elastic properties of tissues using pre- and post- [245]

compression images of objects mimicking properties of tissues

Estimating mechanical response of elastic plates under different [246]

loading conditions

Finding optimal solutions to reference biharmonic problems of [247]

elasticity and elastic plate theory

Simulating elastodynamic problems, e.g., elastic wave propaga- [231]

tion, deflected plate under periodic uniaxial strain, without la-
beled data

Heterogeneous Materials Inferring the spatial variation of compliance coefficients of mate- [227]
rials (e.g., speed of the elastic waves) to identify microstructure

Resolving non-linear stress, displacement and energy fields in [228]

heterogeneous microstructures

Solving coupled thermo-mechanics problems in composite mate- [248]

rials

Predicting the size, shape, and location of the internal structures [249]
(e.g., void, inclusion) using linear elasticity, hyperelasticity, and
plasticity constitutive models

Structural Elements Predicting the small-strain response of arbitrarily curved shells [250]

Solving mechanical problems of elasticity in one-dimensional el- [184]

ements such as rods and beams

Predicting creep-fatigue life of components (316 stainless steel) [251]

at elevated temperatures

Structural Vibrations Estimating and optimizing vibration characteristics and system [252]
properties of structural mechanics and vibration problems

Digital Materials Resolving physical behaviors of digital materials to design next- [230]
generation composites

Fracture Mechanics Reconstructing the solution of displacement field after damage [253]
to predict crack propagation for quasi-brittle materials

Elasto-viscoplasticity Modeling the strain-rate and temperature dependence of the de- [254]
formation fields (i.e., displacement, stress, plastic strain)

Additive Manufacturing Predicting finite fatigue life in materials containing defects [185]

Solid Mechanics Providing a detailed introduction to programming PiNN-based [255]

computational solid mechanics from 1D to 3D problems

20
Table 7: A non-exhaustive list of different variants of PiNN architectures used in computational mechanics
problems.

PiNN Architecture Objectives Reference

PhySRNet Enhancing PiNN using super resolution techniques to reconstruct down- [239]
scaled fields from their upscaled counterparts

Enhanced PiNN Improving the interpolation and extrapolation ability of PiNN by in- [235]
tegrating potential energy, collocation method, and deep learning for
hyperelastic problems

PDDO-PiNN Improving the performance of PiNN in presence of sharp gradient by [241]

integrating PDDO and PiNN methods to solve elastoplastic deformation
problems

PiELM Accelerating PiNN’s training process by integrating PiNN and ELM [243]
methods to model high dimensional shell structures

PiELM Accelerating PiNN’s training process by integrating PiNN and ELM [256]
methods to model biharmonic equations

DPiNN Providing a truly unified framework for addressing problems in solid [50]
mechanics Solving high dimensional inverse in heterogeneous media such
as linear elasticity

A-PiNN Improving PiNN model to solve inverse problems of non-linear integro- [237]
differential equations

PiNN-FEM Hybridizing FEM and PiNN to solve heterogeneous media problem such [244]
as elasticity and poisson equation

media, which are most likely to be encountered in engineering practices. Some of the other solutions to
solve high dimensional inverse problems are Conservative PiNN (cPiNN) [181] and Self-Adaptive PiNN [49].
Further, XPiNN [182], with its intrinsic parallelization capabilities to deploy multiple neural networks in smaller
subdomains, can be used to considerably reduce the computational cost of PiNNs in large (three-dimensional)
domains.
However, these modifications and alternatives do not solve the generalization problem of the PiNN as the
resultant models lack the ability to ”hard-code” the existing physical knowledge. To this end, physics-encoded
neural networks (PeNNs) have started to emerge. In the next section, we will review the recent literature on
physics-encoded neural networks and their different subcategories.

4. Physics-encoded Neural Network, PeNN

Physics-encoded Neural Networks (PeNNs) are another family of mesh-free algorithms used in scientific
computing, mostly in the fluid mechanics and solid mechanics fields, that strive to hard-encode physics (i.e.,
prior knowledge) into the core architecture of the neural networks. The resultant models have enhanced
generalization, interpretability, and computational efficiency compared with PgNN and PiNN [39, 266, 40].
PeNN extends the learning capability of a neural network from instance learning (imposed by PgNN and PiNN
architectures) to continuous learning [266, 39]. For these reasons, PeNN is also gaining considerable attention
despite its implementation complexity. To hard-encode physical laws (in terms of ODEs, PDEs, closure laws,
etc.) into a neural network, different approaches have been proposed [40, 266, 39, 8]. In this section, we review
three of the most prominent algorithms and their applications in fluid and solid mechanics: (i) Physics-encoded
Recurrent Convolutional Neural Network (PeRCNN) [40], (ii) Fourier Neural Operators (FNO) [39], and (iii)
Differential Programming (DP) or Neural Ordinary Differential Equations (NeuralODE) [266, 8].

4.1. Physics-encoded Recurrent Convolutional Neural Network (PeRCNN)

Rao et al. [40] introduced the PeRCNN model that hard encodes prior knowledge governing non-linear
systems into a neural network. The PeRCNN architecture shown in Fig. 8 facilitates learning in a data-driven
manner while forcibly encoding the known physics knowledge. This model exceeds PgNN and PiNN’s capabili-
ties for phenomena in which the explicit formulation of PDEs does not exist and very limited measurement data
is available (e.g., Earth or climate system modeling [51]). The proposed encoding mechanism of physics, which
is fundamentally different from the penalty-based physics-informed learning, ensures the network rigorously

21
Figure 8: An schematic architecture for Physics-encoded Recurrent Convolutional Neural Network [40]. The
architecture consist of X as initial inputs, convolutional layers, Xo as full resolution initial state, unconventional
convolutional block (π block), and the predicted output layer Y’. Further, the π block consist of multiple
parallel convolutional layers whose operations are defined as Sn × Wn , where n is the number of layers; π
carries out element-wise product and + carries out element-wise addition.

obeys the given physics. Instead of using non-linear activation functions, they proposed a novel element-wise
product operation to achieve the non-linearity of the model. Numerical experiments demonstrated that the
resulting physics-encoded learning paradigm possesses remarkable robustness against data noise/scarcity and
generalizability compared with some state-of-the-art models for data-driven modeling.
As shown in Fig. 8, PeRCNN is made of: an input layer, which is constituted by low-resolution noisy
initial state measurements X = [X1 , X2 , X3 , ..., Xn ]; a fully convolutional network, as the initial state generator
(ISG), which downscales/upsamples the low resolution initial state to a full resolution initial state, dubbed as
modified Xo , to be used as input to further recurrent computations. For the purpose of recurrent computing,
an unconventional convolutional block, dubbed as π, is employed [40]. In the π block, which is the core of
PeRCNN, the modified Xo goes through multiple parallel convolutional layers, whose feature maps will then
be fused via an element-wise product layer. Further, a one-by-one (1×1) convolutional layer [267], is appended
after the product operation to aggregate multiple channels into the output of the desired number of channels.
Assuming the output of the 1 × 1 convolution layer approximates the non-linear function, it can be multiplied
by the time spacing δt to obtain the residual of the dynamical system at time tk , i.e., δUk . Ultimately, the
last layer generates predictions Y 0 = [Y10 , Y20 , Y30 , ..., Yn0 ] by element-wise addition. These operations are shown
schematically in Fig. 8.
The PeRCNN architecture was tested on two datasets representing 2D Burgers and 3D Gray-Scott
reaction-diffusion equations [40]. In both cases, PeRCNN was compared with Convolutional Long-Short Term
Memory [268], Deep Residual Network [269], and Deep Hidden Physics Models [171] in terms of accuracy
(root-mean-squared-error, RMSE), data noise/scarcity, and generalization. The comparison for 2D Burgers’
dataset is shown in Fig. 9(a), adapted from [40]. The accumulative RMSE for PeRCNN began with a larger
value in the training region (due to 10 percent Gaussian noise in the data) and reduced as additional time
steps were assessed. The accumulative RMSE for PeRCNN slightly increases in the extrapolation phase (as
a measure of the model’s generalization), but clearly surpasses all other algorithms in terms of long-term
extrapolation.
Ren et al. [270] proposed a hybrid algorithm combining PeRCNN and PiNN to solve the limitations
in low-dimensional spatiotemporal parameterization encountered by PgNNs and PiNNs. In the resultant
physics-informed convolutional-recurrent network, dubbed as PhyCRNet, an encoder-decoder convolutional
long-short term memory network is proposed for low-dimensional spatial feature extraction and temporal
evolution learning. In PhyCRNet, the loss function is specified as aggregated discretized PDE residuals. The
boundary conditions are hard-coded in the network via designated padding, and initial conditions are defined
as the first input state variable for the network. Autoregressive and residual connections that explicitly
simulate time marching were used to enhance the networks. This method ensures generalization to a variety
of initial and boundary condition scenarios and yields a well-posed optimization problem in network training.
Using PhyCRNet, it is also possible to simultaneously enforce known conservation laws into the network (e.g.,
mass conservation can be enforced by applying a stream function as the solution variable in the network for
fluid dynamics) [270]. Ren et al. [270] evaluated and validated the performance of PhyCRNet using several

22
Figure 9: Panel (a) shows a comparison of error propagation in training and extrapolation phases for physics
encoded recurrent convolutional neural network (PeRCNN), deep hidden physics model (DHPM), and convo-
lutional long-short term memory (ConvLSTM) modeling 2D Burgers’ dataset (adapted from Rao et al. [40]).
Panel (b) shows a comparison between the predictions by PhyCRNet and PiNN for 2D Burgers’ equations.
The predicted velocity field in x direction is compared at the training time of t = 1 s, and at the extrapolation
time of t = 3 s (adapted from Ren et al. [270]).

non-linear PDEs compared to state-of-the-art baseline algorithms such as the PiNN and auto-regressive dense
encoder-decoder model [262]. A comparison between PhyCRNet and PiNN to solve Burgers’ equation is shown
in Fig. 9(b) [270]. Results obtained by Ren et al. [270] clearly demonstrated the superiority of the PhyCRNet
methodology in terms of solution accuracy, extrapolability, and generalizability.

4.2. Fourier Neural Operator (FNO)

Most of the scientific deep learning methods discussed so far, e.g., PgNN and PiNN, are designed to map
or learn the solution of a physical phenomenon for a single instance (i.e., a certain spatiotemporal domain to
solve a PDE using PiNN), and thus, cannot be successfully scoped outside that instance (i.e., extrapolation).
To overcome this limitation, a new family of PeNN algorithms has emerged that learns the solution operators
for PDEs [259, 271, 272]. These models use mesh-invariant, infinite-dimensional operators based on neural
networks, dubbed as neural operators (NO), that do not require prior understanding of PDEs.
Neural operators merely work with data to learn the resolution-invariant solution to the problem of interest
[43]. In other words, neural operators can be trained on one spatiotemporal resolution and successfully inferred
on any other [272]. This is achieved using the fact that a neural operator learns continuous functions, rather
than discretized vectors, by parameterizing the model in function space [43, 272]. However, due to the high cost
of evaluating integral operators, neural operators have been unable to develop effective numerical algorithms
capable of replacing convolutional or recurrent neural networks in an infinite-dimensional context [39]. In
order to benefit from neural operators in infinite-dimensional spaces, Li et al. [39] developed a neural operator
in the Fourier space, dubbed as the Fourier Neural Operator (FNO).

Figure 10: A schematic architecture of Fourier Neural Operator (FNO) [39]. Here, X is the inputs; S and M
are the neural network for dimensional space operation, and Y’ is the predicted outputs. Each of the Fourier
layer consist of v(x) as the initial state, F layer that performs Fourier transform, T layer that performs linear
transform, F −1 layer that perform inverse Fourier transform, W layer for local linear transform, + block that
carries out element-wise addition on W and final output from F −1 layer. The final product from element-wise
addition is passed on to a σ layer.

23
 Figure 10 illustrates a schematic representation of FNO’s architecture. The training starts with an input
X, which is subsequently elevated to a higher dimensional space by a neural network S. The second phase
entails the use of several Fourier layers of integral operators and activation functions. In each Fourier layer, the
input is transformed using (i) a Fourier transform, F ; (ii) a linear transform, T , on the lower Fourier modes that
filters out the higher modes; and (iii) an inverse Fourier transform, F −1 . The input is also transformed using a
local linear transform, W , before the application of the activation function, σ. The Fourier layers are designed
to be discretization-invariant due to the fact that they learn from functions that are discretized arbitrarily.
Indeed, the integral operator is applied in a convolution and is represented as a linear transformation in the
Fourier domain, allowing the FNO to learn the mapping over infinite-dimensional spaces. The result of the
Fourier layer is projected back to the target dimension in the third phase using another neural network M ,
which eventually outputs the desired output Y 0 [39]. In contrast to other DL methodologies, the FNO model’s
error is consistent regardless of the input and output resolution (e.g., in PgNN methods, the error grows with
the resolution). Li et al. [39] employed FNO on three different test cases, including the 1D Burgers’ equation,
2D Darcy flow equation, and 2D Navier-Stokes equations. For each test case, FNO was compared with
state-of-the-art models. In particular, for Burgers’ and Darcy’s test cases, the methods used for comparison
were conventional ANN (i.e., PgNN), reduced bias method [273], fully convolutional networks [274], principal
component analysis as an encoder in the neural network [271], graph neural operator [272], and the low rank
decomposition neural operator (also known as unstacked DeepONet [259]). In all test cases, FNO yielded the
lowest relative error. The models’ error comparisons for 1D Burgers’ and 2D Darcy flow equations are depicted
in Fig. 11, adapted from [39].

Figure 11: Error comparison between Fourier Neural Operator (FNO) and other state-of-the-art methods
(reduced biased method (RBM), fully convolutional network (FCN), graph neural operator (GNO), and graph
convolutional network (GCN)) for (a) Burgers’ equation and (b) Darcy’s Flow equation at different resolutions
(adapted from Li et al. [39]).

The FNO model, as stated, can be trained on a specific resolution and tested on a different resolution. Li
et al. [39] demonstrated this claim by training a FNO on the Navier-Stokes equations for a 2D test case with
a resolution of 64×64×20 (nx , ny , nt ) standing for spatial (x, y) and time resolution, and then evaluating it
with a resolution of 256×256×80, as shown in Fig. 12(a). In comparison to other models, the FNO was the
only technique capable of performing resolution downscaling both spatially and temporally [39].
FNOs can also achieve several orders of magnitude speedup factors in comparison to conventional nu-
merical PDE solvers. However, they have only been used for 2D or small 3D problems due to the large
dimensionality of their input data that increases the number of network weights significantly. With this prob-
lem in mind, Grady et al. [275] proposed a parallelized version of FNO based on domain-decomposition to
resolve this limitation. Using this extension, they were able to use FNO in large-scale modeling, e.g., simulat-
ing the transient evolution of the CO2 plume in subsurface heterogeneous reservoirs as a part of the carbon
capture and storage (CCS) technology [276], see Fig. 12(b). The input to the network (with a similar archi-
tecture to the one proposed by Li et al. [39]) was designed to be a tensor containing both the permeability
and topography fields at each 3D spatial position using a 60×60 ×64 (nx , ny , nz ) resolution, and the output
was 60×60×64×nt . For a time resolution of nt = 30 s, they found that the parallelized FNO model was 271x
faster (without even leveraging GPU) than the conventional porous media solver while achieving a comparable
accuracy. Wen et al. [277] also proposed U-FNO, an extension of FNO, to simulate multiphase flows in porous
media, specifically CO2-water multiphase flow through a heterogeneous medium with broad ranges of reservoir
conditions, injection configurations, flow rates, and multiphase flow properties. They compared U-FNO with
original FNO and CNN (i.e., PgNN) and showed that the U-FNO architecture provides the best performance
for both gas saturation and pressure buildup predictions in highly heterogeneous geological formations. They
24
also showed that the U-FNO architecture enhances the training accuracy of the original FNO, but does not
naturally enable the flexibility of training and testing at multiple discretizations.

Figure 12: A qualitative comparison between predictions made by the Fourier Neural Operator (FNO) and
ground truth values. Panel (a) shows the comparison for a FNO model trained on 64×64×20 resolution
(nx , ny , nt ) and evaluated on 256×256×80 resolution to solve 2D Navier-Stokes equations (adapted from Li
et al. [39]). Panel (b) shows the comparison for a parallelized FNO model trained to solve large-scale 3D
subsurface CO2 flow modeling evaluated at 60×60×64×nt resolution (adapted from Grady et al. [275]).

You et al. [278] proposed an implicit Fourier neural operator (IFNO) to model the complex responses
of materials due to their heterogeneity and defects without using conventional constitutive models. The
IFNO model captures the long-range dependencies in the feature space, and as the network becomes deeper, it
becomes a fixed-point equation that yields an implicit neural operator (e.g., it can mimic displacement/damage
fields). You et al. [278] demonstrated the performance of IFNO using a series of test cases such as hyperelastic,
anisotropic, and brittle materials. Fig. 13 depicts a comparison between IFNO and FNO for the transient
propagation of a glass-ceramic crack [278]. As demonstrated, IFNO outperforms FNO (in terms of accuracy)
and conventional constitutive models (in terms of computational cost) to predict the displacement field.
The FNO model has also been hybridized with PiNN to create the so-called physics-informed neural
operator (PiNO) [43]. The PiNO framework is a combination of operating-learning (i.e., FNO) and function-
optimization (i.e., PiNN) frameworks that improves convergence rates and accuracy over both PiNN and
FNO models. This integration was suggested to address the challenges in PiNN (e.g., generalization and
optimization, especially for multiscale dynamical systems) and the challenges in FNO (e.g., the need for
expensive and impractical large training datasets) [43]. Li et al. [43] deployed the PiNO model on several
benchmark problems (e.g., Kolmogorov flow, lid-cavity flow, etc.) to show that PiNO can outperform PiNN
and FNO models, while maintaining the FNO’s exceptional speed-up factor over other solvers.

4.3. Neural Ordinary Differential Equations (NeuralODE)

The neural ordinary differential equations (NeuralODE) method is another family of PeNN models in
which the hidden state of the neural network is transformed from a discrete sequence to a continuous non-
linear function by parametrizing the hidden state derivative using a differentiable function [266]. The output
of the network is then computed using a traditional differential equation solver. During training, the error
is backpropagated through the network as well as through the ODE solver without access to its internal
operations. This architecture is feasible due to the fact that numerical linear algebra is the common underlying
infrastructure for both scientific computing and deep learning, which is bridged by automated differentiation
(AD) [279]. Because differential equations and neural networks are both differentiable, standard optimization
and error backpropagation techniques can be used to optimize the network’s weights during training. Instead
of learning the non-linear transformation directly from the training data, the model in NeuralODE learns the
structures of the non-linear transformation. Therefore, due to the fact that the neural network optimization
equations are differentiable, the physical differential equations can be encoded directly into a layer as opposed
25
Figure 13: Comparison between implicit Fourier neural operator (IFNO) and FNO to model crack prop-
agation and damage field within a region of interest (ROI) in a pre-cracked glass-ceramics experiment with
randomly distributed material property fields (adapted from You et al. [278]).

to adding more layers (e.g., deeper networks). This results in a shallower network mimicking an infinitely deep
model that can be inferred continuously at any desired accuracy at reduced memory and computational cost
[280]).
These continuous-depth models offer features that are lacking in PiNN and PgNNs, such as (i) a re-
duced number of parameters for supervised learning, (ii) constant memory cost as a function of depth, and
(iii) continuous time-series learning (i.e., training with datasets acquired at arbitrary time intervals), just to
name a few [266]. However, the error backpropagation may cause technical difficulties while training such
continuous-depth networks. Chen et al. [266] computed gradients using the adjoint sensitivity method [281]
while considering the ODE solver as a black box. They demonstrated that this method uses minimal memory,
can directly control numerical error, and most importantly, scales linearly with the problem size.
Ma et al. [282] compared the performance of discrete and continuous adjoint sensitivity analysis. They
indicated that forward-mode discrete local sensitivity analysis implemented via AD is more efficient than
the reverse-mode and continuous forward and/or adjoint sensitivity analysis for problems with approximately
fewer than 100 parameters. However, in terms of scalability, they showed that the continuous adjoint method
is more efficient than the discrete adjoint and forward methods.
Several computational libraries have been implemented to facilitate the practical application of Neu-
ralODE. Poli et al. [283] implemented the TorchDyn library to train NeuralODE models and be as accessible
as regular plug–and–play deep learning primitives. Innes et al. [8] and Rackauckas et al. [280] developed GPU-
accelerated Zygote and DiffEqFlux libraries in the Julia coding ecosystem to bring differentiable programming
and universal differential equation solver capabilities together. As an example, they encoded the ordinary dif-
ferential equation of motion, as the transformation function, into a neural network to simulate the trebuchet’s
inverse dynamics [8]. As shown in Fig. 14, the network with classical layers takes the target location and wind
speed as input and estimates the weight and angle of the projectile to hit the target. These outputs are fed
into the ODE solver to calculate the achieved distance. The model compares the predicted value with the
target location and backpropagates the error through the entire chain to adjust the weights of the network.
This PeNN model solves the trebuchet’s inverse dynamics on a personal computer 100x faster than a classical
optimization algorithm for this inverse problem. Once trained, this network can be used to aim at any blind
target, not only the ones it was trained on; hence, the model is both accelerated and predictive.

Figure 14: A NeuralODE architecture that leverages differentiable programming to model the inverse dy-
namics of a trebuchet. This simple network is 100x faster than direct optimization (adapted from Innes et al.
[8]).

26
 NeuralODE has also been integrated with PiNN models, dubbed as PiNODE, in order to further constrain
the network with known governing physics during training. Such an architecture consists of a neural network
whose hidden state is parameterized by an ODE with a loss function similar to the PiNN’s loss function (see
Fig. 5). The loss function penalizes the algorithm based on data and the strong form of the governing ODEs
and backpropagates the error through the application of the adjoint sensitivity methods [282], to update the
learnable parameters in the architecture. PiNODE can be deployed to overcome high bias (due to the use of
first principles in scientific modeling) and high variance (due to the use of pure data-driven models in scientific
modeling) problems. In other words, using PiNODE, prior physics knowledge in terms of ODE is integrated
where it is available, and function approximation (e.g., neural networks) is used where it is not available.
Lai et al. [284] used PiNODE to model the governing equations in the field of structural dynamics (e.g., free
vibration of a 4-degree-of-freedom dynamical system with cubic non-linearity). They showed that PiNODE
provides an adaptable framework for structural health monitoring (e.g., damage detection) problems. Roehrl
et al. [285] tested PiNODE using a forward model of an inverted pendulum on a cart and showed the approach
can learn the non-conservative forces in a real-world physical system with substantial uncertainty.
The application of neural differential equation has also been extended to learn the dynamics of PDE-
described systems. Dulny et al. [286] proposed NeuralPDE by combining the Method of Lines (which represents
arbitrarily complex PDEs by a system of ODEs) and NeuralODE through the use of a multi-layer convolutional
neural network. They tested NeuralPDE on several spatiotemporal datasets generated from the advection-
diffusion equation, Burgers’ equation, wave propagation equation, climate modeling, etc. They found that
NeuralPDE is competitive with other DL-based approaches, e.g., ResNet [287]. The NeuralPDE’s limitations
are set by the limitations of the Method of Lines, e.g., it cannot be used to solve elliptical second-order PDEs.
Table 8 reports a non-exhaustive list of leading studies that leveraged PeNN to model different scientific
problems.

Table 8: A non-exhaustive list of recent studies that leveraged PeNN to model different scientific problems.

PeNN Structure Objective Reference

PeRCNN Hard-encoding prior knowledge into neural network to model non-linear [40]
systems

PhyCRNet Leveraging the benefits of PeRCNN and PiNN into a single architecture [270]

FNO Providing a mesh- and discretization-invariant model to be inferred on [39]

any arbitrary spatiotemporal resolutions

Parallelized FNO Extending FNO based on domain-decomposition to model large-scale [275]

three-dimensional problems

U-FNO Enhancing the training and testing accuracy of FNO for large-scale [277]
highly heterogeneous geological formations

IFNO Proposing an implicit Fourier neural operator to model the complex [278]
responses of materials due to their heterogeneity and defects without
using conventional constitutive models

PINO Integrating PiNN and FNO to remove the FNO’s requirement of large [43]
training dataset while enhancing PiNN’s generalisation and optimization

NeuralODE Developing a continuous-depth network by parametrizing the hidden [266]

state derivative using a differentiable function

Zygote / DiffEqFlux Providing libraries in the Julia coding ecosystem to facilitate the prac- [8, 280]
tical application of NeuralODE.

PiNODE Integrating PiNN and NeuralODE to provide an adaptable framework [284]

for structural health monitoring

NeuralPDE Combining the Method of Lines and NeuralODE to to learn the dynam- [286]
ics of PDE-described systems

4.4. PeNN Limitations

Despite the advancement of numerous PeNN architectures, standalone PeNN and PeNN-PiNN variant
architectures still have some limitations. For instance, the vanilla form of FNO is typically limited to basic
geometry and structured data [288], and experiences over-fitting as the number of trainable parameters in-

27
creases, making the training process more difficult [289]. IFNOs have addressed this challenge to some extent
[278]. In IFNO, the solution operator is first formulated as an implicitly defined mapping and then modeled as
a fixed point. The latter aims to overcome the challenge of network training in the case of deep layers, while
the former minimizes the amount of trainable parameters and memory cost. PeNN-based models also promote
continuous learning using the development of continuous-depth networks, making PeNNs more difficult to
train than PgNNs and PiNNs. Notably, neural operators only need to be trained once, i.e., finding a solution
for a new condition of the physical parameter requires only a forward pass of the network, and not training a
new neural network that is the basis for PgNNs and PiNNs [39]. PeNNs have a complex architecture and their
implementation is not as straightforward as PiNNs or PgNNs. Despite PeNNs’ implementation complexity,
their efficient algorithms in the finite-dimensional setting and their ability to provide transferable solutions
make them a great potential to significantly accelerate traditional scientific computing.

5. Conclusions and Future Research Directions

A considerable number of research topics collectively support the efficacy of combining scientific computing
and deep learning approaches. In particular, this combination improves the efficiency of both forward and
inverse modeling for high-dimensional problems that are prohibitively expensive, contain noisy data, require a
complex mesh, and are governed by non-linear, ill-posed differential equations. The ever-increasing computer
power will continue to further furnish this combination by allowing the use of deeper neural networks and
considering higher-dimensional interdependencies and design space.

Table 9: A comparison of the main characteristics of PgNNs, PiNNs, and PeNNs to model non-linear multi-
scale physiochemical phenomena with sparse datasets.

PgNN PiNN PeNN

Accelerating Capability X X X

Mesh Free Simulation X X X

Straightforward Network Training X 7 7

Training without Labeled Data 7 X 7

Physics-informed Loss Function 7 X X

Continuous Solution 7 X X

Spatiotemporal Interpolation 7 X X

Physics Encoding 7 7 X

Operator Learning 7 7 X

Continuous-depth Models 7 7 X

Spatiotemporal Extrapolation 7 7 X

Solution Transferability 7 7 X

The combination of scientific computing and deep learning approaches also surpasses traditional compu-
tational mechanics solvers in a number of prevalent scenarios in practical engineering. For example, a sparse
dataset obtained experimentally for a complex (i.e., hard-to-acquire data) phenomenon cannot be simply in-
tegrated with traditional solvers. Whereas, with DL the following can be performed: (i) PgNN models can
be applied to the sparse data to extract latent interdependencies and conduct spatiotemporal downscaling
or upscaling (i.e., interpolated data), (ii) PiNN can be applied to the interpolated data to deduce governing
equations and potentially unknown boundary or initial conditions of the phenomena (i.e., strong mathematical
form), and (iii) PeNN can be used to combine the interpolated data and strong mathematical form to conduct
extrapolation exploration. Therefore, this combination provides scientists with a cost-effective toolbox to ex-
plore problems across different scales that were deemed far-fetched computationally. To this end, several other
breakthroughs in DL are required to enable the use of PgNN, PiNN, and PeNN in large-scale three-dimensional
(or multi-dimensional) problems. For instance, the training of complex DL models (e.g., PiNN and PeNN)
should be accelerated using different parallelization paradigms.
Table 9 compares the main characteristics of the PgNN, PiNN, and PeNN models. The PgNN-based
models suffer mainly from their statistical training process, for which they require large datasets. They map

28
curated training datasets only based on correlations in statistical variations, and hence, their predictions
are naturally physics-agnostic. The PiNN-based models suffer mainly from the lack of theoretical proof of
convergence, and the presence of competing loss terms that may destabilize the training process. PiNN is also
a solution learning algorithm with limited generalizability due to its inability to learn the physical operation
of a specific phenomenon. PeNN-based models, on the other hand, have a low convergence rate and require
large, structured datasets.
Considering the effectiveness of this new challenge of combining scientific computing and DL, future studies
can be divided into three distinct categories: (i) Improving algorithms: Developing advanced variants
of PgNN, PiNN, and PeNN algorithms that offer simpler implementation with enhanced convergence rate;
faster training in multi-dimensional and multi-physics problems; higher accuracy and generalization to unseen
conditions while using sparse training datasets; better adaptability to multi-spatiotemporal-resolutions; more
flexibility to encode various types of governing equations (e.g., all PDE types, closure laws, data-driven laws,
etc.), and provide a closer tie with a plethora of traditional solvers; (ii) Considering causalities: Developing
causal training algorithm (e.g., causal Q-learning [290]) that restores physical causality during the training of
PgNN, PiNN, and PeNN models by re-weighting the governing equations (e.g., PDEs) residual loss at each
iteration. This line of research will allow for developing causality-conforming variants of PgNN, PiNN, and
PeNN algorithms that can bring new opportunities for the application of these algorithms to a wider variety
of complex scenarios across diverse domains; (iii) Expanding applications: Leveraging the potentials of
PgNN, PiNN and PeNN in problems with complex anisotropic materials (e.g., flow in highly heterogeneous
porous media, metal and non-metal particulate composites, etc.); problems with multiscale multi-physics
phenomena (e.g., magnetorheological fluids, particle-laden fluids, dry powder dynamics, reactive transport,
unsaturated soil dynamics, etc.); and structural health monitoring (e.g., crack identification and propagation,
hydrogen pipeline leakage, CO2 plume detection, etc.); and (iv) Coupling solvers: Coupling PgNN, PiNN
and PeNN libraries with open-source computational mechanics packages such as OpenIFEM, OpenFOAM,
Palabos, LAMMPS, LIGGGHTS, MOOSE, etc. This line of research will allow for faster surrogate modeling,
and hence, faster development of next-generation solvers. It also expedites the community and industry
adoption to the combined scientific-DL computational paradigm.

6. Acknowledgements
S.A.F. would like to acknowledge supports by DOE Biological and Environmental Research (BER) (award
no. DE-SC0023044), NSF Partnership for Research and Education in Materials (PREM) (award no. DMR-
2122041), and TXST Multidisciplinary Internal Research Grant (MIRG) (award no. 9000003028).
C.F. would like to acknowledge supports by FEDER funds through the COMPETE 2020 Programme and
National Funds through FCT (Portuguese Foundation for Science and Technology) under the projects UID-
B/05256/2020, UID-P/05256/2020 and MIT-EXPL/TDI/0038/2019-APROVA-Deep learning for particle-laden
viscoelastic flow modelling (POCI-01-0145-FEDER-016665) under MIT Portugal program.

References
[1] R. Vinuesa and S. L. Brunton. The potential of machine learning to enhance computational fluid dynamics.
arXiv:2110.02085, 2021.
[2] J. R. Mianroodi, N. H. Siboni, and D. Raabe. Teaching solid mechanics to artificial intelligence - a fast solver for heteroge-
neous materials. NPJ Computational Materials, 7:99, 2021. DOI: https://www.nature.com/articles/s41524-021-00571-z.
[3] Y. Kim, C. Yang, K. Park, G. X. Gu, and R. Seunghwa. Deep learning framework for material design space ex-
ploration using active transfer learning and data augmentation. npj Computational Materials, 7:140, 2021. DOI:
https://doi.org/10.1038/s41524-021-00609-2.
[4] H. I. Dino, S. R. Zeebaree, A. A. Salih, R. R. Zebari, Z. S. Ageed, H. M. Shukur, L. M. Haji, and S. S. Hasan. Impact
of process execution and physical memory-spaces on os performance. Technology Reports of Kansai University, 62(5):
2391–2401, 2020.
[5] S. Im, J. Lee, and M. Cho. Surrogate modeling of elasto-plastic problems via long short-term memory neural networks
and proper orthogonal decomposition. Computer Methods in Applied Mechanics and Engineering, 385:114030, 2021. DOI:
https://doi.org/10.1016/j.cma.2021.114030.
[6] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nature
Reviews Physics, 3(6):422–440, 2021. DOI: https://doi.org/10.1038/s42254-021-00314-5.
[7] S. Arman and W. Anthony. Physics-inspired architecture for neural network modeling of forces and torques in particle-laden
flows. Computers and Fluids, 238:105379, 2022. DOI: https://doi.org/10.1016/j.compfluid.2022.105379.
[8] M. Innes, A. Edelman, K. Fischer, C. Rackauckas, E. Saba, V.B. Shah, and W. Tebbutt. A differentiable programming
system to bridge machine learning and scientific computing. arXiv preprint arXiv:1907.07587, 2019.
[9] S. L. Brunton, B. R. Noack, and P. Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics,
52:477–508, 2020. DOI: https://doi.org/10.1146/annurev-fluid-010719-060214.
[10] S. Cai, Z. Mao, Z. Wang, M. Yin, and G. E. Karniadakis. Physics-informed neural networks (pinns) for fluid mechanics: A
review. Acta Mechanica Sinica, pages 1–12, 2022. DOI: https://doi.org/10.1007/s10409-021-01148-1.
[11] J. N. Kutz. Deep learning in fluid dynamics. Journal of Fluid Mechanics, 814:1–4, 2017. DOI:
https://doi.org/10.1017/jfm.2016.803.
[12] Z. Shi, E. Tsymbalov, M. Dao, S. Suresh, A. Shapeev, and J. Li. Deep elastic strain engineering of bandgap through machine
learning. Proc. Natl. Acad. Sci., 116:4117–4122, 2019. DOI: https://doi.org/10.1073/pnas.1818555116.
[13] E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes. A physics-informed deep learning framework for inversion
and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021.
DOI: https://doi.org/10.1016/j.cma.2021.113741.

29
[14] G. Pilania, C. Wang, Jiang X., S. Rajasekaran, and R. Ramprasad. Accelerating materials property predictions using
machine learning. Sci. Rep., 3:1–6, 2013. DOI: https://doi.org/10.1038/srep02810.
[15] K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh. Machine learning for molecular and materials science.
Nature, 559:547–555, 2018. DOI: https://doi.org/10.1038/s41586-018-0337-2.
[16] S. L. Brunton and J. N. Kutz. Methods for data-driven multiscale model discovery for materials. J. Phys. Mater., 2:044002,
2019. DOI: https://doi.org/10.1088/2515-7639/ab291e.
[17] E. Bedolla, L. C. Padierna, and R. Castaneda-Priego. Machine learning for condensed matter physics. Journal of Physics:
Condensed Matter, 33(5):053001, 2020. DOI: https://doi.org/10.1088/1361-648X/abb895.
[18] D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, and S. Hoyer. Machine learning–accelerated
computational fluid dynamics. Proceedings of the National Academy of Sciences, 118:e2101784118, 2021. DOI:
https://doi.org/10.1073/pnas.2101784118.
[19] D. Tran, M. D. Hoffman, R. A. Saurous, E. Brevdo, K. Murphy, and D.M. Blei. Deep probabilistic programming. arXiv
preprint arXiv:1701.03757, 2017.
[20] K. H. Jin, M.T. McCann, E. Froustey, and M. Unser. Deep convolutional neural network for inverse problems in imaging.
IEEE Transactions on Image Processing, 26(9):4509–4522, 2017. DOI: https://doi.org/10.1109/TIP.2017.2713099.
[21] Z. Lai, Q. Chen, and L. Huang. Machine-learning-enabled discrete element method: Contact detection and resolution of
irregular-shaped particles. International Journal for Numerical and Analytical Methods in Geomechanics, 46(1):113–140,
2022. DOI: https://doi.org/10.1002/nag.3293.
[22] S. A. Faroughi, A. I. Roriz, and C. Fernandes. A meta-model to predict the drag coefficient of a particle translating in
viscoelastic fluids: a machine learning approach. Polymers, 14(3):430, 2022. DOI: https://doi.org/10.3390/polym14030430.
[23] B. Taylor. Methodus incrementorum directa & inversa. Auctore Brook Taylor, LL. D. & Regiae Societatis Secretario. typis
Pearsonianis: prostant apud Gul. Innys ad Insignia Principis in . . . , 1715.
[24] B. J. Alder and T. E. Wainwright. Phase transition for a hard sphere system. The Journal of chemical physics, 27(5):
1208–1209, 1957.
[25] R. W. Clough. The finite element method in plane stress analysis. In Proceedings of 2nd ASCE Conference on Electronic
Computation, Pittsburgh Pa., Sept. 8 and 9, 1960, 1960.
[26] J. Smagorinsky. General circulation experiments with the primitive equations: I. the basic experiment. Monthly weather
review, 91(3):99–164, 1963.
[27] P. A. Cundall and O. Strack. A discrete numerical model for granular assemblies. Géotechnique, 29:47–65, 1979. DOI:
https://doi.org/10.1680/geot.1979.29.1.47.
[28] P. W. McDonald. The computation of transonic flow through two-dimensional gas turbine cascades, volume 79825. American
Society of Mechanical Engineers, 1971.
[29] C. S. Peskin. Flow patterns around heart valves: a numerical method. Journal of computational physics, 10(2):252–271,
1972.
[30] L. B. Lucy. A numerical approach to the testing of the fission hypothesis. The astronomical journal, 82:1013–1024, 1977.
[31] D. D’Humieres, P. Lallemand, and U. Frisch. Lattice gas models for 3d hydrodynamics. EPL (Europhysics Letters), 2(4):
291, 1986.
[32] F. Bassi and S. Rebay. A high-order accurate discontinuous finite element method for the numerical solution of the
compressible navier–stokes equations. Journal of computational physics, 131(2):267–279, 1997.
[33] A. G. Ivakhnenko and V. G. Lapa. Cybernetics and forecasting techniques, volume 8. American Elsevier Publishing
Company, 1967.
[34] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. nature, 323
(6088):533–536, 1986.
[35] K. Andersen, G. E. Cook, G. Karsai, and K. Ramaswamy. Artificial neural networks applied to arc welding process modeling
and control. IEEE Transactions on industry applications, 26(5):824–830, 1990. DOI: https://doi.org/10.1109/28.60056.
[36] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of
the IEEE, 86(11):2278–2324, 1998.
[37] Goodfellow I. J., P. Jean, M. Mehdi, X. Bing, W. David, O. Sherjil, and C. Courville Aaron. Generative adversarial nets.
In Proceedings of the 27th international conference on neural information processing systems, volume 2, pages 2672–2680,
2014.
[38] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear
partial differential equations. arXiv preprint arXiv:1711.10561, 2017.
[39] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator
for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
[40] C. Rao, H. Sun, and Y. Liu. Hard encoding of physics for learning spatiotemporal dynamics. arXiv preprint
arXiv:2105.00557, 2021.
[41] M. Lienen and S. Gunnemann. Learning the dynamics of physical systems from sparse observations with finite el-
ement networks. In The Tenth International Conference on Learning Representations. OpenReview, 2022. DOI:
https://doi.org/10.48550/arXiv.2203.08852.
[42] U. Hasson, S.A. Nastase, and A. Goldstein. Direct fit to nature: an evolutionary perspective on biological and artificial
neural networks. Neuron, 105(3):416–434, 2020. DOI: https://doi.org/10.1016/j.neuron.2019.12.002.
[43] Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics-informed neural
operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
[44] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving
forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:
686–707, 2019. DOI: https://doi.org/10.1016/j.jcp.2018.10.045.
[45] M. A. Nabian and H. Meidani. Adaptive physics-informed neural networks for markov-chain monte carlo. arXiv preprint
arXiv:2008.01604, 2020.
[46] S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-
informed neural networks: Where we are and what’s next. arXiv preprint arXiv:2201.05624, 2022.
[47] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: a survey.
arXiv:1502.05767, 2015.
[48] C. Rao, S. Hao, and L. Yang. Physics-informed deep learning for incompressible laminar flows. Theoretical and Applied
Mechanics Letters, 10:207–212, 2020. DOI: https://doi.org/10.1016/j.taml.2020.01.039.
[49] L. McClenny and U. Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv
preprint arXiv:2009.04544, 2020.
[50] G. K. Yadav, S. Natarajan, and B. Srinivasan. Distributed pinn for linear elasticity—a unified approach for smooth, singular,
compressible and incompressible media. International Journal of Computational Methods, page 2142008, 2022.
[51] P. Bauer, P. D. Dueben, T. Hoefler, T. Quintino, T. C. Schulthess, and N. P. Wedi. The digital revolution of earth-system
science. Nature Computational Science, 1(2):104–113, 2021. DOI: https://doi.org/10.1038/s43588-021-00023-0.

30
[52] H. Chung, S. J. Lee, and J. G. Park. Deep neural network using trainable activation functions. In 2016 International Joint
Conference on Neural Networks (IJCNN), pages 348–352. IEEE, 2016. DOI: https://doi.org/10.1109/IJCNN.2016.7727219.
[53] M. Mattheakis, P. Protopapas, D. Sondak, G. M. Di, and E. Kaxiras. Physical symmetries embedded in neural networks.
arXiv preprint arXiv:1904.08991, 2019.
[54] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436–444, 2015. DOI:
https://www.nature.com/articles/nature14539.
[55] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath. Generative adversarial networks:
An overview. IEEE Signal Processing Magazine, 35(1):53–65, 2018. DOI: https://doi.org/10.1109/MSP.2017.2765202.
[56] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE transactions
on neural networks, 20(1):61–80, 2008. DOI: https://doi.org/10.1109/TNN.2008.2005605.
[57] A. D. Rasamoelina, F. Adjailia, and P. Sinčák. A review of activation function for artificial neural network. In 2020
IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 281–286. IEEE, 2020. DOI:
https://doi.org/10.1109/SAMI48414.2020.9108717.
[58] C. He, M. Ma, and P. Wang. Extract interpretability-accuracy balanced rules from artificial neural networks: A review.
Neurocomputing, 387:346–358, 2020. DOI: https://doi.org/10.1016/j.neucom.2020.01.036.
[59] M. Li, T. Zhang, Y. Chen, and A. J. Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of
the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 661–670, 2014. DOI:
https://doi.org/10.1145/2623330.2623612.
[60] K. Huang, M. Krugener, A. Brown, F. Menhorn, H. J. Bungartz, and D. Hartmann. Machine learning-based optimal mesh
generation in computational fluid dynamics. arXiv preprint: 2102.12923v1, 2021.
[61] S. Kumar and D. M. Kochmann. What machine learning can do for computational solid mechanics. In Current Trends and
Open Problems in Computational Mechanics, pages 275–285. Springer, 2022.
[62] K. Guo, Z. Yang, C-H. Yu, and M. J. Buehler. Artificial intelligence and machine learning in design of mechanical materials.
Materials Horizons, 8(4):1153–1172, 2021. DOI: https://doi.org/10.1039/D0MH01451F.
[63] Z. Zhang, Y. Wang, P. K. Jimack, and H. Wang. MeshingNet: a new mesh generation method based on deep learning.
arXiv preprint: 2004.07016v1, 2020.
[64] T. Wu, X. Liu, W. An, Z. Huang, and H. Lyu. A mesh optimization method using machine learning technique and variational
mesh adaptation. Chinese Journal of Aeronautics, 35:27–41, 2022. DOI: https://doi.org/10.1016/j.cja.2021.05.018.
[65] A. Mendizabal, P. Márquez-Neila, and S. Cotin. Simulation of hyperelastic materials in real-time using deep learning.
Medical image analysis, 59:101569, 2020. DOI: https://doi.org/10.1016/j.media.2019.101569.
[66] L. Lu, X. Gao, J-F. Dietiker, M. Shahnam, and W. A. Rogers. Machine learning accelerated discrete element modeling of
granular flows. Chemical Engineering Science, 245:116832, 2021. DOI: https://doi.org/10.1016/j.ces.2021.116832.
[67] Z. Li, K. Meidani, P. Yadav, and A. B. Farimani. Graph neural networks accelerated molecular dynamics. arXiv preprint
arXiv:2112.03383, 2021.
[68] H. P. Menke, J. Maes, and S. Geiger. Upscaling the porosity–permeability relationship of a microporous carbonate for
darcy-scale flow with machine learning. Scientific Reports, 11(1):1–10, 2021. DOI: https://doi.org/10.1038/s41598-021-
82029-2.
[69] S. Cheng, J. Chen, C. Anastasiou, P. Angeli, O. K. Matar, Y. Guo, C. C. Pain, and R. Arcucci. Generalised latent
assimilation in heterogeneous reduced spaces with machine learning surrogate models. arXiv preprint arXiv:2204.03497,
2022.
[70] M. H. Zawawi, A. Saleha, A. Salwa, N. H. Hassan, N. M. Zahari, M. Z. Ramli, and Z. C. Muda. A review: Fundamentals of
computational fluid dynamics (cfd). In AIP conference proceedings, volume 2030, page 020252. AIP Publishing LLC, 2018.
DOI: https://doi.org/10.1063/1.5066893.
[71] L. He and D. K. Tafti. A supervised machine learning approach for predicting variable drag forces on spherical particles in
suspension. Powder technology, 345:379–389, 2019. DOI: https://doi.org/10.1016/j.powtec.2019.01.013.
[72] L-T. Zhu, J-X. Tang, and Z-H. Luo. Machine learning to assist filtered two-fluid model development for dense gas–particle
flows. AIChE Journal, 66(6):e16973, 2020. DOI: https://doi.org/10.1002/aic.16973.
[73] A. I. Roriz, S. A. Faroughi, G. H. McKinley, and C. Fernandes. Ml driven models to predict the drag coefficient of a sphere
translating in shear-thinning viscoelastic fluids. 2021.
[74] C. Loiro, C. Fernandes, G. H. McKinley, and S. A. Faroughi. Digital-twin for particle-laden viscoelastic fluids: Ml-based
models to predict the drag coefficient of random arrays of spheres. 2021.
[75] M. A. Webb, N. E. Jackson, P. S. Gil, and J. J. de Pablo. Targeted sequence design within the coarse-grained polymer
genome. Science advances, 6(43):eabc6216, 2020. DOI: https://doi.org/10.1126/sciadv.abc6216.
[76] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural
networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
[77] M. M. Bejani and M. Ghatee. A systematic review on overfitting control in shallow and deep neural networks. Artificial
Intelligence Review, 54(8):6391–6438, 2021. DOI: https://doi.org/10.1007/s10462-021-09975-1.
[78] X. Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1168, page 022022.
IOP Publishing, 2019. DOI: https://doi.org/10.1088/1742-6596/1168/2/022022.
[79] Y. Cati, S. aus der Wiesche, and M. Düzgün. Numerical model of the railway brake disk for the temperature and
axial thermal stress analyses. Journal of Thermal Science and Engineering Applications, 14(10):101014, 2022. DOI:
https://doi.org/10.1115/1.4054213.
[80] X. Chen, J. Liu, Y. Pang, J. Chen, L. Chi, and C. Gong. Developing a new mesh quality evaluation method based
on convolutional neural network. Engineering Applications of Computational Fluid Mechanics, 14:391–400, 2020. DOI:
https://doi.org/10.1080/19942060.2020.1720820.
[81] S. Maddu, D. Sturm, B. L. Cheeseman, C. L. Müller, and I. F. Sbalzarini. Stencil-net: Data-driven solution-adaptive
discretization of partial differential equations. arXiv preprint arXiv:2101.06182, 2021.
[82] Y. Bar-Sinai, S. Hoyer, J. Hickey, and M. P. Brenner. Learning data-driven discretizations for partial dif-
ferential equations. Proceedings of the National Academy of Sciences, 116(31):15344–15349, 2019. DOI:
https://doi.org/10.1073/pnas.1814058116.
[83] F. Bernardin, M. Bossy, C. Chauvin, J-F. Jabir, and A. Rousseau. Stochastic lagrangian method for downscaling problems
in computational fluid dynamics. ESAIM: Mathematical Modelling and Numerical Analysis, 44(5):885–920, 2010. DOI:
https://doi.org/10.1051/m2an/2010046.
[84] X. Wei, C. Dong, Z. Chen, K. Xiao, and X. Li. The effect of hydrogen on the evolution of intergranular cracking: a
cross-scale study using first-principles and cohesive finite element methods. RSC advances, 6(33):27282–27292, 2016. DOI:
https://doi.org/10.1039/C5RA26061B.
[85] C. Shu. Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws.
Advanced numerical approximation of nonlinear hyperbolic equations, pages 325–432, 1998.
[86] Z. Zhang, P. K. Jimack, and H. Wang. Meshingnet3d: Efficient generation of adapted tetrahedral meshes for computational
mechanics. Advances in Engineering Software, 157:103021, 2021. DOI: https://doi.org/10.1016/j.advengsoft.2021.103021.

31
 [87] D. G. Triantafyllidis and D. P. Labridis. A finite-element mesh generator based on growing neural networks. IEEE
Transactions on neural networks, 13(6):1482–1496, 2002. DOI: https://doi.org/10.1109/TNN.2002.804223.
[88] K. Srasuay, A. Chumthong, and S. Ruangsinchaiwanich. Mesh generation of fem by ann on iron—core transformer. In 2010
International Conference on Electrical Machines and Systems, pages 1885–1890. IEEE, 2010.
[89] M. J. Lee and J. T. Chen. Fluid property predictions with the aid of neural networks. Industrial & engineering chemistry
research, 32(5):995–997, 1993.
[90] C. Yang, X. Yang, and X. Xiao. Data-driven projection method in fluid simulation. Computer Animation and Virtual
Worlds, 27:415–424, 2016. DOI: https://doi.org/10.1002/cav.1695.
[91] J. Tompson, K. Schlachter, P. Sprechmann, and K. Perlin. Accelerating Eulerian fluid simulation with convolutional
networks, 2016.
[92] D.A.H. Jacobs. Preconditioned conjugate gradient methods for solving systems of algebraic equations. Technical Report
RD/L/N193/80, Central Electricity Research Laboratories, 1980.
[93] J. Chen, J. Viquerat, and E. Hachem. U-net architectures for fast prediction of incompressible laminar flows. arXiv preprint
arXiv:1910.13532, 2019.
[94] Z. Deng, C. He, Y. Liu, and K. Kim. Super-resolution reconstruction of turbulent velocity fields using a generative adversarial
network-based artificial intelligence framework. Physics of Fluids, 31:125111, 2019. DOI: https://doi.org/10.1063/1.5127031.
[95] J. Ling, A. Kurzawski, and J. Templeton. Reynolds averaged turbulence modelling using deep neural networks with
embedded invariance. Journal of Fluid Mechanics, 807:155–166, 2016. DOI: https://doi.org/10.1017/jfm.2016.615.
[96] J. Lévy-Leblond. Galilei group and galilean invariance. In Group theory and its applications, pages 221–299. Elsevier, 1971.
DOI: https://doi.org/10.1016/B978-0-12-455152-7.50011-2.
[97] R. Maulik, O. San, A. Rasheed, and P. Vedula. Subgrid modelling for two-dimensional turbulence using neural networks.
Journal of Fluid Mechanics, 858:122–144, 2019. DOI: https://doi.org/10.1017/jfm.2018.770.
[98] R. H. Kraichnan. Inertial ranges in two-dimensional turbulence. Phys. Fluids, 10:1417–1423, 1967. DOI:
https://doi.org/10.1063/1.1762301.
[99] J. Kim and C. Lee. Prediction of turbulent heat transfer using convolutional neural networks. Journal of Fluid Mechanics,
882:A18, 2020. DOI: https://doi.org/10.1017/jfm.2019.814.
[100] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[101] N-D. Hoang. Image processing-based spall object detection using gabor filter, texture analysis, and adaptive mo-
ment estimation (adam) optimized logistic regression models. Advances in Civil Engineering, 2020, 2020. DOI:
https://doi.org/10.1155/2020/8829715.
[102] I. Priyadarshini and C. Cotton. A novel lstm–cnn–grid search-based deep neural network for sentiment analysis. The
Journal of Supercomputing, 77(12):13911–13932, 2021. DOI: https://doi.org/10.1007/s11227-021-03838-w.
[103] Y. Sun, S. Ding, Z. Zhang, and W. Jia. An improved grid search algorithm to optimize svr for prediction. Soft Computing,
25(7):5633–5644, 2021. DOI: https://doi.org/10.1007/s00500-020-05560-w.
[104] M. Yousif, L. Yu, and H. Lim. Physics-guided deep learning for generating turbulent inflow conditions. Journal of Fluid
Mechanics, 936:A21, 2022. DOI: https://doi.org/10.1017/jfm.2022.61.
[105] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and
video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on
computer vision and pattern recognition, pages 1874–1883, 2016.
[106] M. A. Talab, S. Awang, and S. M. Najim. Super-low resolution face recognition using integrated efficient
sub-pixel convolutional neural network (espcn) and convolutional neural network (cnn). In 2019 IEEE interna-
tional conference on automatic control and intelligent systems (I2CACIS), pages 331–335. IEEE, 2019. DOI:
https://doi.org/10.1109/I2CACIS.2019.8825083.
[107] Z. Huang, W. Xu, and K. Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.
[108] A. Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D:
Nonlinear Phenomena, 404:132306, 2020. DOI: https://doi.org/10.1016/j.physd.2019.132306.
[109] J. Kou and W. Zhang. Data-driven modeling for unsteady aerodynamics and aeroelasticity. Progress in Aerospace Sciences,
125:100725, 2021. DOI: https://doi.org/10.1016/j.paerosci.2021.100725.
[110] Z. Wang, K. Gong, W. Fan, C. Li, and W. Qian. Prediction of swirling flow field in combustor based on deep learning.
Acta Astronautica, 201:302–316, 2022. DOI: https://doi.org/10.1016/j.actaastro.2022.09.022.
[111] K. Chowdhary, C. Hoang, K. Lee, J. Ray, V. G. Weirs, and B. Carnes. Calibrating hypersonic turbulence flow models
with the HIFiRE-1 experiment using data-driven machine-learned models. Computer Methods in Applied Mechanics and
Engineering, 401:115396, 2022. DOI: https://doi.org/10.1016/j.cma.2022.115396.
[112] B. N. Bond and L. Daniel. Guaranteed stable projection-based model reduction for indefinite and unstable linear sys-
tems. In 2008 IEEE/ACM International Conference on Computer-Aided Design, pages 728–735. IEEE, 2008. DOI:
https://doi.org/10.1109/ICCAD.2008.4681657.
[113] D. Beli, J-M. Mencik, P. B. Silva, and J. Arruda. A projection-based model reduction strategy for the wave
and vibration analysis of rotating periodic structures. Computational Mechanics, 62(6):1511–1528, 2018. DOI:
https://doi.org/10.1007/s00466-018-1576-7.
[114] M. F. Siddiqui, T. De Troyer, J. Decuyper, P. Z. Csurcsia, J. Schoukens, and M.C. Runacres. A data-driven nonlinear
state-space model of the unsteady lift force on a pitching wing. Journal of Fluids and Structures, 114:103706, 2022. DOI:
https://doi.org/10.1016/j.jfluidstructs.2022.103706.
[115] X. Wang, J. Kou, and W. Zhang. Unsteady aerodynamic prediction for iced airfoil based on multi-task learning. Physics
of Fluids, 34:087117, 2022. DOI: https://doi.org/10.1063/5.0101991.
[116] B. Stevens and T. Colonius. Enhancement of shock-capturing methods via machine learning. Theoretical and Computational
Fluid Dynamics, 34:483–496, 2020. DOI: https://doi.org/10.1007/s00162-020-00531-1.
[117] N. Pawar and S. A. Faroughi. Complex fluids latent space exploration towards accelerated predictive modeling. Bulletin of
the American Physical Society, 2022.
[118] C. Fernandes, S. A. Faroughi, L. L. Ferrás, and A. M. Afonso. Advanced polymer simulation and processing, 2022.
[119] S. A. Faroughi, C. Fernandes, J. M. Nóbrega, and G. H. McKinley. A closure model for the drag coefficient of
a sphere translating in a viscoelastic fluid. Journal of Non-Newtonian Fluid Mechanics, 277:104218, 2020. DOI:
https://doi.org/10.1016/j.jnnfm.2019.104218.
[120] C. Fernandes, S. A. Faroughi, O. S. Carneiro, J. Miguel Nóbrega, and G. H. McKinley. Fully-resolved simulations of
particle-laden viscoelastic fluids using an immersed boundary method. Journal of Non-Newtonian Fluid Mechanics, 266:
80–94, 2019. DOI: https://doi.org/10.1016/j.jnnfm.2019.02.007.
[121] W. Lin, Z. Wu, L. Lin, A. Wen, and J. Li. An ensemble random forest algorithm for insurance big data analysis. Ieee
access, 5:16568–16575, 2017. DOI: https://doi.org/10.1109/ACCESS.2017.2738069.
[122] T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen, et al. Xgboost: extreme gradient boosting. R
package version 0.4-2, 1(4):1–4, 2015.
[123] K. R. Lennon, G. H. McKinley, and J. W. Swan. Scientific machine learning for modeling and simulating complex fluids.

32
 arXiv preprint arXiv:2210.04431v1, 2022.
[124] Z. Cai, J. Chen, and M. Liu. Least-squares ReLU neural network (LSNN) method for scalar nonlinear hyperbolic conservation
law. Applied Numerical Mathematics, 174:163–176, 2022. DOI: https://doi.org/10.1016/j.apnum.2022.01.002.
[125] G. E. Haber, J. Viquerat, A. Larcher, D. Ryckelynck, J. Alves, A. Patil, and E. Hachem. Deep learning model to assist
multiphysics conjugate problems. Physics of Fluids, 34:015131, 2022. DOI: https://doi.org/10.1063/5.0077723.
[126] F. M. Lara and E. Ferrer. Accelerating high order discontinuous Galerkin solvers using neural networks: 1D Burgers’
equation. Computers & Fluids, 235:105274, 2022. DOI: https://doi.org/10.1016/j.compfluid.2021.105274.
[127] B. List, L. Chen, and N. Thuerey. Learned turbulence modelling with differentiable fluid solvers: Physics-based loss functions
and optimisation horizons. Journal of Fluid Mechanics, 949:A25, 2022. DOI: https://doi.org/10.1017/jfm.2022.738.
[128] A. Vollant, G. Balarac, and C. Corre. Subgrid-scale scalar flux modelling based on optimal estimation theory and machine-
learning procedures. Journal of Turbulence, 18:854–878, 2017. DOI: https://doi.org/10.1080/14685248.2017.1334907.
[129] A. D. Beck, D. G. Flad, and C. D. Munz. Deep neural networks for data-driven turbulence models. arXiv preprint
arXiv:1806.04482, 2018.
[130] V. Sekar, Q. H. Jiang, C. Shu, and B. C. Khoo. Fast flow field prediction over airfoils using deep learning approach. Physics
of Fluids, 31:057103, 2019. DOI: https://doi.org/10.1063/1.5094943.
[131] L. Zhu, W. W. Zhang, J. Kou, and Y. Liu. Machine learning methods for turbulence modeling in subsonic flows around
airfoils. Physics of Fluids, 31:015105, 2019. DOI: https://doi.org/10.1063/1.5061693.
[132] Z. Tadesse, K. Patel, S. Chaudhary, and A. Nagpal. Neural networks for prediction of deflection in composite bridges.
Journal of Constructional Steel Research, 68(1):138–149, 2012. DOI: https://doi.org/10.1016/j.jcsr.2011.08.003.
[133] E. M. Güneyisi, M. D’Aniello, R. Landolfo, K. Mermerdaş, et al. Prediction of the flexural overstrength fac-
tor for steel beams using artificial neural network. Steel and Composite Structures, 17(3):215–236, 2014. DOI:
http://dx.doi.org/10.12989/scs.2014.17.3.215.
[134] T. V. Hung, V. Q. Viet, and T. D. Van. A deep learning-based procedure for estimation of ultimate load carrying of steel
trusses using advanced analysis. Journal of Science and Technology in Civil Engineering (STCE)-HUCE, 13(3):113–123,
2019. DOI: https://doi.org/10.31814/stce.nuce2019-13(3)-11.
[135] G. Chen, T. Li, Q. Chen, S. Ren, C. Wang, and S. Li. Application of deep learning neural network to identify collision
load conditions based on permanent plastic deformation of shell structures. Computational Mechanics, 64(2):435–449, 2019.
DOI: https://doi.org/10.1007/s00466-019-01706-2.
[136] M. Hosseinpour, Y. Sharifi, and H. Sharifi. Neural network application for distortional buckling capacity as-
sessment of castellated steel beams. In Structures, volume 27, pages 1174–1183. Elsevier, 2020. DOI:
https://doi.org/10.1016/j.istruc.2020.07.027.
[137] N. S. Trahair and M. A. Bradford. The behaviour and design of steel structures to AS 4100. CRC Press, 2017.
[138] D. W. White, A. E. Surovek, B. N. Alemdar, C-J. Chang, Y. D. Kim, and G. H. Kuchenbecker. Stability analysis and
design of steel building frames using the 2005 aisc specification. Steel Structures, 6(2):71–91, 2006.
[139] Design of steel structures part 1–1: General rules and rules for buildings. European Committee for Standardization (ECS),
Brussels Belgium., 2005.
[140] D. A. White, W. J. Arrighi, J. Kudo, and S. E. Watts. Multiscale topology optimization using neural net-
work surrogate models. Computer Methods in Applied Mechanics and Engineering, 346:1118–1135, 2019. DOI:
https://doi.org/10.1016/j.cma.2018.09.007.
[141] Y. Zhang, H. Li, M. Xiao, L. Gao, S. Chu, and J. Zhang. Concurrent topology optimization for cellular structures with
nonuniform microstructures based on the kriging metamodel. Structural and Multidisciplinary Optimization, 59(4):1273–
1299, 2019. DOI: https://doi.org/10.1007/s00158-018-2130-0.
[142] O. Sigmund and K. Maute. Topology optimization approaches. Structural and Multidisciplinary Optimization, 48(6):
1031–1055, 2013. DOI: https://doi.org/10.1007/s00158-013-0978-6.
[143] D. W. Abueidda, S. Koric, and N. A. Sobh. Topology optimization of 2d structures with nonlinearities using deep learning.
Computers & Structures, 237:106283, 2020. DOI: https://doi.org/10.1016/j.compstruc.2020.106283.
[144] Y. Yu, T. Hur, J. Jung, and I. G. Jang. Deep learning for determining a near-optimal topological design without any
iteration. Structural and Multidisciplinary Optimization, 59(3):787–799, 2019. DOI: https://doi.org/10.1007/s00158-018-
2101-5.
[145] S. Banga, H. Gehani, S. Bhilare, S. Patel, and L. Kara. 3d topology optimization using convolutional neural networks.
arXiv preprint arXiv:1808.07440, 2018.
[146] B. Li, C. Huang, X. Li, S. Zheng, and J. Hong. Non-iterative structural topology optimization using deep learning.
Computer-Aided Design, 115:172–180, 2019. DOI: https://doi.org/10.1016/j.cad.2019.05.038.
[147] N. Takano and G. Alaghband. Srgan: Training dataset matters. arXiv preprint arXiv:1903.09922, 2019.
[148] Y. Nagano and Y. Kikuta. Srgan for super-resolving low-resolution food images. In Proceedings of the Joint Workshop
on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management, pages 33–37, 2018. DOI:
https://doi.org/10.1145/3230519.3230587.
[149] S. Kumar, S. Tan, L. Zheng, and D. M. Kochmann. Inverse-designed spinodoid metamaterials. npj Computational Materials,
6(1):1–10, 2020. DOI: https://doi.org/10.1038/s41524-020-0341-6.
[150] B. Ni and H. Gao. A deep learning approach to the inverse problem of modulus identification in elasticity. MRS Bulletin,
46(1):19–25, 2021. DOI: https://doi.org/10.1557/s43577-020-00006-y.
[151] M. C. Messner. Convolutional neural network surrogate models for the mechanical properties of periodic structures. Journal
of Mechanical Design, 142(2):024503, 2020. DOI: https://doi.org/10.1115/1.4045040.
[152] D. Tcherniak. Topology optimization of resonating structures using simp method. International Journal for Numerical
Methods in Engineering, 54(11):1605–1622, 2002. DOI: https://doi.org/10.1002/nme.484.
[153] A. Lininger, M. Hinczewski, and G. Strangi. General inverse design of thin-film metamaterials with convolutional neural
networks. arXiv preprint arXiv:2104.01952, 2021.
[154] P. Löper, M. Stuckelberger, B. Niesen, J. Werner, M. Filipič, S. Moon, J-H. Yum, M. Topič, S. De Wolf, and C. Bal-
lif. Complex refractive index spectra of ch3nh3pbi3 perovskite thin films determined by spectroscopic ellipsometry and
spectrophotometry. The journal of physical chemistry letters, 6(1):66–71, 2014. DOI: https://doi.org/10.1021/jz502471h.
[155] K. E. Smith and A O Smith. Conditional gan for timeseries generation. arXiv preprint arXiv:2006.16477, 2020.
[156] Y. Balaji, M. R. Min, B. Bai, R. Chellappa, and H. P. Graf. Conditional gan with discriminative filter generation for
text-to-video synthesis. In IJCAI, volume 1, page 2, 2019.
[157] A. A. Oberai, N. H. Gokhale, and G. R. Feijóo. Solution of inverse problems in elasticity imaging using the adjoint method.
Inverse problems, 19(2):297, 2003.
[158] L. Liang, M. Liu, C. Martin, and W. Sun. A deep learning approach to estimate stress distribution: a fast and ac-
curate surrogate of finite-element analysis. Journal of The Royal Society Interface, 15(138):20170844, 2018. DOI:
https://doi.org/10.1098/rsif.2017.0844.
[159] M. Mozaffar, R. Bostanabad, W. Chen, K. Ehmann, J. Cao, and M. A. Bessa. Deep learning predicts path-dependent
plasticity. PNAS, 116:26414–26420, 2019. DOI: https://doi.org/10.1073/pnas.1911815116.

33
[160] A. Chatterjee. An introduction to the proper orthogonal decomposition. Current science, pages 808–817, 2000. DOI:
https://www.jstor.org/stable/24103957.
[161] Y. C. Liang, H. P. Lee, S. P. Lim, W. Z. Lin, K. H. Lee, and C. Wu. Proper orthogonal decomposition and its applica-
tions—part i: Theory. Journal of Sound and vibration, 252(3):527–544, 2002. DOI: https://doi.org/10.1006/jsvi.2001.4041.
[162] X. Y. Long, S. K. Zhao, C. Jiang, W. P. Li, and C. H. Liu. Deep learning-based planar crack dam-
age evaluation using convolutional neural networks. Engineering Fracture Mechanics, 246:107604, 2021. DOI:
https://doi.org/10.1016/j.engfracmech.2021.107604.
[163] S. Zhu, M. Ohsaki, and X. Guo. Prediction of non-linear buckling load of imperfect reticulated shell us-
ing modified consistent imperfection and machine learning. Engineering Structures, 226:111374, 2021. DOI:
https://doi.org/10.1016/j.engstruct.2020.111374.
[164] B. Miller and L. Ziemiański. Optimization of dynamic behavior of thin-walled laminated cylindrical shells by genetic
algorithms and deep neural networks supported by modal shape identification. Advances in Engineering Software, 147:
102830, 2020. DOI: https://doi.org/10.1016/j.advengsoft.2020.102830.
[165] Z. Nie, H. Jiang, and L. B. Kara. Stress field prediction in cantilevered structures using convolutional neural networks. Jour-
nal of Computing and Information Science in Engineering, 20(1):011002, 2020. DOI: https://doi.org/10.1115/1.4044097.
[166] P. N. Pizarro and L. M. Massone. Structural design of reinforced concrete buildings based on deep neural networks.
Engineering Structures, 241:112377, 2021. DOI: https://doi.org/10.1016/j.engstruct.2021.112377.
[167] C. S. Pathirage, J. Li, L. Li, H. Hao, W. Liu, and P. Ni. Structural damage identification based on autoencoder neural
networks and deep learning. Engineering structures, 172:13–28, 2018. DOI: https://doi.org/10.1016/j.engstruct.2018.05.109.
[168] S. Jiang and J. Zhang. Real-time crack assessment using deep neural networks with wall-climbing unmanned aerial system.
Computer-Aided Civil and Infrastructure Engineering, 35(6):549–564, 2020. DOI: https://doi.org/10.1111/mice.12519.
[169] C. A. Perez-Ramirez, J. P. Amezquita-Sanchez, M. Valtierra-Rodriguez, H. Adeli, A. Dominguez-Gonzalez, and R. J.
Romero-Troncoso. Recurrent neural network model with bayesian training and mutual information for response prediction
of large buildings. Engineering Structures, 178:603–615, 2019. DOI: https://doi.org/10.1016/j.engstruct.2018.10.065.
[170] T. T. Truong, D. Dinh-Cong, J. Lee, and T. Nguyen-Thoi. An effective deep feedforward neural networks (dfnn) method for
damage identification of truss structures using noisy incomplete modal data. Journal of Building Engineering, 30:101244,
2020. DOI: https://doi.org/10.1016/j.jobe.2020.101244.
[171] M. Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine
Learning Research, 19(1):932–955, 2018.
[172] G. Biros, O. Ghattas, M. Heinkenschloss, D. Keyes, B. Mallick, L. Tenorio, B. van Bloemen Waanders, K. Willcox, Y. Mar-
zouk, and L. Biegler. Large-scale inverse problems and quantification of uncertainty. John Wiley & Sons, 2011.
[173] C. R. Vogel. Computational methods for inverse problems. SIAM, 2002.
[174] H. J. Franssen, A. A. Hendricks, M. Riva, M. Bakr, N. Van der Wiel, F. Stauffer, and A. Guadagnini. A comparison of seven
methods for the inverse modelling of groundwater flow. application to the characterisation of well catchments. Advances in
Water Resources, 32(6):851–872, 2009. DOI: https://doi.org/10.1016/j.advwatres.2009.02.011.
[175] E. Randjbaran, R. Zahari, R. Vaghei, and F. Karamizadeh. A review paper on comparison of numerical techniques for
finding approximate solutions to boundary value problems on post-buckling in functionally graded materials. Trends Journal
of Sciences Research, 2(1):1–6, 2015. DOI: https://www.tjsr.org/journal/index.php/tjsr/article/view/9.
[176] H. Triebel. Hybrid function spaces, heat and Navier-Stokes equations. 2015.
[177] D. R. Durran. Numerical methods for wave equations in geophysical fluid dynamics, volume 32. Springer Science & Business
Media, 2013.
[178] G. D. Prato. The stochastic burgers equation. In Kolmogorov Equations for Stochastic PDEs, pages 131–153. Springer,
2004.
[179] D. Medková. The laplace equation. Boundary value problems on bounded and unbounded Lipschitz domains. Springer,
Cham, 2018.
[180] L. Genovese, T. Deutsch, A. Neelov, S. Goedecker, and G. Beylkin. Efficient solution of poisson’s equation with free
boundary conditions. The Journal of chemical physics, 125(7):074105, 2006. DOI: https://doi.org/10.1063/1.2335442.
[181] A. D. Jagtap, E. Kharazmi, and G. E. Karniadakis. Conservative physics-informed neural networks on discrete domains for
conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering,
365:113028, 2020. DOI: https://doi.org/10.1016/j.cma.2020.113028.
[182] A. D. Jagtap and G. E. Karniadakis. Extended physics-informed neural networks (xpinns): A generalized space-time domain
decomposition based deep learning framework for nonlinear partial differential equations. In AAAI Spring Symposium:
MLPS, 2021.
[183] M. Mahmoudabadbozchelou and S. Jamali. Rheology-informed neural networks (rhinns) for forward and inverse metamod-
elling of complex fluids. Scientific reports, 11(1):1–13, 2021. DOI: https://doi.org/10.1038/s41598-021-91518-3.
[184] D. Katsikis, A. D. Muradova, and G. E. Stavroulakis. A gentle introduction to physics informed neural networks, with appli-
cations in static rod and beam problems. J Adv App Comput Math, 9:103–128, 2022. DOI: https://doi.org/10.15377/2409-
5761.2022.09.8.
[185] E. Salvati, A. Tognan, L. Laurenti, M. Pelegatti, and F. De Bona. A defect-based physics-informed machine learning
framework for fatigue finite life prediction in additive manufacturing. Materials & Design, page 111089, 2022. DOI:
https://doi.org/10.1016/j.matdes.2022.111089.
[186] P. Datta, N. Pawar, and S. A. Faroughi. A physics-informed neural network to model the flow of dry particles. In Fall
Meeting 2022. AGU, 2022.
[187] L. Nguyen, M. Raissi, and P. Seshaiyer. Modeling, analysis and physics informed neural network approaches for studying
the dynamics of covid-19 involving human-human and human-pathogen interaction. Computational and Mathematical
Biophysics, 10(1):1–17, 2022. DOI: https://doi.org/10.1515/cmb-2022-0001.
[188] S. Shaier, M. Raissi, and P. Seshaiyer. Data-driven approaches for predicting spread of infectious diseases through dinns:
Disease informed neural networks. Letters in Biomathematics, 9(1):71–105, 2022.
[189] M. Dissanayake and N. Phan-Thien. Neural-network-based approximations for solving partial differential equations. com-
munications in Numerical Methods in Engineering, 10(3):195–201, 1994. DOI: https://doi.org/10.1002/cnm.1640100303.
[190] H. Owhadi. Bayesian numerical homogenization. Multiscale Modeling & Simulation, 13(3):812–828, 2015. DOI:
https://doi.org/10.1137/140974596.
[191] M. Raissi, A. Yazdani, and G. E. Karniadakis. Hidden fluid mechanics: A navier-stokes informed deep learning framework
for assimilating flow visualization data. arXiv preprint arXiv:1808.04327, 2018.
[192] A. Iserles. A first course in the numerical analysis of differential equations. Number 44. Cambridge university press, 2009.
[193] M. A. Nabian, R. J. Gladstone, and H. Meidani. Efficient training of physics-informed neural networks
via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 36(8):962–977, 2021. DOI:
https://doi.org/10.1111/mice.12685.
[194] C. H. Su and C. S. Gardner. Korteweg-de vries equation and generalizations. iii. derivation of the korteweg-de vries equation
and burgers equation. Journal of Mathematical Physics, 10(3):536–539, 1969. DOI: https://doi.org/10.1063/1.1664873.

34
[195] P. Constantin and C. Foias. Navier-stokes equations. University of Chicago Press, 2020.
[196] J. Stiasny, G. S. Misyris, and S. Chatzivasileiadis. Physics-informed neural networks for non-linear system
identification for power system dynamics. In 2021 IEEE Madrid PowerTech, pages 1–6. IEEE, 2021. DOI:
https://doi.org/10.1109/PowerTech46648.2021.9495063.
[197] Z. Mao, A. D. Jagtap, and G.E. Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods
in Applied Mechanics and Engineering, 360:112789, 2020.
[198] A. D. Jagtap, Z. Mao, N. Adams, and G. E. Karniadakis. Physics-informed neural networks for inverse problems in
supersonic flows. arXiv preprint arXiv:2202.11821, 2022.
[199] T. Zhang, B. Dey, P. Kakkar, A Dasgupta, and A. Chakraborty. Frequency-compensated pinns for fluid-dynamic design
problems. arXiv preprint arXiv:2011.01456, 2020.
[200] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng.
Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information
Processing Systems, 33:7537–7547, 2020.
[201] C. Cheng and G-T. Zhang. Deep learning method based on physics informed neural network with resnet block for solving
fluid flow problems. Water, 13(4):423, 2021. DOI: https://doi.org/10.3390/w13040423.
[202] Q. Lou, X. Meng, and G. E. Karniadakis. Physics-informed neural networks for solving forward and inverse
flow problems via the boltzmann-bgk formulation. Journal of Computational Physics, 447:110676, 2021. DOI:
https://doi.org/10.1016/j.jcp.2021.110676.
[203] H. Wessels, C. Weißenfels, and P. Wriggers. The neural particle method–an updated lagrangian physics informed neural
network for computational fluid dynamics. Computer Methods in Applied Mechanics and Engineering, 368:113127, 2020.
DOI: https://doi.org/10.1016/j.cma.2020.113127.
[204] M. Mahmoudabadbozchelou, G. E. Karniadakis, and S. Jamali. nn-pinns: Non-newtonian physics-informed neural networks
for complex fluid modeling. Soft Matter, 18(1):172–185, 2022. DOI: https://doi.org/10.1039/D1SM01298C.
[205] E. Haghighat, D. Amini, and R. Juanes. Physics-informed neural network simulation of multiphase poroelasticity using
stress-split sequential training. arXiv preprint arXiv:2110.03049, 2021.
[206] M. M. Almajid and M. O. Abu-Al-Saud. Prediction of porous media fluid flow using physics informed neural networks.
Journal of Petroleum Science and Engineering, 208:109205, 2022. DOI: https://doi.org/10.1016/j.petrol.2021.109205.
[207] I. Depina, S. Jain, V. S. Mar, and H. Gotovac. Application of physics-informed neural networks to inverse problems in
unsaturated groundwater flow. Georisk: Assessment and Management of Risk for Engineered Systems and Geohazards, 16
(1):21–36, 2022. DOI: https://doi.org/10.1080/17499518.2021.1971251.
[208] A. M. Tartakovsky, C. O. Marrero, P. Perdikaris, G. D. Tartakovsky, and D. Barajas-S. Learning parameters and constitutive
relationships with physics informed deep neural networks. arXiv preprint arXiv:1808.03398, 2018.
[209] S. Thakur, M. Raissi, and A. M. Ardekani. Viscoelasticnet: A physics informed neural network framework for stress
discovery and model selection. arXiv preprint arXiv:2209.06972, 2022.
[210] C. Fernandes, S. A. Faroughi, R. Ribeiro, A. Isabel, and G. H. McKinley. Finite volume simulations of particle-laden
viscoelastic fluid flows: Application to hydraulic fracture processes. Engineering with Computers, pages 1–27, 2022. DOI:
https://doi.org/10.1007/s00366-022-01626-5.
[211] P. Chiu, J. C. Wong, C. Ooi, M. H. Dao, and Y. Ong. Can-pinn: A fast physics-informed neural network based on coupled-
automatic–numerical differentiation method. Computer Methods in Applied Mechanics and Engineering, 395:114909, 2022.
DOI: https://doi.org/10.1016/j.cma.2022.114909.
[212] J. Van Der Hoeven. The truncated fourier transform and applications. In Proceedings of the 2004 international symposium
on Symbolic and algebraic computation, pages 290–296, 2004. DOI: https://doi.org/10.1145/1005285.1005327.
[213] G. Raynaud, S. Houde, and F. P. Gosselin. Modalpinn: an extension of physics-informed neural networks with enforced
truncated fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors. Journal of
Computational Physics, page 111271, 2022. DOI: https://doi.org/10.1016/j.jcp.2022.111271.
[214] J. Oldenburg, F. Borowski, A. Öner, K. Schmitz, and M. Stiehm. Geometry aware physics informed neural network surrogate
for solving navier-stokes equation (gapinn). 2022. DOI: https://doi.org/10.21203/rs.3.rs-1466550/v1.
[215] N. Wandel, M. Weinmann, M. Neidlin, and R. Klein. Spline-pinn: Approaching pdes without data using fast, physics-
informed hermite-spline cnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8529–8538,
2022. DOI: https://doi.org/10.1609/aaai.v36i8.20830.
[216] X. Jin, S. Cai, H. Li, and G. E. Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural net-
works for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021. DOI:
https://doi.org/10.1016/j.jcp.2020.109951.
[217] C. Cheng, P. Xu, Y. Li, and G. Zhang. Deep learning based on pinn for solving 2 d0f vortex induced vibration of cylinder
with high reynolds number. arXiv preprint arXiv:2106.01545, 2021.
[218] H. Eivazi, M. Tahani, P. Schlatter, and R. Vinuesa. Physics-informed neural networks for solving reynolds-averaged navier–
stokes equations. Physics of Fluids, 34(7):075117, 2022. DOI: https://doi.org/10.1063/5.0095270.
[219] J. Wang, J. Wu, and H. Xiao. Physics-informed machine learning approach for reconstructing reynolds
stress modeling discrepancies based on dns data. Physical Review Fluids, 2(3):034603, 2017. DOI:
https://doi.org/10.1103/PhysRevFluids.2.034603.
[220] X. Zhang, Y. Zhu, J. Wang, L. Ju, Y. Qian, M. Ye, and J. Yang. Gw-pinn: A deep learning algorithm for solving groundwater
flow equations. Advances in Water Resources, page 104243, 2022. DOI: https://doi.org/10.1016/j.advwatres.2022.104243.
[221] M. Aliakbari, M. Mahmoudi, P. Vadasz, and A. Arzani. Predicting high-fidelity multiphysics data from low-fidelity fluid flow
and transport solvers using physics-informed neural networks. International Journal of Heat and Fluid Flow, 96:109002,
2022. DOI: https://doi.org/10.1016/j.ijheatfluidflow.2022.109002.
[222] A. M. Tartakovsky, C. O. Marrero, P. Perdikaris, G. D. Tartakovsky, and D. Barajas-Solano. Physics-informed deep neural
networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resources Research, 56
(5):e2019WR026731, 2020. DOI: https://doi.org/10.1029/2019WR026731.
[223] G. Kissas, Y. Yang, E. Hwuang, W. R. Witschey, J. A. Detre, and P. Perdikaris. Machine learning in car-
diovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-
informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358:112623, 2020. DOI:
https://doi.org/10.1016/j.cma.2019.112623.
[224] A. Arzani, J-X. Wang, and R. M. D’Souza. Uncovering near-wall blood flow from sparse data with physics-informed neural
networks. Physics of Fluids, 33(7):071905, 2021. DOI: https://doi.org/10.1063/5.0055600.
[225] A. D. Jagtap, D. Mitsotakis, and G. E. Karniadakis. Deep learning of inverse water waves problems using
multi-fidelity data: Application to Serre–Green–Naghdi equations. Ocean Engineering, 248:110775, 2022. DOI:
https://doi.org/10.1016/j.oceaneng.2022.110775.
[226] E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes. A deep learning framework for solution and discovery in
solid mechanics: linear elasticity. arXiv preprint arXiv:2003.02751, 2020.
[227] K. Shukla, A. D. Jagtap, J. L. Blackshire, D. Sparkman, and G. E. Karniadakis. A physics-informed neural network for

35
 quantifying the microstructural properties of polycrystalline nickel using ultrasound data: A promising approach for solving
inverse problems. IEEE Signal Processing Magazine, 39(1):68–77, 2021. DOI: https://doi.org/10.1109/MSP.2021.3118904.
[228] A. Henkes, H. Wessels, and R. Mahnken. Physics informed neural networks for continuum micromechanics. Computer
Methods in Applied Mechanics and Engineering, 393:114790, 2022. DOI: https://doi.org/10.1016/j.cma.2022.114790.
[229] S. Goswami, M. Yin, Y. Yu, and G. E. Karniadakis. A physics-informed variational deeponet for predicting crack
path in quasi-brittle materials. Computer Methods in Applied Mechanics and Engineering, 391:114587, 2022. DOI:
https://doi.org/10.1016/j.cma.2022.114587.
[230] Z. Zhang and G. X. Gu. Physics-informed deep learning for digital materials. Theoretical and Applied Mechanics Letters,
11(1):100220, 2021. DOI: https://doi.org/10.1016/j.taml.2021.100220.
[231] C. Rao, H. Sun, and Y. Liu. Physics informed deep learning for computational elastodynamics without labeled data. arXiv
preprint arXiv:2006.08472, 2020.
[232] Z. Fang and J. Zhan. Deep physical informed neural networks for metamaterial design. IEEE Access, 8:24506–24513, 2019.
DOI: https://doi.org/10.1109/ACCESS.2019.2963375.
[233] M. Lax and D. F. Nelson. Maxwell equations in material form. Physical Review B, 13(4):1777, 1976. DOI:
https://doi.org/10.1103/PhysRevB.13.1777.
[234] E. Zhang, M. Yin, and G. E. Karniadakis. Physics-informed neural networks for nonhomogeneous material identification in
elasticity imaging. arXiv preprint arXiv:2009.04525, 2020.
[235] D. W. Abueidda, S. Koric, E. Guleryuz, and N. A. Sobh. Enhanced physics-informed neural networks for hyperelasticity.
arXiv preprint arXiv:2205.14148, 2022.
[236] D. W. Abueidda, S. Koric, R. A. Al-Rub, C. M. Parrott, K. A. James, and N. A. Sobh. A deep learning en-
ergy method for hyperelasticity and viscoelasticity. European Journal of Mechanics-A/Solids, 95:104639, 2022. DOI:
https://doi.org/10.1016/j.euromechsol.2022.104639.
[237] L. Yuan, Y. Ni, X. Deng, and S. Hao. A-pinn: Auxiliary physics informed neural networks for forward and in-
verse problems of nonlinear integro-differential equations. Journal of Computational Physics, 462:111260, 2022. DOI:
https://doi.org/10.1016/j.jcp.2022.111260.
[238] L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM
Review, 63(1):208–228, 2021. DOI: https://doi.org/10.1137/19M1274067.
[239] R. Arora. Physrnet: Physics informed super-resolution network for application in computational solid mechanics. arXiv
preprint arXiv:2206.15457, 2022.
[240] E. Madenci, A. Barut, and M. Dorduncu. Peridynamic differential operator for numerical analysis, volume 10. Springer,
2019.
[241] E. Haghighat, A. C. Bekar, E. Madenci, and R. Juanes. A nonlocal physics-informed deep learning framework using the
peridynamic differential operator. Computer Methods in Applied Mechanics and Engineering, 385:114012, 2021. DOI:
https://doi.org/10.1016/j.cma.2021.114012.
[242] G. Huang, Q. Zhu, and C. Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1-3):489–501,
2006. DOI: https://doi.org/10.1016/j.neucom.2005.12.126.
[243] C. A. Yan, R. Vescovini, and L. Dozio. A framework based on physics-informed neural networks and ex-
treme learning for the analysis of composite structures. Computers & Structures, 265:106761, 2022. DOI:
https://doi.org/10.1016/j.compstruc.2022.106761.
[244] S. Rezaei, A. Harandi, A. Moeineddin, B. Xu, and S. Reese. A mixed formulation for physics-informed neural networks
as a potential solver for engineering problems in heterogeneous domains: comparison with finite element method. arXiv
preprint arXiv:2206.13103, 2022.
[245] A. Mallampati and M. Almekkawy. Measuring tissue elastic properties using physics based neural net-
works. In 2021 IEEE UFFC Latin America Ultrasonics Symposium (LAUS), pages 1–4. IEEE, 2021. DOI:
https://doi.org/10.1109/LAUS53676.2021.9639231.
[246] W. Li, M. Z. Bazant, and J. Zhu. A physics-guided neural network framework for elastic plates: Comparison of governing
equations-based and energy-based approaches. Computer Methods in Applied Mechanics and Engineering, 383:113933,
2021. DOI: https://doi.org/10.1016/j.cma.2021.113933.
[247] M. Vahab, E. Haghighat, M. Khaleghi, and N. Khalili. A physics informed neural network approach to solution and
identification of biharmonic equations of elasticity. arXiv preprint arXiv:2108.07243, 2021.
[248] M. Raj, P. Kumbhar, and R. K. Annabattula. Physics-informed neural networks for solving thermo-mechanics problems of
functionally graded material. arXiv preprint arXiv:2111.10751, 2021.
[249] E. Zhang, M. Dao, G. E. Karniadakis, and S. Suresh. Analyses of internal structures and defects in materials using
physics-informed neural networks. Science advances, 8(7):eabk0644, 2022. DOI: https://doi.org/10.1126/sciadv.abk0644.
[250] J. Bastek and D. M. Kochmann. Physics-informed neural networks for shell structures. arXiv preprint arXiv:2207.14291,
2022.
[251] X. Zhang, J. Gong, and F Xuan. A physics-informed neural network for creep-fatigue life predic-
tion of components at elevated temperatures. Engineering Fracture Mechanics, 258:108130, 2021. DOI:
https://doi.org/10.1016/j.engfracmech.2021.108130.
[252] E. Haghighat, A. C. Bekar, E. Madenci, and R. Juanes. Deep learning for solution and inversion of structural mechanics
and vibrations. arXiv preprint arXiv:2105.09477, 2021.
[253] B. Zheng, T. Li, H. Qi, L. Gao, X. Liu, and L. Yuan. Physics-informed machine learning model for computational fracture
of quasi-brittle materials without labelled data. International Journal of Mechanical Sciences, 223:107282, 2022. DOI:
https://doi.org/10.1016/j.ijmecsci.2022.107282.
[254] R. Arora, P. Kakkar, B. Dey, and A. Chakraborty. Physics-informed neural networks for modeling rate-and temperature-
dependent plasticity. arXiv preprint arXiv:2201.08363, 2022.
[255] J. Bai, H. Jeong, C. P. Batuwatta, S. Xiao, Q. Wang, C. M. Rathnayaka, L. Alzubaidi, G. Liu, and Y. Gu. An introduction
to programming physics-informed neural network-based computational solid mechanics. arXiv preprint arXiv:2210.09060,
2022.
[256] V. Dwivedi and B. Srinivasan. Solution of biharmonic equation in complicated geometries with physics informed
extreme learning machine. Journal of Computing and Information Science in Engineering, 20(6), 2020. DOI:
https://doi.org/10.1115/1.4046892.
[257] V. Dwivedi, N. Parashar, and B. Srinivasan. Distributed learning machines for solving forward and inverse problems in
partial differential equations. Neurocomputing, 420:299–316, 2021. DOI: https://doi.org/10.1016/j.neucom.2020.09.006.
[258] O. Fuks and H. A. Tchelepi. Limitations of physics informed machine learning for nonlinear two-phase trans-
port in porous media. Journal of Machine Learning for Modeling and Computing, 1(1), 2020. DOI:
https://doi.org/10.1615/JMachLearnModelComput.2020033905.
[259] L. Lu, P. Jin, and G. E. Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based
on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
[260] Y. Long, X. She, and S. Mukhopadhyay. Hybridnet: integrating model-based and data-driven learning to predict evolution

36
 of dynamical systems. In Conference on Robot Learning, pages 551–560. PMLR, 2018.
[261] Y. Zhu, N. Zabaras, P. Koutsourelakis, and P. Perdikaris. Physics-constrained deep learning for high-dimensional surrogate
modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56–81, 2019. DOI:
https://doi.org/10.1016/j.jcp.2019.05.024.
[262] N. Geneva and N. Zabaras. Modeling the dynamics of pde systems with physics-constrained deep auto-regressive networks.
Journal of Computational Physics, 403:109056, 2020. DOI: https://doi.org/10.1016/j.jcp.2019.109056.
[263] R. Wang, K. Kashinath, M. Mustafa, A. Albert, , and R. Yu. Towards physics-informed deep learning for turbulent flow
prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
pages 1457–1466, 2020. DOI: https://doi.org/10.1145/3394486.3403198.
[264] R. Ranade, C. Hill, and J. Pathak. Discretizationnet: A machine-learning based solver for navier–stokes equations us-
ing finite volume discretization. Computer Methods in Applied Mechanics and Engineering, 378:113722, 2021. DOI:
https://doi.org/10.1016/j.cma.2021.113722.
[265] H. Gao, L. Sun, and J. Wang. Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solv-
ing parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428:110079, 2021. DOI:
https://doi.org/10.1016/j.jcp.2020.110079.
[266] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural
information processing systems, 31, 2018.
[267] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
[268] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional lstm network: A machine learning approach
for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
[269] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of
the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[270] P. Ren, C. Rao, Y. Liu, J. Wang, and Hao Sun. Phycrnet: Physics-informed convolutional-recurrent network for
solving spatiotemporal pdes. Computer Methods in Applied Mechanics and Engineering, 389:114399, 2022. DOI:
https://doi.org/10.1016/j.cma.2021.114399.
[271] K. Bhattacharya, B. Hosseini, N.B. Kovachk, and A.M. Stuart. Model reduction and neural networks for parametric pdes.
arXiv preprint arXiv:2005.03180, 2020.
[272] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Graph
kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020.
[273] R.A. DeVore. The theoretical foundation of reduced basis methods. Model reduction and approximation: theory and
algorithms, 15:137, 2017.
[274] Y. Zhu and N. Zabaras. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty
quantification. Journal of Computational Physics, 366:415–447, 2018. DOI: https://doi.org/10.1016/j.jcp.2018.04.018.
[275] T. J. Grady, R. Khan, M. Louboutin, Z. Yin, P. A. Witte, R. Chandra, R. J. Hewett, and F. J. Herrmann. Towards large-
scale learned solvers for parametric pdes with model-parallel fourier neural operators. arXiv preprint arXiv:2204.01205,
2022.
[276] M. Bui, C. S. Adjiman, A. Bardow, E. J. Anthony, A. Boston, S. Brown, P. S. Fennell, S. Fuss, A. Galindo, L. A. Hackett,
and others. Carbon capture and storage (ccs): the way forward. Energy & Environmental Science, 11(5):1062–1176, 2018.
DOI: https://doi.org/10.1039/C7EE02342A.
[277] G. Wen, Z. Li, K. Azizzadenesheli, A. Anandkumar, and S.M. Benson. U-fno—an enhanced fourier neural
operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022. DOI:
https://doi.org/10.1016/j.advwatres.2022.104180.
[278] H. You, Q. Zhang, C.J. Ross, C-H. Lee, and Y. Yu. Learning deep implicit fourier neural operators (ifnos) with applications
to heterogeneous material modeling. arXiv preprint arXiv:2203.08205, 2022.
[279] A. G. Baydin, B. A Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: a survey.
Journal of Marchine Learning Research, 18:1–43, 2018.
[280] C. Rackauckas, M. Innes, Y. Ma, J. Bettencourt, L. White, and V. Dixit. Diffeqflux. jl-a julia library for neural differential
equations. arXiv preprint arXiv:1902.02376, 2019.
[281] L. S. Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
[282] Y. Ma, V. Dixit, M. J. Innes, X. Guo, and C. Rackauckas. A comparison of automatic differentiation and continuous
sensitivity analysis for derivatives of differential equation solutions. In 2021 IEEE High Performance Extreme Computing
Conference (HPEC), pages 1–9. IEEE, 2021. DOI: https://doi.org/10.1109/HPEC49654.2021.9622796.
[283] M. Poli, S. Massaroli, A. Yamashita, H. Asama, and J. Park. Torchdyn: A neural differential equations library. arXiv
preprint arXiv:2009.09346, 2020.
[284] Z. Lai, C. Mylonas, S. Nagarajaiah, and E. Chatzi. Structural identification with physics-informed neural ordinary differ-
ential equations. Journal of Sound and Vibration, 508:116196, 2021. DOI: https://doi.org/10.1016/j.jsv.2021.116196.
[285] M. A. Roehrl, T. A. Runkler, V. Brandtstetter, M. Tokic, and S. Obermayer. Modeling system dynamics with
physics-informed neural networks based on lagrangian mechanics. IFAC-PapersOnLine, 53(2):9195–9200, 2020. DOI:
https://doi.org/10.1016/j.ifacol.2020.12.2182.
[286] A. Dulny, A. Hotho, and A. Krause. Neuralpde: Modelling dynamical systems from data. arXiv preprint arXiv:2111.07671,
2021.
[287] He K., Zhang X., Ren S., and Sun Jian. Identity mappings in deep residual networks. In European conference on computer
vision, pages 630–645. Springer, 2016.
[288] L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, and G. E. Karniadakis. A comprehensive and fair comparison of two
neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering,
393:114778, 2022. DOI: https://doi.org/10.1016/j.cma.2022.114778.
[289] H. You, Y. Yu, M. D’Elia, T. Gao, and S. Silling. Nonlocal kernel network (nkn): a stable and resolution-independent deep
neural network. arXiv preprint arXiv:2201.02217, 2022.
[290] A. M. Molina, I. F. Avelino, E. F. Morales, and L. E. Sucar. Causal based q-learning. Research in Computing Science, 149:
95–104, 2020.

37