Scribd
Upload
19 views

Final Presentation PML

The document discusses recent advances in machine learning, specifically neural operators and Fourier neural operators. It introduces neural operators as models that can universally approximate continuous operators and are discretization invariant. It then discusses Fourier neural operators and how they achieve resolution invariance through working in the Fourier space. Several experiments are presented applying these techniques to learn the heat equation operator in 2D and 3D.

Uploaded by

Gonzalo Becker
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
19 views

Final Presentation PML

The document discusses recent advances in machine learning, specifically neural operators and Fourier neural operators. It introduces neural operators as models that can universally approximate continuous operators and are discretization invariant. It then discusses Fourier neural operators and how they achieve resolution invariance through working in the Fourier space. Several experiments are presented applying these techniques to learn the heat equation operator in 2D and 3D.

Uploaded by

Gonzalo Becker
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 45

Projects on Recent Advances in

Machine Learning
Neural Operators

Gonzalo BECKER
Alex SZAPIRO
Álvaro SCARRAMBERG

29/03/24
1
1. Neural Operators
Index
2. Fourier Neural Operators

3. Experiments
3.1. Dataset
3.2. Parametric encoding
3.3. Higher dimensions

4. Future improvements

5. Conclusion

6. Appendix
Neural Operators
Neural Operator: Learning Maps Between Function Spaces
With Applications to PDEs - Kovachki et al. (2022)

3
Neural Operators 4
Introduction

Neural networks try to learn mappings between functions.

They satisfy the property of being universal approximators.

Universal approximation: a model that can uniformly approximate any


continuous operator.

Caveat: Typical neural networks do so for finite dimensional spaces.

What if we wanted instead to learn an operator such as a PDE ?


Neural Operators 5
Properties

Neural Operators: models that satisfy both universal approximation and


discretization invariance.

A discretization invariant model satisfies:


1. can act on any discretization of the input function, i.e. accepts any set of points in the
input domain,
2. can be evaluated at any point of the output domain,
3. converges to a continuum operator as the discretization is refined.
Neural Operators 6
Applications

Why is this important?

Burgers, Darcy subsurface flow, Navier-Stokes equations, etc.

Numerical solvers are expensive! This method allows to obtain solutions


in a manner that is several orders of magnitude faster.

Navier-Stokes equation
Neural Operators 7
Problem setting

We aim to estimate an operator (PDE), using a parametric map


between infinite dimension spaces (Neural Network):

The inputs and outputs of the network are functions.

In practical terms they are equivalent to data points, or samplings of


those functions.
Neural Operators 8
Example
Neural Operators 9
Variants

There many variants of neural operators:


● graph-based operators

● low-rank operators

● multipole graph-based operators

● Fourier operators
Fourier Neural Operator
Fourier Neural Operator for parametric partial differential
equations - Li et al.(2021)

10
Fourier Neural Operator 11

What is a pixel-wise or 1x1 convolution? The operation remains


the same no matter the
resolution.
Only pixels from different
channels are combined.
Fourier Neural Operator 12

The Fourier Neural operator


achieves this while also being How?
Resolution Invariant.
Fourier Neural Operator 13

In this case they


The input is translated into a use 1x1 convolutions
latent space with a higher
dimensionality
Fourier Neural Operator 14

This is a 1x1 convolution.

How is this block This layer emulates


resolution invariant? convolutions as matrix
products in the fourier space
Fourier Neural Operator 15

Signal sampled at 66 kHz

The maximum frequency


obtained in the FFT is 33 kHz
Fourier Neural Operator 16

Sample
Rates: Lowering the SR
lowers the maximum
66k frequency...

But it will not change


21k the lower frequency
values of the FFT!
Fourier Neural Operator 17
Resolution invariance in The FNO uses only the base
the Fourier Space modes of the input spectrum.

Doubling the spatial sampling frequency should


yield the same values for the base frequencies.
Fourier Neural Operator 18

From our perspective the main drawback


of the FNO is:

Since the FNO discards higher frequency


modes its output will only recreate the lower
frequency components of the solution.
Dataset definition
for the heat equation

19
Time evolution

Boundary condition:
T = 0 for borders

20
Problem statement 21

Differential equation relating time and spatial variables

Spatial and temporal discretization


Simulation theory 22

Discretization of the problem

Convergence condition:
Initial Conditions 23

➢ Boundary conditions: All grid borders are set to a fixed value (0°)

➢ Random gaussian shaped heat areas are set in the first iteration

➢ To simulate lower resolutions, a subsampling is performed


Dataset Generation 24

Idea: jump every


5 iterations to
accelerate
diffusion
Dataset Generation 25
Dataset Generation 26
Parametric Encoding

27
Time evolution prediction 28

Trained FNO
Positional Embedding 29

Spatial Understanding: Spatial positional


embedding aids deep learning models in
comprehending the spatial relationships
between elements in input data.

https://www.mdpi.com/2227-7390/11/21/4550
Parametric Embedding 30

Data (Temperature)

X coordinates
Normalised
Y coordinates
Constant for
Parameter (Gamma)
every data point
Training 31

➢ Training 𝝲 values: 5 values


between 0.01 and 0.25

➢ Training duration: 2 min. on


GPU

➢ Training size: 2386 input-output


pairs

➢ Batch size: 32

➢ Number of epochs: 100


Hyperparameters for the architecture 32

➢ 8 Fourier layers ➢ 4 input channels

➢ 32 hidden channels ➢ Initial learning rate: 0.008

➢ 64 projection channels ➢ Adam optimizer


Results
16x16

33
Results
What if 𝞬 is outside the training range?

34
Results
32x32

35
Exploring higher dimensions

36
3D heat equation 37
Initial condition

There are no
constraints on the
dimensionality of input
data!

(Only the
computational cost…)
Results

Time propagation of
the 3D heat equation

38
Future improvements

39
Future improvements: 40

● Regression to find the equation parameter.


● Upgrade the model to be N-dimensional.
Conclusion

41
Conclusion 42

● We reviewed the literature on Neural Operators.


● We implemented and tested the Fourier Neural Operators for the 2D
and 3D Heat Diffusion equation.
● We implemented parameter embedding into the model.
Appendix

43
Results

44
Results

45

You might also like