ML Visuals

By dair.ai
Basic components
Softmax

Convolve

Sharpen
Softmax

Convolve

Sharpen
Architectures
C C C CNN
Layer
 Input

Conv3-32 Conv3-64
Maxpool
x4 x2

32x32x3

Maxpool Conv3-128 Maxpool

(2x2) x1 (2x2)

Output

Feature
FC-512
Vector
 SM

FC

AM

L L L L L L L

L L L L L L L Conv

C C C C C C C
 SM

FC

L L L L L L L Conv

C C C C C C C
 SM

FC

L L L L L L L L L L

C C C C C C C C C C
 SM

FC

Conv

C C C C C C C C C C
A) LSTM B) Mixed LSTM / 1D Conv

C) Mixed BiLSTM / 1D Conv D) Mixed Att-BiLSTM / 1D Conv

 SM

FC

AM

L L L L L L L L L L Conv

C C C C C C C C C C
 SM

FC

L L L L L L L L L L Conv

C C C C C C C C C C
 SM

FC

AM

L L L L L L L L L L

L L L L L L L L L L Conv

C C C C C C C C C C
 SM

FC

L L L L L L L L L L Conv

C C C C C C C C C C
Attention
layer

LSTM
layer

CNN C C C C C C C C C C
layer
 Attention
layer

LSTM
layer

CNN C C C C C C C C C C
layer
 SM

FC

AM

L L L L L L L Conv

C C C C C C C
 Hidden
Layers

Input Layer Output

a[1]1 a[2]1 a[3]1
Layer

a[1]2 a[2]2 a[3]2

a[4] Ŷ
X
a[1]3 a[2]3 a[3]3

a[1]n a[2]n a[3]n

X = A[0] A[1] A[2] A[3] A[4]

 Hidden
Layers

Input Layer Output

a1
[1 ]
a[2]1 a[3]1
Layer

a[1]2 a[2]2 a[3]2

a[4] Ŷ
X
a[1]3 a[2]3 a[3]3

a[1]n a[2]n a[3]n

X = A[0] A[1] A[2] A[3] A[4]

 Hidden
Layers

Input Layer Output

a[1]1 a[2]1 a[3]1
Layer

a[1]2 a[2]2 a[3]2

a[4] Ŷ
X
a[1]3 a[2]3 a[3]3

X = A[0] a[1]n a[2]n a[3]n A[4]

A[1] A[2] A[3]

 CONV
operation
MxM ReLU

+b1 +b1 MxMX2

+b2 +b2

NxNx3
MxM ReLU

a[l-1] a[l]
 CONV
operation
MxM ReLU

+b1 +b1 MxMX2

+b2 +b2

NxNx3
MxM ReLU
 CONV
operation
MxM ReLU

+b1 +b1 MxMX2

+b2 +b2

NxNx3
MxM ReLU
Striding in
CONV

S=1

S=2
Inception MaxPool
Module Same s=1

5x5 Same
NxNx192
3x3 Same

1x1 Same

NxNx128
NxNx192

NxNx32

NxNx64
 Softmax

Linear

Add & Norm

Feed
Forward

Add & Norm

Add & Norm
Multi-Head
Feed Attention
Forward
Add & Norm
Add & Norm
Masked
Multi-Head Multi-Head
Attention Attention

Positional Positional
Encoding Encoding

Input Output
Embedding Embedding

Inputs Outputs (shifted right)

 Softmax

Linear

Add & Norm

Feed
Forward

Add & Norm

Add & Norm
Multi-Head
Feed Attention
Forward
Add & Norm
Add & Norm
Masked
Multi-Head Multi-Head
Attention Attention

Positional Positional
Encoding Encoding

Input Output
Embedding Embedding

Inputs Outputs (shifted right)

I love coding and writing

Tokenize

“I love coding and writing”

ML Concepts
 Size
Family
?
X Y
#bed

Walk PRICE
? ŷ Basic Neuron
ZIP
Model

Schoo
l
How does NN
Wealth work (Insprired
from Coursera)

Ŷ = 0

Ŷ = 1
Logistic
Regression
 Linear ReLU(x)
regression

$
$

Size Size
 256
225
56
.

R-G-B
NxN .
NxN .
NxN NxN
214
210
211

Unrolling Feature vectors

 Large NN

Med NN

η Small NN

SVM,LR
etc

Amount of
Data

Why does Deep learning work?

 Hidden
Input Output

a[1]1

One hidden layer neural network

a[1]2

X a[2] Ŷ

a[1]3

a[1]4

X = A[0] A[1] A[2]

 x[1] a[1]1

a[2]

x[2] a[1]2

x[1]

x[2]

x[3]

Neural network templates

 Train Valid Test

Underfitting Good fit Overfitting

x2

x2

x2
x1 x1 x1

Train-Dev-Test vs. Model fitting

x[1]

a[L] DropOut
x[2]

x[3]

x2
r=1
x2

Normalizatio
x1 n

x1
 w2 Early stopping
J

Er
r

w1
w1

Before Dev
w2
Normalization
Train
it
.
J w2

w1 w1

After Normalization
w2
x1

x2 w[L]

w[1] w[2] w[L- w[L- Deep neural

2] 1]
networks

FN TN

TP FP
Understanding
Precision & Recall
 Batch vs. Mini-
batch
Gradient Descent

w2 w2
BGD
SGD

Batch
Gradient Descent
vs. SGD

SGD
w1 w1
x[1]
p[1]

x[2]
p[2]

x[3]

Softmax Prediction
with 2 outputs
Abstract backgrounds
dair.ai
Gradient Backgrounds
ML and Health
 Time slice
in
Theta
Alpha o pa
Conv N

IC
A Conv
Conv L
Conv
Conv S
Alpha Conv
Conv T
Conv M
Conv
Conv
Conv
+
Conv A
Conv
Beta
Conv M
Conv
Conv
Conv

n e

en n
tio tur

s m t io
en t y
ng re
EEG Time Series

s m si
ga a

es ca

t
ni tu
es

re Fe

es en

ss o
ar e a
ag

A in L
Spectral Topography

s s nt
gg a l
Le i a l F
Im

A in I
A por

Pa
Maps
G

Pa
m
at
EE

Te
Sp
Conv
Conv
Conv
Conv
Conv
Conv
Conv
Conv
Conv
Conv
Activations
(U=WX)

Scalp maps
( )
Test S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 Ave
Subje rag
ct e
Acc
ura
cy

Pain 0.8 0.88 0.95 0.96 0.93 0.95 1 0.97 0.98 0.92 0.90 0.89 0.80 0.9
Inten 20 82 69 25 22 63 07 09 26 15 2 94 226
sity 9 230
77

Pain 0.7 0.80 0.93 0.99 0.92 1 0.99 0.90 0.88 0.89 0.88 0.70 0.75 0.8
Locat 24 28 97 51 86 48 88 44 56 16 81 91 786
ion 3 846
15
Level 1 Level 2 Level 3 Level 4

No Low Moderate High

Pain Pain Pain Pain

Signal Segmentation
AEP

Bicubic

PSD
Theta(4~8Hz) Spectral
FFT Topography
Alpha(8~13Hz) Map
Beta(13~30Hz)

Image Generation
 Input

Conv3-32 Maxpool Conv3-64

x4 (2x2) x2

32x32x3

Maxpool Conv3- Maxpool

(2x2) 128 (2x2)

Output

Feature
FC-512
Vector

ConvNet Configuration
 Feature
Vector

Output
FC-512
Max-Pool
Output
Conv3-128
3

ConvNet Configuration
er

FC-512
Lay

ck4

Max-Pool
Sta

Max-Pool
Conv3-64
Conv3-128
ck3

Conv3-64
2

Sta
er

Max-Pool
Lay

Max-Pool
Conv3-64
Conv3-32
Conv3-64
ck2

Conv3-32
Sta

Max-Pool
Conv3-32
Conv3-32
Conv3-32
1er

Conv3-32
Lay

Input
Conv3-32
Conv3-32
ck1
Sta
Softmax
FC
er3 er4
Lay Lay

FC
Max-
Pool
Conv
2
er
Max-
Lay
Pool
Conv

1
er
Lay
Input
 Output
FC-512
ck4
Sta

Max-Pool
Conv3-128
ck3
Sta

Max-Pool
Conv3-64
ck1

Conv3-64
ck2
Sta

Sta

Max-Pool
Conv3-32

Conv3-32

Conv3-32

Conv3-32
 Time
Unbearable
Level 5 Pain

High
Level 4
Pain

Medium
Level 3 Pain

Low
Level 2
Pain

No
Level 1
Pain

(a) (b)
Miscellaneous
 16+3
3 16 16 16 16 1
2
Convolution 3x3 Convolution 1x1 Dropout
0.1
Dropout
Max Pooling 2x2 Skip connection
Up Sampling 0.2
Dropout
Block copied 0.3
2x2

32 32 32+6
32
4

64 64 64+12
64
8

12 12 12
128+256
8 8 8

25 25
6 6
 Feature
Vector

Output
FC-512
Max-Pool
Output
Conv3-128
3er

FC-512
Lay

4er

Max-Pool
Lay

Max-Pool
Conv3-64
Conv3-128
3er

Conv3-64
2

Lay
er

Max-Pool
Lay

Max-Pool
Conv3-64
Conv3-32
Conv3-64
2er

Conv3-32
Lay

Max-Pool
Conv3-32
Conv3-32
Conv3-32
1er

Conv3-32
Lay

Input
Conv3-32
Conv3-32
1er
Lay
Softmax
FC
er3 er4
Lay Lay

FC
Max-
Pool
Conv
2
er
Max-
Lay
Pool
Conv

1
er
Lay
Input
 Filter
concatenation

3x3 convolutions 5x5 convolutions 1x1 convolutions

1x1 convolutions

1x1 convolutions 1x1 convolutions 3x3 max pooling

Previous layer
 Filter
concatenation

3x3 convolutions 5x5 convolutions 1x1 convolutions

1x1 convolutions

1x1 convolutions 1x1 convolutions 3x3 max pooling

Previous layer
 Input
1x11 conv
1x11 conv
Inception 1
Inception 2
Inception 2
Inception 2
1x7 conv
1x7 conv
FC
FC
Output
 Previous layer

1x3 conv,
1 padding 1x3 conv,
1 padding
1x7 conv,
1x3 conv, 3 padding
1 padding
1x5 conv,
2 padding
1x3 conv,
1 padding

Filter
concatenation
 Auxiliary Classifier

Softmax
Auxiliary Classifier

FC
Softmax Conv
FC Avg-Pool
FC Inception
Conv Inception Softmax
Avg-Pool Max-Pool FC
Inception FC
Inception Conv
Inception Avg-Pool
Inception
Inception
Max-Pool
Inception
Inception
Max-Pool
Conv
Max-Pool
Conv
Max-Pool
Conv
Input
 Filter Filter
concatenation concatenation

3x3 1x3 3x1

conv. conv. conv.

1x1 3x3 3x3 1x3 3x1 3x3

1x1
conv. conv. conv. conv. conv. conv.
1x1 conv.
1x1
conv.
1x1 conv.
1x1 1x1
Pool 1x1
conv. conv. Pool conv.
conv.

Previous layer Previous layer

(a) (b)
 Previous input Previous input

x x

F(x) Stacked layers F(x) Stacked layers x

identity
+
y=F(x)
y=F(x)+x

R1

R1 R2

R1

R1 R2 R3 R1 R2 R3
 Dense Block 1 Dense Block 2 Dense Block 3

Avg-Pool
Avg-Pool
Avg-Pool

Softmax
Conv
Conv
Conv
Input

FC
Transition layers
 hi+1

Filter
concatenation

hi+1 add add

Filter 3x3 3x3 3x3

identity
concatenation max conv avg

add add add add add add add add

3x3 3x3 5x5 3x3 3x3 3x3 3x3 5x5 7x7 5x5 3x3 7x7 3x3 5x5
identity identity
conv conv conv avg avg avg conv conv conv conv max conv avg conv

hi hi

... ...

hi-1 hi-1

(a) (b)
 Max(1,1,5,6) = 6

Y
1 1 2 4

5 6 7 8 6 8

3 2 1 0 Pooling performed 3 4
with a 2x2 kernel
and a stride of 2
1 2 3 4

X
Image
Representation
ML System Design / Infrastructure
 疼痛等级 治疗力度

脑电波头盔 APP 疼痛治疗仪

疼痛位置 治疗方案

使用者的治疗时长、治疗
方案、治疗反馈记录