Deep leasi

Modee

Tatoduetion to feed fon oad newal netoles :

Deep feedfouad nett, alo ojten e alled feedBiusad
meual netwciks o mulblaye puceptqons CtLR).
"Peleepton means (t is a &tmple binaly claniie , sic k
talky melhple inputs , Parocetses them and Paoclarcag a
Singe atpat.
.Tu qoal , a feed ooroad netak is to approzi oate
Sonre Puenchion 4*.
Exampla: Paf a clasiyie y- f°(z) mapa an inpat x to a
y=f(1)

" Thee uodals ae called feedfowd becaute ingosnahon

eyaluated forom ,
tlu fanhon bing
olif's f.
through tu tnteme diala conpetatng eeed b
and fhally to tbu output g
" feedforward neulal netb ae called netwoks bec aus
tlay ait typally oprantd by caposig tagetu
dif unt fuunctn . clsected
" nocdel is au ojatedc h a acqelie
ttu tunchne ale Conpoktd togeta
9aph de,eslbing kow funchns f,p)and
Erampl: we nu'glt hau the
)=F20[pl:))
9conneched in a chain to oom
Thee chain Sthuctues Commnly wol
gtaec tus neeat al natks.
-Th tlis case, f) called t e Baat lay
tte
netwolk>fo
)fh ia caeted tla fe cond laye and so n.
tlu oueal length Lle alain
tlat lle n
name
"cleep leatning" ise.
" Jhe fral layet o a feocl forusalol netle is called tue
output layu.
" During neutal netwck taining , we diue fluy to atk
J*) Ta training data prodles
appoonimale Paanpls (y evaluatd at aliy uund
training pots . tach enanpe x is by a

dabd yNf().
dlata oeu not Shou lu olered output f
Taaintng
lach tluu layes, tlee doyes e Calleo ddon layet.

" Netwdks al calledl neesal becaye tleey dootely

iospired by menesiinet.
" Gna way to cendestand feedßomasd netatky ts to beg'
Citte clireas modleli Qnd congidee hous to Oleone

ttuis litahony.
and
 6Aadient - Baed learning
Latooouch'an hy o p l e
a nual metok es not much
Deigntg and toainig
di'steent From taaning any oller machins
nodal

wuith gnad'ent deseaad. betusen tlu lineal neodsly , and

eena
* 7e lalget oit netal nttt.
ineaiyBa
helal netasoky t teu non- becOne non -conveu
fo
. Non-lineazity Causes los Penshont
netlki ale etually t rainsd ung cost faunehon
" Neutal ntodk
ophnis es tat d's ta
diu
qrad'eat -baud lreal equahon
vale ,etate than tle
to a vey low
no greston
folyes eesed to train 9wanleed globul
optinirahon algorithms eeth
convx o Sum).
Covetgen (egs logite rreson
abon cpnueig es gtating from ony irsthial
Convx optaás
Patarnets. applid los
applid too nan-Convn
. Stochae qaint dsent
Con°eigencu quarnte, ane
funchn does not haue Jeuch Porameles.
to tlu alees g tlee inihal
ib Benhue
Conveu' 'non Conwex

pa

foroad nelal netwolka , ti hali're all sergss to

. for feed
Smal manolom alees.
inibaliud o 2uo o Srall pos'htag
"The biases may be
Vales.
 tlu
desund tle cast funehon.
grodas
". inea regrnon and UH moclaly can
train
gmadiant descent ftfol latget training Se.
conpag nusal netdk tluan
thon lines
iabo odals, but can
tiu be done tbicinily and
Conput Lu gmadtieat faty
tlus back -propagam
algotan.
To7o appy gadiat - baud
1. The cot futon
The forrm g tlu outpat cunits

Cot Panctton
" o apy gadint- baud leatnig Coxt tnehon mt be
chonn , and additionally wlat output Bhold be obBoinso
alto mett be cbuotn
The to tal cost fuunchon b wed o train a nusal nett
maximum
" MoBt uodeun meutal netk
meutal netetka
a tratnad uing
le., tlu cost functn is Jirply td negahu
Wkel'hood
Ubel'hood (Cos enoopy betuen lu toraing
olata
and tleu nualel isto bebon ).
mt be larg e, and
" Jhe gaadient g tle cost anetion nut
ti leaining algothm
abl to precdit cnogh (becone l'
atutate y Pht) encletnian llets
. Cactnt tlat
beeceuse
objectiaSmall.
Tey malce tl grad'ent becous y
 Betivahon funchony wd to poroclce tlu outp 3tlu
hidden nits tle owt put cenit Satuat1
. The negaliue log- Lirelibood katp to avors tlus robam
modeli
many Conditonal Dstoibutors ith raxinum

ikelihood

matimm
taained
" Most modn neural netwoks ae
l'kelihood.
meany tat tlu cost tunct'n is Bopy lu negahu
This
tog-tkelthood fquivalently descu'bed as tlu cos- entopby
and the aadel olitnbet'n
btuven be training data
Tu cost funchn ginby
Jle) = - Eay pate logpmadel (y/z)
ceang from
lu cost funchona
Gpeecpe Bam

abou fj uahon byptally yilods Roa

na
The supain g tluu on
te
teu odel parameles and
depnd
teims that olo not
d.
may be llscaldde
Cranph lyla) - N[H: fa;e) ,2) , un
tu mean Squalld ero tost,

J(e) Exywpdata |l 4--f(%;e) | It cont.

a fact g and a teem that olos not
" up to Sooling u
Te olis catoled congtant a baleel on
depen d on O.
tle Gausan listibub on,ieh in
tls
Cape e chose nol b parametne .
" Poevtously, we Saw that tlu fquvalonca btween
maímum ikelihood estnaton tth an output dista but
and nininisaton % e an Squastd ero hold! tot a
ieae neoalel
fqui wal1nu hol ody megasdles tlu flx ;o) yd
o pore cdiet Clu mean ttu Gaunstan.
Advantage te cost funeKon from maimum lceliload
cot funchaa
is that it noes tu bul dn deeiging
oalel pyly'a
PCHl) autrmafeoly
K8 each oalel. Specßy'ng a ndet
deteinines a cost funhon lay Plyl).
tanu toughout neuslal netek del'gn
One recerotn tlu cot furton must be'
ta tat ta q3adiend z
to Suue as agood gd
latge and przdietabk enougl

Lealniny tonal
Cordi &tasties
1.B)
a full Probability clitibuhon pCylx;o)
we osten ant to leosn jest
poeoictl fl;o) tat
Cranpl: we may hae a
predict tlu mean
cuish t netwotk, we
&etbiccenty poveyul neetal
being able to epreent
netal netdk as ceelth
Can tlisk tlu fanchony,
ee'ale clas
any funehon f foom featule! Bah as
ltatthd ony bey
lis beiog
anl ratlu tlan dy hauig
Conhaity tom.
Speit'a palametoia
 tis twiwt u dott
Can jat a tnhin
a tunetonal gatteu than j geal numbas
fonon taotot
fanehiod to
ppg
Caamplei we can lelyn tlu cost funchoal to haus
miimum lie on ttu lun Hat maps te
iTs
Srpected yalue y y gan
phhusahin pooban with mept t a funton
Jolving an Caloukas % atiahont
caued
Nguies a matamatal toot Vatahng is
Calculus
" Ous at reseelt oie d eing
pasbim
that Sotuêng tle ophaisahon

yidds lamplesofoom
couldl raio inginitey
distibtion,maymninmising tu
ttu tru olata -genag funchon glusy funhi that )

Squaied trao cast K

an
mean each alu
neana y fot gtatshes . A Second
Frediet tu ias l%ent
Dibeunt coit fanctns yaráhmp &tat
calculuy
qyelt deiusd eeng
Nalee
prediets tlee median
a tunchon that funchan may be oleCLbed
each ,fo
So long as feueh a
y d
we o firu'e oue Tis coXt
funchont
tu family g mean abolute erao.
mmly Called
funcion is comn
and mea abtolute erroi ogten leac D.
. Mean Swasud gadient - bated phriaata
eud uith
to por retats wlan