AMRUTA V. PATIL. Dr. SNEHAL R. RATHI NEHA S. PATIL GIRISH J. NAVALE SPPU Final Year (B.E.) Degree Course In ARTIFICIAL INTELLIGENCE AND DATA SCIENCE Semester - VIII & , a @ wwwopragationline.com #4 [email protected] G_ wwwfacebook.com/niralibooks PRAKASHAN @nirali.prakashanSYLLABUS 06 Hours, I Foundations of Deep Learning ‘What is machine learning and deep learning? History of deep learning, Advantage and challenges of deep learning, teaming representations from data , Understanding how deep learning works in three figures(input, hidden layer, output), ‘Common Architectural Principles of Deep Network, Architecture Design, Applications of Deep learning, Hyperparemeters Learning Rate, Regularization, Momentum, Sparsity, Hidden Units, cost functions, error back propagation, Gradient-Based Learning, Implementing Gradient Descent, vanishing and Exploding gradient descent, Optimization algorthm(SGD, ‘AdaGred, RMSProp, adam) 06 Hours Unit 1: Deep Neural Networks (ONNs) Introduction to Neural Networks : The Biological Neuron, The Perceptror(AND.OR.NOT.XOR), Deep forward network, Multilayer Feed-Fonward Networks, Training Neural Networks: Backpropagation and Forward propagation Activation Functions: Linear Sigmoid, Tannh, Hard Tanh, Softmax Rectified Linear, Loss Functions: Loss Function Notation , Loss Functions for Regression , Loss Functions for Classification, Loss Functions for Reconstruction. 06 Hours: Unit II: Convolution Neural Network (CNN) Introduction, CNN architecture overview, The Basic Structure of 2 Convolutional Network- Padding, Strides, Typical Settings, the ReLU layer, Pooling, Fully Connected Layers, The Interleaving between Layers, Local Response Normalization, Training a Convolutional Network. 06 Hours Unit 1V: Recurrent Neural Network (RNN) Recurrent and Recursive Nets: Unfolding Computational Graphs, Recurrent Neural Networks, Bidirectional RNNs, Deep Recurrent Networks, Recursive Neural Networks, The for Multiple Time Scales, Encoder-Decoder Sequence-to-Sequence Architecture Challenge of Long-Term Dependencies, Echo State Networks, Leaky Units and Other Stratec ‘The Long Short-Term Memory and Other Gated RNNs, Optimization for Long-Term Dependencies, Explicit Memory, Practical Methodology: Performance Metrics, Default Baseline Models, Determining Whether to Gather More Data, Selecting Hyper parameters. 08 Hours Unit V: Deep Generative Models Introduction to deep generative model, Boltzmann Machine, Deep Belief Networks, Generative adversarial network (GAN) discriminator network, generator network, types of GAN, Applications of GAN networks. 06 Hours Unit Vi Reinforcement Learning Introduction of deep reinforcement learning, Markov Decision Process, basic framework of reinforcement learning, challenges of reinforcement learning, Dynamic programming algorithms for reinforcement learning, Q Learning and Deep Q-Networks, Deep Q recurrent networks, Simple reinforcement learning for Tic-Tac-Toe.aa | 6 6a 6s 66 68 6s 620 620 612 612 616 620 Papa p2P2 ‘The Popular Applications of ML (one) FOUNDATIONS OF DEEP LEARNING WHAT IS MACHINE LEARNING AND DEEP LEARNING? «Machine Learning ond Deep Learning are the two ma Concepts of Data Science and the subsets of Arifial Ineeligence. Most of the people think the machine jearning, deep teaming and as well as artificial intelligence as the same buzzwords. But in actuality, all these terme are cifferent but related to each other. + In this module. we will learn how machine learring ie different from deep learning? But before learning the differences, lets frst have a brief introduction of machine learning and deep learning, Machine Deep learning Fig. 1.1 1_Introduction of Machine Learning Machine learning is part of artificial intelligence and growing technology that enables machines to learn from past data and perform a given task automatically Machine Leaning allows the computers to lear from. the experiences by its own, use statistical methods to improve the performance and predict the output without being explicitly programmed. Email spam filtering Product recommendations Online fraud detection, ete ‘Some Useful ML Algorithms + Decision tee algorithm Naive Bayes + Random Forest = Kemears clustering + KNN algorithm + Apri algorithm, et. How Does Machine Learning Work? ‘The working of machine learning models can be understood by the example of identifying the image of 1, To identify this, the ML model takes a cat oF dot images of both cat and dog as input, extract different features of images such as shape, height rose, eyes, etc, applies the classification algorithm, and predict the output, Consider the below Fig. 12 - by — SS fication output Feature Cl Input Fig. 1.2 1.1.2 Introduction of Deep Learning “= Deep Leaming is the subset of machine learning or can be seid as a special kind of machine learning. It works technically in the same way as machine learning does, but with different capabilities and approaches. ‘+ Its inspired by the functionality of human brain ces, ‘which are called neurons, and leads to the concept of afcial neural networks, Its also called a deep neural network or deep neural learning + In deep learning, models use different layers to learn and discover insights from the data. ‘= Some popular applications of deep learning are selt= driving cars, language translation, natural language processing, etc a)Kenko Fuku‘ource ofthe pretiem tad out to be conan | | Acta fuetons A numberof acta condened tum reducing the ouput | fang in 9 somentat chat ston Tis produced | , ge ares of put mapped ove an exveely sr | inthse a of ipa» ge dange wil be rue | {0.2m change nthe ouput sung ina vanishing | gradient Two solutions 2d to sole thi problem | Ad then the vaning sftare was alowed to At the end ofthe traning one neuton in the highest layer was found to respond strongly a the images ‘ts Andrew Na the project's founder sid "We lsc found a neuron that responded very stongh 80 Unsypenisd learning remains significant goa the fla of dep learn, * fheuce in aula Gah was ans Oe, Sn te a oe ter sing. reduce tk. deme and, ang | LDATA thot dhe fi | 3:1 introduction of Representation Learning the 2 (conor ligence Une) | + Reptsetton lange « 1.22. Advantage and Challenges of Deep Learning Following are the advantages of Deep Learn estes outcome to be exacted ahead of tine Thi olds Consuming methine erring techniques ktonstesy eae + Robustness speach pple to many teen ap + Thee arin ehcp 123 Disadvantages of Deep tearing [IS LEARNING REPRESENTATIONS FROM | Brovidng a better understanding ofthe datas evel ‘asc. Machine leaming tasks such a elsifeation Frequently demand input that is mathematical and| o@UNDERSTANDING HOW DEEP LEARNING | | | WorKS IN-THE FIGURES (INPUT, HIDDEN| || saverH Bsc herelere ae ng Works? 1+ Samwames he nag calle partes of es Objective Function Te predictors a big tan thereto andthe te target + comptes» ditance score what we wanted the eer to cuput by capturing how wel the et tas dove on thi specie example the base esp raring is 0 ubize hs score a8 9 ‘al adjust he ale of the wigs, Feedback he object fncton she oof he es hncon the netvor. Tis sao calle the eects frcton een ot | ae * ee te set Teste ‘te woo | eee eee Step 4 Theta mapped ten highimersion, ei here the aeccn Gouna ago types [atone tothe tance betwen these points n our tears representation space completly bypassing the erp Computation othe re representation | * Thpicay eral fncons are cated by hand rather thon leaned fam data - inthe case ofan SVM ony the separation hyperplane ie learned At the tne te | were developed SiMe ethiited state-of thee | Peromsnce on simple cstcaton problems and vere one af the few machine leaning methods buco by estesve theory” and. amenable to. secur mathemati enaisis mating them well understood and easly interpretable 1 Sif bec eer Bp inb e ang wate) having sar proper I ay be one of the best ‘ongatde deep teaming, i one of the moe Cannon weheche hae eet x rove Pa oct is hon tee om te ee ‘eee on ar ret es ‘hole ht ou tan nyt sc pec rotons Ceol grow Seema et we "teraction of ep lyre nee eters on Gad Rear Tis aie clasifes deep leaning architectures ito supervised and unsupervised leaning and inttodces several popular dep leaning architectures (© Comlitional Neural Networks (CNN) (@ Recurent Neural Neborks (NN) {Log Short-Term Memenry STM)152 Architecture in Unsperised Deep Leaming 1. Sat Orgniced Maps Kovonen in 1982 ans was pop Kotonen map + SOM 1 on unsipeised naval network that cts clstes of the input dats set by rducng we imenionalty ofthe input SOMS tay tom te itera ein eval nett auite saw nye + Set-Organzed Map S04 was inverted by Tete noi as the | thee i no concept of caelting enor and ban ample Aopleatins > Dnensnsy reduc > Radian grade rest > Alister vation Stein 958 Te les continues using backward Propagation, For ths reason. autoencoden are assed asf supeised igortens Example Applications > Dimersionsiy reduction > Datainterpolaion phase, ROMS ee using ® = oe re mspecve hidden and wile Pe ‘bias helps in crate speach + Ae ii ne ae i * Because in an RBM the reconstructed input is abways ‘ferent from te orignal input they are aso known a ‘so, because of the bultin randomness. the same Dredicion est in diferent outputs. In fac ths the mos significant diference fom an autoencoder which is deterministic mode [Example Applications: > Dimensionaiy reduction > ata compression / decompression > cooboratve ering‘changes in the Imatket that they dete *ae biases of 4 neva network, clster cent tisteing thee =) Sonepat aa teri | hy ave ied 0) the Or may rm tere a, > Number of Each 461. Difference between Model Parameter and __Model Hyperparameter Model Parameters Moat parameters ar contortion oles ht se inteal © the model anda mode ur hem on ts | For eae W Weighs or Coins of dependent {arb nthe Hiner regression med or Weahisce oetfientsofindecendent varaies a SM, reg | 2162 categories of iyperparameters_ ei ect ca | Hyperparameterfor Optimization [ v0 J + te process of selecting the bast hyperparametr ue isknown as hperparrete tuin,and the ona, proces ial known as hyptpararater tines peace —> [esters ee Nel ang O_ mam Fig.139 hypeneansi (9 Number fpochs The tumber of esc Hyperparameter for Specific Models Hrperpramaters that 2 yperpsrameters for Thaseare given below ANumber of Hidden Units * iden untae par of neural eta the components conpesng — bette put and ouput un suet of he Specie mee, ks. wich reer vo the lays of processors ‘A desiabe easing rat ow enough forthe network to Converge on soma ‘Ser ean tes necetate mae ning epochs ofthe fewer changes. On the ot larger earings est in fa tr charges Moreover, large lamin fates fequenty rest in suboptial fal et of weights ‘An anaical method cannat be used to caelte the ‘wight of a neral network Instead the weights mn be discovered using stochastic gradent denen on {empirical optimization approach 4m simpler tems. the stochastic gradient. descent algorithm is sed totam deep leaning rate evra networks pon te a wih Hat eaten fr eto | Ere eee el ny ath Cat Oot om wf cette og itier | if mila narnrad)« ee sf [+ Rode hermione | eee — Potro ease cert rea W nae || eee eran fneton or atin rane | impr minty a9 fof the data / | | | | eterotentatt xy bss 560, n=leny) eh _Int_ (et r-0.01, max iterst000, ‘ predict target vl by tong taking | betch sies32; tote) ‘ oling dot radi of dependert on theta vue 79.125: tc rade optnntion put |) Aearing rate of the $60 Optimizer Y.Rred = np dot, ef theta) X: rpranoncanieo 5) # rete careening trget wu bys rendom Moros 2.3,4 509) + rprardomrenst00)* 04 "odes SeD(r=00, max ieri00, bch sn, tote3) ng rect mathe frm mode {ed model prix) ate descent since i ues ony one cage ‘dete porters Memory tfficincy: Since SGD updates the ‘erarates fr each taining ample one ats tne "ememonyefcet and can handle ange dane ‘vet inte memaret ome features ar sparse or expen Bo sarge een | rer te. Ts wit ade theo weigh {othe new weight wich may lead FoutoAnone of PBA, Prthon leeatenethod Ee UStropindex ben 3, nh, opey Inde layers tex} eight -< (pa +1208)" Iodellyer indexes 4208)" a 184 Adam (lh / (arts ‘fhe latest stateofhe-art optimization agtnns borg | lik by many practioners of machine leg. The fx | ADAMa cw rere com Inde: gon sme some! nd | carmen est ola te 08)} aes | .edelimeines bars (oe sats [Sooner “ ee, ewvemepecn ge peck 1) e-pwesonms, 8