[FREE PDF sample] Automated Deep Learning Using Neural Network Intelligence: Develop and Design PyTorch and TensorFlow Models 1st Edition Ivan Gridin ebooks
[FREE PDF sample] Automated Deep Learning Using Neural Network Intelligence: Develop and Design PyTorch and TensorFlow Models 1st Edition Ivan Gridin ebooks
com
https://ebookmeta.com/product/automated-deep-learning-using-
neural-network-intelligence-develop-and-design-pytorch-and-
tensorflow-models-1st-edition-ivan-gridin/
OR CLICK HERE
DOWLOAD NOW
https://ebookmeta.com/product/controlling-her-the-kinky-affair-bound-
and-taken-dark-romance-2-1st-edition-kat-t-scott/
ebookmeta.com
CSM AC Mathematical Methods Year 12 2017th Edition Michael
Evans
https://ebookmeta.com/product/csm-ac-mathematical-methods-
year-12-2017th-edition-michael-evans/
ebookmeta.com
https://ebookmeta.com/product/himachal-the-struggle-for-land-water-
and-forest-uttam-singh/
ebookmeta.com
https://ebookmeta.com/product/vlsi-physical-design-from-graph-
partitioning-to-timing-closure-2nd-edition-andrew-b-kahng/
ebookmeta.com
https://ebookmeta.com/product/engineering-mechanics-statics-and-
dynamics-3rd-edition-michael-plesha-2/
ebookmeta.com
https://ebookmeta.com/product/richard-hooker-the-architecture-of-
participation-1st-edition-paul-anthony-dominiak/
ebookmeta.com
Hannah Arendt Legal Theory and the Eichmann Trial 1st
Edition Peter Burdon
https://ebookmeta.com/product/hannah-arendt-legal-theory-and-the-
eichmann-trial-1st-edition-peter-burdon/
ebookmeta.com
Automated Deep
Learning Using
Neural Network
Intelligence
Develop and Design PyTorch and TensorFlow
Models Using Python
—
Ivan Gridin
Automated Deep Learning
Using Neural Network
Intelligence
Develop and Design PyTorch
and TensorFlow Models Using Python
Ivan Gridin
Automated Deep Learning Using Neural Network Intelligence: Develop and Design
PyTorch and TensorFlow Models Using Python
Ivan Gridin
Vilnius, Lithuania
v
Table of Contents
vi
Table of Contents
vii
Table of Contents
Index��������������������������������������������������������������������������������������������������������������������� 379
ix
About the Author
Ivan Gridin is a researcher, author, developer, and artificial
intelligence expert who has worked on distributive high-
load systems and implemented different machine learning
approaches in practice. One of the primary areas of his
research is the design and development of predictive time
series models. Ivan has fundamental math skills in random
process theory, time series analysis, machine learning,
reinforcement learning, neural architecture search, and
optimization. He has published books on genetic algorithms
and time series forecasting.
He is a loving husband and father and collector of old math books.
You can learn more about him on LinkedIn: https://www.linkedin.com/in/survex/.
xi
About the Technical Reviewer
Andre Ye is a deep learning researcher and writer
working toward making deep learning more accessible,
understandable, and responsible through technical
communication. He is also a cofounder at Critiq, a machine
learning platform facilitating greater efficiency in the peer-
review process. In his spare time, Andre enjoys keeping up
with current deep learning research, reading up on history
and philosophy, and playing the piano.
xiii
Introduction
Machine learning is a big part of our lives in today's world. We cannot even think of a
world without machine learning approaches at the moment, and it has already started
to take a huge part of our daily activities. Websites, mobile applications, self-driving cars,
home devices, and many things surrounding us use machine learning algorithms. The
dawn of computing power, especially the graphics processing unit, was accompanied
by the practical start of the deep learning implementation. Deep learning studies the
design of deep neural networks. This approach shows impressive efficiency and has
experienced explosive growth in recent years.
Not surprisingly, the number of tasks to be solved and the need for machine learning
specialists are constantly growing. At the same time, the number of routine actions that
developers and data scientists execute to solve machine learning problems is increasing.
Meanwhile, researchers developed special techniques to save time and automate the
most common machine learning tasks. These techniques were separated into the
special area called automated machine learning, or AutoML. This book focuses on the
automated deep learning (AutoDL) area, which studies the automation of deep learning
problems. AutoDL considers the issues of creating and designing optimal deep learning
models. This approach has been rapidly developed in recent years and, in some cases,
can completely automate the solution of typical tasks.
This book is about implementing AutoDL methods using Microsoft Neural Network
Intelligence (NNI). NNI is a Python toolkit that contains the most common and
advanced AutoDL methods: Hyperparameter Optimization (HPO), Neural Architecture
Search (NAS), and Model Compression. NNI supports the most popular deep learning
frameworks. This book covers the NNI implementation of various AutoDL techniques
using the PyTorch and TensorFlow frameworks.
Сhapter 1 focuses on automated deep learning basics and why we should put this
approach into practice. We will also install NNI and examine the main basic scenarios
for its use. We will learn how to run simple Hello World Experiments and interact with
NNI via the command line and WebUI.
In Сhapter 2, we will move on to the study of the most common AutoDL task –
Hyperparameter Optimization (HPO). We will learn what Hyperparameter Optimization
xv
Introduction
is, what hyperparameters are, and how to organize an NNI HPO experiment using
PyTorch and TensorFlow. We will also construct three kinds of research that will make
a historical journey to the origins of deep learning. The first one will help us determine
the best LeNet model hyperparameters for the MNIST problem. The second research
integrates a new dropout layer and rectified linear unit (ReLU) activation into the
original LeNet model. And the third one will show us how we can evolve the LeNet
model in AlexNet using simple HPO techniques.
In Сhapter 3, we will study NNI's main search algorithms (Tuners), which aim to
solve HPO tasks. Here, we will consider the practical application and the description of
the following algorithms: Evolution Tuner, Anneal Tuner, and SMBO Tuners.
Chapter 3 provides the creation of a custom Tuner and applies it to the classic Shallow
AutoML problem – building an optimal pipeline using Scikit methods.
In Chapter 4, we will begin to research Neural Architecture Search (NAS). NAS is an
approach that studies the creation and design of neural networks best suited to solve
a specific problem. This chapter covers Multi-trial NAS and its main principles. We'll
discuss the NNI Retiari framework, define Model Spaces and Model Mutators, and set
up experiments that construct optimal neural networks. Also, this chapter introduces
various exploration algorithms that explore Multi-trial NAS Model Space: Regularized
Evolution, TPE Strategy, and RL Strategy. Next, we will build LeNet-based and ResNet-
based Multi-trial NAS experiments to solve the CIFAR-10 problem.
In Chapter 5, we move on to One-shot NAS, one of the latest advances in
AutoDL. This chapter explains how to construct a Supernet, how to design cell-based
neural architectures, and perform Efficient Neural Architecture Search (ENAS) and
Differentiable Architecture Search (DARTS) One-shot NAS algorithms.
In Chapter 6, we will cover the important topic of model pruning. Model pruning
compresses neural network removing redundant weights or even layers. This technique
is crucial for lightweight devices when we need to save computing resources. This
chapter will examine basic One-shot and iterative pruning algorithms.
Chapter 7 will focus on practical recipes for using NNI to organize robust, extensive,
and big data experiments.
This book explores practical NNI applications of AutoDL methods and describes
their theory also. Therefore, this book can be helpful for data scientists who want to get
the idea that underlies various AutoDL techniques and algorithms.
This book requires intermediate deep learning understanding and TensorFlow or
PyTorch knowledge.
xvi
Introduction
xvii
CHAPTER 1
Introduction to Neural
Network Intelligence
There was a great burst of deep learning industry in the past few years. Deep learning
approaches have achieved outstanding results in computer vision, natural language
processing, robotics, time series forecasting, and optimal control theory. However,
there is no “silver bullet model” to solve all kinds of problems. Each problem and
dataset needs a specific model architecture to achieve suitable performance. Machine
learning models, especially deep learning models, have a lot of tunable parameters
that can drastically affect the model performance. Those are model design, training
method, model configuration hyperparameters, etc. The model optimization process
is performed for each application and even each dataset. Data scientists and machine
learning experts often spend a lot of time performing manual model optimization. This
activity can be frustrating because it takes too much time and is usually based on an
expert’s experience and quasi-random search.
However, recent results in automated machine learning and deep learning meta-
optimization make it possible to automate the optimizing process for a specific task.
It is also possible to create brand new model architecture from scratch without having
any experience solving similar problems in the past. The Neural Network Intelligence
(NNI) toolkit provides the latest state-of-the-art techniques to solve the most challenging
automated deep learning problems. We’ll start exploring the basic NNI features in this
chapter.
Poika odotti ääneti ja uhmailevana vain sitä, että Aljoša nyt toki
välttämättömästi hyökkäisi hänen kimppuunsa. Kun hän näki että
tämä nytkään ei aio sitä tehdä, niin hän vimmastui kuin peto: hän
syöksyi paikaltaan ja hyökkäsi Aljošan kimppuun, eikä tämä
ennättänyt liikahtaakaan, kun ilkeä poika taivuttaen päänsä alas ja
tarttuen molemmin käsin hänen vasempaan käteensä puraisi
kipeästi hänen keskisormeaan. Poika pureutui siihen hampaineen
eikä noin kymmeneen sekuntiin päästänyt sitä irti. Aljoša huudahti
kivusta ja nyki kaikin voimin sormeaan. Poika päästi sen viimein irti
ja juoksi pois entisen välimatkan päähän. Sormea hän oli purrut
kipeästi aivan kynnen kohdalta syvälle luuhun asti, ja siitä virtasi
verta. Aljoša otti taskustaan liinan ja kääri sen tiukasti haavoitetun
käden ympäri. Käärimiseen meni melkein kokonainen minuutti. Poika
seisoi kaiken aikaa ja odotti. Viimein Aljoša kohotti häneen
rauhallisen katseensa.
— No, hyvä, — sanoi hän, — näette, kuinka kipeästi olette minua
purrut, kai se jo riittää? Sanokaa nyt, mitä minä olen teille tehnyt?
4.
Hohlakovien luona
— Sain kyllä.
Mutta heti kun Lise näki raosta Aljošan sormen, hän avasi oven
selko selälleen.
— Minä tuon heti kaikki, Lise, älä vain huuda äläkä ole levoton.
Näetkö, kuinka lujana Aleksei Fjodorovitš kantaa onnettomuutensa.
Missä te olettekaan voinut niin kamalasti haavoittua, Aleksei
Fjodorovitš?
— Se jäi sinne.
— Minkä tähden?
— Te loukkaatte minua!
— Enhän minä voinut tietää, että hän tulee tänne sormi purtuna,
muutenhan tosiaankin olisin voinut tehdä sen tahallani. Enkelini,
äitikulta, te alatte puhua tavattoman teräväjärkisesti.
— No, riittää jo, Lise, kenties minä tosiaankin olin liian kärkäs
puhumaan hullusta pojasta ja sinä teit jo omat johtopäätöksesi. Heti
kun Katerina Ivanovna sai kuulla teidän tulleen, Aleksei Fjodorovitš,
hän syöksyi luokseni, hänellä on hirveä, hirveä halu nähdä teitä.
— Ah, äiti! Menkää sinne yksin, hän ei voi nyt heti lähteä, hän
kärsii kovin.
— Ah, Lise, tuo on vain leikkiä sinun puoleltasi, vaan mitäpä, jos
tosiaankin nukkuisit! — huudahti rouva Hohlakov.
— Minä en tiedä, millä olen… Minä jään vielä noin kolmeksi tai, jos
niin tahdotte, viideksi minuutiksi, — mutisi Aljoša.
Mullerrus vierashuoneessa