M ONTRÉAL .

AI ACADEMY: A RTIFICIAL I NTELLIGENCE 101

F IRST W ORLD -C LASS OVERVIEW OF AI FOR A LL
VIP AI 101 C HEAT S HEET

A P REPRINT

Vincent Boucher∗
MONTRÉAL.AI
Montreal, Quebec, Canada
[email protected]

March 2, 2020

A BSTRACT
For the purpose of entrusting all sentient beings with powerful AI tools to learn, deploy and scale AI
in order to enhance their prosperity, to settle planetary-scale problems and to inspire those who, with
AI, will shape the 21st Century, MONTRÉAL.AI introduces this VIP AI 101 CheatSheet for All.

*MONTRÉAL.AI is preparing a global network of education centers.

**ALL OF EDUCATION, FOR ALL. MONTRÉAL.AI is developing a teacher (Saraswati AI) and
an agent learning to orchestrate synergies amongst academic disciplines (Polymatheia AI).
Curated Open-Source Codes and Science: http://www.academy.montreal.ai/.

Keywords AI-First · Artificial Intelligence · Deep Learning · Reinforcement Learning · Symbolic AI

1 AI-First
TODAY’S ARTIFICIAL INTELLIGENCE IS POWERFUL AND ACCESSIBLE TO ALL. AI is capable of transform-
ing industries and opens up a world of new possibilities. What’s important is what you do with AI and how you
embrace it. To pioneer AI-First innovations advantages: start by exploring how to apply AI in ways never thought of.
The Emerging Rules of the AI-First Era: Search and Learning.
"Search and learning are general purpose methods that continue to scale with increased computation, even as the
available computation becomes very great." — Richard Sutton in The Bitter Lesson
The Best Way Forward For AI2 .
"... so far as I’m concerned, system 1 certainly knows language, understands language... system 2... it does involve
certain manipulation of symbols... Gary Marcus ... Gary proposes something that seems very natural... a hybrid
architecture... I’m influenced by him... if you look introspectively at the way the mind works... you’d get to that
distinction between implicit and explicit... explicit looks like symbols." — Nobel Laureate Danny Kahneman at
AAAI-20 Fireside Chat with Daniel Kahneman https://vimeo.com/390814190
In The Next Decade in AI 3 , Gary Marcus proposes a hybrid, knowledge-driven, reasoning-based approach, centered
around cognitive models, that could provide the substrate for a richer, more robust AI than is currently possible.
∗
Founding Chairman at MONTRÉAL.AI http://www.montreal.ai and QUÉBEC.AI http://www.quebec.ai.
2
https://montrealartificialintelligence.com/aidebate/
3
https://arxiv.org/abs/2002.06177v3
 A PREPRINT - M ARCH 2, 2020

2 Getting Started
Tinker with neural networks in the browser with TensorFlow Playground http://playground.tensorflow.org/.

• Deep Learning Drizzle https://deep-learning-drizzle.github.io.

• Papers With Code (Learn Python 3 in Y minutes4 ) https://paperswithcode.com/state-of-the-art.
• Google Dataset Search (Blog5 ) https://datasetsearch.research.google.com.
"Dataset Search has indexed almost 25 million of these datasets, giving you a single place to search for datasets and
find links to where the data is." — Natasha Noy
The Measure of Intelligence (Abstraction and Reasoning Corpus6 ) https://arxiv.org/abs/1911.01547.
v Growing Neural Cellular Automata, Mordvintsev et al. https://distill.pub/2020/growing-ca/.

2.1 In the Cloud

Colab 7 . Practice Immediately 8 . Labs9 : Introduction to Deep Learning (MIT 6.S191)

• Free GPU compute via Colab https://colab.research.google.com/notebooks/welcome.ipynb.
• Colab can open notebooks directly from GitHub by simply replacing "http://github.com" with
"http://colab.research.google.com/github/ " in the notebook URL.
• Colab Pro https://colab.research.google.com/signup.

2.2 On a Local Machine

JupyterLab is an interactive development environment for working with notebooks, code and data 10 .
• Install Anaconda https://www.anaconda.com/download/ and launch ‘Anaconda Navigator’
• Update Jupyterlab and launch the application. Under Notebook, click on ‘Python 3’
"If we truly reach AI, it will let us know." — Garry Kasparov

3 Deep Learning
After the Historical AI Debate11 : "Yoshua Bengio and Gary Marcus on the Best Way Forward for AI" https://
montrealartificialintelligence.com/aidebate/, there have been clarifications on the term "deep learning"12 .
"Deep learning is inspired by neural networks of the brain to build learning machines which discover rich and useful
internal representations, computed as a composition of learned features and functions." — Yoshua Bengio
"DL is constructing networks of parameterized functional modules and training them from examples using
gradient-based optimization." — Yann LeCun
"... replace symbols by vectors and logic by continuous (or differentiable) functions." — Yann LeCun
Deep learning allows computational models that are composed of multiple processing layers to learn REPRESEN-
TATIONS of (raw) data with multiple levels of abstraction[2]. At a high-level, neural networks are either encoders,
decoders, or a combination of both13 . Introductory course http://introtodeeplearning.com. See also Table 1.
4
https://learnxinyminutes.com/docs/python3/
5
https://blog.google/products/search/discovering-millions-datasets-web/
6
https://github.com/fchollet/ARC
7
https://medium.com/tensorflow/colab-an-easy-way-to-learn-and-use-tensorflow-d74d1686e309
8
https://colab.research.google.com/github/madewithml/practicalAI/
9
https://colab.research.google.com/github/aamini/introtodeeplearning_labs
10
https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906
11
https://www.zdnet.com/article/devils-in-the-details-in-bengio-marcus-ai-debate/
12
https://www.zdnet.com/article/whats-in-a-name-the-deep-learning-debate/
13
https://github.com/lexfridman/mit-deep-learning

2
 A PREPRINT - M ARCH 2, 2020

Table 1: Types of Learning, by Alex Graves at NeurIPS 2018

Name With Teacher Without Teacher
Active Reinforcement Learning / Active Learning Intrinsic Motivation / Exploration
Passive Supervised Learning Unsupervised Learning

Figure 1: Multilayer perceptron (MLP).

Deep learning (distributed representations + composition) is a general-purpose learning procedure.

"When you first study a field, it seems like you have to memorize a zillion things. You don’t. What you need is to identify
the 3-5 core principles that govern the field. The million things you thought you had to memorize are various
combinations of the core principles." — J. Reed

"1. Multiply things together

2. Add them up
3. Replaces negatives with zeros
4. Return to step 1, a hundred times."
— Jeremy Howard

v Linear Algebra. Prof. Gilbert Strang14 .

v Dive into Deep Learning http://d2l.ai.
v Minicourse in Deep Learning with PyTorch15 .
v Introduction to Artificial Intelligence, Gilles Louppe16 .
v Deep Learning. The full deck of (600+) slides, Gilles Louppe17 .
v These Lyrics Do Not Exist https://theselyricsdonotexist.com.
v Backward Feature Correction: How Deep Learning Performs Deep Learning18 .
v A Selective Overview of Deep Learning https://arxiv.org/abs/1904.05526.
v The Missing Semester of Your CS Education https://missing.csail.mit.edu.
v fastai: A Layered API for Deep Learning https://arxiv.org/abs/2002.04688.
v Anatomy of Matplotlib https://github.com/matplotlib/AnatomyOfMatplotlib.
v Data project checklist https://www.fast.ai/2020/01/07/data-questionnaire/.
v Using Nucleus and TensorFlow for DNA Sequencing Error Correction, Colab Notebook19 .
14
https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/
15
https://github.com/Atcold/pytorch-Deep-Learning-Minicourse
16
https://glouppe.github.io/info8006-introduction-to-ai/pdf/lec-all.pdf
17
https://github.com/glouppe/info8010-deep-learning/raw/v2-info8010-2019/pdf/lec-all.pdf
18
https://arxiv.org/abs/2001.04413
19
https://colab.research.google.com/github/google/nucleus/blob/master/nucleus/examples/dna_
sequencing_error_correction.ipynb

3
 A PREPRINT - M ARCH 2, 2020

v PoseNet Sketchbook https://googlecreativelab.github.io/posenet-sketchbook/.

v Removing people from complex backgrounds in real time using TensorFlow.js in the web browser20 .
v A Recipe for Training Neural Networks https://karpathy.github.io/2019/04/25/recipe/.
v TensorFlow Datasets: load a variety of public datasets into TensorFlow programs (Blog21 | Colab22 ).
v The Markov-Chain Monte Carlo Interactive Gallery https://chi-feng.github.io/mcmc-demo/.
v NeurIPS 2019 Implementations https://paperswithcode.com/conference/neurips-2019-12.
v Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning23 .
v How to Choose Your First AI Project https://hbr.org/2019/02/how-to-choose-your-first-ai-project.
v Blog | MIT 6.S191 https://medium.com/tensorflow/mit-introduction-to-deep-learning-4a6f8dde1f0c.

3.1 Universal Approximation Theorem

The universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite
number of neurons can solve any given problem to arbitrarily close accuracy as long as you add enough parameters.
Neural Networks + Gradient Descent + GPU24 :

• Infinitely flexible function: Neural Network (multiple hidden layers: Deep Learning)25 .
• All-purpose parameter fitting: Backpropagation2627 . Backpropagation is the key algorithm that makes training
deep models computationally tractable and highly efficient28 . The backpropagation procedure is nothing more
than a practical application of the chain rule for derivatives.

Figure 2: All-purpose parameter fitting: Backpropagation.

• Fast and scalable: GPU.

"You have relatively simple processing elements that are very loosely models of neurons. They have connections coming
in, each connection has a weight on it, and that weight can be changed through learning." — Geoffrey Hinton

"Deep learning has fully solved the curse of dimensionality. It vanished like an RNN gradient!" — Ilya Sutskever

When a choice must be made, just feed the (raw) data to a deep neural network (Universal function approximators).
20
https://github.com/jasonmayes/Real-Time-Person-Removal
21
https://medium.com/tensorflow/introducing-tensorflow-datasets-c7f01f7e19f3
22
https://colab.research.google.com/github/tensorflow/datasets/blob/master/docs/overview.ipynb
23
https://drive.google.com/file/d/1sJvLQwxMyu89t2z4Zf9tD7O7efnbIUyB/view
24
http://wiki.fast.ai/index.php/Lesson_1_Notes
25
http://neuralnetworksanddeeplearning.com/chap4.html
26
https://github.com/DebPanigrahi/Machine-Learning/blob/master/back_prop.ipynb
27
https://www.jeremyjordan.me/neural-networks-training/
28
https://colah.github.io/posts/2015-08-Backprop/

4
 A PREPRINT - M ARCH 2, 2020

3.2 Convolution Neural Networks (Useful for Images | Space)

Richer innate priors : innateness that enables learning.

A significant percentage of Deep Learning breakthroughs comes from reusable constructs and parameters sharing. The
deep convolutional network is a construct that reuses weights in multiple locations (parameters sharing in space)29 .

"Virtually all modern observers would concede that genes and experience work together; it is “nature and nurture”,
not “nature versus nurture”. No nativist, for instance, would doubt that we are also born with specific biological
machinery that allows us to learn. Chomsky’s Language Acquisition Device should be viewed precisely as an innate
learning mechanism, and nativists such as Pinker, Peter Marler (Marler, 2004) and myself (Marcus, 2004) have
frequently argued for a view in which a significant part of a creature’s innate armamentarium consists not of specific
knowledge but of learning mechanisms, a form of innateness that enables learning." — Gary Marcus, Innateness,
AlphaZero, and Artificial Intelligence30

The deep convolutional network, inspired by Hubel and Wiesel’s seminal work on early visual cortex, uses hierarchical
layers of tiled convolutional filters to mimic the effects of receptive fields, thereby exploiting the local spatial correlations
present in images[1]. See Figure 4. Demo https://ml4a.github.io/demos/convolution/.

0 1 1 1×1 0×0 0×1 0

0 0 1 1×0 1×1 0×0 0 1 4 3 4 1
0 0 0 1×1 1×0 1×1 0 1 0 1 1 2 4 3 3
0 0 0 1 1 0 0 ∗ 0 1 0 = 1 2 3 4 1
0 0 1 1 0 0 0 1 0 1 1 3 3 1 1
0 1 1 0 0 0 0 3 3 1 1 0
1 1 0 0 0 0 0
I K I∗K

Figure 3: 2D Convolution. Source: Cambridge Coding Academy

A ConvNet is made up of Layers. Every Layer has a simple API: It transforms an input 3D volume to an output 3D
volume with some differentiable function that may or may not have parameters31 . Reading32 .
In images, local combinations of edges form motifs, motifs assemble into parts, and parts form objects3334 .
Representation learning : the language of neural networks. The visual vocabulary of a convolutional neural network
seems to emerge from low level features such as edges and orientations, and builds up textures, patterns and composites,
. . . and builds up even further into complete objects. This relates to Wittgenstein’s "language-game" in Philosophical
Investigations35 , where a functional language emerge from simple tasks before defining a vocabulary36 .

"DL is essentially a new style of programming – "differentiable programming" – and the field is trying to work out the
reusable constructs in this style. We have some: convolution, pooling, LSTM, GAN, VAE, memory units, routing units,
etc." — Thomas G. Dietterich

v CS231N : Convolutional Neural Networks for Visual Recognition37 .

v Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches. Yang et al.38 .
v TensorSpace (https://tensorspace.org) offers interactive 3D visualizations of LeNet, AlexNet and Inceptionv3.
29
https://twitter.com/iamtrask/status/949439556499230720
30
https://arxiv.org/abs/1801.05667
31
http://cs231n.github.io/convolutional-networks/
32
https://ml4a.github.io/ml4a/convnets/
33
http://yosinski.com/deepvis
34
https://distill.pub/2017/feature-visualization/
35
https://en.wikipedia.org/wiki/Philosophical_Investigations
36
https://media.neurips.cc/Conferences/NIPS2018/Slides/Deep_Unsupervised_Learning.pdf
37
https://www.youtube.com/playlist?list=PLzUTmXVwsnXod6WNdg57Yc3zFx_f-RYsq
38
https://arxiv.org/abs/2001.02890

5
 A PREPRINT - M ARCH 2, 2020

Figure 4: Architecture of LeNet-5, a Convolutional Neural Network. LeCun et al., 1998

3.3 Recurrent Neural Networks (Useful for Sequences | Time)

Recurrent neural networks are networks with loops in them, allowing information to persist39 . RNNs process an input
sequence one element at a time, maintaining in their hidden units a ‘state vector’ that implicitly contains information
about the history of all the past elements of the sequence[2]. For sequential inputs. See Figure 5.

ht h0 h1 h2 h3 ht

A = A A A A ... A

xt x0 x1 x2 x3 xt

Figure 5: RNN Layers Reuse Weights for Multiple Timesteps.

Figure 6: Google Smart Reply System is built on a pair of recurrent neural networks. Diagram by Chris Olah

"I feel like a significant percentage of Deep Learning breakthroughs ask the question “how can I reuse weights in
multiple places?” – Recurrent (LSTM) layers reuse for multiple timesteps – Convolutional layers reuse in multiple
locations. – Capsules reuse across orientation." — Andrew Trask
v CS224N : Natural Language Processing with Deep Learning40 .
v Long Short-Term-Memory (LSTM), Sepp Hochreiter and Jürgen Schmidhuber41 .
39
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
40
https://www.youtube.com/playlist?list=PLU40WL8Ol94IJzQtileLTqGZuXtGlLMP_
41
https://www.bioinf.jku.at/publications/older/2604.pdf

6
 A PREPRINT - M ARCH 2, 2020

v The Unreasonable Effectiveness of Recurrent Neural Networks, blog (2015) by Andrej Karpathy42 .
v Understanding LSTM Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/.
v Can Neural Networks Remember? Slides by Vishal Gupta: http://vishalgupta.me/deck/char_lstms/.

3.4 Transformers

Figure 7: Attention Is All You Need. Vaswani et al., 2017 : https://arxiv.org/abs/1706.03762.

Transformers are generic, simples and exciting machine learning architectures designed to process a connected set of
units (tokens in a sequence, pixels in an image, etc.) where the only interaction between units is through self-attention.
Transformers’ performance limit seems purely in the hardware (how big a model can be fitted in GPU memory)43 .
The fundamental operation of transformers is self-attention (a sequence-to-sequence operation, Figure 8): an attention
mechanism relating different positions of a single sequence in order to compute a representation of the same sequence44 .
Let’s call the input vectors (of dimension k) :
x1 , x2 , ..., xt (1)

Let’s call the corresponding output vectors (of dimension k) :

y1 , y2 , ..., yt (2)

The self attention operation takes a weighted average over all the input vectors :
X
yi = wij xj (3)
j

The weight wij is derived from a function over xi and xj . The simplest option is the dot product (with softmax) :

T
exi xj
wij = P xT x (4)
ei j
j

Transformers are Graph Neural Networks45 .

42
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
43
http://www.peterbloem.nl/blog/transformers
44
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html
45
https://graphdeeplearning.github.io/post/transformers-are-gnns/

7
 A PREPRINT - M ARCH 2, 2020

Figure 8: Self-attention. By Peter Bloem : http://www.peterbloem.nl/blog/transformers.

v Making Transformer networks simpler and more efficient46 .

v AttentioNN: All about attention in neural networks described as colab notebooks47 .
v Attention Is All You Need, Vaswani et al. https://arxiv.org/abs/1706.03762.
v How to train a new language model from scratch using Transformers and Tokenizers48 .
v The Illustrated Transformer http://jalammar.github.io/illustrated-transformer/.
v The annotated transformer (code) http://nlp.seas.harvard.edu/2018/04/03/attention.html.
v Attention and Augmented Recurrent Neural Networks https://distill.pub/2016/augmented-rnns/.
v Transformer model for language understanding. Tutorial showing how to write Transformer in TensorFlow 2.049 .
v Transformer in TensorFlow 2.0 (code) https://www.tensorflow.org/beta/tutorials/text/transformer.
v Write With Transformer. By Hugging Face: https://transformer.huggingface.co.

3.4.1 Natural Language Processing (NLP) | BERT: A New Era in NLP

BERT (Bidirectional Encoder Representations from Transformers)[6] is a deeply bidirectional, unsupervised language
representation, pre-trained using only a plain text corpus (in this case, Wikipedia)50 .

• Reading: Unsupervised pre-training of an LSTM followed by supervised fine-tuning[7].

• TensorFlow code and pre-trained models for BERT https://github.com/google-research/bert.
• Better Language Models and Their Implications51 .

"I think transfer learning is the key to general intelligence. And I think the key to doing transfer learning will be the
acquisition of conceptual knowledge that is abstracted away from perceptual details of where you learned it from." —
Demis Hassabis

v Towards a Conversational Agent that Can Chat About. . . Anything52 .

v How to Build OpenAI’s GPT-2: "The AI That’s Too Dangerous to Release"53 .
v A Primer in BERTology: What we know about how BERT works, Rogers et al., 202054 .
v Play with BERT with your own data using TensorFlow Hub https://colab.research.google.com/github/
google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb.
46
https://ai.facebook.com/blog/making-transformer-networks-simpler-and-more-efficient/
47
https://github.com/zaidalyafeai/AttentioNN
48
https://huggingface.co/blog/how-to-train
49
https://www.tensorflow.org/tutorials/text/transformer
50
https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
51
https://blog.openai.com/better-language-models/
52
https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html
53
https://blog.floydhub.com/gpt2/
54
https://arxiv.org/abs/2002.12327

8
 A PREPRINT - M ARCH 2, 2020

Figure 9: The two steps of how BERT is developed. Source https://jalammar.github.io/illustrated-bert/.

3.5 Unsupervised Learning

True intelligence will require independent learning strategies.

"Give a robot a label and you feed it for a second; teach a robot to label and you feed it for a lifetime." — Pierre
Sermanet

Unsupervised learning is a paradigm for creating AI that learns without a particular task in mind: learning for the
sake of learning55 . It captures some characteristics of the joint distribution of the observed random variables (learn the
underlying structure). The variety of tasks include density estimation, dimensionality reduction, and clustering.[4]56 .

"The unsupervised revolution is taking off!" — Alfredo Canziani

Figure 10: A Simple Framework for Contrastive Learning of Visual Representations, Chen et al., 2020
55
https://deepmind.com/blog/unsupervised-learning/
56
https://media.neurips.cc/Conferences/NIPS2018/Slides/Deep_Unsupervised_Learning.pdf

9
 A PREPRINT - M ARCH 2, 2020

Self-supervised learning is derived form unsupervised learning where the data provides the supervision. E.g.
Word2vec57 , a technique for learning vector representations of words, or word embeddings. An embedding is a
mapping from discrete objects, such as words, to vectors of real numbers58 .

"The next revolution of AI won’t be supervised." — Yann LeCun

v Self-Supervised Image Classification, Papers With Code59 .

v Self-supervised learning and computer vision, Jeremy Howard60 .
v Momentum Contrast for Unsupervised Visual Representation Learning, He et al.61
v Data-Efficient Image Recognition with Contrastive Predictive Coding, Hénaff et al.62
v A Simple Framework for Contrastive Learning of Visual Representations, Chen et al.63
v FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, Sohn et al.64
v Self-Supervised Learning of Pretext-Invariant Representations, Ishan Misra, Laurens van der Maaten65 .

3.5.1 Generative Adversarial Networks

Simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model
D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is
to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game[3].

min max[IEx∼pdata (x) [logDθd (x)] + IEz∼pz (z) [log(1 − Dθd (Gθg (z)))]] (5)
θg θd

"What I cannot create, I do not understand." — Richard Feynman

Goodfellow et al. used an interesting analogy where the generative model can be thought of as analogous to a team of
counterfeiters, trying to produce fake currency and use it without detection, while the discriminative model is analogous
to the police, trying to detect the counterfeit currency. Competition in this game drives both teams to improve their
methods until the counterfeits are indistiguishable from the genuine articles. See Figure 11.

Figure 11: GAN: Neural Networks Architecture Pioneered by Ian Goodfellow at University of Montreal (2014).

StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks

• Paper http://stylegan.xyz/paper | Code https://github.com/NVlabs/stylegan.

57
https://jalammar.github.io/illustrated-word2vec/
58
http://projector.tensorflow.org
59
https://paperswithcode.com/task/self-supervised-image-classification
60
https://www.fast.ai/2020/01/13/self_supervised/
61
https://arxiv.org/abs/1911.05722
62
https://arxiv.org/abs/1905.09272
63
https://arxiv.org/abs/2002.05709
64
https://arxiv.org/abs/2001.07685
65
https://arxiv.org/abs/1912.01991

10
 A PREPRINT - M ARCH 2, 2020

• StyleGAN for art. Colab https://colab.research.google.com/github/ak9250/stylegan-art.

• This Person Does Not Exist https://thispersondoesnotexist.com.
• Which Person Is Real? http://www.whichfaceisreal.com.
• This Resume Does Not Exist https://thisresumedoesnotexist.com.
• This Waifu Does Not Exist https://www.thiswaifudoesnotexist.net.
• Encoder for Official TensorFlow Implementation https://github.com/Puzer/stylegan-encoder.
• How to recognize fake AI-generated images. By Kyle McDonald66 .

v 100,000 Faces Imagined by a GAN https://generated.photos.

v Introducing TF-GAN: A lightweight GAN library for TensorFlow 2.067 .
v Generative Adversarial Networks (GANs) in 50 lines of code (PyTorch)68 .
v Few-Shot Adversarial Learning of Realistic Neural Talking Head Models69 .
v Wasserstein GAN http://www.depthfirstlearning.com/2019/WassersteinGAN.
v GANpaint Paint with GAN units http://gandissect.res.ibm.com/ganpaint.html.
v A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications. Gui et al.70 .
v CariGANs: Unpaired Photo-to-Caricature Translation. Cao et al.: https://cari-gan.github.io.
v Infinite-resolution (CPPNs, GANs and TensorFlow.js) https://thispicturedoesnotexist.com.
v PyTorch pretrained BigGAN https://github.com/huggingface/pytorch-pretrained-BigGAN.
v GANSynth: Generate high-fidelity audio with GANs! Colab http://goo.gl/magenta/gansynth-demo.
v SC-FEGAN: Face Editing Generative Adversarial Network https://github.com/JoYoungjoo/SC-FEGAN.
v Demo of BigGAN in an official Colaboratory notebook (backed by a GPU) https://colab.research.google.
com/github/tensorflow/hub/blob/master/examples/colab/biggan_generation_with_tf_hub.ipynb

3.5.2 Variational AutoEncoder

Variational Auto-Encoders71 (VAEs) are powerful models for learning low-dimensional representations See Figure 12.
Disentangled representations are defined as ones where a change in a single unit of the representation corresponds to a
change in single factor of variation of the data while being invariant to others (Bengio et al. (2013).

Figure 12: Variational Autoencoders (VAEs): Powerful Generative Models.

v Colab72 : "Debiasing Facial Detection Systems." AIEthics

v Reading: Disentangled VAE’s (DeepMind 2016) https://arxiv.org/abs/1606.05579.
v Slides: A Few Unusual Autoencoders https://colinraffel.com/talks/vector2018few.pdf.
v MusicVAE: Learning latent spaces for musical scores https://magenta.tensorflow.org/music-vae.
v Generative models in Tensorflow 2 https://github.com/timsainb/tensorflow2-generative-models/.
v SpaceSheet: Interactive Latent Space Exploration with a Spreadsheet https://vusd.github.io/spacesheet/.
66
https://medium.com/@kcimc/how-to-recognize-fake-ai-generated-images-4d1f6f9a2842
67
https://medium.com/tensorflow/introducing-tf-gan-a-lightweight-gan-library-for-tensorflow-2-0-36d767e1abae
68
https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f
69
https://arxiv.org/abs/1905.08233
70
https://arxiv.org/abs/2001.06937
71
https://arxiv.org/abs/1906.02691v2
72
https://colab.research.google.com/github/aamini/introtodeeplearning_labs/blob/master/lab2/Part2_
debiasing_solution.ipynb

11
 A PREPRINT - M ARCH 2, 2020

3.5.3 Capsule
Stacked Capsule Autoencoders. The inductive biases in this unsupervised version of capsule networks give rise to
object-centric latent representations, which are learned in a self-supervised way—simply by reconstructing input images.
Clustering learned representations is enough to achieve unsupervised state-of-the-art classification performance on
MNIST (98.5%). Reference: blog by Adam Kosiorek.73 Code74 .
Capsules learn equivariant object representations (applying any transformation to the input of the function has the same
effect as applying that transformation to the output of the function).

Figure 13: Stacked Capsule Autoencoders. Image source: Blog by Adam Kosiorek.

4 Autonomous Agents
We are on the dawn of The Age of Artificial Intelligence.

"In a moment of technological disruption, leadership matters." — Andrew Ng

An autonomous agent is any device that perceives its environment and takes actions that maximize its chance of
success at some goal. At the bleeding edge of AI, autonomous agents can learn from experience, simulate worlds and
orchestrate meta-solutions. Here’s an informal definition75 of the universal intelligence of agent π 76 :
X
Υ(π) := 2−K(µ) Vµπ (6)
µ∈E

"Intelligence measures an agent’s ability to achieve goals in a wide range of environments." — Shane Legg

4.1 Deep Reinforcement Learning

Reinforcement learning (RL) studies how an agent can learn how to achieve goals in a complex, uncertain environment
(Figure 14) [5]. Recent superhuman results in many difficult environments combine deep learning with RL (Deep
Reinforcement Learning). See Figure 15 for a taxonomy of RL algorithms.
v An Opinionated Guide to ML Research77 .
v CS 188 : Introduction to Artificial Intelligence78 .
v Introduction to Reinforcement Learning by DeepMind79 .
v "My Top 10 Deep RL Papers of 2019" by Robert Tjarko Lange80 .
v Deep tic-tac-toe https://zackakil.github.io/deep-tic-tac-toe/.
v CS 287: Advanced Robotics81 . https://people.eecs.berkeley.edu/~pabbeel/cs287-fa19/.
73
http://akosiorek.github.io/ml/2019/06/23/stacked_capsule_autoencoders.html
74
https://github.com/google-research/google-research/tree/master/stacked_capsule_autoencoders
75
https://arxiv.org/abs/0712.3329
76
Where µ is an environment, K is the Kolmogorov complexity function, E is the space of all computable reward summable
environmental measures with respect to the reference machine U and the value function Vµπ is the agent’s “ability to achieve”.
77
http://joschu.net/blog/opinionated-guide-ml-research.html
78
https://inst.eecs.berkeley.edu/~cs188/fa18/
79
https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ
80
https://roberttlange.github.io/posts/2019/12/blog-post-9/
81
https://people.eecs.berkeley.edu/~pabbeel/cs287-fa19/exam/cs287-fa19-exam-study-handout.pdf

12
 A PREPRINT - M ARCH 2, 2020

Figure 14: An Agent Interacts with an Environment.

Figure 15: A Taxonomy of RL Algorithms. Source: Spinning Up in Deep RL by Achiam et al. | OpenAI

4.1.1 Model-Free RL | Value-Based

The goal in RL is to train the agent to maximize the discounted sum of all future rewards Rt , called the return:

Rt = rt + γrt+1 + γ 2 rt+2 + . . . (7)

The Q-function captures the expected total future reward an agent in state s can receive by executing a certain action a:

Q(s, a) = E[Rt ] (8)

The optimal policy should choose the action a that maximizes Q(s,a):

π ∗ (s) = argmaxa Q(s, a) (9)

• Q-Learning: Playing Atari with Deep Reinforcement Learning (DQN). Mnih et al, 2013[10]. See Figure 17.

"There’s no limit to intelligence." — David Silver

v Q-Learning in enormous action spaces via amortized approximate maximization, de Wiele et al.82 .
v TF-Agents (DQN Tutorial) | Colab https://colab.research.google.com/github/tensorflow/agents.
82
https://arxiv.org/abs/2001.08116

13
 A PREPRINT - M ARCH 2, 2020

Figure 16: Open-Source RL Algorithms https://docs.google.com/spreadsheets/d/1EeFPd-XIQ3mq_

9snTlAZSsFY7Hbnmd7P5bbT8LPuMn0/

Figure 17: DQN Training Algorithm. Volodymyr Mnih, Deep RL Bootcamp

4.1.2 Model-Free RL | Policy-Based

An RL agent learns the stochastic policy function that maps state to action and act by sampling policy.
Run a policy for a while (code: https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5):

τ = (s0 , a0 , r0 , s1 , a1 , r1 , . . . , sT −1 , aT −1 , rT −1 , sT ) (10)

Increase probability of actions that lead to high rewards and decrease probability of actions that lead to low rewards:
"T −1 #
X
∇θ Eτ [R(τ )] = Eτ ∇θ log π(at |st , θ)R(τ ) (11)
t=0

14
 A PREPRINT - M ARCH 2, 2020

Figure 18: Policy Gradient Directly Optimizes the Policy.

πθ (s, α1 )
πθ (s, α2 )
πθ (s, α3 )
πθ (s, α4 )
πθ (s, α5 )
s

Vψ (s)

Figure 19: Asynchronous Advantage Actor-Critic (A3C). Source: Petar Velickovic

• Policy Optimization: Asynchronous Methods for Deep Reinforcement Learning (A3C). Mnih et al, 2016[8].
• Policy Optimization: Proximal Policy Optimization Algorithms (PPO). Schulman et al, 2017[9].

v Deep Reinforcement Learning for Playing 2.5D Fighting Games. Li et al.83 .

4.1.3 Model-Based RL
In Model-Based RL, the agent generates predictions about the next state and reward before choosing each action.

Figure 20: World Model’s Agent consists of: Vision (V), Memory (M), and Controller (C). | Ha et al, 2018[11]

83
https://arxiv.org/abs/1805.02070

15
 A PREPRINT - M ARCH 2, 2020

• Learn the Model: Recurrent World Models Facilitate Policy Evolution (World Models84 ). The world model
agent can be trained in an unsupervised manner to learn a compressed spatial and temporal representation of
the environment. Then, a compact policy can be trained. See Figure 20. Ha et al, 2018[11].
• Learn the Model: Learning Latent Dynamics for Planning from Pixels https://planetrl.github.io/.
• Given the Model: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
(AlphaZero). Silver et al, 2017[14]. AlphaGo Zero Explained In One Diagram85 .

v Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Schrittwieser et al.86 .

4.1.4 Toward a General AI-Agent Architecture: SuperDyna (General Dyna-style RL Agent)

"Intelligence is the computational part of the ability to predict and control a stream of experience." — Rich Sutton

SuperDyna.87 The ambition: a general AI agent for Artificial Biological Reinforcement Learning.

1. Interact with the world: sense, update state and take an action
2. Learn from what just happened: see what happened and learn from it
3. Plan: (while there is time remaining in this time step) imagine hypothetical states and actions you might take
4. Discover : curate options and features and measure how well they’re doing

Figure 21: Inner Loop of a General Dyna-Style RL Agent (SuperDyna).

The first complete and scalable general AI-agent architecture that has all the most important capabilities and desiderata:

• Acting, learning, planning, model-learning, subproblems, and options.

• Function approximation, partial observability, non-stationarity and stochasticity.
• Discovery of state features, and thereby of subproblems, options and models.
• All feeding back to motivate new, more-abstract features in a virtuous cycle of discovery.
84
https://worldmodels.github.io
85
https://applied-data.science/static/main/res/alpha_go_zero_cheat_sheet.png
86
https://arxiv.org/abs/1911.08265
87
https://insidehpc.com/2020/02/video-toward-a-general-ai-agent-architecture/

16
 A PREPRINT - M ARCH 2, 2020

Figure 22: SuperDyna: Virtuous cycle of discovery.

Presentation by Richard Sutton (starts at 15 min.)88 .

"In practice, I work primarily in reinforcement learning as an approach to artificial intelligence. I am exploring ways to
represent a broad range of human knowledge in an empirical form–that is, in a form directly in terms of experience–and
in ways of reducing the dependence on manual encoding of world state and knowledge." — Richard S. Sutton

4.1.5 Improving Agent Design

Via Reinforcement Learning: Blog89 . arXiv90 . ASTool https://github.com/hardmaru/astool/.
Via Evolution: Video91 . Evolved Creatures http://www.karlsims.com/evolved-virtual-creatures.html.

Figure 23: A comparison of the original LSTM cell vs. two new good generated. Top left: LSTM cell. [19]

"The future of high-level APIs for AI is... a problem-specification API. Currently we only search over network weights,
thus "problem specification" involves specifying a model architecture. In the future, it will just be: "tell me what data
you have and what you are optimizing"." — François Chollet
88
https://slideslive.com/38921889/biological-and-artificial-reinforcement-learning-4
89
https://designrl.github.io
90
https://arxiv.org/abs/1810.03779
91
https://youtu.be/JBgG_VSP7f8

17
 A PREPRINT - M ARCH 2, 2020

v Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments92 .

4.1.6 OpenAI Baselines

High-quality implementations of reinforcement learning algorithms https://github.com/openai/baselines.
Colab https://colab.research.google.com/drive/1KKq9A3dRTq1q6bJmPyFOgg917gQyTjJI.

4.1.7 Google Dopamine and A Zoo of Agents

Dopamine is a research framework for fast prototyping of reinforcement learning algorithms.93 .
A Zoo of Atari-Playing Agents: Code94 , Blog95 and Colaboratory notebook https://colab.research.google.
com/github/uber-research/atari-model-zoo/blob/master/colab/AtariZooColabDemo.ipynb.

4.1.8 TRFL : TensorFlow Reinforcement Learning

TRFL ("truffle"): a library of reinforcement learning building blocks https://github.com/deepmind/trfl.

4.1.9 bsuite : Behaviour Suite for Reinforcement Learning

A collection of experiments that investigate core capabilities of RL agents http://github.com/deepmind/bsuite.

4.2 Evolution Strategies (ES)

In her Nobel Prize in Chemistry 2018 Lecture "Innovation by Evolution: Bringing New Chemistry to Life" (Nobel
Lecture)†96 , Prof. Frances H. Arnold said :
"Nature ... invented life that has flourished for billions of years. (...) Equally awe-inspiring is the process by which
Nature created these enzyme catalysts and in fact everything else in the biological world. The process is evolution, the
grand diversity-generating machine that created all life on earth, starting more than three billion years ago. (...)
evolution executes a simple algorithm of diversification and natural selection, an algorithm that works at all levels
of complexity from single protein molecules to whole ecosystems." — Prof. Frances H. Arnold
Evolution and neural networks proved a potent combination in nature.
"Evolution is a slow learning algorithm that with the sufficient amount of compute produces a human brain." —
Wojciech Zaremba
Natural evolutionary strategy directly evolves the weights of a DNN and performs competitively with the best deep
reinforcement learning algorithms, including deep Q-networks (DQN) and policy gradient methods (A3C)[21].

Figure 24: https://colab.research.google.com/github/karpathy/randomfun/blob/master/es.ipynb.

Neuroevolution, which harnesses evolutionary algorithms to optimize neural networks, enables capabilities that are
typically unavailable to gradient-based approaches, including learning neural network building blocks, architectures
and even the algorithms for learning[12].
92
https://arxiv.org/abs/1910.07224
93
https://github.com/google/dopamine
94
https://github.com/uber-research/atari-model-zoo
95
https://eng.uber.com/atari-zoo-deep-reinforcement-learning/
96
https://onlinelibrary.wiley.com/doi/epdf/10.1002/anie.201907729

18
 A PREPRINT - M ARCH 2, 2020

". . . evolution — whether biological or computational — is inherently creative, and should routinely be expected to
surprise, delight, and even outwit us." — The Surprising Creativity of Digital Evolution, Lehman et al.[22]
The ES algorithm is a “guess and check” process, where we start with some random parameters and then repeatedly:
1. Tweak the guess a bit randomly, and
2. Move our guess slightly towards whatever tweaks worked better.
Neural architecture search has advanced to the point where it can outperform human-designed models[13].
"Caterpillar brains LIQUIFY during metamorphosis, but the butterfly retains the caterpillar’s memories!" — M. Levin
"Open-ended" algorithms are algorithms that endlessly create. Brains and bodies evolve together in nature.
"We’re machines," says Hinton. ""We’re just produced biologically (...)" — Katrina Onstad, Toronto Life
v Evolution Strategies97 .
v VAE+CPPN+GAN98 .
v Demos: ES on CartPole-v199 and ES on LunarLanderContinuous-v2100 .
v Spiders Can Fly Hundreds of Miles Riding the Earth’s Magnetic Fields101 .
v A Visual Guide to ES http://blog.otoro.net/2017/10/29/visual-evolution-strategies/.
v Xenobots A scalable pipeline for designing reconfigurable organisms, Kriegman et al.102 . Learn103 . Evolve104 .

4.3 Self Play

Silver et al.[15] introduced an algorithm based solely on reinforcement learning, without human data, guidance or
domain knowledge. Starting tabula rasa (and being its own teacher!), AlphaGo Zero achieved superhuman performance.
AlphaGo Zero showed that algorithms matter much more than big data and massive amounts of computation.
"Self-Play is Automated Knowledge Creation." — Carlos E. Perez
Self-play mirrors similar insights from coevolution. Transfer learning is the key to go from self-play to the real world105 .
"Open-ended self play produces: Theory of mind, negotiation, social skills, empathy, real language understanding." —
Ilya Sutskever, Meta Learning and Self Play
v How To Build Your Own MuZero AI Using Python106 .
v TensorFlow.js Implementation of DeepMind’s AlphaZero Algorithm for Chess. Live Demo107 . | Code108 .
v An open-source implementation of the AlphaGoZero algorithm https://github.com/tensorflow/minigo.
v ELF OpenGo: An Open Reimplementation of AlphaZero, Tian et al.: https://arxiv.org/abs/1902.04522.

4.4 Multi-Agent Populations

"We design a Theory of Mind neural network – a ToMnet – which uses meta-learning to build models of the agents it
encounters, from observations of their behaviour alone." — Machine Theory of Mind, Rabinowitz et al.[25]
Cooperative Agents. Learning to Model Other Minds, by OpenAI[24], is an algorithm which accounts for the fact that
other agents are learning too, and discovers self-interested yet collaborative strategies. Also: OpenAI Five109 .
97
https://lilianweng.github.io/lil-log/2019/09/05/evolution-strategies.html
98
https://colab.research.google.com/drive/1_OoZ3z_C5Jl5gnxDOE9VEMCTs-Fl8pvM
99
https://colab.research.google.com/drive/1bMZWHdhm-mT9NJENWoVewUks7cGV10go
100
https://colab.research.google.com/drive/1lvyKjFtc_C_8njCKD-MnXEW8LPS2RPr6
101
https://www.cell.com/current-biology/fulltext/S0960-9822(18)30693-6
102
https://www.pnas.org/content/early/2020/01/07/1910837117
103
https://cdorgs.github.io
104
https://github.com/skriegman/reconfigurable_organisms
105
http://metalearning-symposium.ml
106
https://medium.com/applied-data-science/how-to-build-your-own-muzero-in-python-f77d5718061a
107
https://frpays.github.io/lc0-js/engine.html
108
https://github.com/frpays/lc0-js/
109
https://blog.openai.com/openai-five/

19
 A PREPRINT - M ARCH 2, 2020

Figure 25: Facebook, Carnegie Mellon build first AI that beats pros in 6-player poker https://ai.facebook.
com/blog/pluribus-first-ai-to-beat-pros-in-6-player-poker

"Artificial Intelligence is about recognising patterns, Artificial Life is about creating patterns." — Mizuki Oka et al.

Active Learning Without Teacher. In Intrinsic Social Motivation via Causal Influence in Multi-Agent RL, Jaques et
al. (2018) https://arxiv.org/abs/1810.08647 propose an intrinsic reward function designed for multi-agent RL
(MARL), which awards agents for having a causal influence on other agents’ actions. Open-source implementation 110 .
"Open-ended Learning in Symmetric Zero-sum Games," Balduzzi et al.: https://arxiv.org/abs/1901.08106
v Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks,
Suarezet al.111 Project Page https://jsuarez5341.github.io, Video112 and Slides113 .
v Neural MMO: A massively multiagent env. for simulations with many long-lived agents. Code114 and 3D Client115 .

4.5 Deep Meta-Learning

Learning to Learn[16].

"The notion of a neural "architecture" is going to disappear thanks to meta learning." — Andrew Trask

v Stanford CS330: Multi-Task and Meta-Learning, Finn et al., 2019116 .

v Meta Learning Shared Hierarchies[18] (The Lead Author is in High School!).
v Causal Reasoning from Meta-reinforcement Learning https://arxiv.org/abs/1901.08162.

4.5.1 MAML: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

The goal of model-agnostic meta-learning for fast adaptation of deep networks is to train a model on a variety of
learning tasks, such that it can solve new learning tasks using only a small number of training samples[20].
X

θ ← θ − β∇θ LTi fθi0 (12)
Ti ∼p(T )

110
https://github.com/eugenevinitsky/sequential_social_dilemma_games
111
https://arxiv.org/abs/2001.12004
112
https://youtube.com/watch?v=DkHopV1RSxw
113
https://docs.google.com/presentation/d/1tqm_Do9ph-duqqAlx3r9lI5Nbfb9yUfNEtXk1Qo4zSw/edit?usp=
sharing
114
https://github.com/openai/neural-mmo
115
https://github.com/jsuarez5341/neural-mmo-client
116
http://youtube.com/playlist?list=PLoROMvodv4rMC6zfYmnD7UG3LVvwaITY5

20
 A PREPRINT - M ARCH 2, 2020

A meta-learning algorithm takes in a distribution of tasks, where each task is a learning problem, and it produces a
quick learner — a learner that can generalize from a small number of examples[17].

Figure 26: Diagram of Model-Agnostic Meta-Learning (MAML)

v How to Train MAML (Model-Agnostic Meta-Learning)117 .

v Meta-Learning with Implicit Gradients https://arxiv.org/abs/1909.04630.
v Colaboratory reimplementation of MAML (Model-Agnostic Meta-Learning) in TF 2.0118 .
v Torchmeta: A Meta-Learning library for PyTorch119 https://github.com/tristandeleu/pytorch-meta.

4.5.2 The Grand Challenge for AI Research | AI-GAs: AI-Generating Algorithms, an Alternate Paradigm for
Producing General Artificial Intelligence
In AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence120 , Jeff
Clune describes an exciting path that ultimately may be successful at producing general AI. The idea is to create an
AI-generating algorithm (AI-GA), which automatically learns how to produce general AI.
Three Pillars are essential for the approach: (1) Meta-learning architectures, (2) Meta-learning the learning algo-
rithms themselves, and (3) Generating effective learning environments.

• The First Pillar, meta-learning architectures, could potentially discover the building blocks : convolution, re-
current layers, gradient-friendly architectures, spatial tranformers, etc.
• The Second Pillar, meta-learning learning algorithms, could potentially learn the building blocks : intelligent
exploration, auxiliary tasks, efficient continual learning, causal reasoning, active learning, etc.
• The Third Pillar, generating effective and fully expressive learning environments, could learn things like :
co-evolution / self-play, curriculum learning, communication / language, multi-agent interaction, etc.

On Earth,

"( . . . ) a remarkably simple algorithm (Darwinian evolution) began producing solutions to relatively simple
environments. The ‘solutions’ to those environments were organisms that could survive in them. Those organism often
created new niches (i.e. environments, or opportunities) that could be exploited. Ultimately, that process produced all
of the engineering marvels on the planet, such as jaguars, hawks, and the human mind." — Jeff Clune

Turing Complete (universal computer) : an encoding that enables the creation any possible learning algorithm.
Darwin Complete : an environmental encoding that enables the creation of any possible learning environment.
v Learning to Continually Learn. Beaulieu et al. https://arxiv.org/abs/2002.09571.
v Fully Differentiable Procedural Content Generation through Generative Playing Networks. Bontrageret et
al.121
117
https://medium.com/towards-artificial-intelligence/how-to-train-maml-model-agnostic-meta-learning-90aa093f8e46
118
https://colab.research.google.com/github/mari-linhares/tensorflow-maml/blob/master/maml.ipynb
119
https://medium.com/pytorch/torchmeta-a-meta-learning-library-for-pytorch-f76c2b07ca6d
120
https://arxiv.org/abs/1905.10985
121
https://arxiv.org/abs/2002.05259

21
 A PREPRINT - M ARCH 2, 2020

5 Symbolic AI
v On neural-symbolic computing: suggested readings on foundations of the field. Luis Lamb122 .
v Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. Besold et al.123
v Neural Module Networks for Reasoning over Text. Gupta et al.124 Code.125
v The compositionality of neural networks: integrating symbolism and connectionism. Hupkes et al.126
v Neuro-symbolic A.I. is the future of artificial intelligence. Here’s how it works. Luke Dormehl127 .
v DDSP: Differentiable Digital Signal Processing. Engel et al. Blog128 , Colab129 , Paper130 and Code131 .
v Differentiable Reasoning on Large Knowledge Bases and Natural Language. Minervini et al.132 Open-source
neuro-symbolic reasoning framework, in TensorFlow https://github.com/uclnlp/gntp.

6 Environments
Platforms for training autonomous agents.
"Run a physics sim long enough and you’ll get intelligence." — Elon Musk

6.1 OpenAI Gym

"Situation awareness is the perception of the elements in the environment within a volume of time and space, and the
comprehension of their meaning, and the projection of their status in the near future." — Endsley (1987)
The OpenAI Gym https://gym.openai.com/ (Blog133 | GitHub134 ) is a toolkit for developing and comparing
reinforcement learning algorithms. What makes the gym so great is a common API around environments.

Figure 27: Robotics Environments https://blog.openai.com/ingredients-for-robotics-research/

"By framing the approach within the popular OpenAI Gym framework, design firms can create more realistic
environments – for instance, incorporate strength of materials, safety factors, malfunctioning of components under
stressed conditions, and plug existing algorithms into this framework to optimize also for design aspects such as energy
usage, easy-of-manufacturing, or durability." — David Ha135
How to create new environments for Gym136 . Minimal example with code and agent (evolution strategies on foo-v0):
122
https://twitter.com/luislamb/status/1218575842340634626
123
https://arxiv.org/abs/1711.03902
124
https://arxiv.org/abs/1912.04971
125
https://nitishgupta.github.io/nmn-drop
126
https://arxiv.org/abs/1908.08351
127
https://www.digitaltrends.com/cool-tech/neuro-symbolic-ai-the-future/
128
http://magenta.tensorflow.org/ddsp
129
http://g.co/magenta/ddsp-demo
130
http://g.co/magenta/ddsp-paper
131
http://github.com/magenta/ddsp
132
https://arxiv.org/abs/1912.10824
133
https://blog.openai.com/openai-gym-beta/
134
https://github.com/openai/gym
135
https://designrl.github.io
136
https://github.com/openai/gym/blob/master/docs/creating-environments.md

22
 A PREPRINT - M ARCH 2, 2020

1. Download gym-foo https://drive.google.com/file/d/1r2A8J9CJjIQNwss246gATeD0LLMzpUT-/

view?usp=sharing
2. cd gym-foo
3. pip install -e .
4. python ES-foo.py

He’re another more difficult (for the agent!) new environment for Gym (evolution strategies on foo-v3):

1. Download gym-foo-v3137
2. cd gym-foo-v3
3. pip install -e .
4. python ES-foo-v3.py

v OpenAI Gym Environment for Trading138 .

v Fantasy Football AI Environment https://github.com/njustesen/ffai.
v Create custom gym environments from scratch — A stock market example139 .
v IKEA Furniture Assembly Environment https://clvrai.github.io/furniture/.
v Minimalistic Gridworld Environment https://github.com/maximecb/gym-minigrid.
v DoorGym: A Scalable Door Opening Environment and Baseline Agent, Urakami et al., 2019140 .
v gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo, Lopez et al., 2019141 .
v OFFWORLD GYM Open-access physical robotics environment for real-world reinforcement learning142 .
v Safety Gym: environments to evaluate agents with safety constraints https://github.com/openai/safety-gym.

6.2 DeepMind Lab

DeepMind Lab: A customisable 3D platform for agent-based AI research https://github.com/deepmind/lab.

• DeepMind Control Suite https://github.com/deepmind/dm_control.

• Convert DeepMind Control Suite to OpenAI Gym Envs https://github.com/zuoxingdong/dm2gym.

6.3 Unity ML-Agents

Unity ML Agents allows to create environments where intelligent agents (Single Agent, Cooperative and Competitive
Multi-Agent and Ecosystem) can be trained using RL, neuroevolution, or other ML methods https://unity3d.ai.

• Getting Started with Marathon Environments for Unity ML-Agents143 https://github.com/

Unity-Technologies/marathon-envs.
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence144 .

6.4 AI Habitat

AI Habitat enables training of embodied AI agents (virtual robots) in a highly photorealistic and efficient 3D simulator,
before transferring the learned skills to reality. By Facebook AI Research https://aihabitat.org/.
Why the name Habitat? Because that’s where AI agents live!
137
https://drive.google.com/file/d/1cGncsXJ56UUKCO9MaRWJVTnxiQEnLuxS/view?usp=sharing
138
https://github.com/hackthemarket/gym-trading
139
https://towardsdatascience.com/creating-a-custom-openai-gym-environment-for-stock-trading-be532be3910e
140
https://arxiv.org/abs/1908.01887
141
https://arxiv.org/abs/1903.06278
142
https://gym.offworld.ai
143
https://towardsdatascience.com/gettingstartedwithmarathonenvs-v0-5-0a-c1054a0b540c
144
https://arxiv.org/abs/1905.08085

23
 A PREPRINT - M ARCH 2, 2020

6.5 POET: Paired Open-Ended Trailblazer

Diversity is the premier product of evolution. Endlessly generate increasingly complex and diverse learning environ-
ments145 . Open-endedness could generate learning algorithms reaching human-level intelligence[23].

• Implementation of the POET algorithm https://github.com/uber-research/poet.

7 Deep-Learning Hardware

Figure 28: Edge TPU - Dev Board https://coral.ai/products/dev-board/

Figure 29: The world’s largest chip : Cerebras Wafer Scale Engine https://www.cerebras.net

v Which GPU(s) to Get for Deep Learning, by Tim Dettmers146 .

v A Full Hardware Guide to Deep Learning, by Tim Dettmers147 .
v Jetson Nano. A small but mighty AI computer to create intelligent systems148 .
v Build AI that works offline with Coral Dev Board, Edge TPU, and TensorFlow Lite, by Daniel Situnayake149 .
145
https://eng.uber.com/poet-open-ended-deep-learning/
146
http://timdettmers.com/2019/04/03/which-gpu-for-deep-learning/
147
http://timdettmers.com/2018/12/16/deep-learning-hardware-guide/
148
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-nano/
149
https://medium.com/tensorflow/build-ai-that-works-offline-with-coral-dev-board-edge-tpu-and-tensorflow-lite-70

24
 A PREPRINT - M ARCH 2, 2020

8 Deep-Learning Software
8.1 TensorFlow
• TensorFlow Lite for Microcontrollers150 .
• TensorBoard in Jupyter Notebooks151 . Colab152 .
• TensorFlow 2.0 + Keras Crash Course. Colab153 .
• tf.keras (TensorFlow 2.0) for Researchers: Crash Course. Colab154 .
• TensorFlow Tutorials https://www.tensorflow.org/tutorials.
• TensorFlow 2.0: basic ops, gradients, data preprocessing and augmentation, training and saving. Colab155 .

8.2 PyTorch
• PyTorch primer. Colab156 .
• PyTorch internals http://blog.ezyang.com/2019/05/pytorch-internals/

9 AI Art | A New Day Has Come in Art Industry

Figure 30: On October 25, 2018, the first AI artwork ever sold at Christie’s auction house fetched USD 432,500.

The code (art-DCGAN) for the first artificial intelligence artwork ever sold at Christie’s auction house (Figure 23) is a
modified implementation of DCGAN focused on generative art: https://github.com/robbiebarrat/art-dcgan.

• TensorFlow Magenta. An open source research project exploring the role of ML in the creative process.157 .
• Magenta Studio. A suite of free music-making tools using machine learning models!158 .
• Style Transfer Tutorial https://colab.research.google.com/github/tensorflow/docs/blob/
master/site/en/r2/tutorials/generative/style_transfer.ipynb
• AI x AR Paper Cubes https://experiments.withgoogle.com/paper-cubes.
• Photo Wake-Up https://grail.cs.washington.edu/projects/wakeup/.
• COLLECTION. AI Experiments https://experiments.withgoogle.com/ai.

"The Artists Creating with AI Won’t Follow Trends; THEY WILL SET THEM." — The House of Montréal.AI Fine Arts
150
https://www.tensorflow.org/lite/microcontrollers
151
https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks
152
https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/tensorboard_in_
notebooks.ipynb
153
https://colab.research.google.com/drive/1UCJt8EYjlzCs1H1d1X0iDGYJsHKwu-NO
154
https://colab.research.google.com/drive/14CvUNTaX1OFHDfaKaaZzrBsvMfhCOHIR
155
https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/TF_2_0.ipynb
156
https://colab.research.google.com/drive/1DgkVmi6GksWOByhYVQpyUB4Rk3PUq0Cp
157
https://magenta.tensorflow.org
158
https://magenta.tensorflow.org/studio

25
 A PREPRINT - M ARCH 2, 2020

v Tuning Recurrent Neural Networks with Reinforcement Learning159 .

v MuseNet. Generate Music Using Many Different Instruments and Styles!160 .
v Infinite stream of machine generated art. Valentin Vieriu https://art42.net.
v Deep Multispectral Painting Reproduction via Multi-Layer, Custom-Ink Printing. Shi et al.161 .
v Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning. Shen et al.162 .
v Synthesizing Programs for Images using Reinforced Adversarial Learning, Ganin et al., 2018163 . Agents164 .

10 AI Macrostrategy: Aligning AGI with Human Interests

Montréal.AI Governance: Policies at the intersection of AI, Ethics and Governance.

Figure 31: A Map of Ethical and Right-Based Approaches https://ai-hr.cyber.harvard.edu/primp-viz.html

"(AI) will rank among our greatest technological achievements, and everyone deserves to play a role in shaping it." —
Fei-Fei Li

v AI Index. http://aiindex.org.
v Malicious AI Report. https://arxiv.org/pdf/1802.07228.pdf.
v ADDRESS PREPARED BY POPE FRANCIS FOR THE PLENARY ASSEMBLY OF THE PONTIFICAL
ACADEMY FOR LIFE165 .
v Artificial Intelligence and Human Rights. https://ai-hr.cyber.harvard.edu.
v Ethically Aligned Design, First Edition166 . From Principles to Practice https://ethicsinaction.ieee.org.

"It’s springtime for AI, and we’re anticipating a long summer." — Bill Braun

References
[1] Mnih et al. Human-Level Control Through Deep Reinforcement Learning. In Nature 518, pages 529–533. 26
February 2015. https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf
[2] Yann LeCun, Yoshua Bengio and Geoffrey Hinton. Deep Learning. In Nature 521, pages 436–444. 28 May 2015.
https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf
159
https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-networks-with-reinforcement-learning
160
https://openai.com/blog/musenet/
161
http://people.csail.mit.edu/liangs/papers/ToG18.pdf
162
https://arxiv.org/pdf/1903.02678.pdf
163
http://proceedings.mlr.press/v80/ganin18a.html
164
https://github.com/deepmind/spiral
165
http://w2.vatican.va/content/francesco/en/speeches/2020/february/documents/papa-francesco_
20200228_accademia-perlavita.html
166
https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/other/ead1e.pdf

26
 A PREPRINT - M ARCH 2, 2020

[3] Goodfellow et al. Generative Adversarial Networks. arXiv preprint arXiv:1406.2661, 2014. https://arxiv.
org/abs/1406.2661
[4] Yoshua Bengio, Andrea Lodi, Antoine Prouvost. Machine Learning for Combinatorial Optimization: a Method-
ological Tour d’Horizon. arXiv preprint arXiv:1811.06128, 2018. https://arxiv.org/abs/1811.06128
[5] Brockman et al. OpenAI Gym. 2016. https://gym.openai.com
[6] Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint
arXiv:1810.04805, 2018. https://arxiv.org/abs/1810.04805
[7] Dai et al. Semi-supervised Sequence Learning. arXiv preprint arXiv:1511.01432, 2015. https://arxiv.org/
abs/1511.01432
[8] Mnih et al. Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint arXiv:1602.01783, 2016.
https://arxiv.org/abs/1602.01783
[9] Schulman et al. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017. https:
//arxiv.org/abs/1707.06347
[10] Mnih et al. Playing Atari with Deep Reinforcement Learning. DeepMind Technologies, 2013. https://www.cs.
toronto.edu/~vmnih/docs/dqn.pdf
[11] Ha et al. Recurrent World Models Facilitate Policy Evolution. arXiv preprint arXiv:1809.01999, 2018. https:
//arxiv.org/abs/1809.01999
[12] Kenneth et al. Designing neural networks through neuroevolution. In Nature Machine Intelligence VOL 1, pages
24–35. January 2019. https://www.nature.com/articles/s42256-018-0006-z.pdf
[13] So et al. The Evolved Transformer. arXiv preprint arXiv:1901.11117, 2019. https://arxiv.org/abs/1901.
11117
[14] Silver et al. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. arXiv
preprint arXiv:1712.01815, 2017. https://arxiv.org/abs/1712.01815
[15] Silver et al. AlphaGo Zero: Learning from scratch. In DeepMind’s Blog, 2017. https://deepmind.com/blog/
alphago-zero-learning-scratch/
[16] Andrychowicz et al. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv:1606.04474,
2016. https://arxiv.org/abs/1606.04474
[17] Nichol et al. Reptile: A Scalable Meta-Learning Algorithm. 2018. https://blog.openai.com/reptile/
[18] Frans et al. Meta Learning Shared Hierarchies. arXiv preprint arXiv:1710.09767, 2017. https://arxiv.org/
abs/1710.09767
[19] Zoph and Le, 2017 Neural Architecture Search with Reinforcement Learning. arXiv preprint arXiv:1611.01578,
2017. https://arxiv.org/abs/1611.01578
[20] Finn et al., 2017 Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv preprint
arXiv:1703.03400, 2017. https://arxiv.org/abs/1703.03400
[21] Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. 2017. https:
//blog.openai.com/evolution-strategies/
[22] Lehman et al. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary
Computation and Artificial Life Research Communities. arXiv preprint arXiv:1803.03453, 2018. https://arxiv.
org/abs/1803.03453
[23] Wang et al. Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse
Learning Environments and Their Solutions. arXiv preprint arXiv:1901.01753, 2019. https://arxiv.org/abs/
1901.01753
[24] Foerster et al. Learning to Model Other Minds. 2018. https://blog.openai.com/
learning-to-model-other-minds/
[25] Rabinowitz et al. Machine Theory of Mind. arXiv preprint arXiv:1802.07740, 2018. https://arxiv.org/abs/
1802.07740

27