Deep Learning

COMP 5329

Dr Chang Xu

[email protected]
School of Computer Science

The University of Sydney Page 1

 AI’s Big Breakthroughs in 2020

GPT-3, the 3rd version of Generative Pre-Trained Transformer model

released by OpenAI.
GPT-3 是OpenAI发布的⽣成式预训练变压器模型的第三版。

建筑毫升模型。Keras代码是由GPT-3编写的，⽽输⼊是我们想要为什么ML模型编写代码的简单纯⽂本，然后它就⽣成了模型。
开发Web应⽤程序。输⼊⼀个描述⾕歌主⻚布局的句⼦，这⾥您将看到GPT-3为它⽣成代码。
Develop Web apps. Enter a sentence Building ML Models. Keras code written by
describing Google home page layout, and GPT-3 while input was simple plain text of
here you see GPT-3 generating the code for what ML model do we want to write code for,
it.The University of Sydney and boom, it generated the model. Page 2
 AI’s Big Breakthroughs in 2020 蛋⽩质是⼀种⾮常复杂的分⼦，在细胞内完成所
有的⼯作。为了了解蛋⽩质是如何⼯作的，我们
需要知道它们折叠成最终构象时采⽤的结构。

⽣物学界50年⼤挑战的解决⽅案。
AlphaFold:
Proteins are very complex molecules that do
AlphaFold: a solution to a 50-year-old grand all the work in a cell. In order to understand
challenge in biology. how proteins work, one needs to know what
structure they adopt when they fold into their
final conformation.

The University of Sydney Page 3

Artificial Intelligence:
Human Intelligence
Exhibited by Machines.

Machine Learning :
An Approach to
Achieve Artificial
Intelligence
⼈⼯智能:由机器表现出来的⼈类智能。

机器学习:⼀种实现⼈⼯智能的⽅法

深度学习:⼀种实现机器学习的技术

Deep Learning: A
Technique for
Implementing Machine
Learning

The University of Sydney Page 4

 What is Deep Learning?

The short answers

1. ‘Deep Learning’ means using a neural network
with several layers of nodes between input and output
深度学习”意味着使⽤⼀个在输⼊和输出之间有多层节点的神经⽹络
1. “

2. 输⼊和输出之间的⼀系列层次进⾏特征识别和处理的⼀系列阶段，就像我们的⼤脑看起来。

2. the series of layers between input & output to do

feature identification and processing in a series of stages,
just as our brains seem to.

The University of Sydney Page 5

 What is Deep Learning?

We have always had good algorithms for learning the

weights in networks with 1 hidden layer;
我们⼀直有很好的算法来学习⼀个隐含层的⽹络中的权值;

但这些算法不善于学习隐含层较多的⽹络的权值

But these algorithms are not good at learning the

weights for networks with more hidden layers
Image credit to becominghuman.ai
The University of Sydney Page 6
 What to Learn?
https://canvas.sydney.edu.au/courses/30917/pages/course-contents

1. Introduction
2. Multilayer Neural Networks
3. Optimization for Deep Models 2.
1. 介绍

多层神经⽹络

4. Regularization for Deep Models 3.深度模型优化

4. 深度模型的正则化

5. 假期

5. Holiday 6. 卷积神经⽹络

7. 神经⽹络结构

8. 复发性神经⽹络

6. Convolutional Neural Networks 9. 变压器神经⽹络

10. 图卷积⽹络

11. 深度学习的应⽤

7. Neural Network Architectures 12. 深层⽣成模型

13. 审查

8. Recurrent Neural Networks

9. Transformer Neural Networks
10. Graph Convolutional Networks
11. Deep Learning Applications
12. Deep Generation Models
13. Review

The University of Sydney Page 7

Enrollment of COMP5329

Enrollment of COMP5329
400
300
200
100
0
2018 2019 2020 2021

The University of Sydney Page 8

 Online Lectures and Tutorials

– Lecture: Friday 18:00 – 20:00

– Tutorial: Thursday 17:00 – 18:00; Friday 20:00 – 21:00 (from the 2nd week)

– Consultation: Friday, 15:00-16:00

https://uni-sydney.zoom.us/j/7228816221?pwd=RmQ3dEgzY25ETXRrSkh6ekVBSyt6Zz09
Password: 135109

– Teaching Assistant: Mr Gary Jiajun Huang <[email protected]>

The University of Sydney Page 9
 Ed Discussion
https://edstem.org/courses/5205/discussion/385710

The University of Sydney Page 10

 COMP5329 Reference book

Goodfellow I J, Bengio Y, Courville A, Deep Learning.

MIT Press, 2016.

LeCun Y, Bengio Y, Hinton G. Deep learning[J]. nature,

2015, 521(7553): 436-444.

• We will use PyTorch in labs. PyTorch documentation can be

found at https://pytorch.org/docs/stable/index.html

The University of Sydney Page 11

 COMP5329 Reference book
Applied Math and Machine Learning Basics:

Goodfellow I J, Bengio Y, Courville A, Deep Learning. MIT Press, 2016.

The University of Sydney Page 12
 Assessment Overview

– Assignment 1: 20%
– Week-7 (23-April) 18:00
– Groups of 2 or 3 students
– Classification task

– Assignment 2: 20%
– Week-12 (28-May) 18:00
– Groups of 2 or 3 students for Competition Track
– Groups of 2 or 3 students for Research Track

– Final exam: 60%

– date to be defined

The University of Sydney Page 13

 Call for research track abstracts for Assignment 2
(see Canvas)
Ø This research track calls for brave new ideas on deep learning. In this
track, you are encouraged to propose and investigate new algorithms
or problems in deep learning.

Ø You must not use the project (e.g. your capstone or SSP project) that
you have already done or are currently doing in other units to
participate in this track.

Ø An extended abstract (up to one page) to highlight the importance

and novelty of your research problem, the major idea of your
candidate solution, the feasibility of the proposed research, and the
expected experimental achievements (if applicable).
Ø 这⼀研究⽅向呼吁对深度学习提出⼤胆的新想法。在本课程中，⿎励您提出并研究深度学习中的新算法或问题。
Ø你不能使⽤你已经完成或正在其他单元进⾏的项⽬(例如你的顶点项⽬或SSP项⽬)来参与这个轨道。

Ø⼀个扩展摘要(最多⼀⻚)，以突出你的研究问题的重要性和新颖性，你的候选解决⽅案的主要思想，拟议研究的可⾏性，和预期的实验成果(如果适⽤)。

Ø 10 groups will be selected.

Ø将选择10组。

Ø摘要提交截⽌⽇期:第4周(3⽉26⽇)

Ø摘要决定:第6周(4⽉16⽇)

Ø Abstract Submission Deadline: Week 4 (26 March)

Ø Abstract Decision: Week 6 (16 April)
The University of Sydney Page 14
 Expectations

– Students attend scheduled classes, and devote an extra 6-9 hrs

per week
– doing assessments - 学⽣参加预定的课程，每周额外投⼊6-9⼩时
——做评估

– preparing and reviewing for classes -准备和复习课程

-修改和整合思想

-练习和⾃我评估

– revising and integrating the ideas -学⽣是负责任的学习者

建设性地参与课堂活动
•尊重彼此(批评想法，⽽不是⼈)

– practice and self-assess

•谦逊:没有⼈知晓⼀切;我们每个⼈都知道有价值的东⻄

-检查画布⽹站⾄少每周⼀次!

-有困难时通知学者

-及时、诚实地通知集团合作伙伴遇到的困难

– Students are responsible learners

– Participate in classes, constructively
• Respect for one another (criticize ideas, not people)
• Humility: none of us knows it all; each of us knows valuable things
– Check Canvas site at least once a week!
– Notify academics whenever there are difficulties
– Notify group partners honestly and promptly about difficulties

The University of Sydney Page 15

 Special Consideration (University policy)

– If your performance on assessments is affected by illness or

misadventure
– Follow proper bureaucratic procedures
– Have professional practitioner sign special USyd form
– Submit application for special consideration online, upload scans
– Note you have only a quite short deadline for applying
– http://sydney.edu.au/current_students/special_consideration/
– Also, notify coordinator by email as soon as anything begins
to go wrong
– There is a similar process if you need special arrangements
eg for religious observance, military service, representative
sports
The University of Sydney Page 16
 Late assessments in COMP5329

– Suppose you hand in work after the deadline:

– If you have not been granted special consideration or
arrangements
– A penalty of 5% of the maximum marks will be taken per day (or part)
late. After ten days, you will be awarded a mark of zero.
– e.g. If an assignment is worth 40% of the final mark and you are
one hour late submitting, then the maximum marks possible
would be 38%.
– e.g. If an assignment is worth 40% of the final mark and you are
28 hours late submitting, then the maximum marks possible
marks would be 36%.
– Warning: submission sites get very slow near deadlines
– Submit early; you can resubmit if there is time before the deadline

The University of Sydney Page 17

 Academic Integrity (University policy)

– “The University of Sydney is unequivocally opposed to, and intolerant of,

plagiarism and academic dishonesty.
– Academic dishonesty means seeking to obtain or obtaining
academic advantage for oneself or for others (including in the
assessment or publication of work) by dishonest or unfair means.
– Plagiarism means presenting another person’s work as one’s own
work by presenting, copying or reproducing it without appropriate
acknowledgement of the source.” [from site below]
– http://sydney.edu.au/elearning/student/EI/index.shtml
– Submitted work is compared against other work (from students, the
internet, etc)
– TurnitIn for textual tasks (through Canvas), other systems for code
– Penalties for academic dishonesty or plagiarism can be severe
– Complete self-education AHEM1001 (required to pass INFOxxxx)

The University of Sydney Page 18

 COMP5329: Resources

– Canvas: https://canvas.sydney.edu.au/
– Login using Unikey and password
– Link to Units website: https://sydney.edu.au/units/
• Official schedule, list of learning outcomes, etc
– Copies of slides
– Lab instructions
– Assignment instructions
– Lecture videos
– Submit official assignment work here;
– see your grades; etc

– Discussion forum: on edstem

The University of Sydney Page 19
WHS Induction

School of Computer Science

The University of Sydney Page 20

 General Housekeeping – Use of Labs

– Keep work area clean and orderly

– Remove trip hazards around desk area
– No food and drink near machines
– No smoking permitted within University buildings
– Do not unplug or move equipment without permission

The University of Sydney Page 21

 EMERGENCIES – Be prepared

https://sydney.edu.au/campus-life/safety-security.html

The University of Sydney Page 22

 EMERGENCIES

Assembly Area:
Cadigal Green

The University of Sydney Page 23

 EMERGENCIES

The University of Sydney Page 24

l
MEDICAL EMERGENCY

– If a person is seriously ill/injured:

1. call an ambulance 0-000
2. notify the closest Nominated First Aid Officer
If unconscious– send for Automated External Defibrillator (AED)
AED locations.
NEAREST to CS Building (J12)
- Electrical Engineering Building, L2 (ground) near lifts
- Seymour Centre, left of box office
- Carried by all Security Patrol vehicles
3. call Security - 9351-3333
4. Facilitate the arrival of Ambulance Staff (via Security)

Nearest Medical Facility

University Health Service in Level 3, Wentworth Building

First Aid kit – SIT Building (J12)

kitchen area adjacent to Lab 110
The University of Sydney Page 25
 School of Computer Science Safety Contacts
CHIEF WARDEN
Greg Ryan
Level 1W 103
9351 4360
0411 406 322

FIRST AID OFFICERS

OR
Julia Ashworth
Level 2E Reception Undergraduates: to Katie Yang
9351 3423 9351 4918
Coursework
Will Calleja Postgraduates: to Cecille Faraizi
Level 1W 103
9351 6060
9036 9706
0422 001 964 or Keiko Narushima
8627 0872
Katie Yang CS School
Level 2E 237
Manager: Priyanka Magotra
9351 4918
8627 4295

The University of Sydney Page 26

 Assistance

– There are a wide range of support services available for

students: https://sydney.edu.au/campus-life/health-
wellbeing-success.html
– Please make contact, and get help
– You are not required to tell anyone else about this
– If you are willing to inform the unit coordinator, they may be
able to work with other support to reduce the impact on this
unit
– eg provide advice on which tasks are most significant

The University of Sydney Page 27

 DISABILITY SERVICES

Do you have a disability?

– You may not think of yourself as having a ‘disability’ but the definition under the
Disability Discrimination Act is broad and includes temporary or chronic medical
conditions, physical or sensory disabilities, psychological conditions and learning
disabilities.
– The types of disabilities we see include:
– anxiety, arthritis, asthma, asperger's disorder, ADHD, bipolar disorder, broken bones, cancer, cerebral palsy,
chronic fatigue syndrome, crohn’s disease, cystic fibrosis, depression, diabetes, dyslexia, epilepsy, hearing
impairment, learning disability, mobility impairment, multiple sclerosis, post traumatic stress, schizophrenia , vision
impairment, and much more.

– Students needing assistance must register with Disability Services –

– it is advisable to do this as early as possible.
– http://sydney.edu.au/study/academic-support/disability-support.html

The University of Sydney Page 28

 Do you have a disability?
You may not think of yourself as having a ‘disability’ but
the definition under the Disability Discrimination Act
(1992) is broad and includes temporary or chronic
medical conditions, physical or sensory disabilities,
psychological conditions and learning disabilities.

The types of disabilities we see include:

Anxiety // Arthritis // Asthma // Autism // ADHD
Bipolar disorder // Broken bones // Cancer
Cerebral palsy // Chronic fatigue syndrome
Crohn’s disease // Cystic fibrosis // Depression
Diabetes // Dyslexia // Epilepsy // Hearing impairment //
Learning disability // Mobility impairment // Multiple
sclerosis // Post-traumatic stress // Schizophrenia //
Vision impairment
and much more.

Students needing assistance must register with

Disability Services. It is advisable to do this as early as
possible. Please contact us or review our website to
find out more.

Disability Services Office

sydney.edu.au/disability
The University of Sydney
02-8627-8422 Page 29
 Other support

– Learning support
– http://sydney.edu.au/study/academic-support/learning-support.html
– International students
– http://sydney.edu.au/study/academic-support/support-for-international-students.html
– Aboriginal and Torres Strait Islanders
– http://sydney.edu.au/study/academic-support/aboriginal-and-torres-strait-islander-
support.html
– Student organization (can represent you in academic appeals etc)
– http://srcusyd.net.au/ or http://www.supra.net.au/
– Please make contact, and get help
– You are not required to tell anyone else about this
– If you are willing to inform the unit coordinator, they may be able to work
with other support to reduce the impact on this unit
– eg provide advice on which tasks are most significant
The University of Sydney Page 30
 Advice

– Metacognition
– Pay attention to the learning outcomes in Canvas
– Self-check that you are achieving each one
– Think how each assessment task relates to these
– Time management
– Watch the due dates
– Start work early, submit early
– Networking and community-formation
– Make friends and discuss ideas with them
– Know your tutor, lecturer, coordinator
– Keep them informed, especially if you fall behind
• Don’t wait to get help
– Enjoy the learning!

The University of Sydney Page 31

Invited talks

- Xikun Zhang - "On Dropping Clusters to Regularize Graph

Convolutional Neural Networks"

- Kwon Nung Choi - "VideoTrain: A Generative Adversarial

Framework for Synthetic Video Traffic Generation"

- Xinqi Zhu - "Learning Disentangled Representations with Latent

Variation Predictability"

- Yanxi Li - "Adapting Neural Architectures Between Domains"

The University of Sydney Page 32