Chapter 5

Augmented Reality (AR)

1
 The fundamental idea of AR is to combine, or mix,
the view of the real environment with additional,
virtual content that is presented through computer
graphics.
 Its convincing effect is achieved by ensuring that the
virtual content is aligned and registered with the
real objects.
2
As a person moves in an environment and
their perspective view of real objects changes,
the virtual content should also be presented
from the same perspective

3
Augmented reality (AR) is a form of emerging
technology that allows users to overlay computer-
generated content in the real world.
AR refers to a live view of a physical real-world
environment whose elements are merged with
augmented computer-generated images creating a
mixed reality.
4
⁑ The augmentation is typically done in real-time and in
semantic context with environmental elements.
⁑ By using the latest AR techniques and technologies, the
information about the surrounding real world becomes
interactive and digitally usable.
⁑ Through this augmented vision, a user can digitally interact
with and adjust information about their surrounding
environment.
5
↓ Augmented Reality (AR) as a real-time direct or
indirect view of a physical real-world environment
that has been enhanced/augmented by adding
virtual computer-generated information to it.
↓ Augmented reality is the integration of digital
information with the user's environment in real-
time.
6
◊ Unlike virtual reality, which creates a totally artificial

environment, augmented reality uses the existing

environment and overlays new information on top of it.
◊ A live direct or indirect view of a physical, real-world

environment whose elements are augmented by

computer-generated sensory input such as sound, video,
graphics or GPS data.
7
5.2. Virtual reality (VR), Augmented Reality (AR) vs Mixed
reality (MR)
► With constant development in computer vision and the exponential
advancement of computer processing power, virtual reality (VR),
augmented reality (AR), and mixed reality (MR) technology is becoming
more and more prominent.
► With some overlap in the applications and functions of these emerging
technologies, sometimes these terms get confused or are used
incorrectly.
► The main differences between them are explained below (see Figure 5.1).
8
9
 5.2.1. Virtual Reality (VR)
↔VR is fully immersive, which tricks your senses into thinking
you’re in a different environment or world apart from the real
world.
↔Using a head-mounted display (HMD) or headset, you’ll
experience a computer-generated world of imagery and
sounds in which you can manipulate objects and move around
using haptic controllers while tethered to a console or PC.

10
 It is also called a computer-simulated reality.
It refers to computer technologies using
reality headsets to generate realistic sounds,
images and other sensations that replicate a
real environment or create an imaginary
world.

11
Advanced VR environment will engage all five senses
(taste, sight, smell, touch, sound), but it is important
to say that this is not always possible.
 It refers to computer technologies using reality
headsets to generate realistic sounds, images and
other sensations that replicate a real environment or
create an imaginary world.

12
• Advanced VR environment will engage all five
senses (taste, sight, smell, touch, sound), but
it is important to say that this is not always
possible
• Using VR devices such as HTC Vive, Oculus Rift
or Google Cardboard, users can be
transported into a number of real-world and
imagined environments.

13
 The most advanced VR experiences even provide freedom of
movement – users can move in a digital environment and
hear sounds.
 Moreover, special hand controllers can be used to enhance
VR experiences.

14
⌂ Most VR headsets are connected to a computer (Oculus
Rift) or a gaming console (PlayStation VR) but there are
standalone devices (Google Cardboard is among the most
popular) as well.
⌂ Most standalone VR headsets work in combination with
smart phones – you insert a smart phone, wear a headset,
and immerse in the virtual reality (see Figure 5.3).

15
16
17
 5.2.2. Augmented Reality (AR)
 In augmented reality, users see and interact with the real
world while digital content is added to it.
 If you own a modern smart phone, you can easily download
an AR app and try this technology.

18
There’s a different way to experience augmented reality,
though – with special AR headsets, such as Google Glass,
where digital content is displayed on a tiny screen in front of
a user’s eye.
AR adds digital elements to a live view often by using the
camera on a smart phone.
Examples of augmented reality experiences include Snap chat
lenses and the game Pokemon Go.
19
 Augmented Reality (AR) is a live, direct or indirect view of a
physical, real-world environment whose elements are
augmented (or supplemented) by computer-generated
sensory input such as sound, video, graphics or GPS data
(see Figure 5.4).

20
21
22
 5.2.3. Mixed Reality (MR)
 Mixed Reality (MR), sometimes referred to as hybrid reality, is
the merging of real and virtual worlds to produce new
environments and visualizations where physical and digital
objects co-exist and interact in real-time.

23
 It means placing new imagery within a real space in such a
way that the new imagery is able to interact, to an extent,
with what is real in the physical world we know

24
 For example, with MR, you can play a virtual video game,
grab your real-world water bottle, and smack an imaginary
character from the game with the bottle.
 MR is that the synthetic content and the real-world content
are able to react to each other in real-time.

25
•

26
 In mixed reality, you interact with and manipulate both
physical and virtual items and environments, using next-
generation sensing and imaging technologies.
 MR allows you to see and immerse yourself in the world
around you even as you interact with a virtual environment
using your own hands—all without ever removing your
headset.

27
28
 It provides the ability to have one foot (or hand) in the real
world, and the other in an imaginary place, breaking down
basic concepts between real and imaginary, offering an
experience that can change the way you game and work
today.

29
 One of the most obvious differences among augmented
reality, virtual reality, and mixed reality is the hardware
requirements and also VR is content which is 100% digital
and can be enjoyed in a fully immersive environment, AR
overlays digital content on top of the real-world.
 and MR is a digital overlay that allows interactive virtual
elements to integrate and interact with the real-world
environment.
30
 Numerous augmented reality apps and games can run on
almost every smart phone on the market.
 On the other hand, virtual reality programs require
specialized VR headsets, noise-canceling headphones,
cameras to track room space and boundaries, and sometimes
even motion capture technology.

31
 Some of the biggest names in VR tech today are the
Oculus Rift, HTC Vive, and PlayStation VR.
 For the enjoyment of simple VR videos, there are
affordable makeshift VR headsets like the Google
Cardboard, which work by running a video in 360 modes
on your smart phone and inserting the phone into the
headset.

32
 --
Mixed reality hardware is still emerging and hasn’t quite broken into the
mainstream consumer market, most likely due to the price.
The consumer releases of the Microsoft HoloLens and Magic Leap One
retail for over $2000 USD, which is 3 to 4 times the cost of the
PlayStation VR and HTC Vive VR headsets.
However, mixed reality applications sometimes require exponentially
more processing power and thus require more powerful hardware.

33
 -
For example, the Microsoft HoloLens includes a built-in
microphone array, binaural sound capabilities, a built-in
camera for recording, a depth sensor, head-tracking cameras,
and an inertial measurement unit which helps track your head
movement.
On top of the traditional CPU and GPU, Microsoft also created
a Holographic Processing Unit to help track where the user is
looking and understand command gestures.
34
 5.3. The architecture of AR Systems
 The first Augmented Reality Systems (ARS) were usually designed with a
basis on three main blocks, as is illustrated in Figure 5.7: (1) Infrastructure
Tracker Unit, (2) Processing Unit, and (3) Visual Unit.
 The Infrastructure Tracker Unit was responsible for collecting data from
the real world, sending them to the Processing Unit, which mixed the
virtual content with the real content and sent the result to the Video Out
module of the Visual Unit.
 Some designs used a Video In, to acquire required data for the
Infrastructure Tracker Unit.
35
36
 The Visual Unit can be classified into two types of system,
depending on the followed visualization technology:
1. Video see-through: It uses a Head-Mounted Display (HMD)
that employs a video-mixing and displays the merged images
on a closed-view HMD.

37
2. Optical see-through: It uses an HMD that employs optical
combiners to merge the images within an open-view HMD.
 HMDs are currently the dominant display technology in the

AR field.
 However, they lack in several aspects, such as ergonomics,

high prices and relatively low mobility due to their sizes

and connectivity features.
38
 An additional problem involving HMD is the interaction with
the real environment, which places virtual interactive zones to
the user, making the collision with these zones hard due to
the difficulty to interact with multiple points in different
depths.
 Alternative approaches to developing ARS involve the use of
monitors and tablets.
 Monitors are used as an option for indirect view since the
user does not look directly into the mixed world.
 Tablets are used in direct view since the user points the
camera to the scene and loo

39
 5.4. Applications of AR Systems
 Technology is ever-changing and ever-growing.
 One of the newest developing technologies is augmented
reality (AR), which can be applied to many different disciplines
such as education, medicine, entertainment, military, etc.
 Let us see some of its applications:

40
 5.4.1. AR In education
 Augmented reality allows flexibility in use that is attractive to education.
 AR technology can be utilized through a variety of mediums including
desktops, mobile devices, and smart phones.
 The technology is portable and adaptable to a variety of scenarios.
 AR can be used to enhance content and instruction within the
traditional classroom, supplement instruction in the special
education classroom, extend content into the world outside
the classroom, and be combined with other technologies to
enrich their individual applications.

41
More importantly, the following reasons for using
augmented reality in education:
Affordable learning materials - posters, digital
illustrations, physical models, prototypes are very
expensive and it’s impossible for schools to find
enough money to purchase all the supplementary
materials they would like to.
42
 Using AR technology allows for avoiding investments
in physical materials.
 Besides, students can get access to learning
materials and interact with them anytime.
 Interactive lessons - when AR technology is used in
classrooms, students can view models on their own
smart phones and get a better idea of the concepts
they are studying.
 That increases engagements and reinforces the
learning.
43
Higher engagement - when teachers integrate augmented
reality into their lectures, they attract the attention of their
students and make lessons more effective.
When students are interested, it is much easier to make them
work more productively.

44
 Higher retention - using the AR app, students can get

access to augmented models that represent any real

objects from a famous monument or work of art to a
molecule. Besides, students can get access to a website
with specific information.
 When learning with AR technology, students use different

senses and retain more knowledge for a long time.

45
Boost intellectual curiosity - augmented reality makes students more
excited about learning certain subjects.
 Modern students were born in a digital era so they will always be
excited with innovative technologies that can help them learn new
ideas and develop their critical thinking skills.
 When using AR technology in the classroom, teachers can create an
authentic learning environment for students with different learning
styles.

46
 5.4.2. AR In Medicine
 The purpose of any invention and technology is to simplify our life.
 Augmented reality has the potential to play a big role in improving the
healthcare industry.
 Only a few years since the first implementations of augmented reality in
medicine, it has already filled an important place in doctors’ and nurses’
routine, as well as patients’ lives.
 This new technology is enhancing medicine and healthcare towards more
safety and efficiency.

47
For now, augmented reality has already made significant changes
in the following medical areas:
 surgery (minimally invasive surgery);
 education of future doctors;
 diagnostics;
 AR tools may also aid to detect the signs of depression and
other mental illnesses by reading from facial expressions,
voice tones, and physical gestures.
48
In medicine, AR has the following applications:
1) Describing symptoms - Have you ever been in a situation when
it was hard to describe to the doctor what was bothering you? It
is a common problem for all us, the roots of which extend to
overreacting and lack of knowledge.
And what is most important, it impacts on finding out the
accurate diagnosis.

49
 The first steps to find the solutions are already made.

 To increase patients’ education, medical app AyeDecide

is using augmented reality to show the simulation of

the vision, harmed by the different diseases.
 It helps patients to understand their conditions and

describe correctly their symptoms.

50
2) Nursing care - About 40% of the first intravenous

injections fail, and this ratio is even higher in the case of

children and elderly patients.
The AccuVein uses augmented reality to cope with this
negative statistic.

51
A handheld scanner projects on the skin and shows
the patients’ veins.
 It increases the successful finding of the vein from the
first try in 3,5 times.
That is why this invention got the greatest recognition
among the general public and medical staff .

52
3) Surgery - In no sphere augmented reality does not have such
practical application as in the medicine, especially in surgery,
where it literally helps to save lives.
Three-dimensional reconstructions of organs or tumors will help
surgeons become more efficient at surgery operations.

53
 For example, spinal surgery, as usually, is a long and difficult

process.
 But with the use of AR, it can reduce the time, cut the risks and

improve the results.

 The Israeli startup Augmedics had created an augmented reality

headset for spine surgeons.

 This technology overlays a 3D model of the CT-scan on the spine,

so, the surgeon gets some kind of “X-ray” vision.

54
4) Ultrasounds - Some time ago ultrasound made a small
revolution in medicine.
 Today, it has another one chance to make the same with using
augmented reality.
 Already a few AR software companies developed handy
ultrasound scanner, which with the help of smart glasses
works as a traditional one.

55
5) Diabetes management - In 2017, the number of people struggle with
diabetes reached up to 425 million adults worldwide.
 And the amount of diagnosed people is increasing every year.
 In 2014, Google revealed the plans for creating a smart contact lens
(Google Contact Lens), in which the main function will be to measure the
glucose levels in the tears.
 It will help people with this disease to live the life they used to, without
permanent worries about sugar level in the blood.

56
6) Navigation - The using AR in navigation apps has already
become a “traditional” way.
 By pointing your phone to the city landscape, you get the
information about nearby objects of your interest (museums,
hotels, shops, metro stations, etc.).

57
 The same way AR can be useful to provide information about
the nearest hospitals.
 For example, the EHBO app helps to find the nearest to you
AEDs (automated external defibrillators).

58
Generally, AR provides the following benefits to patients and
healthcare workers:
 Reduce the risks associated with minimally invasive surgery.
 Better informed decisions about the right treatment and
illness prevention.
 Make procedures more tolerable.
 Better aftercare
 Medical training and education.
 Assistance in medical procedures and routine tasks.

59
 5.4.3. AR In Entertainment

 Augmented reality can be used in various

“entertainment” industries as entertainment covers
quite a number of different industries – music,
movies, live shows, games – and all of them can
benefit from using augmented reality.

60
 AR in games

 the AR games were praised for increasing physical activity in

people you actually have to move around to find your target, for

example, Pokémon.

 At the same time, there are complaints that players could cause

various incidents and accidents being too engrossed in the game.

 In any case, Pokémon GO has rightfully earned its popularity and

opened the world of AR games to us.

61
 AR in music
 music is not only about listening to favorite tracks and

putting together playlists.

 When we like a piece, we often want to find out more

about its background: the performers’ bios, the lyrics of

the song, the making of the recording or the music video .

62
 Augmented reality can do all that and much more providing
complete information on the track or its performer.
 Augmented reality can enhance live performances by
illustrating the story told by a track or displaying the way it
was created by the band.

63
 AR on TV
 this may seem a bit far-fetched, as television already shows a virtual
world, thus adding augmented reality will raise it to the second power.
 However, some experiments of fusing augmented reality on TV are already
being made with the promise of future enhancements.
 One way of integrating augmented reality in television is adding
supplementary information to what is going on the TV screen – such as
match scores, betting options, and the like

64
 AR in Sports
 Recently, the industry of Sports has been gaining popularity in
all parts of the globe.
 Competitive online gaming has become as fascinating as
real sports, and the technology is following it closely with
new solutions and unusual implementations.
 Augmented reality turns eSports shows into interactive
experiences allowing the watchers to become participants.
65
 AR in the theater
 in this sector, augmented reality can serve not only for
entertainment purposes but also for the purposes of
accessibility.
 The possibility to overlay virtual objects over the real
environment can be used, for example, for subtitling in
various theater shows.

66
 Now, many theaters use LED displays either to

provide subtitles for translation or to assist

hearing-impaired visitors.
 However, LED equipment is not available in each

theater and even when it is, it can distract the

viewers from the show.
67