2023

AI-Course Study Material

Prescribed Syllabus:

[1]
[2]
What is AI?
Simply put, AI is the creation of software that imitates human behaviors and capabilities.
Key workloads include:

 Machine learning - This is often the foundation for an AI system, and is the way
we "teach" a computer model to make prediction and draw conclusions from data.
 Anomaly detection - The capability to automatically detect errors or unusual
activity in a system.
 Computer vision - The capability of software to interpret the world visually
through cameras, video, and images.
 Natural language processing - The capability for a computer to interpret written
or spoken language, and respond in kind.
 Knowledge mining - The capability to extract information from large volumes of
often unstructured data to create a searchable knowledge store.

Understand machine learning

Machine Learning is the foundation for most AI solutions.

Let's start by looking at a real-world example of how machine learning can be used to
solve a difficult problem.

[3]
Sustainable farming techniques are essential to maximize food production while
protecting a fragile environment. The Yield, an agricultural technology company based
in Australia, uses sensors, data and machine learning to help farmers make informed
decisions related to weather, soil and plant conditions.

How machine learning works

So how do machines learn?

The answer is, from data. In today's world, we create huge volumes of data as we go
about our everyday lives. From the text messages, emails, and social media posts we
send to the photographs and videos we take on our phones, we generate massive
amounts of information. More data still is created by millions of sensors in our homes,
cars, cities, public transport infrastructure, and factories.

Data scientists can use all of that data to train machine learning models that can make
predictions and inferences based on the relationships they find in the data.

For example, suppose an environmental conservation organization wants volunteers to

identify and catalog different species of wildflower using a phone app. The following
animation shows how machine learning can be used to enable this scenario.

1. A team of botanists and scientists collect data on wildflower samples.

2. The team labels the samples with the correct species.
3. The labeled data is processed using an algorithm that finds relationships between
the features of the samples and the labeled species.
4. The results of the algorithm are encapsulated in a model.
5. When new samples are found by volunteers, the model can identify the correct
species label.

Machine learning in Microsoft Azure

Microsoft Azure provides the Azure Machine Learning service - a cloud-based platform
for creating, managing, and publishing machine learning models. Azure Machine
Learning provides the following features and capabilities:

Feature Capability

[4]
Feature Capability
Automated machine This feature enables non-experts to quickly create an effective machine
learning learning model from data.
Azure Machine Learning A graphical interface enabling no-code development of machine learning
designer solutions.
Data and compute Cloud-based data storage and compute resources that professional data
management scientists can use to run data experiment code at scale.
Pipelines Data scientists, software engineers, and IT operations professionals can define
pipelines to orchestrate model training, deployment, and management tasks.

Understand anomaly detection

Imagine you're creating a software system to monitor credit card transactions and
detect unusual usage patterns that might indicate fraud. Or an application that tracks
activity in an automated production line and identifies failures. Or a racing car telemetry
system that uses sensors to proactively warn engineers about potential mechanical
failures before they happen.

These kinds of scenario can be addressed by using anomaly detection - a machine

learning based technique that analyzes data over time and identifies unusual changes.

Let's explore how anomaly detection might help in the racing car scenario.

1. Sensors in the car collect telemetry, such as engine revolutions, brake temperature,
and so on.
2. An anomaly detection model is trained to understand expected fluctuations in the
telemetry measurements over time.
3. If a measurement occurs outside of the normal expected range, the model reports
an anomaly that can be used to alert the race engineer to call the driver in for a pit
stop to fix the issue before it forces retirement from the race.

Anomaly detection in Microsoft Azure

In Microsoft Azure, the Anomaly Detector service provides an application
programming interface (API) that developers can use to create anomaly detection
solutions.

[5]
Understand computer vision

Computer Vision is an area of AI that deals with visual processing. Let's explore some of
the possibilities that computer vision brings.

The Seeing AI app is a great example of the power of computer vision. Designed for the
blind and low vision community, the Seeing AI app harnesses the power of AI to open
up the visual world and describe nearby people, text and objects.

Computer Vision models and capabilities

Most computer vision solutions are based on machine learning models that can be
applied to visual input from cameras, videos, or images. The following table describes
common computer vision tasks.

Task Description

Image classification

Image classification involves training a machine learning model to classify images based on
their contents. For example, in a traffic monitoring solution you might use an image
classification model to classify images based on the type of vehicle they contain, such as
taxis, buses, cyclists, and so on.

[6]
Task Description

Object detection

Object detection machine learning models are trained to classify individual objects within an
image, and identify their location with a bounding box. For example, a traffic monitoring
solution might use object detection to identify the location of different classes of vehicle.

Semantic
segmentation

Semantic segmentation is an advanced machine learning technique in which individual pixels

in the image are classified according to the object to which they belong. For example, a
traffic monitoring solution might overlay traffic images with "mask" layers to highlight
different vehicles using specific colors.

[7]
Task Description

Image analysis

You can create solutions that combine machine learning models with advanced image
analysis techniques to extract information from images, including "tags" that could help
catalog the image or even descriptive captions that summarize the scene shown in the
image.

Face detection,
analysis, and
recognition

Face detection is a specialized form of object detection that locates human faces in an
image. This can be combined with classification and facial geometry analysis techniques to
recognize individuals based on their facial features.

[8]
Task Description

Optical character
recognition (OCR)

Optical character recognition is a technique used to detect and read text in images. You can
use OCR to read text in photographs (for example, road signs or store fronts) or to extract
information from scanned documents such as letters, invoices, or forms.

Computer vision services in Microsoft Azure

Microsoft Azure provides the following cognitive services to help you create computer
vision solutions:

Service Capabilities

Computer Vision You can use this service to analyze images and video, and extract descriptions, tags, objects,
and text.

Custom Vision Use this service to train custom image classification and object detection models using your
own images.

Face The Face service enables you to build face detection and facial recognition solutions.

Form Recognizer Use this service to extract information from scanned forms and invoices.

[9]
Understand natural language processing

Natural language processing (NLP) is the area of AI that deals with creating software
that understands written and spoken language.

NLP enables you to create software that can:

 Analyze and interpret text in documents, email messages, and other sources.
 Interpret spoken language, and synthesize speech responses.
 Automatically translate spoken or written phrases between languages.
 Interpret commands and determine appropriate actions.

For example, Starship Commander, is a virtual reality (VR) game from Human Interact,
that takes place in a science fiction world. The game uses natural language processing
to enable players to control the narrative and interact with in-game characters and
starship systems.

Natural language processing in Microsoft Azure

In Microsoft Azure, you can use the following cognitive services to build natural
language processing solutions:

Service Capabilities

Language Use this service to access features for understanding and analyzing text, training language models
that can understand spoken or text-based commands, and building intelligent applications.

Translator Use this service to translate text between more than 60 languages.

Speech Use this service to recognize and synthesize speech, and to translate spoken languages.

Azure Bot This service provides a platform for conversational AI, the capability of a software "agent" to
participate in a conversation. Developers can use the Bot Framework to create a bot and manage it
with Azure Bot Service - integrating back-end services like Language, and connecting to channels for
web chat, email, Microsoft Teams, and others.

[10]
Understand knowledge mining

Knowledge mining is the term used to describe solutions that involve extracting
information from large volumes of often unstructured data to create a searchable
knowledge store.

Knowledge mining in Microsoft Azure

One of these knowledge mining solutions is Azure Cognitive Search, a private,
enterprise, search solution that has tools for building indexes. The indexes can then be
used for internal only use, or to enable searchable content on public facing internet
assets.

Azure Cognitive Search can utilize the built-in AI capabilities of Azure Cognitive Services
such as image processing, content extraction, and natural language processing to
perform knowledge mining of documents. The product's AI capabilities makes it
possible to index previously unsearchable documents and to extract and surface insights
from large amounts of data quickly.

Challenges and risks with AI

Artificial Intelligence is a powerful tool that can be used to greatly benefit the world.
However, like any tool, it must be used responsibly.

The following table shows some of the potential challenges and risks facing an AI
application developer.

Challenge or Risk Example

Bias can affect results A loan-approval model discriminates by gender due to bias
in the data with which it was trained
Errors may cause harm An autonomous vehicle experiences a system failure and
causes a collision
Data could be exposed A medical diagnostic bot is trained using sensitive patient
data, which is stored insecurely
Solutions may not work for A home automation assistant provides no audio output for
everyone visually impaired users
Users must trust a complex system An AI-based financial tool makes investment
recommendations - what are they based on?
Who's liable for AI-driven An innocent person is convicted of a crime based on
decisions? evidence from facial recognition – who's responsible

[11]
Understand Responsible AI

At Microsoft, AI software development is guided by a set of six principles, designed to

ensure that AI applications provide amazing solutions to difficult problems without any
unintended negative consequences.

Fairness
AI systems should treat all people fairly. For example, suppose you create a machine
learning model to support a loan approval application for a bank. The model should
predict whether the loan should be approved or denied without bias. This bias could be
based on gender, ethnicity, or other factors that result in an unfair advantage or
disadvantage to specific groups of applicants.

Azure Machine Learning includes the capability to interpret models and quantify the
extent to which each feature of the data influences the model's prediction. This
capability helps data scientists and developers identify and mitigate bias in the model.

Another example is Microsoft's implementation of Responsible AI with the Face service,

which retires facial recognition capabilities that can be used to try to infer emotional
states and identity attributes. These capabilities, if misused, can subject people to
stereotyping, discrimination or unfair denial of services.

Reliability and safety

AI systems should perform reliably and safely. For example, consider an AI-based
software system for an autonomous vehicle; or a machine learning model that
diagnoses patient symptoms and recommends prescriptions. Unreliability in these kinds
of systems can result in substantial risk to human life.

AI-based software application development must be subjected to rigorous testing and

deployment management processes to ensure that they work as expected before
release.

Privacy and security

AI systems should be secure and respect privacy. The machine learning models on which
AI systems are based rely on large volumes of data, which may contain personal details

[12]
that must be kept private. Even after the models are trained and the system is in
production, privacy and security need to be considered. As the system uses new data to
make predictions or take action, both the data and decisions made from the data may
be subject to privacy or security concerns.

Inclusiveness
AI systems should empower everyone and engage people. AI should bring benefits to all
parts of society, regardless of physical ability, gender, sexual orientation, ethnicity, or
other factors.

Transparency
AI systems should be understandable. Users should be made fully aware of the purpose
of the system, how it works, and what limitations may be expected.

Accountability
People should be accountable for AI systems. Designers and developers of AI-based
solutions should work within a framework of governance and organizational principles
that ensure the solution meets ethical and legal standards that are clearly defined.

[13]
Summary

Artificial Intelligence enables the creation of powerful solutions to many kinds of

problems. AI systems can exhibit human characteristics to analyze the world around
them, make predictions or inferences, and act on them in ways that we could only
imagine a short time ago.

With this power, comes responsibility. As developers of AI solutions, we must apply

principles that ensure that everyone benefits from AI without disadvantaging any
individual or section of society.

[14]
Microsoft Azure AI Fundamentals: Explore visual tools for machine
learning

Introduction

Machine Learning is the foundation for most artificial intelligence solutions. Creating an
intelligent solution often begins with the use of machine learning to train predictive
models using historic data that you have collected.

Azure Machine Learning is a cloud service that you can use to train and manage machine
learning models.

In this module, you'll learn to:

 Identify the machine learning process.

 Understand Azure Machine Learning capabilities.
 Use automated machine learning in Azure Machine Learning studio to train and
deploy a predictive model.

What is machine learning?

Machine learning is a technique that uses mathematics and statistics to create a model
that can predict unknown values.

[15]
For example, suppose Adventure Works Cycles is a business that rents cycles in a city.
The business could use historic data to train a model that predicts daily rental demand
in order to make sure sufficient staff and cycles are available.

To do this, Adventure Works could create a machine learning model that takes
information about a specific day (the day of week, the anticipated weather conditions,
and so on) as an input, and predicts the expected number of rentals as an output.

Mathematically, you can think of machine learning as a way of defining a function (let's
call it f) that operates on one or more features of something (which we'll call x) to
calculate a predicted label (y) - like this:

f(x) = y

In this bicycle rental example, the details about a given day (day of the week, weather,
and so on) are the features (x), the number of rentals for that day is the label (y), and the
function (f) that calculates the number of rentals based on the information about the
day is encapsulated in a machine learning model.

The specific operation that the f function performs on x to calculate y depends on a

number of factors, including the type of model you're trying to create and the specific
algorithm used to train the model. Additionally in most cases, the data used to train the
machine learning model requires some pre-processing before model training can be
performed.

Types of machine learning

There are two general approaches to machine learning, supervised and unsupervised
machine learning. In both approaches, you train a model to make predictions.

The supervised machine learning approach requires you to start with a

dataset with known label values. Two types of supervised machine learning tasks include
regression and classification.

 Regression: used to predict a continuous value; like a price, a sales total, or some
other measure.
 Classification: used to determine a class label; an example of a binary class label is
whether a patient has diabetes or not; an example of multi-class labels is
classifying text as positive, negative, or neutral.

[16]
The unsupervised machine learning approach starts with a dataset without known
label values. One type of unsupervised machine learning task is clustering.

 Clustering: used to determine labels by grouping similar information into label

groups; like grouping measurements from birds into species.
What is Azure Machine Learning studio?

Training and deploying an effective machine learning model involves a lot of work,
much of it time-consuming and resource-intensive. Azure Machine Learning is a cloud-
based service that helps simplify some of the tasks it takes to prepare data, train a
model, and deploy a predictive service.

Most importantly, Azure Machine Learning helps data scientists increase their efficiency
by automating many of the time-consuming tasks associated with training models; and
it enables them to use cloud-based compute resources that scale effectively to handle
large volumes of data while incurring costs only when actually used.

Azure Machine Learning workspace

To use Azure Machine Learning, you first create a workspace resource in your Azure
subscription. You can then use this workspace to manage data, compute resources,
code, models, and other artifacts related to your machine learning workloads.

After you have created an Azure Machine Learning workspace, you can develop
solutions with the Azure machine learning service either with developer tools or the
Azure Machine Learning studio web portal.

Azure Machine Learning studio

Azure Machine Learning studio is a web portal for machine learning solutions in Azure.
It includes a wide range of features and capabilities that help data scientists prepare
data, train models, publish predictive services, and monitor their usage. To begin using
the web portal, you need to assign the workspace you created in the Azure portal to
Azure Machine Learning studio

[17]
Azure Machine Learning compute
At its core, Azure Machine Learning is a service for training and managing machine
learning models, for which you need compute on which to run the training process.

Compute targets are cloud-based resources on which you can run model training and
data exploration processes.

In Azure Machine Learning studio, you can manage the compute targets for your data
science activities. There are four kinds of compute resource you can create:

 Compute Instances: Development workstations that data scientists can use to work with
data and models.
 Compute Clusters: Scalable clusters of virtual machines for on-demand processing of
experiment code.
 Inference Clusters: Deployment targets for predictive services that use your trained
models.

[18]
  Attached Compute: Links to existing Azure compute resources, such as Virtual Machines
or Azure Databricks clusters.
What is Azure Automated Machine Learning?

Azure Machine Learning includes an automated machine learning capability that

automatically tries multiple pre-processing techniques and model-training algorithms in
parallel. These automated capabilities use the power of cloud compute to find the best
performing supervised machine learning model for your data.

Automated machine learning allows you to train models without extensive data science
or programming knowledge. For people with a data science and programming
background, it provides a way to save time and resources by automating algorithm
selection and hyperparameter tuning.

You can create an automated machine learning job in Azure Machine Learning studio.

In Azure Machine Learning, operations that you run are called jobs. You can configure
multiple settings for your job before starting an automated machine learning run. The
run configuration provides the information needed to specify your training script,

[19]
compute target, and Azure ML environment in your run configuration and run a training
job.

Understand the AutoML process

Completed100 XP

 5 minutes

You can think of the steps in a machine learning process as:

1. Prepare data: Identify the features and label in a dataset. Pre-process, or clean and
transform, the data as needed.
2. Train model: Split the data into two groups, a training and a validation set. Train a
machine learning model using the training data set. Test the machine learning model for
performance using the validation data set.
3. Evaluate performance: Compare how close the model's predictions are to the known
labels.
4. Deploy a predictive service: After you train a machine learning model, you can deploy
the model as an application on a server or device so that others can use it.

These are the same steps in the automated machine learning process with Azure
Machine Learning.
[20]
Prepare data
Machine learning models must be trained with existing data. Data scientists expend a lot
of effort exploring and pre-processing data, and trying various types of model-training
algorithms to produce accurate models, which is time consuming, and often makes
inefficient use of expensive compute hardware.

In Azure Machine Learning, data for model training and other operations is usually
encapsulated in an object called a dataset. You can create your own dataset in Azure
Machine Learning studio.

Train model
The automated machine learning capability in Azure Machine Learning
supports supervised machine learning models - in other words, models for which the

[21]
training data includes known label values. You can use automated machine learning to
train models for:

 Classification (predicting categories or classes)

 Regression (predicting numeric values)
 Time series forecasting (predicting numeric values at a future point in time)

In Automated Machine Learning you can select from several types of

tasks:

In Automated Machine Learning, you can select configurations for the primary metric,
type of model used for training, exit criteria, and concurrency limits.

[22]
Importantly, AutoML will split data into a training set and a validation set. You can
configure the details in the settings before you run the job.

[23]
Evaluate performance
After the job has finished you can review the best performing model. In this case, you
used exit criteria to stop the job. Thus the "best" model the job generated might not be
the best possible model, just the best one found within the time allowed for this
exercise.

The best model is identified based on the evaluation metric you specified, Normalized
root mean squared error.

A technique called cross-validation is used to calculate the evaluation metric. After the
model is trained using a portion of the data, the remaining portion is used to iteratively
test, or cross-validate, the trained model. The metric is calculated by comparing the
predicted value from the test with the actual known value, or label.

The difference between the predicted and actual value, known as the residuals, indicates
the amount of error in the model. The performance metric root mean squared
error (RMSE), is calculated by squaring the errors across all of the test cases, finding the
mean of these squares, and then taking the square root. What all of this means is that
smaller this value is, the more accurate the model's predictions. The normalized root

[24]
mean squared error (NRMSE) standardizes the RMSE metric so it can be used for
comparison between models which have variables on different scales.

The Residual Histogram shows the frequency of residual value ranges. Residuals
represent variance between predicted and true values that can't be explained by the
model, in other words, errors. You should hope to see the most frequently occurring
residual values clustered around zero. You want small errors with fewer errors at the
extreme ends of the scale.

The Predicted vs. True chart should show a diagonal trend in which the predicted value
correlates closely to the true value. The dotted line shows how a perfect model should
perform. The closer the line of your model's average predicted value is to the dotted
line, the better its performance. A histogram below the line chart shows the distribution
of true values.

[25]
After you've used automated machine learning to train some models, you can deploy
the best performing model as a service for client applications to use.

Deploy a predictive service

In Azure Machine Learning, you can deploy a service as an Azure Container Instances
(ACI) or to an Azure Kubernetes Service (AKS) cluster. For production scenarios, an AKS
deployment is recommended, for which you must create an inference cluster compute
target. In this exercise, you'll use an ACI service, which is a suitable deployment target
for testing, and does not require you to create an inference cluster.

[26]
Microsoft Azure AI Fundamentals: Explore computer vision
Introduction

Computer vision is one of the core areas of artificial intelligence (AI), and focuses on
creating solutions that enable AI applications to "see" the world and make sense of it.

Of course, computers don't have biological eyes that work the way ours do, but they are
capable of processing images; either from a live camera feed or from digital
photographs or videos. This ability to process images is the key to creating software that
can emulate human visual perception.

Some potential uses for computer vision include:

 Content Organization: Identify people or objects in photos and organize them based on
that identification. Photo recognition applications like this are commonly used in photo
storage and social media applications.
 Text Extraction: Analyze images and PDF documents that contain text and extract the text
into a structured format.
 Spatial Analysis: Identify people or objects, such as cars, in a space and map their
movement within that space.

To an AI application, an image is just an array of pixel values. These numeric values can
be used as features to train machine learning models that make predictions about the
image and its contents.

Training machine learning models from scratch can be very time intensive and require a
large amount of data. Microsoft's Computer Vision service gives you access to pre-
trained computer vision capabilities.

[27]
Learning objectives
In this module you will:

 Identify image analysis tasks that can be performed with the Computer Vision service.
 Provision a Computer Vision resource.
 Use a Computer Vision resource to analyze an image.
Get started with image analysis on Azure
Completed100 XP

 3 minutes

The Computer Vision service is a cognitive service in Microsoft Azure that provides pre-
built computer vision capabilities. The service can analyze images, and return detailed
information about an image and the objects it depicts.

Azure resources for Computer Vision

To use the Computer Vision service, you need to create a resource for it in your Azure
subscription. You can use either of the following resource types:

 Computer Vision: A specific resource for the Computer Vision service. Use this resource
type if you don't intend to use any other cognitive services, or if you want to track
utilization and costs for your Computer Vision resource separately.
 Cognitive Services: A general cognitive services resource that includes Computer Vision
along with many other cognitive services; such as Text Analytics, Translator Text, and
others. Use this resource type if you plan to use multiple cognitive services and want to
simplify administration and development.

Whichever type of resource you choose to create, it will provide two pieces of
information that you will need to use it:

 A key that is used to authenticate client applications.

 An endpoint that provides the HTTP address at which your resource can be accessed.
Note

If you create a Cognitive Services resource, client applications use the same key and
endpoint regardless of the specific service they are using.

Analyzing images with the Computer Vision service

[28]
After you've created a suitable resource in your subscription, you can submit images to
the Computer Vision service to perform a wide range of analytical tasks.

Describing an image

Computer Vision has the ability to analyze an image, evaluate the objects that are
detected, and generate a human-readable phrase or sentence that can describe what
was detected in the image. Depending on the image contents, the service may return
multiple results, or phrases. Each returned phrase will have an associated confidence
score, indicating how confident the algorithm is in the supplied description. The highest
confidence phrases will be listed first.

To help you understand this concept, consider the following image of the Empire State
building in New York. The returned phrases are listed below the image in the order of
confidence.

 A black and white photo of a city

 A black and white photo of a large city
 A large white building in a city

Tagging visual features

The image descriptions generated by Computer Vision are based on a set of thousands
of recognizable objects, which can be used to suggest tags for the image. These tags
can be associated with the image as metadata that summarizes attributes of the image;

[29]
and can be particularly useful if you want to index an image along with a set of key
terms that might be used to search for images with specific attributes or contents.

For example, the tags returned for the Empire State building image include:

 skyscraper
 tower
 building

Detecting objects

The object detection capability is similar to tagging, in that the service can identify
common objects; but rather than tagging, or providing tags for the recognized objects
only, this service can also return what is known as bounding box coordinates. Not only
will you get the type of object, but you will also receive a set of coordinates that indicate
the top, left, width, and height of the object detected, which you can use to identify the
location of the object in the image, like this:

Detecting brands

This feature provides the ability to identify commercial brands. The service has an
existing database of thousands of globally recognized logos from commercial brands of
products.

[30]
When you call the service and pass it an image, it performs a detection task and
determine if any of the identified objects in the image are recognized brands. The
service compares the brands against its database of popular brands spanning clothing,
consumer electronics, and many more categories. If a known brand is detected, the
service returns a response that contains the brand name, a confidence score (from 0 to 1
indicating how positive the identification is), and a bounding box (coordinates) for
where in the image the detected brand was found.

For example, in the following image, a laptop has a Microsoft logo on its lid, which is
identified and located by the Computer Vision service.

Detecting faces

The Computer Vision service can detect and analyze human faces in an image, including
the ability to determine age and a bounding box rectangle for the location of the
face(s). The facial analysis capabilities of the Computer Vision service are a subset of
those provided by the dedicated Face Service. If you need basic face detection and
analysis, combined with general image analysis capabilities, you can use the Computer
Vision service; but for more comprehensive facial analysis and facial recognition
functionality, use the Face service.

The following example shows an image of a person with their face detected and
approximate age estimated.

[31]
Categorizing an image

Computer Vision can categorize images based on their contents. The service uses a
parent/child hierarchy with a "current" limited set of categories. When analyzing an
image, detected objects are compared to the existing categories to determine the best
way to provide the categorization. As an example, one of the parent categories
is people_. This image of a person on a roof is assigned a category of people_.

A slightly different categorization is returned for the following image, which is assigned
to the category people_group because there are multiple people in the image:

[32]
Review the 86-category list here.

Detecting domain-specific content

When categorizing an image, the Computer Vision service supports two specialized
domain models:

 Celebrities - The service includes a model that has been trained to identify thousands of
well-known celebrities from the worlds of sports, entertainment, and business.
 Landmarks - The service can identify famous landmarks, such as the Taj Mahal and the
Statue of Liberty.

For example, when analyzing the following image for landmarks, the Computer Vision
service identifies the Eiffel Tower, with a confidence of 99.41%.

[33]
Optical character recognition

The Computer Vision service can use optical character recognition (OCR) capabilities to
detect printed and handwritten text in images. This capability is explored in the Read
text with the Computer Vision service module on Microsoft Learn.

Additional capabilities

In addition to these capabilities, the Computer Vision service can:

 Detect image types - for example, identifying clip art images or line drawings.
 Detect image color schemes - specifically, identifying the dominant foreground,
background, and overall colors in an image.
 Generate thumbnails - creating small versions of images.
 Moderate content - detecting images that contain adult content or depict violent, gory
scenes

[34]
Microsoft Azure AI Fundamentals: Explore natural language processing

Natural language processing supports applications that can see, hear, speak with, and
understand users. Using text analytics, translation, and language understanding services,
Microsoft Azure makes it easy to build applications that support natural language.

Introduction
Completed100 XP

 3 minutes

Analyzing text is a process where you evaluate different aspects of a document or

phrase, in order to gain insights into the content of that text. For the most part, humans
are able to read some text and understand the meaning behind it. Even without
considering grammar rules for the language the text is written in, specific insights can
be identified in the text.

As an example, you might read some text and identify some key phrases that indicate
the main talking points of the text. You might also recognize names of people or well-
known landmarks such as the Eiffel Tower. Although difficult at times, you might also be
able to get a sense for how the person was feeling when they wrote the text, also
commonly known as sentiment.

Text Analytics Techniques

Text analytics is a process where an artificial intelligence (AI) algorithm, running on a
computer, evaluates these same attributes in text, to determine specific insights. A
person will typically rely on their own experiences and knowledge to achieve the
insights. A computer must be provided with similar knowledge to be able to perform the
task. There are some commonly used techniques that can be used to build software to
analyze text, including:

 Statistical analysis of terms used in the text. For example, removing common "stop words"
(words like "the" or "a", which reveal little semantic information about the text), and
performing frequency analysis of the remaining words (counting how often each word
appears) can provide clues about the main subject of the text.
 Extending frequency analysis to multi-term phrases, commonly known as N-grams (a two-
word phrase is a bi-gram, a three-word phrase is a tri-gram, and so on).

[35]
  Applying stemming or lemmatization algorithms to normalize words before counting them
- for example, so that words like "power", "powered", and "powerful" are interpreted as
being the same word.
 Applying linguistic structure rules to analyze sentences - for example, breaking down
sentences into tree-like structures such as a noun phrase, which itself
contains nouns, verbs, adjectives, and so on.
 Encoding words or terms as numeric features that can be used to train a machine learning
model. For example, to classify a text document based on the terms it contains. This
technique is often used to perform sentiment analysis, in which a document is classified as
positive or negative.
 Creating vectorized models that capture semantic relationships between words by
assigning them to locations in n-dimensional space. This modeling technique might, for
example, assign values to the words "flower" and "plant" that locate them close to one
another, while "skateboard" might be given a value that positions it much further away.

While these techniques can be used to great effect, programming them can be complex.
In Microsoft Azure, the Language cognitive service can help simplify application
development by using pre-trained models that can:

 Determine the language of a document or text (for example, French or English).

 Perform sentiment analysis on text to determine a positive or negative sentiment.
 Extract key phrases from text that might indicate its main talking points.
 Identify and categorize entities in the text. Entities can be people, places, organizations, or
even everyday items such as dates, times, quantities, and so on.

In this module, you'll explore some of these capabilities and gain an understanding of
how you might apply them to applications such as:

 A social media feed analyzer to detect sentiment around a political campaign or a product
in market.
 A document search application that extracts key phrases to help summarize the main
subject matter of documents in a catalog.
 A tool to extract brand information or company names from documents or other text for
identification purposes.

These examples are just a small sample of the many areas that the Language service can
help with text analytics

Get started with text analysis

Completed100 XP

 10 minutes

[36]
The Language service is a part of the Azure Cognitive Services offerings that can
perform advanced natural language processing over raw text.

Azure resources for the Language service

To use the Language service in an application, you must provision an appropriate
resource in your Azure subscription. You can choose to provision either of the following
types of resource:

 A Language resource - choose this resource type if you only plan to use natural language
processing services, or if you want to manage access and billing for the resource
separately from other services.
 A Cognitive Services resource - choose this resource type if you plan to use the Language
service in combination with other cognitive services, and you want to manage access and
billing for these services together.

Language detection
Use the language detection capability of the Language service to identify the language
in which text is written. You can submit multiple documents at a time for analysis. For
each document submitted to it, the service will detect:

 The language name (for example "English").

 The ISO 6391 language code (for example, "en").
 A score indicating a level of confidence in the language detection.

For example, consider a scenario where you own and operate a restaurant where
customers can complete surveys and provide feedback on the food, the service, staff,
and so on. Suppose you have received the following reviews from customers:

Review 1: "A fantastic place for lunch. The soup was delicious."

Review 2: "Comida maravillosa y gran servicio."

Review 3: "The croque monsieur avec frites was terrific. Bon appetit!"

You can use the text analytics capabilities in the Language service to detect the
language for each of these reviews; and it might respond with the following results:

[37]
Document Language Name ISO 6391 Score
Code

Review 1 English en 1.0

Review 2 Spanish es 1.0

Review 3 English en 0.9

Notice that the language detected for review 3 is English, despite the text containing a
mix of English and French. The language detection service will focus on
the predominant language in the text. The service uses an algorithm to determine the
predominant language, such as length of phrases or total amount of text for the
language compared to other languages in the text. The predominant language will be
the value returned, along with the language code. The confidence score may be less
than 1 as a result of the mixed language text.

Ambiguous or mixed language content

There may be text that is ambiguous in nature, or that has mixed language content.
These situations can present a challenge to the service. An ambiguous content example
would be a case where the document contains limited text, or only punctuation. For
example, using the service to analyze the text ":-)", results in a value of unknown for the
language name and the language identifier, and a score of NaN (which is used to
indicate not a number).

Sentiment analysis
The text analytics capabilities in the Language service can evaluate text and return
sentiment scores and labels for each sentence. This capability is useful for detecting
positive and negative sentiment in social media, customer reviews, discussion forums
and more.

Using the pre-built machine learning classification model, the service evaluates the text
and returns a sentiment score in the range of 0 to 1, with values closer to 1 being a
positive sentiment. Scores that are close to the middle of the range (0.5) are considered
neutral or indeterminate.

For example, the following two restaurant reviews could be analyzed for sentiment:

[38]
"We had dinner at this restaurant last night and the first thing I noticed was how
courteous the staff was. We were greeted in a friendly manner and taken to our table right
away. The table was clean, the chairs were comfortable, and the food was amazing."

and

"Our dining experience at this restaurant was one of the worst I've ever had. The service
was slow, and the food was awful. I'll never eat at this establishment again."

The sentiment score for the first review might be around 0.9, indicating a positive
sentiment; while the score for the second review might be closer to 0.1, indicating a
negative sentiment.

Indeterminate sentiment

A score of 0.5 might indicate that the sentiment of the text is indeterminate, and could
result from text that does not have sufficient context to discern a sentiment or
insufficient phrasing. For example, a list of words in a sentence that has no structure,
could result in an indeterminate score. Another example where a score may be 0.5 is in
the case where the wrong language code was used. A language code (such as "en" for
English, or "fr" for French) is used to inform the service which language the text is in. If
you pass text in French but tell the service the language code is en for English, the
service will return a score of precisely 0.5.

Key phrase extraction

Key phrase extraction is the concept of evaluating the text of a document, or
documents, and then identifying the main talking points of the document(s). Consider
the restaurant scenario discussed previously. Depending on the volume of surveys that
you have collected, it can take a long time to read through the reviews. Instead, you can
use the key phrase extraction capabilities of the Language service to summarize the
main points.

You might receive a review such as:

"We had dinner here for a birthday celebration and had a fantastic experience. We were
greeted by a friendly hostess and taken to our table right away. The ambiance was
relaxed, the food was amazing, and service was terrific. If you like great food and attentive
service, you should try this place."

[39]
Key phrase extraction can provide some context to this review by extracting the
following phrases:

 attentive service
 great food
 birthday celebration
 fantastic experience
 table
 friendly hostess
 dinner
 ambiance
 place

Not only can you use sentiment analysis to determine that this review is positive, you
can use the key phrases to identify important elements of the review.

Entity recognition
You can provide the Language service with unstructured text and it will return a list
of entities in the text that it recognizes. The service can also provide links to more
information about that entity on the web. An entity is essentially an item of a particular
type or a category; and in some cases, subtype, such as those as shown in the following
table.

Type SubType Example

Person "Bill Gates", "John"

Location "Paris", "New York"

Organization "Microsoft"

Quantity Number "6" or "six"

Quantity Percentage "25%" or "fifty percent"

Quantity Ordinal "1st" or "first"

Quantity Age "90 day old" or "30 years old"

Quantity Currency "10.99"

[40]
Type SubType Example

Quantity Dimension "10 miles", "40 cm"

Quantity Temperature "45 degrees"

DateTime "6:30PM February 4, 2012"

DateTime Date "May 2nd, 2017" or "05/02/2017"

DateTime Time "8am" or "8:00"

DateTime DateRange "May 2nd to May 5th"

DateTime TimeRange "6pm to 7pm"

DateTime Duration "1 minute and 45 seconds"

DateTime Set "every Tuesday"

URL "https://www.bing.com"

Email "[email protected]"

US-based Phone "(312) 555-0176"

Number

IP Address "10.0.1.125"

The service also supports entity linking to help disambiguate entities by linking to a
specific reference. For recognized entities, the service returns a URL for a
relevant Wikipedia article.

For example, suppose you use the Language service to detect entities in the following
restaurant review extract:

"I ate at the restaurant in Seattle last week."

Entity Type SubType Wikipedia URL

Seattle Location https://en.wikipedia.org/wiki/Seattle

last week DateTime DateRange

[41]
Create conversational AI solutions
Conversational AI solutions are based on interactions between human users and AI agents
called bots. In this learning path, you'll learn how to build bots that can be delivered on
Microsoft Azure.

ntroduction
Completed100 XP

 1 minute

A bot is an application with a conversational interface.

While there are many ways you can implement a bot, some common features of bots
include:

 Users interact with a bot by initiating activities in turns.

 Activities are events, such as a user joining a conversation or sending a message.
 Messages can be text, speech, or visual interface elements (such as cards or buttons).

[42]
  A flow of activities can form a dialog, in which state is maintained to manage a multi-turn
conversation.
 Activities are exchanged across channels, such as web chat, email, Microsoft Teams, and
others.

Introduce principles of bot design

Completed100 XP

 5 minutes

Before embarking on the development of a bot, it's worth spending some time
considering some principles for effective bot design.

Factors influencing a bot's success

Ultimately, factors that lead to a successful bot all revolve around creating a great user
experience.

 Is the bot discoverable? If users are not able to discover the bot, they will be
unable to use it. Discoverability can be achieved through integration with the
proper channels. As an example, an organization may make use of Microsoft
Teams for collaboration. Integrating with the Teams channel will make your bot
available in the Teams app.

In some cases, making your bot discoverable is as simple as integrating it directly

into a web site. For example, your company's support web site could make a
question and answer bot the primary mechanism that customers interact with on
the initial support page.

 Is the bot intuitive and easy to use? The more difficult or frustrating a bot
interaction is, the less use it will receive. Users will not return to a bad user
experience.
 Is the bot available on the devices and platforms that users care
about? Knowing your customer-base is a good start to address this consideration.
If you only make your bot available on Microsoft Teams but most of your target
audience is using Slack, the bot will not be successful. It would require users to
install a new and unfamiliar software application.
 Can users solve their problems with minimal use and bot
interaction? Although it may seem counter-intuitive, success doesn't equate to

[43]
 how long a user interacts with the bot. Users want answers to their issues or
problems as quickly as possible. If the bot can solve the user's issue in the minimal
number of steps, the user experience will be a pleasant one and users are more
likely to come back to the bot again, or even help to promote the use of the bot
on your behalf.
 Does the bot solve the user issues better than alternative experiences? If a
user can reach an answer with minimal effort through other means, they are less
likely to use the bot. For example, most company switchboards use an automated
system of messages and options to choose when you call. Many users continue to
press 0 or some other key on the keypad in an attempt to bypass the options. The
rational is to go directly to an operator or support technician.

Factors that do not guarantee success

When designing a bot, you might want to create the smartest bot in the market.
Perhaps you want to ensure you have support for speech so that users don't have to
type text for the interaction. Demonstrating factors such as these, may impress fellow
developers, but are less likely to impress users. They could lead to user experience issues
as well.

Consider the concept of simplicity. The more complex your bot is, in terms of AI or
machine learning features, the more open it may be to issues and problems. Consider
adding advanced machine learning features to the bot if they are necessary to solve the
problems the bot is designed to address.

Adding natural language features may not always make the bot experience great. Again,
the conversation returns to whether the bot is addressing the problems the user needs
solved. A simple bot, that solves the user's problem without any conversational aspects,
is still a successful bot.

You might also believe that using speech for bot interactions would make the bot more
successful. There are many areas where it can be problematic. The ability to support
every language and dialect is not possible at this time. Speaker pronunciation and speed
can greatly impact the accuracy. A user interacting with the bot in language that is not
their native language can create issues in recognition. Other factors where speech
enabled bots can be problematic are in noisy environments. Background noise will
impact the accuracy of speech recognition and could create issues for the user in
hearing the bot responses. Use voice only where it truly makes sense for bot user
interaction.

[44]
Considerations for responsible AI
In addition to optimizing the user experience with the bot, you should consider how
your bot's implementation relates to principles for responsible AI development.
Microsoft provides guidance for responsible bot development
at https://www.microsoft.com/research/publication/responsible-bots, describing ten
guidelines for developers of conversational AI solutions. These guidelines include:

 Articulate the purpose of your bot and take special care if your bot will support
consequential use cases.
 Be transparent about the fact that you use bots as part of your product or service.
 Ensure a seamless hand-off to a human where the human-bot exchange leads to
interactions that exceed the bot's competence.
 Design your bot so that it respects relevant cultural norms and guards against misuse.
 Ensure your bot is reliable.
 Ensure your bot treats people fairly.
 Ensure your bot respects user privacy.
 Ensure your bot handles data securely.
 Ensure your bot is accessible.
 Accept responsibility for your bots operation and how it affects people

Get started with the Bot Framework SDK

Completed100 XP

 5 minutes

Bot solutions on Microsoft Azure are supported by the following technologies:

 Azure Bot Service. A cloud service that enables bot delivery through one or more
channels, and integration with other services.
 Bot Framework Service. A component of Azure Bot Service that provides a REST API for
handling bot activities.
 Bot Framework SDK. A set of tools and libraries for end-to-end bot development that
abstracts the REST interface, enabling bot development in a range of programming
languages.

[45]
Developing a Bot with the Bot Framework SDK
The Bot Framework SDK provides an extensive set of tools and libraries that software
engineers can use to develop bots. The SDK is available for multiple programming
languages, including Microsoft C# (.NET Core), Python, and JavaScript (Node.js)

Bot templates

The easiest way to get started with the Bot Framework SDK is to base your new bot on
one the templates it provides:

 Empty Bot - a basic bot skeleton.

 Echo Bot - a simple "hello world" sample in which the bot responds to messages by
echoing the message text back to the user.
 Core Bot - a more comprehensive bot that includes common bot functionality, such as
integration with the Language Understanding service.

Bot application classes and logic

The template bots are based on the Bot class defined in the Bot Framework SDK, which
is used to implement the logic in your bot that receives and interprets user input, and

[46]
responds appropriately. Additionally, bots make use of an Adapter class that handles
communication with the user's channel.

Conversations in a bot are composed of activities, which represent events such as a user
joining a conversation or a message being received. These activities occur within the
context of a turn, a two-way exchange between the user and bot. The Bot Framework
Service notifies your bot's adapter when an activity occurs in a channel by calling
its Process Activity method, and the adapter creates a context for the turn and calls the
bot's Turn Handler method to invoke the appropriate logic for the activity.

Testing with the Bot Framework Emulator

Bots developed with the Bot Framework SDK are designed to run as cloud services in
Azure, but while developing your bot, you'll need a way to test it before you deploy it
into production.

The Bot Framework Emulator is an application that enables you to run your bot a local
or remote web application and connect to it from an interactive web chat interface that
you can use to test your bot. Details of activity events are captured and shown in the
testing interface, so you can monitor your bots behavior as you submit messages and
review the responses.

Implement activity handlers and dialogs

Completed100 XP

 5 minutes

The logic for processing the activity can be implemented in multiple ways. The Bot
Framework SDK provides classes that can help you build bots that manage
conversations using:

 Activity handlers: Event methods that you can override to handle different kinds of
activities.
 Dialogs: More complex patterns for handling stateful, multi-turn conversations.

Activity handlers
For simple bots with short, stateless interactions, you can use Activity Handlers to
implement an event-driven conversation model in which the events are triggered by
activities such as users joining the conversation or a message being received. When an

[47]
activity occurs in a channel, the Bot Framework Service calls the bot adapter's Process
Activity function, passing the activity details. The adapter creates a turn context for the
activity and passes it to the bot's turn handler, which calls the individual, event-specific
activity handler.

The ActivityHandler base class includes event methods for the many kinds of common
activity, including:

 Message received
 Members joined the conversation
 Members left the conversation
 Message reaction received
 Bot installed
 Others...

You can override any activity handlers for which you want to implement custom logic.

Turn context

An activity occurs within the context of a turn, which represents a single two-way
exchange between the user and the bot. Activity handler methods include a parameter
for the turn context, which you can use to access relevant information. For example, the
activity handler for a message received activity includes the text of the message.

More Information

For more information about activity handlers, see the Bot Framework SDK
documentation.

[48]
Dialogs
For more complex conversational flows where you need to store state between turns to
enable a multi-turn conversation, you can implement dialogs. The Bot Framework SDK
dialogs library provides multiple dialog classes that you can combine to implement the
required conversational flow for your bot.

There are two common patterns for using dialogs to compose a bot conversation:

Component dialogs

A component dialog is a dialog that can contain other dialogs, defined in its dialog set.
Often, the initial dialog in the component dialog is a waterfall dialog, which defines a
sequential series of steps to guide the conversation. It's common for each step to be
a prompt dialog so that conversational flow consists of gathering input data from the
user sequentially. Each step must be completed before passing the output onto the next
step

For example, a pizza ordering bot might be defined as a waterfall dialog in which the
user is prompted to select a pizza size, then toppings, and finally prompted for payment.

[49]
Adaptive dialogs

An adaptive dialog is another kind of container dialog in which the flow is more flexible,
allowing for interruptions, cancellations, and context switches at any point in the
conversation. In this style of conversation, the bot initiates a root dialog, which contains
a flow of actions (which can include branches and loops), and triggers that can be
initiated by actions or by a recognizer. The recognizer analyzes natural language input
(usually using the Language Understanding service) and detects intents, which can be
mapped to triggers that change the flow of the conversation - often by starting new
child dialogs, which contain their own actions, triggers, and recognizers.

For example, the pizza ordering bot might start with a root dialog that simply welcomes
the user. When the user enters a message indicating that they want to order a pizza, the
recognizer detects this intent and uses a trigger to start another dialog containing the
flow of actions required to gather information about the pizza order. At any point
during the pizza order dialog, the user might enter a message indicating that they want
to do something else (for example, cancel the order), and the recognizer for the pizza
ordering dialog (or its parent dialog) can be used to trigger an appropriate change in
the conversational flow.

Note

[50]
Using dialogs offers greater flexibility than is possible with a bot based purely on activity
handlers, but can be more complex to program. While you can use the Bot Framework
SDK to implement dialogs in code, you may want to consider using the Bot Framework
Composer to build bots with complex dialogs, taking advantage of its visual design
interface

Deploy a bot
Completed100 XP

 5 minutes

After you've completed development of your bot, you can deploy it to Azure. The
specific details of how the bot is hosted varies, depending on the programming
language and underlying runtime you have used; but the basic steps for deployment are
the same.

Create the Azure resources required to support your

bot
Your will need to create an Azure application registration to give your bot an identity it
can use to access resources, and a bot application service to host the bot.

Register an Azure app

You can create the application registration by using the az ad app create Azure
command-line interface (CLI) command, specifying a display name and password for
your app identity. This command registers the app and returns its registration
information, including a unique application ID that you will need in the following step.

Create a bot application service

Your bot requires a Bot Channels Registration resource, along with associated
application service and application service plan. To create these resources, you can use
the Azure resource deployment templates provided with the Bot Framework SDK
template you used to create your bot. Just run the az deployment group
create command, referencing the deployment template and specifying your bot
application registration's ID (from the az ad app create command output) and the
password you specified.

[51]
Prepare your bot for deployment
The specific steps you need to perform to prepare your bot depend on the
programming language used to create it. For C# and JavaScript bots, you can use the az
bot prepare-deploy command to ensure your bot is properly configured with the
appropriate package dependencies and build files. For Python bots, you must include
a requirements.txt file listing any package dependencies that must be installed in the
deployment environment.

Deploy your bot as a web app

The final step is to package your bot application files in a zip archive, and use the az
webapp deployment source config-zip command to deploy the bot code to the Azure
resources you created previously.

After deployment has completed, you can test and configure your bot in the Azure
portal.

…000…

[52]