Deep Learning with Multiple GPUs

How to Use Multiple GPUs for Deep Learning

Parallel processing enables multiple data objects
Deep learning is a subset of machine learning that to be processed at the same time, drastically
does not rely on structured data to develop reducing training time. This parallel processing is
accurate predictive models. This method uses typically accomplished through the use of graphical
networks of algorithms modeled after neural processing units (GPUs). GPUs are specialized
networks in the brain to distill and correlate large processors created to work in parallel. These units
amounts of data. The more data you feed your can provide significant advantages over traditional
network, the more accurate the model becomes. CPUs, including up to 10x more speed. Typically,
You can functionally train deep learning models multiple GPUs are built into a system in addition to
using sequential processing methods. However, CPUs. While the CPUs can handle more complex or
the amount of data needed and the length of data general tasks, the GPUs can handle specific, highly
processing make it impractical if not impossible to repetitive processing tasks.
train models without parallel processing.

All rights reserved to Run:ai.

No part of this content may be used

1
without express permission of Run:ai. www.run.ai
In this guide, you will learn:

Multi GPU Distributed Deep Learning Strategies 3

How Does Multi GPU Work in Common Deep Learning Frameworks? 4

TensorFlow Multiple GPU 4

PyTorch Multi GPU 4

Multi GPU Deployment Models 5

GPU Server 5

GPU Cluster 5

Also refer to our other detailed guides about:

Machine Learning Operations (MLops)

Deep Learning GPU

All rights reserved to Run:ai.

No part of this content may be used

2
without express permission of Run:ai. www.run.ai
Multi GPU Deep Learning Strategies

Once multiple GPUs are added to your systems, Data Parallelism

you need to build parallelism into your deep learning Data parallelism is a method that uses duplicates of
processes. There are two main methods to add your model across GPUs. This method is useful when
parallelism—models and data. the batch size used by your model is too large to fit
Model Parallelism on a single machine, or when you want to speed up
the training process. With data parallelism, each copy
Model parallelism is a method you can use when
of your model is trained on a subset of your dataset
your parameters are too large for your memory
simultaneously. Once done, the results of the models
constraints.
are combined and training continues as normal.
Using this method, you split your model training
processes across multiple GPUs and perform each
process in parallel (as illustrated in the image
Data Parallelism
below) or in series. Model parallelism uses the same
dataset for each portion of your model and requires
synchronizing data between the splits.

Model Parallelism

Machine 4

Machine 1 Machine 2

Machine 2 Machine 3

Machine 1
Machine 3 Machine 4

All rights reserved to Run:ai.

No part of this content may be used

3
without express permission of Run:ai. www.run.ai
How Does Using Multiple Replicas of your model are created and assigned to a
GPU. Then, a subset of the dataset is assigned to that

GPUs Work in Common

replica

Deep Learning Frameworks? The subset for each GPU is processed and gradients
are produced

When working with deep learning models, there The gradients from all model replicas are averaged
are several frameworks you may use, including and the result is used to update the original model
Keras, PyTorch and TensorFlow. Depending on the
framework you choose, there are different ways to
implement multi GPU systems. The process repeats until your model is fully trained

TensorFlow PyTorch
TensorFlow is an open source framework, created PyTorch is an open source scientific computing
by Google, that you can use to perform machine framework based on Python. You can use it
learning operations. The library includes a variety of to train machine learning models using tensor
machine learning and deep learning algorithms and computations and GPUs. This framework supports
models that you can use as a base for your training. distributed training through the torch.distributed
It also includes built-in methods for distributed backend.
training using GPUs.

Through the API, you can use the tf.distribute. With PyTorch, there are three parallelism (or
distribution) classes that you can perform with
Strategy method to distribute your operations
GPUs. These include:
across GPUs, TPUs or machines. This method
enables you to create and support multiple user
segments and to switch between distributed Data Parallel: enables you to distribute model
strategies easily. replicas across multiple GPUs in a single machine.
You can then use these models to process different
subsets of your data set.
Two additional strategies that extend the distribute
method are MirroredStrategy and TPUStrategy.
Both of these enable you to distribute your Distributed Data Parallel: extends the DataParallel
workloads, the former across multiple GPUs and class to enable you to distribute model replicas across
machines in addition to GPUs. You can also use this
the latter across multiple Tensor Processing Units
class in combination with model_parallel to perform
(TPUs). TPUs are units available through Google both model and data parallelism.
Cloud Platform that are specifically optimized for
training with TensorFlow.
Model Parallel: enables you to split large models
across multiple GPUs with partial training happening
on each. This requires syncing training data
Both of these methods use roughly the same between the GPUs since operations are performed
data-parallel process, summarized as follows: sequentially.

Your dataset is segmented so data is distributed as

evenly as possible.

All rights reserved to Run:ai.

No part of this content may be used

4
without express permission of Run:ai. www.run.ai
Multi GPU Deployment Replicas of your model are created and assigned to a
GPU. Then, a subset of the dataset is assigned to that

Models replica

The subset for each GPU is processed and gradients

are produced
There are three main deployment models you
can use when implementing machine learning
operations that use multiple GPUs. The model you The gradients from all model replicas are averaged
use depends on where your resources are hosted and the result is used to update the original model

and the size of your operations.

The process repeats until your model is fully trained

GPU Server
GPU servers are servers that incorporate GPUs
in combination with one or more CPUs. When Using Multiple GPUs with Run:ai
workloads are assigned to these servers, the Run:ai automates resource management and
CPUs act as a central management hub for the workload orchestration for machine learning
GPUs, distributing tasks and collecting outputs as infrastructure. With Run:ai, you can automatically
available. run as many deep learning experiments as needed
on multi-GPU infrastructure.

GPU Cluster
GPU clusters are computing clusters with Here are some of the capabilities you gain when
nodes that contain one or more GPUs. These using Run:ai:
clusters can be formed from duplicates of the
same GPU (homogeneous) or from different Advanced Visibility: create an efficient pipeline of
GPUs (heterogeneous). Each node in a cluster resource sharing by pooling GPU compute resources
is connected via an interconnect to enable the
transmission of data.
No More Bottlenecks: you can set up guaranteed
quotas of GPU resources, to avoid bottlenecks and
optimize billing.
GPU Cluster
Kubernetes is an open source platform you
can use to orchestrate and automate container A Higher Level of Control: Run:ai enables you to
deployments. This platform offers support for dynamically change resource allocation, ensuring
each job gets the resources it needs at any given time
the use of GPUs in clusters to enable workload
acceleration, including for deep learning.
Run:ai simplifies machine learning infrastructure
When using GPUs with Kubernetes, you can pipelines, helping data scientists accelerate their
deploy heterogeneous clusters and specify your productivity and the quality of their models.
resources, such as memory requirements. You
can also monitor these clusters to ensure reliable Learn more about the Run:ai GPU virtualization
performance and optimize GPU utilization. platform.

All rights reserved to Run:ai.

No part of this content may be used

5
without express permission of Run:ai. www.run.ai