Implementation and analysis of a federated learning architecture

using CIFAR-10 dataset

Shubham1,∗ , Albert Sharma2,∗ , Arjun Arora3,∗ , P ukhraj Gupta4,∗

1 [email protected], 2 [email protected], 3 [email protected], 4 [email protected]
∗
Department of M athematics & Scientif ic Computing, N IT, Hamirpur, (HP ) 177005

Abstract—This paper investigates the application of federated et al. (2018) [4] proposed FedProx, an extension that integrates
learning architecture in conjunction with various convolutional the closest term into the optimization objective to address the
network models, namely VGG13, VGG16, and VGG19, utiliz- distribution of non-IID data among client devices. This adaptation
has been shown to be important for improving the convergence of
ing different hyperparameters. The primary focus is on evalu-
topology models in situations, where device data exhibit different
ating the accuracy of these models across two distinct dataset statistical properties. In the context of privacy protection, Shokri
distributions: Independent and Identically Distributed (IID) et al. (2015) [5] introduced the concept of differential privacy in
and (non-IID) Throughout experimentation and analysis, the federated learning. Their work demonstrated the application of
CIFAR-10 data set serves as the benchmark data set [1]. The differentiated security mechanisms to ensure that personal data
implementation employs a client-server architecture, where the contributions do not compromise user privacy. This has become
dataset is manually distributed among the participating clients. the foundation for improving the security and privacy aspects of
federated image classification.
Future research directions are outlined, highlighting the poten-
In the context of privacy protection, Shokri et al. (2015) [5]
tial for real-world data implementation sourced directly from introduced the concept of differential privacy in federated learning.
client devices, as opposed to a centralized dataset distribution Their work demonstrated the application of differentiated security
approach. Additionally, it is suggested to extend the study to mechanisms to ensure that personal data contributions do not
encompass different data sets to assess performance variations compromise user privacy. This has become the foundation for
with varying class distributions for further investigation. improving the security and privacy aspects of federated image
Keywords— Federated learning, non-IID, CIFAR-10, VGG classification.

1. Introduction 3. Preliminaries
The fundamentals of FL involve training machine learning
Federated learning (FL) represents a paradigm shift during
models distributed across a network of decentralized devices, such
machine learning, the training process takes place for models are
as smartphones, IoT devices, or Edge servers. Instead of sending
trained collaboratively across decentralized devices without the
raw data to a central server, devices calculate model updates
exchange of raw data between the server and clients [2]. This
locally based on their data and only share the updates [6]. This
innovative approach Tackles privacy issues linked to centralized
decentralized learning method minimizes the risk of data exposure,
data storage while enabling the creation of robust and accurate
making it especially suitable for applications with strict privacy
models. This paper provides an overview of federated learning, its
requirements [7].
fundamentals, and it’s step by step application for image classifica-
The principles underlying federated learning require an ex-
tion using deep learning models. Federated learning is emerging as
ploration of its fundamental components. At the core of this
a response to the growing demand for privacy-preserving machine
concept are several key factors. Firstly, the inclusion of client
learning solutions [3]. Traditional centralized models often require
devices, each housing local data, is paramount. These devices
consolidation of sensitive data on a central server, raising privacy
engage in collaborative training of a model, either concurrently
concerns and legal challenges. FL, on the other hand, allows
or asynchronously. Secondly, a central server assumes the role of
training models on distributed devices, allowing users to maintain
orchestrator within the federated learning framework. Manages the
control over their data.
coordination of the process, aggregates model updates from client
devices, and oversees the evolution of the collective model. Finally,
2. Previous work the implementation of specialized federated learning algorithms is
essential. These algorithms are meticulously crafted to harmonize
Previous work in federated learning for image classification and synchronize model updates across various client devices while
has made significant progress, demonstrating the effectiveness of maintaining the accuracy of the overall global model. Through
this approach in maintaining privacy while achieving high model the harmonious interplay of these elements, federated learning
performance. McMahan et al. (2017) conducted pioneering re- emerges as a powerful paradigm for decentralized model training
search by introducing Federated Averaging (FedAvg), an efficient and advancement.
communication algorithm for training deep neural networks on While addressing current situations, Federated Learning finds
decentralized devices. Their work laid the foundation for further applications in various fields including healthcare, finance, and per-
research on linked image classification. Based on FedAvg, Li sonalized services. In healthcare, FL enables collaborative training
 Figure 1. Centralised learning

of models on patient data without compromising individual privacy. techniques introduce random variations, augmenting the dataset,
In finance, it facilitates the training of fraud detection models and diversifying the training samples.
across multiple banks without sharing sensitive transaction details Following data augmentation, the next step involves converting
[8]. the PIL image to a PyTorch tensor. Additionally, we normalize
Despite its promising benefits, federated learning faces chal- the tensor values, adjusting the mean and standard deviation to
lenges such as communication overhead, non-IID (non-identical match the specific requirements of the CIFAR-10 data set. By
distribution) data, and model convergence. Researchers have pro- adhering to these meticulous transformations, we optimize the
posed solutions such as efficient communication algorithms, adap- input data to facilitate the training of deep learning models,
tive aggregation techniques, and non-IID data distribution process- ultimately enhancing their performance and accuracy.
ing methods.

4. Dataset
5. Proposed methodology
The dataset used in this study is the CIFAR-10 dataset [9]
which is known as a benchmark dataset for machine learning In our proposed federated learning system, we describe nu-
research. CIFAR-10 includes his 32x32 colour image patches merous innovative contributions that improve collaborative model
categorized into 10 different classes: airplanes, cars, birds, cats, training in the CIFAR-10 data. First, we develop a customized data
deer, dogs, frogs, horses, ships, and trucks. Figure 1 shows example splitting mechanism that assigns data to clients based on random
images for each of these classes. This study is done on 50,000 weights, resulting in a realistic and diversified image distribution.
training samples, each class consisting of his 5,000 images. Model To react to changing client learning dynamics, we use a dynamic
evaluation is then performed on another set of 10,000 test images learning rate adjustment strategy using the ReduceLROnPlateau
to evaluate the performance of the trained model. callback for each optimizer. The chosen VGG19 neural network
design, which is known for its simplicity and effectiveness, serves
4.1. Data Transformation as the foundation for both the global server and individual client
models. Our aggregation technique uses a weighted average of
Data transformation is a critical aspect of enhancing the effi- model weights across clients to encourage collaborative learning
cacy of the training process, particularly in deep learning models while taking into account each client’s dataset size. Additionally,
applied to datasets such as CIFAR-10 [10] . In our implementation, we incorporate a baseline retraining phase on the global server
we meticulously follow a series of steps to ensure that the input using a fixed dataset, showcasing the adaptability of the global
data aligns with the specific requirements of the models. Firstly, we model to specific baseline data characteristics. Throughout multiple
convert the input array, typically an image, into a Python Imaging communication rounds, we monitor training progress, reporting
Library (PIL) image. Subsequently, we employ data augmentation average train losses, test losses, and test accuracy, providing in-
techniques, as outlined by Shorten [11], which involve random sights into the convergence and generalization performance of
cropping of the image to dimensions of 32x32 pixels with a the federated learning system. Finally, our adaptive learning rate
padding of 4 pixels, followed by a random horizontal flip. These reduction mechanism dynamically adjusts learning rates when the
 The network topology embraces a federated architecture, facilitat-
ing conversation among the server and the federated nodes.

6.1. VGG Model Configuration:

The decision to design neural network (CNN) models for
image classification in our research stems from the transformative
effect that CNNs have tested within the subject of computer vision.
The introduction of CNNs has marked a paradigm shift in the realm
of computer vision, in image classification tasks. Pioneering works
together with AlexNet, VGGNet, and next architectures such as
GoogLeNet and ResNet have showcased the remarkable function-
ality of deep convolutional architectures in capturing hierarchical
and summary functions from visual data [10]. The choice of the
VGG architecture for our research is rooted in its balanced design,
placing an equilibrium among model complexity and simplicity.
VGGNet, with its consistent 3x3 convolutional filters and depth,
has tested latest overall performance in numerous computer vision
tasks [12]. This simplicity allows ease of understanding, training,
and transferability of the discovered features.
In this paper, we have worked with Visual Geometry Group
(VGG), especially we have tested our architecture using the
VGG19 model, where we first constructed the dictionary consist-
Figure 2. Illustration of Data Augmentation : Cropping and Flipping ing of the architecture configurations for different VGG models
(VGG11, VGG13, VGG16, and VGG19). After the established
VGG architecture with specific configurations for convolutional
test accuracy plateaus, aiming to improve convergence in scenarios and pooling layers. The fully connected layers in the classifier
where model performance becomes stagnant. Together, these inno- section contribute to the final classification output that has an
vations contribute to an effective and adaptive federated learning output size of 10, suitable for classification tasks with 10 classes
approach on CIFAR-10. for CIFAR-10 dataset.
In Figure 3, we can clearly see for the CNN model VGG19 we
get a fully connected layer network of 1x1x512 pixels where in the
5.1. Experimental setup connected layer we get 512 neurons with the Rectified Linear Unit
(ReLU) activation function and for the last layer we get 10 neurons,
In configuring the experimental setup, both hardware and soft-
i.e. 10 output classes. In the final layer we have implemented the
ware specifications were meticulously outlined to ensure robustness
Log-Softmax activation to the output of the classifier.
and accuracy in the federated learning architecture, particularly
concerning IID and non-IID scenarios. The hardware configuration
encompassed an Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz,
featuring 8 cores and 16 logical processors, alongside an NVIDIA
GeForce RTX 2060 GPU, and 16 GB DDR3 RAM. On the soft-
ware front, the experimental environment operated on Microsoft
Windows 11 Home Single Language, leveraging Python 3.9.6 and
CUDA 11.7 for programming tasks. Deep learning frameworks Py-
Torch 2.0.1 and TensorFlow 2.8.0 were also integral components of
the software configuration, ensuring compatibility and proficiency
within the experimental paradigm. Such meticulous delineation
of hardware and software elements underscores the precision and
reliability of the experimental setup, thereby facilitating rigorous
investigation and analysis within the federated learning framework.

6. Federated Learning Framework:

In this research, we elucidate the elaborate details of the Feder-
ated Learning (FL) framework employed in our research, aiming to
explain the orchestration of collaborative model training throughout
decentralized gadgets. Federated Learning, as a dispensed device
studying paradigm, fosters model training without the need of cen-
tralizing raw information. The framework followed on this look at
is pivotal to harnessing the collaborative strength of diverse devices
while ensuring data privacy. Our Federated Learning framework
is constructed upon a decentralized network topology, comprising
a heterogeneous ensemble of client devices. Each client device,
denoted as a federated node, performs an important position in Figure 3. VGG 19 Model Architecture
locally training the VGG model on its respective dataset subset.
 determined after aggregation of client updates, measures the global
model’s performance on a fixed baseline dataset, indicating its
6.2. Client/Node Configuration: generalization ability. The loss and accuracy of the test, evaluated
in an independent test set, provide further assessment of the
To visualize the real-world scenario, we have cleared a dummy generalization and correctness of the model, respectively, offering
client-server model within our system to test the classification insight into the prediction error and the accuracy of the model.
techniques using federated learning. The CIFAR-10 dataset is split Communication rounds, representing the total iterations of feder-
amongst a specified range of clients at the same time as making ated learning, indicate the frequency of update and aggregation of
sure every client has a subset of classes. This mimics the situation model parameters across the client network, thereby reflecting the
in which clients keep different portions of the general dataset. overall learning process.

6.3. Training Procedure: 7.2. Performance metrics

In this research we at first randomly distributes the data In the context of federated learning, performance metrics such
amongst clients, simulating a real-world situation wherein cus- as accuracy, average train loss, and test loss play crucial roles in
tomers may have different amounts of records [13]. This feature evaluating the effectiveness and efficiency of the learning process.
generates random values for each client, representing the variety
of images that each client possesses. 7.2.1. Accuracy: . Accuracy measures the correctness of the
We prepare data loaders for each client and the test set, where prediction of the model [14]. In federated learning, accuracy re-
the client receives a subset of the entire dataset based on the deter- flects how well the model performs on unseen data from distributed
mined distribution. During the training period, data augmentation devices or clients. Higher accuracy indicates that the model has
and normalization transformations are applied. Clients’ models are learned meaningful patterns and can generalize well to new data
updated locally on their respective datasets for a set number of [15]. Evaluating accuracy helps assess the overall performance of
epochs and, the training is done using the client’s local data loader. the federated learning model in terms of its predictive power across
The loss is computed using negative logarithmic likelihood loss for the distributed dataset [16].
image classification. After local training, models from different
clients participating in a particular round are synchronized with 7.2.2. Average Train Loss: . The average train loss represents
the global model on the server. This step is generally where the the average discrepancy between the prediction of the model and
global model aggregates the knowledge learned by an individual the actual labels during the training phase. It measures how well
client without sharing the data. The aggregating technique used the model fits the training data. In federated learning, since training
for research purposes is the weighted mean aggregation method, occurs locally on each client device, the average train loss reflects
where the contribution of any individual client is measured by the the convergence of the model’s parameters across the distributed
number of images it possesses. The aggregation model is evaluated devices. Monitoring the average train loss helps ensure that the
on the test set, where we calculate the test loss and accuracy to get federated learning process is effectively updating the model’s pa-
insights onto the performance of federated model on the unseen rameters to minimize errors on the local training data.
data.
While working in the non-IID, we also formed the function 7.2.3. Test Loss: . Test loss, also known as validation loss,
for baseline data, which generates a data loader contain a subset measures the model’s performance on a separate validation dataset
of the CIFAR-10 dataset [Li, T., Sahu, A.K., Talwalkar, A. and that it has not been trained on. This metric helps evaluate the
Smith, V., 2020. Federated learning: Challenges, methods, and model’s generalization capability and its ability to make accurate
future directions. IEEE signal processing magazine, 37(3), pp.50- predictions on unseen data. In federated learning, test loss provides
60.]. The model uses this data during the retraining phase on the insights into how well the model performs across the distributed
global server that is provided in a controlled and consistent subset devices in real-world scenarios. A lower test loss indicates that the
of data. Generally, it is observed that retraining on a fixed set model can generalize well to new data and is not overfitting to the
of baseline data helps maintain the stability of the model and its local training data.
consistency in various retraining rounds, especially in a diverse
group of client data. These performance metrics collectively provide a comprehen-
sive evaluation of the federated learning process [6]. By analysing
accuracy, average train loss, and test loss, researchers and prac-
7. Result Analysis titioners can assess the model’s performance, identify potential
issues such as overfitting or underfitting, and make informed
Upon comparing the federated learning model‘s overall perfor- decisions to improve the federated learning system . [17]. Addi-
mance, we delve right into a complete analysis of diverse metrics tionally, monitoring these metrics over time allows for continuous
and overall performance indicators. These metrics offer treasured optimization of the federated learning process to achieve better
insights into the convergence, generalization, and predictive accu- performance and scalability across distributed environments.
racy of the model across dispensed datasets.
Through the implementation of federated learning with the
7.1. Evaluation Metrics: VGG 19 model, we conducted a comparative analysis utilizing
two distinct data distribution techniques: IID and non-IID datasets.
Various metrics play crucial roles in assessing the perfor- Employing a learning rate of 0.01, we evaluated various perfor-
mance and progress of federated learning models. The train loss, mance metrics including accuracy, average train loss, and test loss,
calculated during local training on randomly selected clients in as illustrated in Figure 5. Our findings indicate that the VGG 19
each communication round, offers insights into the convergence model exhibits higher accuracy on the IID dataset compared to the
of the model and the progression of learning. Retraining loss, non-IID dataset, aligning with prior research findings as discussed
 Figure 4. VGG 19 Model on IID Dataset
Figure 6. Comparison of VGG 19 Model on IID and Non-IID Dataset

in [paper reference], which suggests that the IID dataset tends to

yield such outcomes more consistently.

In our exploration of the implementation of federated learning,

we used the VGG 16 model in three different learning rates to
investigate the impact on accuracy. As depicted in Figure 6, our
observations reveal increased fluctuations and decreased accuracy
with higher learning rates than 0.01. This phenomenon is consistent
with the principles of stochastic gradient descent, where a high
learning rate often leads to the model bypassing local minima
more frequently, thus deviating from optimal convergence. Further
research is warranted to ascertain if this behaviour persists with
further increases in the learning rate, suggesting the potential for Figure 7. Learning Rates Comparison on VGG 16 Model
mathematical proofs within this domain of inquiry.
In this study, we conducted an experiment using the VGG16
model, employing three distinct learning rates: 0.01, 0.04, and 0.08.
The purpose was to examine and analyze the variations in the
results that arise from the manipulation of this critical parameter.
This investigation aimed to provide information on the impact of
different learning rates on the performance of the VGG16 model
[10]. This finding makes it even tough to choose a suitable learning
rate for the model, thus we implemented the concept of Learning
Rate Schedulers, where we initialized the optimizer with an initial
learning rate and the scheduler monitors a specified metric (vali-
dation accuracy) and reduces the learning rate when the monitored
metric stops improving for a certain number of consecutive epochs
(determined by patience). To carry out this, we also set the value
for minimum learning rate that the scheduler can reduce to. The
figure below shows how this approach can help in getting the same
accuracy in much lesser number of communication rounds.
In the following figure we have compared the result for non-
IID using VGG13, VGG16, and VGG19 models for 100 com-
munication rounds. It can be observed that even VGG19 model
has a greater number of dense layered as compared to the other
two model, then also VGG13 and VGG16 perform better, where
Figure 5. VGG 19 Model on Non-IID Dataset we infer that in our experiment, we have input image size of
32x32 that might have hindered the performance of VGG model
as the architecture of VGG19 is not adapted for smaller input
sizes. Reducing the input size from 224x224 to 32x32 results in a
significant loss of spatial information.
 addressing demanding situations related to data privacy, scalability,
and efficiency, consequently paving the manner for transformative
improvements in machine learning and artificial intelligence.

References
[1] Yuanshao Zhu, Shuyu Zhang, Yi Liu, Dusit Niyato, and JQ James.
Robust federated learning approach for travel mode identification
from non-iid gps trajectories. In 2020 IEEE 26th International
Conference on Parallel and Distributed Systems (ICPADS), pages
585–592. IEEE, 2020.
[2] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P
Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe
Morency. Think locally, act globally: Federated learning with local
and global representations. arXiv preprint arXiv:2001.01523, 2020.
[3] Howard H Yang, Zuozhu Liu, Tony QS Quek, and H Vincent Poor.
Scheduling policies for federated learning in wireless networks. IEEE
Figure 8. Fixed Learning Rate vs. Learning Rate Schedulers on Non-IID transactions on communications, 68(1):317–333, 2019.
Data
[4] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith.
Federated learning: Challenges, methods, and future directions. IEEE
signal processing magazine, 37(3):50–60, 2020.
[5] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning.
In Proceedings of the 22nd ACM SIGSAC conference on computer
and communications security, pages 1310–1321, 2015.
[6] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet,
Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary
Charles, Graham Cormode, Rachel Cummings, et al. Advances and
open problems in federated learning. Foundations and trends® in
machine learning, 14(1–2):1–210, 2021.
[7] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson,
and Blaise Aguera y Arcas. Communication-efficient learning of
deep networks from decentralized data. In Artificial intelligence and
statistics, pages 1273–1282. PMLR, 2017.
[8] Ashish Rauniyar, Desta Haileselassie Hagos, Debesh Jha, Jan Erik
Håkegård, Ulas Bagci, Danda B Rawat, and Vladimir Vlassov. Fed-
erated learning for medical applications: A taxonomy, current trends,
challenges, and future research directions. IEEE Internet of Things
Figure 9. Comparison of VGG 13, VGG 16 and VGG 19 model Journal, 2023.
[9] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of
features from tiny images. 2009.
8. Conclusion [10] Karen Simonyan and Andrew Zisserman. Very deep convolu-
tional networks for large-scale image recognition. arXiv preprint
In conclusion, the implementation and evaluation of a federated arXiv:1409.1556, 2014.
learning architecture making use of the CIFAR-10 dataset exhibit [11] Debarshi Datta, Safiya George Dalmida, Laurie Martinez, David
promising consequences and insights into the efficacy of decen- Newman, Javad Hashemi, Taghi M Khoshgoftaar, Connor Shorten,
tralized learning paradigms. Through rigorous experimentation and Candice Sareli, and Paula Eckardt. Using machine learning to identify
assessment, we have located the convergence behavior, generaliza- patient characteristics to predict mortality of in-patients with covid-19
tion capabilities, and predictive accuracy of the federated model in south florida. Frontiers in Digital Health, 5:1193467, 2023.
across distributed devices. [12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet
Our findings underscore the potential of federated learning in classification with deep convolutional neural networks. Communica-
leveraging decentralized data sources even as keeping user privacy tions of the ACM, 60(6):84–90, 2017.
and records locality. By harnessing the collective intelligence of [13] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson,
distributed devices, federated learning demonstrates resilience to and Blaise Aguera y Arcas. Communication-efficient learning of deep
records heterogeneity and scalability to large-scale datasets, as networks from decentralized data. pages 1273–1282, 2017.
evidenced through its overall performance at the CIFAR-10 dataset. [14] Aurélien Géron. Hands-on machine learning with Scikit-Learn,
In addition, the evaluation of the evaluation metrics, including Keras, and TensorFlow. ” O’Reilly Media, Inc.”, 2022.
train loss, test loss, and accuracy, provides a complete understand- [15] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine
ing of the model‘s learning dynamics and predictive capabilities of learning: From theory to algorithms. Cambridge university press,
the model ‘. These insights contribute to refining federated learning 2014.
methodologies and optimizing model performance in real-world [16] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson,
scenarios. and Blaise Aguera y Arcas. Communication-efficient learning of
As we navigate toward the technology of decentralized system deep networks from decentralized data. In Artificial intelligence and
learning, the exploration and refinement of federated learning statistics, pages 1273–1282. PMLR, 2017.
architectures remain pivotal. Through continued studies and inno- [17] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learn-
vation, we will release the entire capacity of federated learning in ing. MIT Press, 2016. http://www.deeplearningbook.org.