Presented by Babacar Ndao

Big Data Tools

under the supervision of
Professor Marie Ndiaye.

Comparative study
 • Big Data Challenge

Introduction •
•
•
Data Sources
Key Focus
Objective

• Big Data Challenge

Managing and analyzing large volumes of structured, semi- structured,
and unstructured data.
• Data Sources
Social media, transactions, IoT devices, healthcare systems, etc.
• Big Data Tools
Solutions for storage, processing, analysis, and visualization
of large datasets.
• Key Focus
Explore top Big Data tools and their capabilities in:

•Data Integration and Compatibility

•Scalability
•Real-time vs. Batch Processing
•Fault Tolerance and Reliability
•Data Security and Governance
• Objective
Comparative study of tools to identify the best solutions for modern data challenges.
01
Plan

• Introduction
• Big Data Concepts
• Big Data Tools Categories
• Big Data Processing Frameworks
• Comparative Study
• Top 10 Big Data Tools
• Comparison
• Conclusion

02
 • Definition

Big Data Concepts •

•
Types of Big Data
Importance of Big Data

• Definition
• Types of Big Data
• Importance of Big Data

03
 • Definition
Big Data Concepts • Types of Big Data
• Importance of Big Data

• What is Big Data ?

• Big Data refers to the massive collection of structured, semi-structured, and

unstructured data generated from various sources like social media, transactions,
and smart devices.

• It is often characterized by three Vs:

• Volume : Refers to the large amounts of data collected from various sources
such as transactions, IoT devices, videos, and social media.
• Velocity : Represents the speed at which data is generated and processed, often in
real-time, especially with IoT devices like sensors and smart meters.
• Variety : Describes the different formats of data, ranging from structured (numeric
data) to unstructured (texts, videos, emails, etc.).

04
 • Definition

• Types of Big
Big Data Concepts Data
• Importance of Big Data

The types of Big Data are typically categorized into three main types:

• Structured Data:
Data that is highly organized and easily searchable in databases, such as SQL.

• Unstructured Data:
Data that lacks a predefined structure, including text, audio, video, and social media posts.

• Semi-Structured Data:
Data that does not fit fully into structured databases but has some organizational properties, like
JSON, XML, or CSV files.

05
 • Definition
• Types of Big Data

Big Data Concepts • Importance of

Big Data

Importance of Big Data:

• Big Data is crucial for extracting insights that drive decision-making,

enhance customer experiences, and improve business processes.

• It helps in cost savings, understanding market trends, and speeding up

decision-making processes.

06
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

• Data Storage and Management Tools

• Data Processing and Analytics Tools
• ETL (Extract, Transform, Load) Tools
• Data Warehousing and Querying Tools
• Data Ingestion Tools
• Machine Learning and Data Mining Tools
• Data Visualization Tools
• Data Governance and Security Tools

07
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

1. Data Storage and Management Tools

• Definition:
These tools are responsible for storing large datasets, ensuring that data is accessible, scalable, and manageable.
They often provide distributed storage systems to handle massive volumes of structured and unstructured data.
• Examples:
• Hadoop Distributed File System (HDFS)
• Amazon S3
• Google Cloud Storage
• Apache Cassandra
• MongoDB
• Characteristics:
• Scalability to store petabytes of data
• Support for both structured and unstructured data
• Fault tolerance and data replication
• Efficient retrieval mechanisms
• Distributed architecture to handle large-scale data

08
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

2. Data Processing and Analytics Tools

Definition:
These tools perform computations and transformations on Big Data, often supporting both real-time and batch
processing to derive insights and analytics.
Examples:
1.Apache Spark
2.Aache Flink
3.Hadoop MapReduce
4.Google BigQuery
5.Databricks
Characteristics:

•Distributed processing for large datasets

•Support for both batch and real-time processing
•In-memory computation for faster data processing
•Integration with various data storage platforms
•Advanced analytics support like machine learning algorithms
09
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

4. Data Warehousing and Querying Tools

Definition:
These tools provide storage optimized for analytical queries, making it easier to run complex queries and generate business
insights from large datasets.
Examples:
Amazon Redshift
Google BigQuery
Snowflake
Apache Hive
Microsoft Azure Synapse
Characteristics:
Optimized for high-performance querying
Support for SQL-like querying languages
Massively parallel processing (MPP) for faster analytics
Efficient data compression and partitioning
Integration with business intelligence (BI) tools for reporting

10
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

5. Data Ingestion Tools

Definition:
These tools are responsible for collecting, importing, and preparing data from multiple sources into a Big Data ecosystem for processing and analysis.
Examples:
Apache Kafka
Apache Flume
Amazon Kinesis
Google Cloud Pub/Sub
Apache Sqoop
Characteristics:
Real-time data streaming or batch data ingestion
Scalability for large volumes of incoming data
Fault tolerance and reliable data delivery
Compatibility with various data sources (databases, logs, IoT devices)
Data buffering and aggregation capabilities

11
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

6. Machine Learning and Data Mining Tools

Definition:
These tools enable the development, training, and deployment of machine learning models and algorithms on large datasets, as
well as discovering patterns and insights from data.
Examples:
Apache Mahout
TensorFlow
H2O.ai
Scikit-learn
Google AI Platform
Characteristics:
Support for a wide range of machine learning algorithms (supervised, unsupervised, etc.)
Scalable model training for large datasets
Integration with data storage and processing systems
Tools for hyperparameter tuning, model evaluation, and optimization
Support for deep learning and neural networks

12
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

7. Data Visualization Tools

Definition:
These tools allow the representation of Big Data in graphical formats to make insights, patterns, and trends more
understandable and actionable.
Examples:
Tableau
Microsoft Power BI
Qlik Sense
Google Data Studio
Grafana
Characteristics:
Interactive dashboards for visual data exploration
Integration with various data sources
Support for different types of charts, graphs, and visualizations
Real-time data monitoring and reporting
User-friendly drag-and-drop interfaces

13
 • Definition
Big Data Tools Categories •
•
Examples
Characteristics

8. Data Governance and Security Tools

Definition:
These tools help in managing data privacy, security, and compliance within an organization by setting policies, auditing access, and
monitoring for breaches.
Examples:
Apache Ranger
IBM Guardium
Collibra
Talend Data Fabric
Varonis
Characteristics:
Centralized management of data policies and permissions
Support for compliance with regulations (GDPR, HIPAA, etc.)
Data access auditing and tracking
Encryption and data masking for security
Alerts and monitoring for potential data breaches

14
 • Definition
Big Data Processing Frameworks •
•
Examples
Characteristics

• Batch Processing Frameworks

• Stream Processing Frameworks
• Hybrid Processing Frameworks

15
 • Definition
Big Data Processing Frameworks •
•
Examples
Characteristics

Batch Processing Frameworks

• Definition:
Batch processing frameworks handle data in large, discrete chunks or batches, allowing
scheduled processing of accumulated data without the need for immediate real-time
action.
• Examples:
• Apache Hadoop
• Apache Spark (batch mode)
• Amazon EMR
• Characteristics:
• High throughput for large datasets
• Suitable for processing historical data
• Jobs are scheduled and executed periodically (e.g., daily or weekly)
• Involves significant latency between data input and output
• Efficient for tasks like ETL and large-scale computations

16
 • Definition
Big Data Processing Frameworks •
•
Examples
Characteristics

Stream Processing Frameworks

• Definition:
Stream processing frameworks allow for the continuous ingestion and processing of real-
time data. They provide low-latency processing, enabling real-time analysis and insights.
• Examples:
• Apache Flink
• Apache Kafka Streams
• Amazon Kinesis
• Characteristics:
• Real-time, low-latency data processing
• Continuous data input and output streams
• Suitable for event-driven applications (e.g., fraud detection, live monitoring)
• Provides immediate insights and analytics
• More complex architecture than batch systems due to continuous processing

17
 • Definition
Big Data Processing Frameworks •
•
Examples
Characteristics

Hybrid Processing Frameworks

• Definition:
Hybrid processing frameworks combine both batch and stream processing capabilities, enabling
organizations to handle real-time streaming data while also processing historical batch data.
• Examples:
• Apache Spark (Structured Streaming)
• Apache Flink (both batch and stream)
• Google Dataflow
• Characteristics:
• Supports both real-time and batch processing
• Flexible architecture for a wide range of use cases
• Allows for immediate insights (streaming) while handling large volumes of historical data (batch)
• Unified programming model for developers
• Ideal for applications needing real-time insights along with historical trend analysis

18
 Comparative Study •
•
Definition
Examples
• Characteristics

REFERENCES
A Comparative Study on Different Big Data Tools
https://hdl.handle.net/10365/31657

A Comparative Study of Big Data Tools and Deployement

Platforms
https://www.researchgate.net/publication/340307931_A_Comparative_Study_of_Big_Da
ta_Tools_and_Deployment_PIatforms?enrichId=rgreq-
ee1266ab49c234027ea237f1f102ec75-
XXX&enrichSource=Y292ZXJQYWdlOzM0MDMwNzkzMTtBUzo5MjQ4ODE0OTU1NDc5MDRAMT
U5NzUyMDM0MDcyOA%3D%3D&el=1_x_3&_esc=publicationCoverPdf

Compare the features & pricing of 2023's best big

data tools.
https://www.fivetran.com/learn/big-data-tools

Top Big Data Tools You Need to Know in 2024

https://www.knowledgehut.com/blog/big-data/big-data-tools

19
 • Evaluating criteria
Comparative Study • Ranking

A Comparative Study on Different Big Data Tools

https://hdl.handle.net/10365/31657

NDSU is a public research university in the United States, known for its programs in agriculture, engineering,
and technology. It also conducts extensive research in various fields, including Big Data, and contributes to
academic and professional communities through its research publications, conferences, and collaborations.

• Ranking
· MapReduce
• Evaluating criteria
· Pig
· Performance
· Sqoop
· Efficiency · Apache Flume
· Scalability · Apache Hadoop (HDFS + YARN)
· Processing Paradigms · Hive
· Data Flow Management
· Real-time vs Batch Processing · Apache Kafka
· Ease of Use · Apache Tez
· Apache Spark 20
 • Evaluating criteria
Comparative Study • Ranking

A Comparative Study of Big Data Tools and Deployement

Platforms
https://www.researchgate.net/publication/340307931_A_Comparative_Study_of_Big_Da
ta_Tools_and_Deployment_PIatforms?enrichId=rgreq-
ee1266ab49c234027ea237f1f102ec75-
XXX&enrichSource=Y292ZXJQYWdlOzM0MDMwNzkzMTtBUzo5MjQ4ODE0OTU1NDc5MDRAMT
U5NzUyMDM0MDcyOA%3D%3D&el=1_x_3&_esc=publicationCoverPdf

ResearchGate is a social networking site for scientists and researchers to share their publications, ask questions, and
collaborate on research projects. It serves as a hub for academic resources, facilitating access to scientific papers, data sets,
and discussions across disciplines, including data science, machine learning, and Big Data analytics.

• Evaluating criteria • Ranking

Apache Spark
Data Processing Model Apache Flink
Scalability Apache Kafka
Fault Tolerance Apache Hadoop
Latency Apache Samza
Throughput Apache Storm
Ease of Use Apache Cassandra
Real-time Processing Apache HBase
Batch Processing Apache Hive
Integration with Other Apache Pig.
Systems 21
 • Evaluating criteria
Comparative Study • Ranking

Compare the features & pricing of 2023's best big data tools.
https://www.fivetran.com/learn/big-data-tools
Fivetran is a company that provides automated data integration solutions, specializing in extracting, transforming, and loading
(ETL) data from various sources into centralized data warehouses. Its tools are used by data engineers and analysts to
streamline data ingestion and transformation processes for business intelligence and analytics.

• Ranking
Apache Spark
Apache Kafka
Fivetran
• Evaluating criteria Cloudera
Apache Hadoop
Apache Cassandra
Apache Hive
· Organization Use Case & Objectives Zoho Analytics
· Pricing Apache Kylin
· Ease of Use RapidMiner
Apache Storm
· Integration Support
Lumify
· Scalability Trino
· Data Governance & Security OpenRefine
Apache Samza 22
 • Evaluating criteria
Comparative Study • Ranking

Top Big Data Tools You Need to Know in 2024

https://www.knowledgehut.com/blog/big-data/big-data-tools

UpGrad is an online education platform that offers professional courses, particularly in technology, management, and data
science. It partners with universities and industry experts to deliver high-quality educational content, including certifications
and degrees in areas such as Big Data, AI, and business analytics.

• Ranking
• Evaluating criteria Apache Hadoop
Cloudera (CDH)
Business Objectives Alignment
Cost
Apache Cassandra
Ease of Use KNIME
Advanced Analytics Capabilities Lumify
Security Storm (Apache Storm)
Scalability
Integration Capabilities Apache SAMOA
RapidMiner. 23
 Top 10 Big Data Tools

Based on the comparative study section of our document and the references to the studies from NDSU, ResearchGate,
Fivetran, and UpGrad, here are the top 10 Big Data tools considering key criteria such as performance, scalability, ease of
use, real-time vs. batch processing, integration support, and more:

1. Apache Spark
• Featured in NDSU, ResearchGate, Fivetran, and UpGrad.
• Known for its fast, in-memory data processing and support for both batch and stream processing.
2. Apache Kafka
• Mentioned in all four studies.
• Highly scalable real-time data streaming tool with strong fault tolerance.
3. Hadoop (HDFS + YARN)
• Covered in NDSU, ResearchGate, Fivetran, and UpGrad.
• The foundational distributed storage and processing system, strong in batch processing and scalability.
4. Apache Flink
• NDSU, ResearchGate, and UpGrad highlight this tool for its stream processing capabilities, supporting real-time and hybrid
processing.
24
 Top 10 Big Data Tools

5. Amazon Kinesis
• Fivetran and UpGrad emphasize it for real-time data streaming and integration with AWS.
6. Google BigQuery
• Highlighted in Fivetran and UpGrad for its serverless, highly scalable data warehousing capabilities with real-time
querying.
7. Databricks
• Fivetran and UpGrad list Databricks for its optimized Apache Spark implementation with collaboration and machine
learning capabilities.
8. Microsoft Azure Synapse
• Featured in Fivetran for its integrated data analytics and querying platform that supports both structured and unstructured
data.
9. Snowflake
• Found in Fivetran and UpGrad, it is praised for its ease of use, scalability, and support for SQL-based data warehousing.
10. Apache Hive
• Mentioned in NDSU and ResearchGate for large-scale data querying with its SQL-like interface on top of Hadoop.

These tools are ranked based on their overall performance across several criteria, such as real-time and
batch processing capabilities, scalability, ease of integration, and advanced analytics features. Let me know
if you'd like further breakdowns or detailed comparisons.
25
 •Data Integration and Compatibility
Comparison •Scalability
•Real-time vs. Batch Processing criteria
•Fault Tolerance and Reliability
•Data Security and Governance
Key Criteria for Big Data Platform

1. Data Integration and Compatibility 2. Scalability

• Seamless integration with Orthanc for DICOM data • Handle large, growing medical datasets
• Support for machine learning and Big Data analysis formats • Supports both horizontal and vertical scaling
• Relevant Tools: Apache NiFi, Apache Kafka • Manage increasing data load from multiple PACS systems

3. Real-time vs. Batch Processing 4. Fault Tolerance and Reliability

• Real-time: Urgent medical insights (e.g., Apache Flink, Apache Kafka) • Ensures data integrity and no data loss in ETL pipeline
• Batch Processing: Less time-sensitive operations (e.g., Apache Spark) • Reliable Tools: Hadoop, Apache Kafka, Apache Spark

5. Data Security and Governance These criteria will help build a robust,
• Compliance with regulations (HIPAA, GDPR)
• Ensure encryption, access control, and audit trails scalable, and secure Big Data platform for
• Security Tools: Apache Ranger, IBM Guardium medical data migration.
26
 1--> is low performance
Comparison 2-->
3-->
is poor performance
is average performance
4--> is good performance
Let's assign scores on a scale of 1 to 5 for each criterion. 5--> is excellent performance

Data Integration and Real-time vs. Batch Fault Tolerance and Data Security and
Compatibility(ORTHANC) Scalability Processing Reliability Governance Total
Apache Hadoop 4 5 3 5 4 21
Apache kafka 5 4 5 5 4 23
Apache spark
4 5 4 5 4 22
Apache flink 4 4 5 4 3 20
Amazon Kinesis 4 5 5 4 4 22
Microsoft Azure
Synapse 4 5 3 4 5 21
Databricks
4 5 4 4 4 21
Snowflake
4 5 3 4 5 21
Apache Hive
3 4 3 4 3 17 27
 • Chosen Tool: Apache Kafka

Conclusion •
•
•
Categories Covered by Apache Kafka
Big Data Processing Framework
Conclusion

• Chosen Tool: Apache Kafka

• Why Apache Kafka?
• Excellent compatibility for managing unstructured and semi-structured data such as medical images (DICOM).
• Efficient in handling real-time and streaming data, essential for medical environments that require timely insights.
• High fault tolerance and reliability, crucial for maintaining the integrity of sensitive medical data.
• Categories Covered by Apache Kafka
• Data Ingestion Tools: Kafka excels at collecting and importing data from multiple sources into a Big Data ecosystem.
• Data Processing: Kafka integrates well with real-time processing tools like Apache Flink and provides support for hybrid processing.
• Data Governance & Security: Kafka offers capabilities for data security, including encryption and access control, essential for
compliance with GDPR and HIPAA.
• Big Data Processing Framework
• Stream Processing: Kafka supports real-time data processing with low latency, enabling immediate insights—ideal for healthcare
applications.
• Hybrid Processing: Kafka can work in conjunction with batch frameworks like Apache Spark, ensuring flexibility between real-time
and historical data analysis.
• Conclusion
• Apache Kafka is the best choice for migrating medical data from Orthanc, as it covers critical aspects of Big Data types, tool
categories, and processing frameworks.
28
• Its scalability, security, and real-time data capabilities make it the most suitable solution for our Big Data project in healthcare.