Scribd
Upload
17 views

Big Data File Formats For Data Engineers

Uploaded by

techinsight579
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views

Big Data File Formats For Data Engineers

Uploaded by

techinsight579
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Big Data File Formats for Data Engineers www.linkedin.

com/in/ggnanasekaran

A file format in the context of big data refers to the structure and organization in which
data is stored within files. It determines how data is encoded, stored, and represented,
influencing factors like storage efficiency, data processing speed, and compatibility with
various tools and systems.

How you store the data in your data lake is critical and you need to consider the format,
compression and especially how you partition your data.Therefore, processing big data
comes with its own high cost to store data. Moreover, adding the storage cost together
with the cost of CPU to process data, IO, and network costs, you’re left with a
money-swallowing whirlpool. In other words, larger datasets mean more expense.

Why do we need different file formats in big data?


In the realm of big data, where datasets are vast, varied, and processed at an immense
scale, the choice of file formats plays a pivotal role in optimizing storage, processing
speed, compatibility, and data analysis.
● Text-based Formats (e.g., CSV, JSON, XML): These are human-readable formats,
often used for simple data interchange. However, they can be inefficient for
large-scale data processing due to their lack of optimization and data type
information.
● Columnar Formats (e.g., Parquet, ORC): These formats store data column-wise
instead of row-wise, enabling better compression and efficient selective data
retrieval. This suits analytical queries and reduces I/O operations during
processing.
● Binary Formats (e.g., Avro, Apache Arrow): These formats encode data in binary,
providing efficient storage and faster serialization/deserialization. They often
include schema information, enhancing data's self-description and compatibility
across languages.
Choosing the right file format depends on factors such as the type of data, processing
requirements, storage capabilities, and the tools used for analysis. A well-chosen file
format can significantly impact data processing efficiency and overall performance in
big data environments.
Big Data File Formats for Data Engineers www.linkedin.com/in/ggnanasekaran

AVRO:
● Avro is a row-based, language-neutral, schema-based serialization technique
and object container file format. It is an open-source format. The Avro format
stores data definitions (the schema) in JSON and is easily read and interpreted.
The data within the file is stored in binary format, making it compact and
space-efficient.
● One of Avro's key features is its support for schema evolution. As data
requirements change over time, Avro permits the evolution of schemas without
breaking compatibility with existing data. New fields can be added, fields can be
renamed, and default values can be specified, all while maintaining the ability to
read older data
● Avro's binary encoding, coupled with the ability to use various compression
codecs, contributes to efficient storage and reduced data transfer times

Parquet:
● Parquet is a columnar storage file format widely utilized in the domain of big data
processing and analytics. Developed within the Apache Hadoop ecosystem,
Parquet optimizes data storage and query performance.
● Parquet organizes data by columns rather than rows, enabling efficient
compression and faster analytical queries.
● Parquet supports schema evolution, allowing changes to the data schema
without compromising backward compatibility. This flexibility is crucial as data
structures evolve over time, ensuring seamless data processing.
● While Parquet is excellent for analytical use cases, it might not be as well-suited
for transactional workloads or scenarios where frequent updates and deletes are
required due to its append-only nature
.
ORC file format:
● ORC (Optimized Row Columnar) is a columnar storage file format designed for
high-performance data processing in big data environments. Developed within
the Apache Hive project, ORC improves storage efficiency and query speed by
storing data in columns rather than rows, enabling efficient compression and
faster data access.
● The ORC file format stores collections of rows in a single file, in a columnar
format within the file. This enables parallel processing of row collections across a
Big Data File Formats for Data Engineers www.linkedin.com/in/ggnanasekaran

cluster. Due to the columnar layout, each file is optimal for compression,
enabling skipping of data and columns to reduce read and decompression loads
● It significantly reduces I/O operations and boosts overall data processing
efficiency..

Here’s a quick comparison of the three main big data file formats:

Options Parquet ORC Avro

Schema Evolution Good Better Best

File Compression Better Best Good

Splitability Support Good Best Good

Row or Column Column Column Row

Read or Write Read Write Write

You might also like