Approaches in ETL Process

INTRODUCTION:

There are several approaches that can be used in the ETL process:

1. Batch ETL: This approach processes data in batches, typically at regular intervals such as daily, weekly, or monthly. This approach is suitable for handling large volumes of data and is commonly used for loading data into data warehouses.
2. Real-time ETL: This approach processes data in real-time, as soon as it is available. This approach is suitable for handling high-volume, high-velocity data streams and is commonly used for real-time data integration and analytics.
3. Incremental ETL: This approach processes only new or changed data, rather than processing all data every time. This approach is more efficient and faster than full load ETL, and is commonly used for data warehousing and incremental data integration.
4. Data Profiling: This approach is used to understand the data that is being extracted from the source systems. Data profiling techniques can be used to discover data quality issues, data completeness, and to ensure that the data is in the correct format for loading into the data warehouse.
5. Data Cleansing: This approach is used to clean and validate the data before it is loaded into the data warehouse. This may involve removing duplicate records, correcting data errors, and standardizing data.
6. Data Integration: This approach is used to combine data from multiple sources and systems, making it more accessible and usable. This may involve creating new data fields, joining data from multiple tables, and creating new views of the data.
7. Each of these approaches has its own set of advantages and disadvantages, and the choice of approach will depend on the specific requirements of the organization and the data that is being processed.

Prerequisite - ETL (Extraction, Transformation, and Loading) Process ETL stands for Extraction, Transform and Load. These are three database functions that are incorporated into one tool to pull data out from one database and to put data into another database. Big Data encompasses a wide range of enormous data that can either be structured or unstructured. RDBMS finds it challenging to handle huge volumes of data. Also, RDBMS is designed for steady data retention rather than rapid growth. This is where data warehouses come in. Data warehouse supports all types of data and can also handle the rapid growth of data. Thus, for data analysis, data needs to be shifted from databases to data warehouses. The working of the ETL process can be well explained with the help of the following diagram. 
 

Applications of the ETL process are :

• To move data in and out of data warehouses. Databases are not suitable for big data analytics therefore, data needs to be moved from databases to data warehouses which is done via the ETL process.
• Data strategies are more complex than they have ever been. ETL facilitates to transform vast quantities of data into actionable business intelligence.

There are two approaches in ETL :

1. Top Down Approach : The data flow in the top-down OLAP environment begins with data extraction from the operational data sources. This data is loaded into the staging area and validated and consolidated for ensuring a level of correctness and then moved to the Operational Data Store (ODS). The ODS stage is sometimes skipped if it is another copy of the operational databases. Data is loaded into the Data warehouse in a parallel to avoid extracting it from the ODS. Data is routinely extracted from the ODS and temporarily hosted in the staging area for aggregation, summarization and then extracted and loaded into the Data warehouse. The need to have an ODS is determined by the business requirements. If there is a need for detailed data in the Data warehouse then ODS must be created. Once the Data warehouse aggregation and summarization processes are complete, the data mart will extract the data from the Data warehouse into the staging area and perform a new set of transformations on them. This will help organize the data in particular structures as required by data marts. Afterward, the data marts can be loaded with the data and the OLAP environment becomes available to the users. The data in a data warehouse is historical data. A top‐down model approach was proposed by Inmon, to create a centralized Enterprise Data Warehouse using traditional database modeling techniques (ER Model), where the data is stored in 3NF. The data warehouse now acts as a data source for the new data marts.
2. Kimball Methodology (Bottom-Up Approach) : The bottom‐up approach reverses the positions of the Datawarehouse and the Data marts. Data marts are directly loaded with the data through the staging area. The existence of ODS depends on business requirements. The data flow in the bottom-up approach starts with the extraction of data from operational databases into the staging area where it is processed and consolidated and then loaded into the ODS. The data in the ODS is either appended to or replaced by the fresh data being loaded. Once the ODS is refreshed, the present data is once again extracted into the staging area and processed. The data from data mart is pulled to the staging area aggregated, summarized, and so on and loaded into the Data Warehouse and made available to the end-user for analysis.

ETL Tools : Some of the most commonly used ETL tools are MarkLogic, Oracle, Sybase, Hevo, and Xplenty. 

Advantages of ETL Tools :

• Easy to use.
• Load data from different targets at same time.
• Performs data transformation as per need.
• Better for complex rules and transformations.
• Inbuilt Error handling functionality.
• Based on GUI and offer visual flow.
• Save Cost and generate higher revenue.
• Automation: ETL tools automate the process of extracting, transforming, and loading data, reducing the time and effort required to load and update data in the warehouse.
• Improved Data Quality: ETL tools can help to ensure that the data in the data warehouse is accurate, complete, and up-to-date by validating, cleaning and transforming data.
• Increased Productivity: ETL tools can increase productivity by allowing users to schedule and automate data loads, and by providing a user-friendly interface for managing and manipulating data.
• Better Data Integration: ETL tools can help to integrate data from multiple sources and systems, making it more accessible and usable.
• Increased Scalability: ETL tools can improve scalability by providing a way to manage and analyze large amounts of data.
• Better Data Governance: ETL tools can help to improve data security by controlling access to the data warehouse and ensuring that only authorized users can access the data.
• Better Data Warehousing: ETL tools can help to improve data warehousing by providing a way to manage and analyze large amounts of data.
• Data lineage : ETL tools provides the ability to track the data from where it came from, where it is currently and where it is going to. It allows to trace the data if any issues occurs.

Overall, ETL tools offer a range of advantages to organizations, including automation, improved data quality, increased productivity, and better data integration. ETL tools can help organizations to effectively manage and analyze large amounts of data, allowing them to make more informed decisions and gain a competitive advantage.

Disadvantages of ETL Tools :

• Not suitable for near real-time data access.
• Inclined more towards batch data processing
• Difficult to keep up with changing requirements.
• High cost: ETL tools can be expensive to purchase, implement, and maintain, especially for organizations with limited budgets.
• Complexity: ETL tools can be complex to set up and maintain, requiring specialized knowledge and skills.
• Limited scalability: ETL tools may not be able to handle large volumes of data, especially in real-time.
• Limited flexibility: ETL tools may not be able to handle all types of data or handle data in all formats, which can limit their flexibility.
• Limited data governance: ETL tools may not provide robust data security or data governance capabilities, which can be a concern for organizations handling sensitive data.
• Limited data lineage: Some ETL tools may not provide the ability to track the data from where it came from, where it is currently and where it is going to. It may not allow to trace the data if any issues occurs.
• Limited data integration: ETL tools may not be able to integrate data from all possible sources, which can limit their usefulness for organizations with complex data integration requirements.
• Limited real-time data processing: Some ETL tools might not be able to handle real-time data processing which can be a concern for organizations handling high-volume, high-velocity data streams.

Overall, ETL tools can be costly and complex to implement, and may not be able to handle all types of data or data volumes. Additionally, they may not provide robust data security or data governance capabilities, which can be a concern for organizations handling sensitive data. It's important to carefully evaluate the needs and capabilities of your organization before investing in an ETL tool.