Build+Data+Pipeline+Using+Azure+Medallion+Architecture+Approach
Uploaded by
gopichandrecruiter66Build+Data+Pipeline+Using+Azure+Medallion+Architecture+Approach
Uploaded by
gopichandrecruiter66Build Data Pipeline using Azure
Medallion Architecture
Approach
CookBook
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Table of Content:
1.Introduction …………………………………………………………………………………………… 2
2.Use-case depicted in this project……………………………………………………………. 4
3.Architecture…………………………………………………………………………………………… 6
4.P rerequisites………………………………………………………………………………………… 7
.Services…………………………………………………………………………………………………..
5
8
6.Execution………………………………………………………………………………………………… 17
i)Creating Resource Groups………………………………………………………………… 17
ii)Creating Azure SQL Database………………………………………………………….. 20
iii)Importing Data in Azure SQL Database………………………………………….. 33
iv)Creating Azure Blob Storage…………………………………………………………… 47
v) Creating Logic App…………………………………………………………………………. 53
vi)Creating Ingestion workflow in Azure LogicApp………………………….. 65
vii)Creating Azure Data Lake Storage Account (ADLSGen2)…………….. 82
viii)Creating Azure Data Factory………………………………………………………… 87
ix) Creating Azure Databricks…………………………………………………………….. 103
x) Azure Databricks Medallion Architecture……………………………………… 109
Bronze Layer…………………………………………………..
● 12
1
● Silver Layer……………………………………………………. 115
● Gold Layer…………………………………………………….. 122
xi)Connection to Power BI and Visualizations………………………… 130
xii)Summary…………………………………………………………………… 137
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Introduction
In this training, we shall look at and execute a pipeline that is comprehensively designed to analyze
complex datasets, i.e. water sensor data. The pipeline leverages Azure services, which are seamlessly
integrated to enable efficient data movement and processing. Each component works together to
ensure smooth data flow and accurate analysis, showcasing the power of Azure's advanced data
integration capabilities. The use cases for the proposed pipeline can be as follows:
● T he Authoritative and government organizations monitor the concentration levels of water
bodies which were collected using sensors. By analyzing this data, they aim to assess the
s afety and potential hazards of these water sources.
● A unified system for all government-related organizations and concerned authorities to address
water-related issues effectively and implement immediate safety measures. This kind of
system would enable streamlined collaboration and decision-making.
● The real-time recording of water quality data through sensors emphasizes the necessity of a
robust and constructive framework. The ultimate goal is to reduce harmful contaminants in
water, ensuring it remains safe and sustainable within the ecosystem.
T his use case focuses on"Aggregated Sensor Water Data"which is collected across various time
zones, countries, water bodies, and vegetation types. This gets crucial when climatic and habitat
conditions vary, such historical data becomes vital in providing a comprehensive overview to address
and resolve complex or uncertain issues. Every single record is crucial, as it contributes to the
measurement of determinands and informs proactive measures to ensure water safety.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Use-case depicted in this project
T his projectleverages"Aggregated Sensor Water Data"as the main data source to implement a
comprehensive Azure data pipeline. The pipeline is designed to ingest sensor data, process it, and
move it to the destination, utilizing various Azure services such asAzure SQL Database, Azure Logic
Apps, Azure Storage Accounts, Azure Data Factory, and Azure Databricks. The final transformed
data is visualized usingPower BI, the most demandingBusiness Intelligence tool. The pipeline is
s tructured to be modular and customizable, which allows additional Azure services to be seamlessly
integrated to expand its functionality.
TheAzure services chosen for this project are scalable,robust, user-friendly, and cost-effective.
The project uses the “Aggregrated Sensor Water Data” which includes different columns as follows:
. countryCode
1
2. monitoringSiteIdentifier
3. monitoringSiteIdentifierScheme
4. parameterWaterBodyCategory
5. observedPropertyDeterminandCode
6. observedPropertyDeterminandLabel
7. procedureAnalysedMatrix
8. resultUom
9. phenomenonTimeReferenceYear
10.parameterSamplingPeriod
11.procedureLOQValue
12.resultNumberOfSamples
13.resultQualityNumberOfSamplesBelowLOQ
14.resultQualityMinimumBelowLOQ
15.resultMinimumValue
16.resultQualityMeanBelowLOQ
17.resultMeanValue
18.resultQualityMaximumBelowLOQ
19.resultMaximumValue
20.resultQualityMedianBelowLOQ
21.resultMedianValue
22.resultStandardDeviationValue
23.procedureAnalyticalMethod
24.parameterSampleDepth
25.resultObservationStatus
26.remarks
27.metadata_versionId
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
8.metadata_beginLifeSpanVersion
2
29.metadata_statusCode
30.metadata_observationStatus
31.UID
T he following columns represent the level of detail or drill-down of parameters at which water
readings were taken by the sensors.
F or demonstration purposes, the data is initially loaded from a local source to anAzure Managed SQL
DatabaseusingSQL Server Management Studio. Thisapplication allows for seamless connection to
the Azure Cloud Database and enables the import of data from an Excel file (the sensor water data
file) into Azure. The Azure pipeline is designed to handle data of any size, minimizing latency and
ensuring that the processed data is quickly made available for analysis or any defined purpose.
T he goal of this project is to solve water quality problem using Azure Pipeline by incorporating these
endpoints:
1. R aw Data:The raw data, which is available in an Excels heet, is first loaded into the Azure
Managed SQL Database using SQL Server Management Studio (SSMS). This enables the data to
be used in the Azure pipeline, and ensures it’s ready for further processing.
2. Data Movement:After loading raw data into the SQLdatabase, it is then moved to Azure Blob
Storage using Logic Apps. This component helps automate data transfer from the Azure SQL
Database to Blob Storage in JSON format. Logic Apps contain different variety of actions to
accomplish such intensive tasks.
3. Orchestration: Once the data is in Blob Storage,it is transferred to Azure Data Lake Gen2 for
integration with Azure Databricks. This is achieved through Azure Data Factory, which includes
a copy pipeline to move data into the lake.
4. P rocessing Layer:The data then undergoes transformationin Azure Databricks, following the
Medallion Architecture with three layers:
● Bronze Layer: Raw data is ingested from Data Lake Gen2 into a data frame using Spark
jobs.
● Silver Layer: The data is processed and refined for quality.
● Gold Layer: Final processing happens, and the transformed data is stored in a
Databricks table.
5. Data Visualization: The processed data from theGoldLayeris then used as a source for
Power BI, which enables dashboarding and detailedanalysis.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Architecture
The processed data from this pipeline will be visualized using business intelligence tool like Power BI.
The architecture of our project is structured in a way to handle large volume of historical sensor data.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Prerequisites
Breakdown:
● Setup required:
○ Create an Azure account
○ Download the dataset
○ Download the Code
○ This is an OS-independent project, as the whole pipeline is set up on Azure
Cloud
○ SQL Server Management Studio and Access Database Enginefor integration with
Azure Managed SQL database
● Ways to interact with the Azure services:
○ Console- Using the Web-based UI
○ CLI- Using the Azure CLI tool in Terminal/CMD
○ For this project, we will be using the UI Console of Azure to work
with different Azure Services
● Best Practices for Azure Account
○ Enable Multi-Factor Authentication (MFA)
○ Protect the account by adding only authorized users/ groups to Resource Groups
○ Choose the region where services work
○ Create a Budget and set up an alert notification
○ Keep track of Unused ongoing Services
○ Delete Databricks Cluster after execution to cut down charges
○ Rotate all keys and passwords periodically
○ Follow the least privilege principle
○ Follow a naming convention for resources for better understanding, name after
project aim.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Services
Services used in the project:
Azure SQL Database:
zure Managed SQL Database service is fully managed by Microsoft, which means it handles
A
maintenance, backups, and updates, enabling users to focus on their applications rather than
database management. It is cost-sensitive which means it’s important to monitor the cost analysis
constantly. In the context of this project, an Azure SQL Server was created first, followed by setting up
an Azure Managed SQL Database within that server.
S QL Server Management Studio (SSMS) was used to establish a connection to the Azure SQL Server.
This connection was set up using credentials like Microsoft login, password, and SQL Server
authentication. Once connected, the large Excel file was imported into the Azure Managed SQL
Database.
Advantages of Azure SQL Database:
● It supports and processes both relational data and non-relational structures, such as graphs,
JSON, spatial, and XML.
● It is managed by Microsoft and data is spread across data centers, and with 24*7 support
teams. It also creates a high-performance data storage layer for applications and solutions,
with high-speed connectivity.
● Azure SQL Managed Database takes care of database infrastructure, patches, and
maintenance automatically. This significantly reduces the overhead of managing hardware and
s oftware updates.
● We can scale the database's compute and storage resources dynamically based on workload
demands.
● With data being stored across different data centers managed by Microsoft, there is no loss of
data even in the case of outages or disasters
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Disadvantages of Azure SQL Database:
● W hile Azure SQL Managed Database offers flexibility in pricing, it can become complex for
users who are not familiar with the pricing structure. If you select the geo region by mistake, it
will incur more than standard charges.
● Since Azure SQL Managed Database is a fully managed service, you have limited control over
the underlying operating system and database settings compared to an on-premise SQL Server
● For workloads with very high transaction rates or extreme performance demands, Azure SQL
Managed Database might face limitations.
● Azure SQL Managed Database is tightly integrated with Azure, which could lead to vendor
lock-in.
● Azure SQL Managed Database is designed for relational data and SQL workloads.
Alternatives:
A
● mazon RDS (Relational Database Service)
● Google Cloud SQL
● MySQL
Azure Logic App:
hether it’s managing data, coordinating processes, or connecting disparate systems, Azure Logic
W
Apps offers a robust, user-friendly solution to streamline every data operation. Its low-code, visual
designer interface makes it easy to build and manage complex workflows without needing extensive
programming skills. It has become an indispensable source for enterprises to fetch a variety of data
from different sources. Azure logic app has more than 200 connectors which makes it a best choice for
data integration and retrieval.
In the context of this project, Azure Logic Apps was utilized to automate the process of fetching data
from an Azure Managed SQL Database and storing it in Azure Blob Storage. The workflow was set up
with an HTTP request trigger, allowing the Logic App to be invoked via a URL for versatile integration.
Advantages of Azure Logic App:
● E asy to Use:Azure Logic Apps provides a visual designinterface that allows non-developers to
create workflows easily.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
● A PI and Custom Connectors:Logic Apps can also integrate with custom APIs, enabling you to
extend the system to meet unique integration needs.
● Elastic Scalability: Logic Apps scale automatically based on the workflow demand. If your
workflow requires higher processing power
● Environment Support: Logic Apps supports differentenvironments (development, staging,
and production).
● Automate Repetitive Tasks: Logic Apps are ideal forautomating business processes such as
approvals, data synchronization, notifications, and more.
Disadvantages of Azure Logic App:
● D ifficulty in Managing Complex Workflows:As workflowsgrow in complexity, they can
become harder to manage and maintain.
● Cost Predictability: Although Logic Apps follows aconsumption-based pricing model, it can be
challenging to predict costs
● Limited Debugging Capabilities: While Logic Apps provideslogging and monitoring,
debugging complex workflows or issues in real time can be challenging
● Complexity:As our project will inevitably get biggerand bigger, it becomes increasingly
difficult to keep track of which branch we are working on and find where we left off as it
s tarted to resemble something of a spider web.
● Limited Fine-Grained Control: While Logic Apps providesa wide range of triggers and
actions, it may not have complete control over execution sequences or the internal state of a
workflow.
Alternatives:
A
● WS Step Functions
● Google Cloud Workflows
● Apache Airflow
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Azure Storage Account:
zure Storage Account is a fundamental service in Azure that provides highly scalable, durable, and
A
s ecure cloud storage for a variety of data types. It serves as the container for storing data in different
formats such as blobs, files, queues, and tables. There are several types of storage accounts in Azure,
each tailored to specific needs, such as Blob Storage, and Azure Data Lake Storage Gen2.
a) Blob Storage Account:
● P urpose: The storage account comes under the AzureStorage account service which is
designed for storing large amounts of unstructured data, such as text or binary data.
It's typically used for storing files like images, videos, documents, backups, and logs. In
our project, the container is created inside the blob storage account to store the JSON
data coming from the execution of Logic Apps.
b) Azure Data Lake Storage Gen22 (ADLS):
● P urpose: It isBuilt on top of Azure Blob Storage,but Data Lake Storage Gen2 is
optimized for big data analytics workloads and provides more advanced features for
s toring and managing data at scale. In Our project, ADLS acts as a data source in Azure
Databricks during the execution of spark jobs.
Advantages of Azure Storage Account:
● W ith the ability to scale horizontally across both Blob Storage and Data Lake Storage Gen2, the
system can handle increasing data loads from both unstructured raw data
● Using Blob Storage for frequently accessed raw data and Data Lake Storage Gen2 for
large-scale data processing ensures we pay only for the storage you need while maximizing
cost efficiency.
● Both storage types provide a robust set of security features, but Data Lake Storage Gen2 offers
more granular control over who can access specific parts of data.
● By using both storage accounts, it is ensured that raw data in Blob Storage is highly available
and resilient to failures
● Raw JSON data stored in Blob Storage can be easily moved to Data Lake Storage Gen2, where
it can be processed using Spark jobs, machine learning models
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Disadvantages of Azure Storage Account:
● U sing Blob Storage for raw data storage and then moving data to Data Lake Gen2 can
introduce performance overheads, especially with frequent file writes and reads
● Both Blob Storage and Data Lake Storage Gen2 require external tools to perform data
transformations or processing tasks.
● Neither Blob Storage nor Data Lake Storage Gen2 provides native querying capabilities
s ufficient for advanced analytics and querying directly from the storage.
● Depending on the volume and frequency of data transfers, extra costs can add up.
● BothBlob StorageandData Lake Storage Gen2requireexternal tools to handle advanced
data transformations, processing, and analytics
● There is no snapshot mechanism or automated backup for Azure Files.
Alternatives:
A
● mazon S3 (Simple Storage Service)
● Google Cloud Storage
● IBM Cloud Object Storage
Azure Data Factory(ADF):
zure Data Factory (ADF) is a cloud-based data integration service provided by Microsoft Azure. It
A
allows you to create, schedule, and orchestrate data workflows. ADF helps move and transform data
from one location to another, using a visual interface or a code-based approach. In this project,Azure
Data Factory (ADF)will be used to orchestrate themovement of raw data stored inAzure Blob
StoragetoAzure Data Lake Storage Gen2 (ADLS). ADF'sCopy Pipeline will be used, and the JSON
data in Blob Storage will be transferred to ADLS.
Advantages of Azure Data Factory:
● S calability:ADF is highly scalable, making it suitablefor both small and large-scale data
integration projects.
● Integration with Azure Services: ADF integrates seamlesslywith other Azure services, such
as Azure Synapse Analytics, Azure Data Lake Storage, and Azure Machine Learning.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
● C ost-Effective: ADF offers a pay-as-you-go pricingmodel, which means you only pay for the
resources you use.
● Security: ADF provides robust security features, includingdata encryption, network isolation
● Ease of Use: The intuitive visual interface of ADFallows users to create and manage data
pipelines without extensive coding knowledge.
Disadvantages of Azure Data Factory:
● L imited Transformation Options:While ADF is greatfor data movement, its data
transformation capabilities are less flexible compared to other services like Azure Databricks
or SQL-based solutions.
● Complex Error Handling: While ADF has error-handlingmechanisms, they can be cumbersome
to configure.
● Data Movement Restrictions: Although ADF supportsvarious data sources and destinations,
it may not always offer the same level of integration as other services
● User Interface Limitations: The ADF user interfacecan sometimes be unintuitive, especially
for users unfamiliar with the platform.
● Limited Customization for Transformations: ADF’s built-intransformation capabilities are
s omewhat limited, especially for complex data transformations.
Alternatives:
A
● WS Glue
● dbt
● Apache NiFi
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Azure Databricks:
zureDatabricks,developedincollaborationwithMicrosoft,isamanagedversionofDatabricksthat
A
allows Azure customers to s et up with a s ingle click, s treamline workflows, and access s hared
collaborative interactive workspaces. It facilitates rapid collaboration among data s cientists, data
engineers, and business analysts through the Databricks platform. Azure Databricks is closely
integrated with Azure s torage and compute resources, s uch as AzureBlobStorage,DataLakeStore,
SQL Data Warehouse, and HDInsights.
Advantages of Azure Databricks:
● W hile AzureDatabricksisSpark-based,itallowscommonlyusedprogramminglanguageslike
Python,R,andSQLtobeused.TheselanguagesareconvertedinthebackendthroughAPIs,to
interact with Spark.
● DeployingworkfromNotebooksintoproductioncanbedonealmostinstantlybyjusttweaking
the data sources and output directories.
● Aside from those Azure-based s ources mentioned, Databricks easily connects to s ources
including on-premise SQL servers, CSVs, and JSONs.
● While Azure Databricks is ideal for massive jobs, it can also beusedfors maller-scalejobs,
deployment, and development/ testing work.
● Databricksnotebooksallowforreal-timecollaborationamongdataengineers,datas cientists,
and analysts using version control.
Disadvantages of Azure Databricks:
● D espite its detailed documentation and intention to simplify data processing, many users find
Databricks’ lakehouse platform daunting to master.
● Commands issued in non-JVM languages need extra transformations to run on a JVM process.
● While it offers a secure, collaborative environment with different services and integrations,
these enhancements come at a cost.
● Due to its cloud-native architecture, certain workloads might experience performance
overhead.
● While Databricks integrates well with Azure storage options, its native storage capabilities are
not as extensive
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Alternatives:
A
● mazon EMR (Elastic MapReduce)
● Google Cloud Dataproc
● Apache Spark on Kubernetes
Power BI:
icrosoft Power BI isa business intelligence toolthat helps users analyze and visualize data to make
M
informed decisions. It allows users to connect to a variety of data sources, transform the data into
meaningful visualizations, and create dashboards to help make data-driven decisions. Power BI is
widely used for its ease of use, interactivity, and ability to handle large datasets from multiple
s ources in real time.
Advantages of Power BI:
● E ase of Use: Power BI is user-friendly with a drag-and-dropinterface, making it accessible to
both technical and non-technical users.
● Cost-Effective: Power BI offers a free version withcore features and affordable pricing for the
Pro and Premium versions, making it accessible for businesses of all sizes.
● Integration with Multiple Data Sources: Power BI canconnect to a wide variety of data
s ources including databases, cloud storage, spreadsheets, and even live data streams.
● Real-Time Data Access: It supports real-time datarefreshes, enabling users to work with the
latest data and make timely decisions.
● Customizable Dashboards: Users can create personalizeddashboards and interactive reports,
giving flexibility in visualizing data the way they want.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Disadvantages of Power BI:
● L imited Data Capacity (Free Version): Restrictionson data storage and sharing in the free
version.
● Performance with Large Datasets: Can slow down withvery large datasets.
● Steep Learning Curve: Advanced features and customizationscan be difficult to learn.
● Limited Customization for Visualizations: Less flexibilitycompared to other BI tools.
● Data Refresh Limits: Limited refresh frequency, especiallyin the free version.
Alternatives:
T ableau
●
● Qlik Sense
● Google Data Studio
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Execution:
1. Creating Azure Resource Group
ResourceGroupiscreatedinAzuretomanageallAzureServiceseffectivelyinoneplace.Itis
A
easier to monitor, update, or delete all Azure resources together as a single unit.
a) In the search bar, enter"Resource Groups"and selectit from the suggestions.
b) In the top-left corner, click onCreate.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
c) In the Resource Group Name field, enter the desired name for your resource group. Then,
click onNext:Tags
d) U nder Tags, in the Name field, enter s omething likedeveloper.IntheValuefield,enteryour
name.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
e) Then Click on the Create option and a resource group will be created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) N ow, the resource group will be used as a central location to create Azure Service in the
project.
2. Creating Azure SQL Database
a ) Click on the Resource Group you just created, then s elect the “Create” option within the
resource group.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) In the search bar, enter"Azure SQL Database"ands elect it from the suggestions.
c) In the top-left corner, click onCreate.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) IntheResourceGroupfield,clickonthedrop-downmenuands electtheresourcegroupyoujust
d
created.
e) In theDatabasefield, enter the name of the databaseas you want.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) Now, click onNewin theServerfield to createa new server for your Azure SQL database.
g) In theServer Detailspage, under theServer Namefield, enter the desired name for your server.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) Select an available location in the Location field, and if youencountererrors,chooseadifferent
h
location from the dropdown.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
i)IntheAuthenticationMethodfield,s elect“UsebothSQLandMicrosoftEntraAuthentication”.
The reason behind this is to give access to both SQL authentication and MS-authorized users.
j) Select"Set admin"to grant the Azure authorizeduser administrative access to the SQL Server.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
k) In theMicrosoft Entra IDtab, select your user in Azure, then clickSelect.
l) In the Username field, enter the username you created for yourSQLServer,andinthePassword
field, enter the corresponding password.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) UsernameandpasswordforyourSQLServer,s avetheminas ecurelocation,s uchasaNotepad
m
file.
n) After specifying your SQL Server username and password, click on"OK".
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
o) Now, the SQL Server you created will be automatically specified in the server field.
) Now, s et the Workload to Development, as the project does not include production-level
p
workloads.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
)FromBackupStorageRedundancy,chooseLocallyRedundantBackupStorages inceitdoesnot
q
incur high costs.
r) Then, click onNext: Networking
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
s ) Under the Connectivity Method, choose P ublic Endpoint, as it will allow your Azure SQL
Database to connect to your local environment.
t ) In the Firewall Rules, s elect both options as"Yes"toallowaccesstoyourAzureSQLServerfrom
both Azure services and specific IP addresses.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
u) Keep connection policy and Encrypted connections by default and then click on“Review + Create”
v )Afterthat,validationmaytakes ometime.Oncecomplete,yourAzureSQLServeranddatabasewill
be successfully created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) After deployment of Azure SQL Database and Azure SQL Server is complete, click on “Go to
w
Resouce”.
x) The Azure SQL Server and Azure SQL Database have now been successfully created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
3. Importing Data in Azure SQL Database
a)ToimportdatafromanExcelfileintoanAzureSQLDatabase,youneedtheSQLServerManagement
Studio (SSMS) application.
) Navigate to your browser, type "Download SQL Server Management Studio"inthes earchbar,
b
and click on the first website that appears.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
c )Oncetheapplicationisdownloaded,installands etituponyourlocalsystem.Alsomakes urethat
Access Database Engine is also downloaded on your local system. It can be downloaded from the
browser
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) OpentheSSMSapplication,andundertheServerNamefield,s pecifythenameofyourAzureSQL
d
Server that you created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) From the dropdown of the Authentication Method, choose SQL Server Authentication. It helps to
e
access Azure SQL Database, especially when using non-Azure-based tools like SSMS.
f )IntheLoginandPasswordfields,s pecifytheusernameandpasswordofyourAzureSQLServerthat
you saved earlier. After that, Click onConnect.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
g) In the left-side panel, the specification of Azure SQL Database and the server listed there.
h) ExpandDatabasesby clicking on the+icon, andthen your Azure SQL Database will appear there.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
i) To import data from a local ExcelfiletoAzureSQLDatabase,right-clickonthedatabase,clickon
Tasks, and then selectImport Data.
j)UndertheChooseaDataSourcepanel,s electMicrosoftExcelfromthedropdownlistasthedata
s ource.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
k) Then, click onBrowseto select the Excel file available on your local system, and choose your file.
After selecting the file, click on open to select the file.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l) Click on next after selecting the data source
)FromtheChooseaDestinationpanel,s crolldownands electMicrosoftOLEDBP roviderforSQL
m
ServerunderDestination,asthedestinationisyourAzureSQLDatabasewhereyouneedtoloadthe
data.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
n) In theServer Namefield, specify the name of yourAzure SQL Server.
)IntheAuthenticationMethod,chooseUseSQLServerAuthentication,andintheUsernameand
o
Passwordfields, specify the login and password youcreated for your SQL Server. Then click on “Next”
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
p) Choose the first option “Copy Data from one or more tables” and then click on Next.
) Click on the Destination and change the name of the destination to Table in the Azure SQL
q
Database. You can name it according to your preference.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
r )Oncethenameofthetableischanged,clickonthe“editmappings”tochangethelengthofwords
in columns of the dataset.
s ) Change the mappings of the last columns from 255 to 1200. This allows for a larger character
length, accommodating data that exceeds the default limit of 255 characters.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
t) Click on 255 and then type 1200. Once the mapping is changed, then click on “OK”
)Then,clickonNexttwicetoproceedthroughthetwoconfigurations teps,andfinallyclickonFinish
u
to complete the import process.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
v ) Once you click on “Finish”, the process of loading data from the Excel file to the Azure SQL
database gets started.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
)Followingthis,allrowshavebeens uccessfullyuploadedtotheAzureSQLDatabase,andtheresult
w
will show asSucceeded.
x )Tocheck,whetherthedatahasbeenuploadedtothetableinAzureSQLDatabaseornot.As imple
SQL query can be executed on top of the table.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
4.Creating Azure Blob Storage
a )Inthes earchbar,typeAzureStorageAccountsands electtheStorageAccountsoptionfromthe
s uggestions.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) Click onCreateto begin the process of creatinga new storage account.
c )IntheCreateaStorageAccountpanel,undertheResourceGroups ection,clickonthedropdown
and select the resource group you just created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) Specify the name of your storage account as desired. Ensure the name is unique, as storage
d
account names must be globally unique.
e ) UnderPerformance, specifyStandard, as it doesn'trequire premium performance for workload,
which is relatively low.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) Click on the“Next”two times for the next twopages and then Click on“Review + Create”
g) Then click on“Create”to create the Azure BlobStorage account.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
h) Validation might take some time. After the validation is complete, click onGo to the resource.
i) Now, on theBlob Storage Accountinterface, inthe left-side panel, click onData storage. Under
Data storage, click onContainersto create a containerfor Azure Blob Storage.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j) click on+ Containerto create a container. Thepurpose of creating a container is to store all the CSV
and log files inside the Blob container.
k ) In the right-side pop-up panel, specify the name of the container as per your need and then click on
“Create”. Make sure the name is unique within thes torage account.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l) Click on the container that you created to open it and manage its contents.
5. Creating Azure Logic App
a ) In the s earch bar, type Logic Apps and s elect Logic Apps from the s uggestions.Thepurposeof
Logic Apps is to create aworkflowthatfetchesdatafromtheAzureSQLDatabaseandloadsitinto
Azure Blob Storage.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) Click onAddto create a new Logic App.
c ) Under theStandardplan, click onWorkflow serviceplanto create the Logic App with lower
charges, as you're handling a lighter workload. Then click on “select”
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) In theCreating Logic Appinterface, under theResource Groups ection, click on the dropdown and
d
s elect the resource group you created.
e ) In theLogic App Namefield, specify the nameof the Logic App as desired. Ensure the name is
unique. Then click on NEXT: Storage
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) In theStorage Accounts ection, select the storageaccount you created from the dropdown.
y specifying this account, you are ensuring that all logs and artifacts generated by the Logic App will
B
be stored in the selected storage account.
g) Then click onNext: networking
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
h)Enable public access for the Logic App to allowexternal access. Then click on“Review+ Create”
i) Validation might take some time. Once complete, click onGo to resource.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j)In the Logic App interface, in the left-side panel,click onWorkflows. Then, under Workflows, click
on theworkflow.
T his will allow you to create the workflow to load data from the Azure SQL Database to Azure Blob
Storage.
k)Then, click on + Add, and then click Add againto create the workflow.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l)Now, in the right-side panel, specify the nameof theLogic App. Also, set the workflow toStateful.
Then click on “Create” and this will create the workflow of Logic App.
T he reason for selecting Stateful is that it provides better efficiency when uploading large data, as it
retains the state across workflow runs.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
m) Then click on the Workflow you just created.
) In the workflow panel, on the left-hand side, click onDesigner.This will allow you to create the
n
workflow using drag-and-drop actions from the available options.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) This is the interface of the workflow using theDesigner, where actions and triggers will be dragged
o
and dropped for the execution
→ Creating Logic App user in Azure SQL Database
T o fetch data from Azure SQL Database and load it into Azure Blob Storage, Azure Logic Apps needs
access to the Azure SQL Database. A connection must be established between the Logic App and the
Azure SQL Database. The mechanism involvesassociatinga user with the Logic Appthat has the
necessary permissions to access the database. To achieve this,a useris created within the Azure SQL
Database using a SQL query. Then, the Logic App user is granted access to the Azure SQL Database to
allow the workflow to execute.
) Open a new tab in your browser and navigate to the Azure portal. Then, access yourAzure SQL
p
Databaseby searching for it in the portal
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
q)Click on the azure sql database you created.
r)Now, click onQuery Editor.You will have two optionsto access your Azure SQL Database:
U
● se the username and password you created for the SQL Server.
● Log in directly using your Azure email ID.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
s )Click onTables, and you will see the table thatwas populated using SSMS (SQL Server Management
Studio) displayed here.
t ) First Query
First, Run the SQL query to create a user for theLogic App and enable it within your Azure
environment.
reakdown of query:The queryCREATE USER [waterQualityLA1]FROM EXTERNAL PROVIDERis used
B
to create a user in Azure SQL Database that is linked to an external authentication provider for logic
App.
→ CREATE USER [waterQualityLA1] FROM EXTERNAL PROVIDER
REATE USER: This command is used to create a newuser in the Azure SQL Database.
C
waterQualityLA1- Name of Logic App (Replace withthe name of the Logic app you created)
FROM External Provider- This indicates that the useris being authenticated by an external provider
(such as Azure Active Directory), rather than using a traditional SQL authentication
(username/password). In this case, it's the Logic App user being created through Azure AD.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Second Query:
Run this query in the editor:
→ ALTER ROLE db_datareader ADD MEMBER [WaterQualityLA1]
reakdown of Query:This query allows the Logic Appto read data from the Azure SQL Database.
B
Since the Logic App will fetch data and move it to Azure Blob Storage, it only requires read
permissions and permission was assigned by changing the role of db_reader. Logic App user
[WaterQualityLA1] was added as a member to read the data from Azure SQL Database.
ALTER ROLE: Modifies the membership of a databaserole.
db_datareader Role: A built-in role granting read-onlyaccess to all tables and views in the database.
ADD MEMBER: Adds a user or group to the specifiedrole.
db_datareaderrole for
[ WaterQualityLA1]: The Logic App's Azure AD user,added to the
read-only access to the database.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
6. Creating Ingestion Workflow in Azure Logic App
In this segment, a workflow will be created in the Logic App. Actions and triggers will be dragged and
dropped to design a comprehensive workflow. This workflow will fetch data from the Azure SQL
Database and store it in the Azure Blob Storage container (in JSON format) you created earlier.
a ) Return to the tab where the Logic App workflow was created or navigate to the Logic App in the
Azure portal, and click on the workflow you previously created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) Then click on“Designer”from the left side panel.
c) The first step is to create an HTTP trigger in the workflow. For that Click on the “trigger” icon
T his will generate an API-like URL that can be used to call the workflow on demand, reducing the need
for manual intervention.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) In the search bar of the Logic App Designer, type“When an HTTP Request is received"and select
d
it from the options.
e) Then click on + icon to add another action “Execute query” in the workflow.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) Type“Execute Query action”and then click on action“Execute Query”under SQL server.
The Execute query action is added in the workflow to query the entire data from Azure SQL Database.
g ) Once theExecute queryaction is added, you willneed to specify the configurations of Azure SQL
database in execute query field.
For that, click onchange connection.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
h) Now, in the connection details, three configurations need to be specified:
. C
1 onnection name: Give a name to the connection asper your preference.
2. Authentication type: From the dropdown, Choose “Managedidentity”
3. Managed Identity Type:From dropdown, choose“systemassigned”
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
. S erver name:In this field, specify the name of theazure SQL Server you created
4
5. Database name:In this filed, specify the name ofthe Azure SQL Database you created
The name of the AZzure SQL Server and Azure SQL Database can also be copied by navigated to the
→
Azure SQL Datavase service and then click on the Azure SQL Server which you created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
6. O
nce the connection details are specified, click onCreate New andthis will successfully
establish the connection to Azure SQL Database using this action.
i) After specifying the connection details, theSQLqueryprompt page opens. Here, aSQL query will
be executedthat retrieves all the data from the AzureSQL Database.
Query:Select * from [dbo].[Waterqualitydata]
→ Breakdown of Query:
SELECT *:
● Retrieves all columns from the table.
FROM [dbo].[Waterqualitydata]:
● [ dbo]: Default schema in SQL Server and Azure SQLDatabase.dbo (Database Owner)is the
default schema in SQL Server and Azure SQL Database.
● [Waterqualitydata]: The table name from which datais fetched.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
fter writing the query here, just click on the + icon to add another action after the“Execute Query”
A
Action
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j) In the search bar, type“blob”and click on the“see more”option corresponding tothe Azure Blob
Storageaction.
k) Then scroll down, until you find this action“Uploadblob to storage container”. Select this action.
T he"Upload Blob to Storage"action in Azure LogicApps is used to take data retrieved from an Azure
SQL Database using the Execute Query action. Once the data is fetched, it is converted into JSON
format. This JSON data(blob)is then uploaded toan Azure Blob Storage container.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l) After that, click on the“Upload blob to storagecontainer”action. In the right panel, it is asking
for details. But first, we need to establish the connection with theblob storage containerthat we
created.
Click on“Change Connection”in the bottom right cornerand adda new connection.
m) Then, specify the details as mentioned on the connection page.
. C
1 onnection name:Give a name as your preference.
2. Authentication type:From the dropdown, selectStorageaccount connection string.
( To establish a connection with Azure Blob Storage and upload the data into the Blob container, use
Azure Blob Storage Access Keys or a Connection String for authentication)
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
3. Storage Account Connection string:Here, specify yourBlob Storage account access key.
→ Here are the steps to get the Azure Blob Storage account access key:
. O
1 pen another tab in browser and In the Azure portal, go to yourStorage Account.
2. In the left-hand panel, underSecurity + networking,click onAccess keys.
3. Click on show forconnection string, copy it.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
. Navigate back to the tab where Azure Logic App is open and paste theconnection stringinthe
4
Storage Account Connection String.
Then click on“Create New”and it will create a connectionwith the Azure Blob Storage.
n) Now, once the connection is created. specify the following:
. C
1 ontainer Name:In this field, specify the name ofthe blob container you created earlier.
2. Blob Name:In this filed, specify the name of theblob/ file that will be loaded to the Blob
Container. This is nothing just a name that you will give to your file which is going to be loaded
to the blob container (in json format).
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
3. In the Content field, choose the icon , this icon indicates thisupload blob to container
action will store the results of the "Execute Query" action in the Blob container in JSON format.
T hen click on theResult item(as blob action storesthe output of the execute query action and
uploads the results to the blob container in JSON form).
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
ote:As soon asthe result itemis selected,eachloop kind of action gets added to the“upload
N
blob to storage container”action. It means that eachresult is going to loop through the data and
upload to the blob container in JSON format.
o) Now, click onSaveoption to save the workflow
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) After the workflow is saved successfully, click on theoverviewoption in the left side panel toget
p
back to the main interface of your workflow.
) Click on theRunoption, then click onRunagain.This will trigger the workflow and start its
q
execution.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
r ) Wait for a few seconds, then click ontrigger idto check the results. If the action is markedwitha
green check, it means the workflow executed successfullywithout errors.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
s ) Then navigate to theBlob containeryou createdand check whether the JSON file is uploaded to
the Blob Storage container or not.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
S ince the JSON file is in the Blob Storage container, it will now be copied to the ADLS container using
the Copy Pipeline in Azure Data Factory (ADF). To do this, first create an ADLS storage account and an
ADLS container.
7. Creating Azure Data Lake Storage Account (ADLS Gen2) and its container
a ) Navigate to the Azure home portal, typeAzure StorageAccountsin the search bar, and select
Storage Accountsfrom the suggestions.
b) Click on Create
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
c ) In the resource group field. From the dropdown, select theresource groupyou created and also
s pecify the name of your ADLS Storage account as you want and it should be unique.
) In the primary Servcie, from the dropdown - choose“Azure Blob Storage or Azure Data Lake
d
Storage”option.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
e ) Click on next and in the advanced tab, enablehierarchicalnamespaceto create Azure Data Lake
Storage Gen2.
f ) Then click on “Review+Create” andthen click on“create” to create ADLS Gen2 Accountand then
wait until the validation is done.
g) Once the deployment of ADLS account is done, click on“Go to Resource”.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
h) In the left-side panel, click onData Storage,and underStorage, click onContainers.
i) Click on+Container. In the right-side panel, specifythe name of the ADLS container as desired,
then click onCreate.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j) Navigate to theADLS Storage Account (not storagecontainer)and in the left-side panel of the
ADLS Storage Account, scroll down and click onDataManagement. Under theData Management
option, click onData Protection.
Once you find theData Protectionoption, in the right-hands ide panel, uncheck the two options:
. E nable soft delete for blobs
1
2. Enable soft delete for containers
This will disable soft delete for both blobs and containers in the ADLS Storage Account.
T he reason behind disabling this option is to get more control on when we want to terminate
the present in the Storage Account and container.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
8. Creating Azure Data Factory
T he copy pipeline will be created in Azure Data Factory (ADF) to orchestrate the movement of the JSON file
from the Azure Blob Storage container to the ADLS container. ADF will be responsible for initiating and
managing the data movement in this project.
a ) Navigate back to the Azure portal and in the search bar, typeAzure Data Factory. SelectData
Factoriesfrom the suggestions.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) Click on+Create
c) In theResource Groupfield, select the resourcegroup you created from the dropdown.
Also, specify the name of the Azure Data Factory as you want.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
d) Then click onNext and in the Networking tab,makes ure the public endpoint is selected.
e ) ClickNexttwo times, then clickReview + Create.The validation of deployment might take some
time. Once the deployment is successful, clickGoto Resource.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) Click onLaunch Studioto open the Azure Data Factoryinterface.
g) Click onNewand then click+Piplineoption.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) In theActivitiess ection, click on theMove &Transformoption, then click on theCopy Data
h
pipeline and drag it to the white canvas on the right side.
i) Click on theCopy Pipelineon the white canvas,then in theGenerals ection, specify the name of
the pipeline as desired.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j) Click on theSources ection, then click onNewto set the new source.
In this project, the source is theBlob Storage containercontaining the JSON data file.
k )In the right-side panel, under theSelect a datasourcefield, typeBloband select theAzure Blob
Storageoption and click oncontinue
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l) From the select format option, click onJSON Formatand click onContinue.
Since the data file in the Azure Blob Storage container is in JSON format.
) Give a name to the source and then click on+Newto create a linked service for the Azure Blob
m
Storage container.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) In theNew Linked Servicewindow, give any nameto the linked service and keep the default
n
s ettings, and then, from theStorage Accountsdropdown,s elect the Azure Blob Storage account you
created.
Then click on“Create”
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Doing this, the linked service has been created successfully between Azure blob storage and ADF.
) Now,set the propertiesto define the exact locationwhere the JSON file is located in the Azure blob
o
s torage container.
Click onfile iconto browse the location of the AzureBlob container which hasa JSON file.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
p) Then click on theAzure blob containeryou createdand then click onOK
q) Now, click on the file that is present in theAzureBlob Storage Container.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
r) Then clickOk and this will set up the source forAzure Data Factory
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
s ) Click on Sink to set up the ADLS Gen2 container as a sink to store JSON file which is currently in the
Azure Blob Storage Container.
t ) In the right-side panel, under theSelect a datasourcefield, typeAzure Data Lake Storage Gen2
s elect theADLS Gen2option, and click oncontinue.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
u) From select format, selectJSONas a new format,and then click onContinue.
v ) In the linked service, give a name to thelinkedserviceas you want and then click on+newto
create a new Linked Service.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
) Specify a name for the linked service, select theADLS storage accountyou created from the
w
dropdown and then click oncreate.
x ) Then from this panel, specify the details, click on the file icon, and browse to the ADLS container
that you created earlier.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
y) Select theADLS Comatinerthat you created andthen click onok.
z ) Then click on theValidateoption. Validation inADF ensures that the pipeline configurations are
correct and that all connections are properly established before execution.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
fter that click ondebugoption and it will trigger the“Copy pipeline”. As a result, the pipeline waill
A
executed successfully.
T he successful pipeline execution moves the JSON file from Azure Blob Storage to the ADLS container.
To verify, navigate to the ADLS container, and you will see the JSON file there.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
8. Creating Azure Databricks
a ) Navigate to the Azure portal, type "Azure Databricks"in the search bar, and select"Azure
Databricks"from the suggestions.
b) Click on "+Create" to create an Azure Databricksworkspace.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
c) In the"Resource Group"field, select the resourcegroup you created from the dropdown.
) In the"Workspace Name"field, specify a uniquename for your Databricks workspace. Keep all
d
other settings as default. From the pricing tier, select“standard one”as premium is costly.
Then, click on"Review + Create"and finally click"Create"to deploy the workspace.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
e ) Wait for the deployment of Databricks to complete. Once finished, click on"Go to resource"to
access your Databricks workspace.
f ) Click on theLaunch workspaceto open the Databricksinterface. Here, creating a cluster and
execution of code in a notebook to process the data will be done.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
g) In the Databricks interface, the left panel includes the following sections:
● orkspace
W
● Compute
● Job Runs
● Dashboards
● Catalog and so much
) Firstly, a cluster will be created inside Databricks Workspace. In the left side panel, click on
h
Compute.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
i)Click onCreate Compute.
cluster is necessary for Databricks to run code and process data during the running of jobs, as it
A
provides the computational resources.
j)In the policy, from the dropdown - select“unrestricted”orgo with“Personal Compute”if face any
error.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
k )Keep all other configurations as shown in the screenshot,this willreduce costs and save compute
resources.
Click on“Create Compute”to create the cluster.
T he cluster may take some time to create. Once it is ready, aright signalwill appear next to the
cluster name.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
8. Azure Databricks Medallion Architecture
S ince the raw JSON Data file is in the ADLS Container, it will be processed and structured using the
medallion architecture, flowing throughbronze, silver,and gold layers. Each layer will have a
dedicated Databricks notebook in the same workspace for processing, analyzing, and cleaning data
with distinct purposes. After the final transformation is done in the gold layer, the data will be
uploaded to the Hive metastore in Databricks. This data will then be connected to Power BI to create
visualizations.
hy is Medallion Architecture considered the best standard in data engineering for processing raw
W
data and delivering high-quality data?
medallion architecture is a data design pattern used to logically organize data, with the goal of
A
incrementally and progressively improving the structure and quality of data as it flows through each
layer of the architecture (from Bronze ⇒ Silver ⇒ Gold layer tables).
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
a) In the left-side panel, click onWorkspace.
) In the top-right corner, click onCreate, thens electNotebook. Repeat this process three times to
b
create separate notebooks for each layer: Bronze, Silver, and Gold. All notebook will be created in this
s ame workspace.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
ll three notebooks (Bronze, Silver, and Gold Layer) will be in the same workspace as shown above.
A
The reason is to ensure seamless integration, easier collaboration, centralized management, and
s treamlined data flow across the Bronze, Silver, and Gold layers.
c) Instead of creating notebooks, import all three notebooks into the same workspace
It can be done by clicking the (:)icon, selectingImport, and uploading notebooks from your local
system to the Databricks workspace.
d) Click on browse to upload all three notebooks from your local system to the Databricks workspace.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
The execution of the notebooks will follow a sequential order:
Bronze Layer Notebook→Silver Layer Notebook→GoldLayer Notebook.
e)Bronze layer (raw data): Bronze Notebook
T he Bronze layer is where we land all the data from external source systems. In this project, the
Bronze Layer Notebookwill be used to ingest the rawJSON data from the ADLS container and load it
into a DataFrame for further processing.
Code Breakdown:
The code will be explained step by step, cell by cell.
1. T his code imports essential libraries to set up a Spark session and enables the use of
DataFrame operations.SparkSessionis the entry pointfor Spark, and colhelps reference
columns. The execution of this cell sets up the environment for processing data using Spark.
2. It creates a Spark session named"WaterQualityPipeline"(give a name as per your
preference). The execution of this cell will initializethe Spark environment in the Databricks
notebook.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
3. It sets two variables:adls_storage_nameholds thename of the ADLS (Azure Data Lake
Storage) account, andadls_storage_access_keys toresthe access key for authenticating and
accessing the ADLS account. Replaceadls_storage_namewith the ADLS Storage account
name and<storage_access_key>with the actual accesskey to access the data.
T o get the<storage_access_key> of the ADLS storageaccount
Navigate to the Azure portal and go to the ADLS storage account you created.
a ) In the left-hand panel, click on Security + Networking.
b) Under Security + Networking, select Access keys.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
c) C
lick on Show and then copy the key and paste the key in the cell for variable
(adls_storage_access_key)
4. It configures the Spark session to use the ADLS storage account by setting the account key. It
allows Spark to authenticate and access the ADLS storage.
5. It reads a JSON file from the Azure Blob Storage into a DataFrame.
● fi le_location =
"wasbs://<adls-container-name>@<adls-storage-account>.blob.core.windows.net": This
s pecifies the path to the JSON file. Replace <adls-container-name>> withAzure Data Lake
Storage containerand <adls-storage-account> withyourAzure Data Lake Storage account
name.
● bronze_df = spark.read.json(file_location): This reads the JSON file from the specified location
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
and loads it into a DataFrame (bronze_df).
6. T he code sets up writing the DataFramebronze_df(whichcontains the raw data from the JSON
file) into a Delta Lake table in theBronze Layerof the medallion architecture.
b
● ronze_df:The DataFrame contains the raw data loadedfrom Azure Blob Storage.
● .write:This is an operation to write the DataFrameto a storage location.
● .format("delta"):Specifies that the data will bewritten in the Delta format, and is optimized
for large-scale data processing.
● .mode("overwrite"):This means that if the data alreadyexists at the specified location, it will
be overwritten.
● .save("/mnt/datalake/bronze/water_quality"):Thiss pecifies the path where the data will
be saved, i.e., theBronze Layerlocation in the /mnt/datalake/bronze/water_qualitydirectory.
7. This lists the files in the“/mnt/datalake”directoryof DBFS of Databricks.
f)Silver layer (cleansed and conformed data) - SilverNotebook
In this Silver layer, data from the Bronze layer is cleaned, merged, and processed to provide a more
refined and consistent view of key business data. This layer transforms raw data from the Bronze layer
into usable form for analysis in the gold layer.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
1. T his code snippet is the same as explained at the beginning of the bronze layer as it is
necessary for working with spark in the Databricks notebook.
2. It loads the data from the Bronze layer stored in Delta format in the specified location
(/mnt/datalake/bronze/water_quality)into a data framecalledbronze_df. The execution of
this cell will allow to access the raw data that was ingested into the Bronze layer
bronze_df.count()counts the total number ofrows in the DataFrame
3. This bronze_df
.
4. T he functioncount_missings(spark_df, sort=True) -bronze_df is your input Spark DataFrame
whichcounts the number of missing (null or NaN) valuesin each column of a given Spark
DataFrame (s park_df). It does this by using Sparkfunctions to identify and count null or NaN
values in each column. The function checks forNULLor NaN valuesin each column (excluding
timestamp,s tring, anddatecolumns). For each column,it counts the occurrences of NaN or
NULL values usingF.count(F.when(F.isnan(c) | F.isnull(c),c)).
Ifs ortisTrue, it returns the sorted DataFrame.
T he function is called(count_missings = bronze_df)to count the missing values in each
column ofbronze_df.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
5. T his code snippet removes columns from the DataFrame that do not contribute to the use case
and stores the data frombronze_df to silver_df (dataframe of the silver layer).It ensures
that only the relevant columns are kept for further processing
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
6. It renames the necessary columns to make them more meaningful for easier processing and
analysis.
7. T his code maps country abbreviations to their full names in the Countrycode column for better
clarity.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
8. T his code adds a new columnCountry_Nameto thesilver_dfDataFrame by mapping each
Country_Code to its corresponding country name using when-otherwise conditions.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
9. It maps the values in thewater_bodycolumn of thes ilver_df DataFrame to full water body
names (e.g.,"GW" to "Ground Water", "RW" to "RiverWater"). It then adds a new column,
Water_Body_Name, with these full names
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
10. It maps the values in theAnalyzed_Matrixcolumnof the silver_df DataFrame to full names
(e.g.,"W" to "Water", "W-DIS" to "Water - Dissolved").It then adds a new column,
Analyzed_Matrix_Name, with these full names.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
11. It writes the cleanedsilver_dfDataFrame to a DeltaLake table in the Silver layer.
It enables schema evolution(mergeSchema = true),which allows the schema to
automatically adapt to any new columns or changes in the data.
T he data is written in"overwrite"mode, it will replaceany existing data at the specified path
(/mnt/datalake/silver/water_quality_cleaned)in DBFS(Databricks file system).
After cleaning and filtering in the Silver layer, the data will be further processed in the Gold layer to
→
be refined and ready for use in the use case.
g)Gold layer:(curated for use-case) - Gold LayerNotebook
T he final layer of data transformations and data quality rules are applied here. The Gold layer is for
reporting and uses more de-normalized and read-optimized data.
ata from the Gold layer is typically used for dashboards, reporting, and model building, as it
D
represents the final transformed and high-quality data.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
1. T his code snippet is the same as explained at the beginning of the bronze layer as it is
necessary for working with spark in the Databricks notebook.
2. It loads the data from the Bronze layer stored in Delta format in the specified location
(/mnt/datalake/silver/water_quality_cleaned)intoa data frame calledsilver_df. The
execution of this cell will allow to access the cleaned data that was processed using the Silver
layer
3. T he line of codegold_df = silver_df.dropDuplicates()removes any duplicate rows from the
s ilver_df DataFrame, creating the final, clean dataset for the Gold layer.
4. Finding outliers in theMinimum_Valuecolumn
Breakdown:
A)
mean_val =
gold_df.select(F.mean("Minimum_Value")).first()[0]
stddev_val =
gold_df.select(F.stddev("Minimum_Value")).first()[0]
It calculates the average(mean)and the spread(standard deviation)of the values in the
→
Minimum_Valuecolumn and these values are necessary to calculate how far each value in
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
the column is from the average.
B)
gold_df_with_zscore = gold_df.withColumn(
"z_score", (F.col("Minimum_Value") - mean_val) / stddev_val
)
Minimum_Valuecolumn and The Z-score
It calculates theZ-scorefor each value in the
→
tells us how far away each value is from the average in terms of standard deviations. A high
Z-score means the value is far from the average, which might make it an outlier.
ote:(Z-score = z-scores are typically defined asdata points with z-scores
N
beyond a certain threshold, often±2 or ±3.)
C)
gold_df_with_outliers = gold_df_with_zscore.withColumn(
"MinimumValue_outlier", F.when( F.abs(F.col("z_score")) >
3, 1 ).otherwise(0) )
> It checks if the Z-score is greater than 3 orless than -3 (which means the value is
—
far from the average). If it is, the value is marked as an outlier (1), otherwise it's
marked as normal (0). Values with Z-scores beyond3 are considered outliers because
they are far away from the average.
D)
outlier_rows
=gold_df_with_outliers.filter(gold_df_with_outliers.Minimu
mValue_outlier == 1) print("Rows with Minimumvalue_outlier
= 1:") outlier_rows.display()
It is simply filtering and displaying the rows that are considered outliers based on
→
the conditionMinimumValue_outlier == 1.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
fter filtering out, the linegold_df = gold_df_with_outliersassigns the DataFrame
A
gold_df_with_outliers to the gold_df, essentially making the data from gold_df_with_outliers the final
version for the Gold layer.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
5. Finding outliers in theMaximum_Valuecolumn.
F ollow the same process used for theMinimum_Valuecolumn, but replace the column name
withMaximum_Valuein the code.
fter filtering out, the linegold_df = gold_df_with_outliersassigns the DataFrame
A
gold_df_with_outliers to the gold_df, essentially making the data from gold_df_with_outliers the final
version for the Gold layer.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
6. Finding outliers in theMean_Valuecolumn.
F ollow the same process used for theMinimum_Valueand Maximum_Valuecolumns, but
replace the column name withMean_Valuein the code.
fter filtering out, the linegold_df = gold_df_with_outliersassigns the DataFrame
A
gold_df_with_outliers to the gold_df, essentially making the data from gold_df_with_outliers the final
version for the Gold layer.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
7. T his line drops/ deletes thez_scorecolumn from the data frame. The purpose is we have
already calculated the outliers and now the z_score column is not needed.
8. This code splits theSampling_Periodcolumn into twonew columns:
a) Start_Date: It Extracts the first 10 characters (assumedto be the start date).
b) End_Date: It Extracts the last 10 characters (assumedto be the end date).
c) It uses thes ubstr()function to extract these portionsfrom theSampling_Periodand
creates the new columnsStart_DateandEnd_Date.
gold_dfdata to theGold layerin a Delta format:
9. This code writes the final
) f ormat("delta"): Saves the data in Delta format (fordata processing).
a
b) mode("overwrite"): It replaces or overwrites any existingdata at the specified location.
c) save("/mnt/datalake/gold/water_quality_aggregated"):Saves the data to the Gold
layer path in the data lake.
10. To write the final data frame (
gold_df
) to theHive metastore, follow these steps:
a) C
reate Database: In the notebook, create a databasethat will be visible in the Hive
metastore.
Code:
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Give a name to the database as you want (waterdb is the name of the database in this case).
b) R
un the following code snippet to load the data from thegold_dfdata frame into a
table (gold_table) within the database you created.
Code:
c) O
n the left side panel, click onCatalog, then underHive Metastore, select the
database you created.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
d) Y ou'll see the table (gold_table) that was createdin the Hive metastore, which contains
data in CSV format.
9. Connection to Power BI and visualizations
a) Open Power BI Desktop and click on “Get Data” and from the dropdown, click on“More”
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
b) In the search bar, typeAzure Databricksand clickonAzure Databricksfrom the suggestions.
c) Specify the details such asserver hostnameandHTTPpathof your Databricks environment.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
To get theserver hostnameandHTTP pathof your Databricksenvironment, follow these steps:
a ) In the Azure portal, navigate to yourAzure Databricksworkspace.
b) Once inside the Databricks workspace, click on theComputeicon on the left panel.
c) Select the cluster you want to use.
d) In theCluster details page, under theAdvanced Optionss ection, you will see theJDBC/ODBC
tab.
e) In this tab, you'll find theServer HostnameandHTTPPath.
Copy these values to specify them in your connection settings in Power BI
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
f) Once you specify both credentials, click on“OK”
g) In theAzure Databricks connection window, click onAzure Active Directoryfor
authentication.
Click onSign In.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
h) Click on the email id through which you have created your Databricks workspace.
i) Click onConnect.This will establish a connectionbetween Databricks and Power BI
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
j) In theNavigatorwindow, click onHive Metastoreandthen expand theDatabasess ection.
● Select thedatabaseyou created earlier.
● Find and select thegold_tableyou created to storedata in the Hive Metastore.
● ClickLoadto load the data from thegold_tableintoPower BI for visualization.
k) Now, in the right panel under theDatas ection, yourgold_tablewill be listed.
This indicates that the data fromgold_tablehasbeen successfully loaded into Power BI.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
l) N
ow, you can createvisualizationson top of the cleanedand high-quality data in Power BI to
extract valuable insights and information, such as:
● The level of determinands in water over the years.
● The concentration levels of determinands in different water bodies over time.
● Other key parameters for assessing water quality, include trends, anomalies, and
comparisons across various locations and timeframes.
‘’’’
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
Summary
● In this training, we created a comprehensive data pipeline using Azure Cloud, enabling
organizations and concerned authorities to take measurable actions in no time.
● The flexibility of the pipeline allows it to cover a vast number of use cases.
● Initially, we discussed the water sensor data (ourmain data source)and emphasized the
importance of treating every single record generated by the sensors, highlighting the criticality
of handling data for accurate analysis.
● A batch-processing approach was employed to extract the entire dataset from the Azure-based
SQL database.
● For this project, we used aggregated water sensor data collected across different terrains,
water bodies, vegetation, and countries.
● The architecture of our project is quite comprehensive and can accommodate additional
functionalities. The architecture encompasses the flow in this way:
○ Initially, we loaded the data from local storage into the cloud-based SQL database
using pipelines, as the entire database was utilized for a historical approach.
○ In the storage layer, we used ADLS storage containers and Blob Storage to store the raw
data files.
○ In the data movement layer, we leveraged Azure Logic Apps and Azure Data Factory to
facilitate the movement of data between services, ensuring it reached its final
destination.
○ In the transformation layer, we adhered to the Medallion Architecture, an
industry-standard framework, to transform the data rigorously and deliver a
high-quality dataset.
○ Finally, we wrote the processed dataset from the Gold Hive Metastore to Power BI and
visualization development was explained on this data to create comprehensive
dashboards, enabling government and private organizations to make timely decisions
and prevent hazardous activities.
● Various Azure Services were used to implement the project and cost configurations were also
taken care of to slack the unwanted charges.
BUILDDATA PIPELINE USING AZURE MEDALLION ARCHITECTURE APPROACH
You might also like
- InfoWater Student Analysis and Design WorkbookNo ratings yetInfoWater Student Analysis and Design Workbook80 pages
- Big Book of Data Engineering 2nd Edition FinalNo ratings yetBig Book of Data Engineering 2nd Edition Final97 pages
- Microsoft Azure Exam Az-900 Certification WhizcardNo ratings yetMicrosoft Azure Exam Az-900 Certification Whizcard29 pages
- Google Cloud Platform for Data Engineering: From Beginner to Data Engineer using Google Cloud PlatformFrom EverandGoogle Cloud Platform for Data Engineering: From Beginner to Data Engineer using Google Cloud Platform5/5 (1)
- 20210430-WP-Stream Processing Made Easy With Confluent Cloud and KSQLDBNo ratings yet20210430-WP-Stream Processing Made Easy With Confluent Cloud and KSQLDB9 pages
- Guide to Data Warehousing in the Lakehouse 1731468863No ratings yetGuide to Data Warehousing in the Lakehouse 173146886355 pages
- Geo SCADA Expert Design Guidelines v3.0 - EN - 2021No ratings yetGeo SCADA Expert Design Guidelines v3.0 - EN - 202127 pages
- Oracle To Azure Database For PostgreSQL Migration CookbookNo ratings yetOracle To Azure Database For PostgreSQL Migration Cookbook13 pages
- Yang Fengming - 2019213704 - DraftReportNo ratings yetYang Fengming - 2019213704 - DraftReport31 pages
- Integration of Heterogeneous Databases Into XML Format With TranslatorNo ratings yetIntegration of Heterogeneous Databases Into XML Format With Translator28 pages
- Choosing Your Database Migration Path To AzureNo ratings yetChoosing Your Database Migration Path To Azure61 pages
- The Big Book of Data Engineering: A Collection of Technical Blogs, Including Code Samples and Notebooks100% (2)The Big Book of Data Engineering: A Collection of Technical Blogs, Including Code Samples and Notebooks57 pages
- Oracle To Azure Database For Postgresql Migration Cookbook: Prepared byNo ratings yetOracle To Azure Database For Postgresql Migration Cookbook: Prepared by13 pages
- Oracle To Azure PostgreSQL Migration CookbookNo ratings yetOracle To Azure PostgreSQL Migration Cookbook13 pages
- Power Consumptionof Virtual Machinesin Cloud ComputingNo ratings yetPower Consumptionof Virtual Machinesin Cloud Computing54 pages
- Data Center Design With Cisco Nexus Switches and Virtual Portchannel: OverviewNo ratings yetData Center Design With Cisco Nexus Switches and Virtual Portchannel: Overview22 pages
- Dynamic Load Balancing in SDN Based Data Center NetworkNo ratings yetDynamic Load Balancing in SDN Based Data Center Network27 pages
- Data Engineering 101 - Azure Synapse AnalyticsNo ratings yetData Engineering 101 - Azure Synapse Analytics45 pages
- Build and Deploy A Multi-Container Application in Azure Container ServiceNo ratings yetBuild and Deploy A Multi-Container Application in Azure Container Service52 pages
- Building Batch Data Pipelines On Google CloudNo ratings yetBuilding Batch Data Pipelines On Google Cloud18 pages
- 737256182-Student-List-Dec-23-to-Dec-24No ratings yet737256182-Student-List-Dec-23-to-Dec-2450 pages
- Request_Processing_Interconnection_Data_Elements_in_USA_Staffing_(May_2024)No ratings yetRequest_Processing_Interconnection_Data_Elements_in_USA_Staffing_(May_2024)99 pages
- Vanja Anisha 12 Monhts of Employment Agreement CUM APPOINTMENT LETTER_Progressive RPONo ratings yetVanja Anisha 12 Monhts of Employment Agreement CUM APPOINTMENT LETTER_Progressive RPO1 page
- Infoblox Deployment Guide Implementing Infoblox Reporting and AnalyticsNo ratings yetInfoblox Deployment Guide Implementing Infoblox Reporting and Analytics32 pages
- Informatica PowerCenter Real Time EditioNo ratings yetInformatica PowerCenter Real Time Editio8 pages
- Emerging Technologies - Lecture Notes - CH 1 & 2No ratings yetEmerging Technologies - Lecture Notes - CH 1 & 273 pages
- Welcome To Unity OE v4.1 - Scalability and Performance.No ratings yetWelcome To Unity OE v4.1 - Scalability and Performance.47 pages
- Info Sphere DataStage Parallel Framework Standard PracticesNo ratings yetInfo Sphere DataStage Parallel Framework Standard Practices460 pages
- Google Wide Profiling: A Continuous Profiling Infrastructure For Data CentersNo ratings yetGoogle Wide Profiling: A Continuous Profiling Infrastructure For Data Centers15 pages
