Azure Data Lake Store
Azure Data Lake Analytics
A technical overview
and introduction to U-SQL
Gary Hope
Cloud Data Solution Architect
Microsoft South Africa
[email protected]
Proudly
brought to you by
Platinum
Gold
Silver
Bronze
Azure Data Lake
HDInsight Azure Data Lake Analytics
Analytics (“managed clusters”)
Azure Data Lake Storage
Storage
Azure Data Lake
as part of Cortana Intelligence Suite
Information Big Data Stores Machine Learning Intelligence
Data Management and Analytics
People
Sources
SQL Data Machine Cognitive
Data Factory
Warehouse Learning Services
Data Lake Bot Web
Data Catalog Analytics Framework
Data Lake Store
Apps HDInsight
Event Hubs (Hadoop and Cortana Mobile
Spark) Apps
Stream Analytics Bots
Dashboards &
Visualizations
Sensors Automated
and Power BI Systems
devices
Data Intelligence Action
Azure Data Lake
Analytics HDInsight
U-SQL
(extensible by C#, R and Python)
YARN
WebHDFS
Store
Demo – Lets Create The Services
6
?
?
Why data lakes? ?
?
Traditional business analytics process
1. Start with end-user requirements to identify desired reports
and analysis
2. Define corresponding database schema and queries
3. Identify the required data sources
4. Create a Extract-Transform-Load (ETL) pipeline to extract
required data (curation) and transform it to target schema
(‘schema-on-write’)
5. Create reports. Analyze data
Dedicated ETL tools (e.g. SSIS)
Relational Queries
ETL pipeline
LOB Results
Applications Defined schema
All data not immediately required is discarded or archived
8
New big data thinking: All data has value
All data has potential value
Data hoarding
No defined schema—stored in native format
Schema is imposed and transformations are done at query time (schema-on-read).
Apps and users interpret the data as they see fit
Iterate
Gather data
Store indefinitely Analyze See results
from all sources
Data Lake Store: Technical Requirements
Secure Must be highly secure to prevent unauthorized access (especially as all data is in one place).
Scalable Must be highly scalable. When storing all data indefinitely, data volumes can quickly add up
Reliable Must be highly available and reliable (no permanent loss of data).
Throughput Must have high throughput for massively parallel processing via frameworks such as Hadoop and Spark
Details Must be able to store data with all details; aggregation may lead to loss of details.
Native format Must permit data to be stored in its ‘native format’ to track lineage & for data provenance.
All sources Must be able ingest data from a variety of sources-LOB/ERP, Logs, Devices, Social NWs etc.
Multiple analytic Must support multiple analytic frameworks—Batch, Real-time, Streaming, ML etc.
frameworks No one analytic framework can work for all data and all types of analysis.
Big Data analytics workloads
A highly scalable, distributed, parallel file system in the cloud
specifically designed to work with a variety of big data analytics workloads
Devices Azure Data Lake Analytics
Batch
LOB Applications U-SQL
Web
HDInsight
Relational
Batch NoSQL
MapReduce HBase
Social
Azure Data Lake Store
Video Script In-Memory
Pig Spark
Sensors
SQL Predictive
Clickstream
Hive R Server
Azure Data Lake Store
Scale, Performance, Reliability
Azure Data Lake Store: No Scale Limits
Azure Data Lake Store integrates with
Azure Active Directory (AAD) for:
Amount of data stored Seamlessly scales
How long data can be stored from a few KBs
Number of files to several PBs
Size of the individual files
Ingestion throughput
Azure Data Lake Store: How it works
Each file in ADL Store is sliced into blocks
Blocks are distributed across multiple data Azure Data Lake Store file
nodes in the backend storage system
Block 1 Block 2 … Block 2
With sufficient number of backend storage
data nodes, files of any size can be stored
Backend storage runs in the Azure cloud
which has virtually unlimited resources Block Block Block Block Block Block
Metadata is stored about each file Data node Data node Data node Data node Data node Data node
No limit to metadata either. Backend Storage
14
Azure Data Lake Store: Massive throughput
Read operation
Through read parallelism ADL Store provides Azure Data Lake Store file
massive throughput
Block 1 Block 2 … Block 2
Each read operation on a ADL Store file results
in multiple read operations executed in
parallel against the backend storage data
nodes
Block Block Block Block Block Block
Data node Data node Data node Data node Data node Data node
Backend storage
15
ADL Store: High Availability and Reliability
Azure maintains 3 replicas of each data object Write Commit
per region across three fault and upgrade Replica 1
domains Fault/upgrade
domains
Each create or append operation on a replica
is replicated to other two
Writes are committed to application only after
all replicas are successfully updated
Replica 2 Replica 3
Read operations can go against
any replica
Data is never lost or unavailable
even under failures
16
Azure Data Lake Store
Security
Azure Data Lake Store Security: AAD integration
Multi-factor authentication based on OAuth2.0
Integration with on-premises AD for federated authentication
Role-based access control
Privileged account management
Application usage monitoring and rich auditing
Security monitoring and alerting
Fine-grained ACLs for AD identities
18
Azure Data Lake Store Security: Role-based access
Each file and directory is
associated with an owner and a
group
Files or directories have separate
permissions (read(r), write(w),
execute(x)) for owners, members
of the group, and for all other
users
Fine-grained access control lists
(ACLs) rules can be specified for
specific named users or named
groups
19
Granular control of file and folder access
POSIX-Style ACLs with full compatibility with HDFS/WebHDFS
Generate default ACLs for files Child File Folder
and folders
Customize for fine-tuned control
Access ACLS Access ACLS
Access ACLs control how a user
can access to the file or folder Default ACLS
Default ACLs used to construct the
Access ACL of new children
Default ACLs copied to the Default
ACL of new child folders New Child Folder New Child File
Default ACLS
Access ACLS Access ACLS
IP address ACLs
64.34.55.130
–
64.34.55.135
IP range whitelist
Encryption of data at rest
Audit logs for data access
GitHub
[T1] Alice, Write
[T2] Bob, Read
Demo – Lets Upload Some Data
Azure Data Lake Store
Hadoop integration and
Data Movement
Azure Data Lake Store is HDFS-compatible
With a WebHDFS endpoint Azure Data Lake Store is a Hadoop-compatible file system
that integrates seamlessly with Azure HDInsight
HBase
Map reduce Hive query Spark queries Any HDFS application
transactions
Azure HDInsight
Hadoop WebHDFS client
WebHDFS-compatible REST API
Azure Data Lake Store
26
Azure Data Lake Store: Ingress
Data can be ingested into Azure Data Lake Store from a variety of sources
SQL
Azure SQL DB
ADL built-in copy service
Azure Data Factory
Hadoop DistCp
Azure Data Factory Azure Storage Blobs
Azure SQL DW
ADL Store
.NET SDK
CLI
Azure Portal
Table Storage Azure PowerShell
Azure tables
Azure Stream Analytics
On-premises databases Azure Event Hubs Custom programs
27
Azure Data Lake Store: Egress
Data can be exported from Azure Data Lake Store into numerous targets/sinks
Azure Data Factory
SQL
Hadoop DistCp
Azure SQL DB Azure Storage Blobs
Azure Data Factory
Apache Sqoop
Azure SQL DW
ADL Store
.NET SDK
Table Storage
CLI
Azure Tables Azure Portal
Azure PowerShell
On-premises databases Custom programs
28
Lambda architecture
On Premises Cloud Consumption
Cleansing ADL Analytics Analysis
Alerts
Relational DB
Power BI
On-Prem HDFS SQL
Active Incoming DB/DW
Data Lake Store
Data
Azure Data
Factory DMG Data Factory
Event data
EventHubs Web Portals
Streaming
Event data Kafka
Azure
Data Lake Store
Costs
Costs breakdown by stage
Get all the advantages
of ADL Store with
cost concepts
you are familiar with
Azure Data Lake
Analytics
Azure Built on Apache YARN
Data Lake Scales dynamically with the turn of a dial
Analytics Service Pay by the query
Supports Azure AD for access control,
A new distributed roles, and integration with on-prem
analytics service identity systems
Built with U-SQL to unify the benefits of
SQL with the power of C#
Processes data across Azure
Azure Data Lake Analytics
All data Productivity Easy and Limitless scale Enterprise-
from day one powerful data grade
preparation
0100101001000101010100101001000
10101010010100100010101010010100
10001010101001010010001010101001
0100100010101010010100100010101
0100101001000101010100101001000
10101010010100100010101010010100
10001010101001010010001010101001
0100100010101010010100100010101
0100101001000101010100101001000
10101010010100100010101010010100
Developing big data apps
Author, debug, & optimize
big data apps
in Visual Studio
Multiple Languages
U-SQL, Hive, & Pig
Seamlessly integrate .NET
Work across all cloud data
Azure Data Lake
Analytics
Azure Azure SQL DB in an
Azure SQL DW Azure SQL DB Data Lake Store Storage Blobs Azure VM
Azure Data Lake
U-SQL
38
What is A hyper-scalable, highly extensible
language for preparing, transforming
U-SQL? and analyzing all data
Allows users to focus on the what—
not the how—of business problems
Built on familiar languages (SQL and
C#) and supported by a fully integrated
development environment
Built for data developers & scientists
The Origins of U-SQL
SCOPE
Next generation large-scale data processing U-SQL
language combining
The declarative, optimizable and parallelizability of SQL
The extensibility, expressiveness and familiarity of C# Hive
T-SQL
High performance Scalable Affordable Easy to program Secure
Usage scenarios
Achieve the same programming experience in batch or interactive
Schematizing unstructured data
(Load-Extract-Transform-Store) for analysis
Cook data for other users (LETS & Share)
As unstructured data
As structured data
Large-scale custom processing with custom code
Augment big data with high-value data from where it lives
Expression-flow programming style
Automatic "in-lining" of U-SQL
expressions – whole script leads to a
single execution model
Execution plan that is optimized out-of-
the-box and w/o user intervention
Per-job and user-driven parallelization 010010
100100
Detail visibility into execution steps, for
debugging
Heat map functionality to identify
performance bottlenecks 010101
U-SQL Queries: General pattern
Read Process Store
Azure Azure
Storage EXTRACT OUTPUT Storage
Blobs Blobs
Azure RowSet SELECT… RowSet
SQL SELECT FROM…
DB WHERE…
Azure EXTRACT OUTPUT Azure
Data Data
Lake INSERT Lake
SELECT
Anatomy of a U-SQL query
REFERENCE ASSEMBLY WebLogExtASM;
• U-SQL types are the same as
@rs = C# types
EXTRACT
UserID string, • The structure (schema) is first
Start DateTime, imposed when the data is first
Rowset: Conceptually is like End DatetTime, extracted/read from the file
an intermediate table… Region string, (schema-on-read)
is how U-SQL passes data SitesVisited string,
between statements PagesVisited string
FROM "swebhdfs://Logs/WebLogRecords.txt" Input is read from this file in ADL
USING WebLogExtractor(); Custom function to read from
@result = SELECT UserID, input file
(End.Subtract(Start)).TotalSeconds AS Duration
FROM @rs ORDER BY Duration DESC FETCH 10; C# Expression
OUTPUT @result TO "swebhdfs://Logs/Results/top10.txt" Output is stored in this file in ADL
USING Outputter.Tsv();
Built-in function that writes the output
in TSV format
U-SQL data types
Category Types
byte, byte? short, short?
sbyte, sbyte? ushort, ushort?
int, int? ulong, unlong?
Numeric uint, unint? float, float?
long, long? double, double?
decimal,
decimal?
char, char?
Text
string
MAP<K>
Complex
ARRAY<K,T>
Temporal DateTime, DateTime?
bool, bool?
Other Guid, Guid?
Byte[]
Federated queries: Query data where it lives
Easily query data in multiple Azure data stores without moving it to a single store
Benefits
Avoid moving large amounts of data across the
network between stores
Azure
Single view of data irrespective of physical location Storage Blobs
Minimize data proliferation issues caused by
maintaining multiple copies Query
U-SQL
Single query language for all data Query
Result
Each data store maintains its own sovereignty Azure Data Azure SQL
Lake Analytics in VMs
Design choices based on the need
Azure
SQL DB
46
Demo – Lets Run Some Queries
Azure Data Lake Analytics
Billing
Azure Data Lake Analytics Billing
Accounts are FREE!
Pay for the compute resources you want
for your queries
Pay for storage separately
(query_minutes * parallelism * parallelism_cost_per_minute)+ per_job_charge
Get started
today!
For more information visit:
http://azure.com/datalake
49
Where to learn more…
50
Email: [email protected]
Twitter: @GaryHope
Mobile: +27 82 7778886
