Scribd
Upload
53 views

Azure Data Engineer

This document describes the responsibilities of an Azure data engineer including designing and implementing Azure data solutions, building data pipelines, ensuring data security, and staying up to date with technologies. The preferred candidate should have experience with Azure services and tools as well as skills in programming languages for data manipulation.

Uploaded by

RAjesh Boyapally
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
53 views

Azure Data Engineer

This document describes the responsibilities of an Azure data engineer including designing and implementing Azure data solutions, building data pipelines, ensuring data security, and staying up to date with technologies. The preferred candidate should have experience with Azure services and tools as well as skills in programming languages for data manipulation.

Uploaded by

RAjesh Boyapally
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 2

Azure data engineer

Job description
Responsibilities :

 Design, develop, and implement Azure-based data solutions, including data lakes, data
warehouses, and data processing systems.
 Build and optimize data pipelines for extracting, transforming, and loading (ETL) data from
various sources into Azure environments.
 Collaborate with data architects, analysts, and other stakeholders to understand data requirements
and translate them into technical solutions.
 Ensure data security, integrity, and performance by implementing appropriate Azure data services
and best practices.
 Perform data modeling, schema design, and database optimization for efficient data storage and
retrieval.
 Develop and maintain documentation for data processes, configurations, and procedures.
 Monitor and troubleshoot data solutions to identify and resolve performance issues or bottlenecks.
 Stay updated with emerging Azure technologies and trends to recommend enhancements and
improvements.

Preferred candidate profile


 Proven experience as a Data Engineer or similar role, with expertise in Azure data services and
tools.
 Hands-on experience with Azure Data Factory, Azure Databricks, Azure Synapse Analytics,
Azure SQL Database, Azure Cosmos DB, etc.
 Proficiency in programming languages such as SQL, Python, or Scala for data manipulation
and scripting.
 Strong understanding of data modeling, data warehousing concepts, and ETL processes.
 Knowledge of data security, encryption, and compliance standards in Azure environments.
 Excellent problem-solving skills and the ability to work collaboratively in a team environment.
 Strong communication skills with the ability to convey complex technical concepts to non-
technical stakeholders.

Description :
Snowflake Data Engineer who can perform the activities listed below.
 Knowledge of Snowflake platform
 Creating objects (Tables, Views, Procedures (using Javascript, SQL), User-defined Functions)
 Advanced SQL Skills.
 Good understanding of advanced Snowflake concepts
 Tasks, Streams and Dynamic Tables.
 Different Types of Integrations
 Snowflake Costs
 Role-based Access Control(RBAC)
 Snowflake Security
 Streamlit in Snowflake
 Proficiency in at least one modern data stack tool - Matillion, Azure Data Factory, Fivetran+dbt
 Intermediate Python Skills.

Any combination of below technical skills

· Cloud : Azure/AWS/GCP Big data native services

· Lakehouse : Synapse, Redshift, BigQuery, Snowflake, Databricks

· Spark : Databricks/Spark in AWS EMR/Spark in Azure HDInsight

· NoSQL : Cassandra, MongoDB, Hbase, AWS Dynamodb, DocumentDb

· Visualization : Tableau, PowerBI, MSTR, Qliksense

· Advance Analytics : Axure ML, AWS sagemaker, Vertex, MLOps

· Data Governance & Data observability

Responsibilities:
 Design, build, and maintain scalable data pipelines using PySpark and Databricks
 Optimize data processing and storage for maximum performance and efficiency
 Troubleshoot and debug data-related issues, and implement solutions to prevent reoccurrence
 Collaborate with data scientists, software engineers, and other stakeholders to ensure that data
solutions are aligned with business goals
Requirements:
 Strong experience in Python programming or PySpark, and SparkSQL
 Clear understanding of Spark Data structures, RDD, Dataframe, dataset
 Expertise in Databricks and ADLS
 Expertise handling data type, from dictionaries, lists, tuples, sets, arrays, pandas dataframes, and
spark dataframes
 Expertise working with complex data types such as, structs, and JSON strings.
 Clear understanding of Spark Broadcast, Repartition, Bloom index filters
 Experience with ADLS optimization, partitioning, shuffling and shrinking
 Ideal experience with disk caching
 Ideal Experience with cost based optimizer
 Experience with data modeling, data warehousing, data-lake, delta-lake and ETL/ELT processes
in ADF
 Strong analytical and problem-solving skills
 Excellent documentation, communication and collaboration skills

You might also like