Module Lead - Azure Data Factory (ADF) Job

Experience

6-8 Years of Relevant Experience

Job Description

Lead a Team of Data Engineers on Azure  to deliver the expected business results and mentor Junior consultants in team and provide assistance in case needed.

6 to 8 yrs of experience in ADF, Data Bricks and Azure utilities is mandatory.

Knowledge on both ADF and DATA BRICKS using various transformations, ETL Framework and Design/ Development  is mandatory 

Good knowledge on Python Scripting for data processing and validations in building pipelines is mandatory

Develop ETL Solutions using ADF Data Integration/ Application Integration handling large data volumes

Azure - ADF Experience with the business/data analysis, integration, quality controls and governance of golden source data management. 

Experience in Python, PySpark scripting for data transformations and validations in ADF and Data Bricks respectively

Knowledge in handling various source/target systems including XML/Flat files/Databases using ADF File Ingestion.

Understand the business requirement and propose Azure - ADF design solutions accordingly Strong ETL knowledge who can develop the complex mapping as per given requirement.

Experience in change data capture, Data warehousing and experience in handling admin and support activities of tickets/incidents

Good understanding of Data Warehousing concept with hands on experience with slowly changing dimensions and fact tables.

Develop design specifications, unit test plans, and handle the complex scenario Advance knowledge of SQL query who can develop complex query Ability to work in a fast-paced environment and meet deadlines

Excellent with SQL coding - performance optimization, table indexing, stored procedures, functions, data partition/distribution

Provide the solutioning and design the prototype and present.

Build, test, and maintain database pipeline architectures and collaborate with management to understand company objectives

Ensure compliance with data governance and security policies

Work closely with business and dev teams to translate the business/functional requirements into technical specifications that drive Big Data solutions to meet functional requirements.

Create new data validation methods and data analysis tools and Ensure compliance with data governance and security policies

Acquire datasets that align with business needs and develop algorithms to transform data into useful, actionable information

Maintain application stability and data integrity by monitoring key metrics and improving code base accordingly.    
 

Required Technical/ Functional Competencies

Domain/ Industry Knowledge: 

• Basic knowledge of customer's business processes- relevant technology platform or product.

• Able to prepare process maps, workflows, business cases and simple business models in line with customer requirements with assistance from SME and apply industry standards/ practices in implementation with guidance from experienced team members.

Requirement Gathering and Analysis:

• Working knowledge of requirement management processes and requirement analysis processes, tools & methodologies.

• Able to analyse the impact of change requested/ enhancement/ defect fix and identify dependencies or interrelationships among requirements & transition requirements for engagement.

Product/ Technology Knowledge:

• Working knowledge of technology product/platform standards and specifications.

• Able to implement code or configure/customize products and provide inputs in design and architecture adhering to industry standards/ practices in implementation.

• Analyze various frameworks/tools, review the code and provide feedback on improvement opportunities.

Architecture tools and frameworks:

• Working knowledge of architecture Industry tools & frameworks.

• Able to identify pros/ cons of available tools & frameworks in market and use those as per Customer requirement and explore new tools/ framework for implementation.

Architecture concepts and principles:

• Working knowledge of architectural elements, SDLC, methodologies.

• Able to provides architectural design/ documentation at an application or function capability level and implement architectural patterns in solution & engagements and communicates architecture direction to the business.

Analytics Solution Design:

• Knowledge of statistical & machine learning techniques like classification, linear regression modelling, clustering & decision trees.

• Able to identify the cause of errors and their potential solutions.

Tools & Platform Knowledge:

• Familiar with wide range of mainstream commercial & open-source data science/analytics software tools, their constraints, advantages, disadvantages, and areas of application.

Required Behavioral Competencies

Accountability: Takes responsibility for and ensures accuracy of own work, as well as the work and deadlines of the team.

Collaboration: Shares information within team, participates in team activities, asks questions to understand other points of view.

Agility: Demonstrates readiness for change, asking questions and determining how changes could impact own work.

Customer Focus: Identifies trends and patterns emerging from customer preferences and works towards customizing/ refining existing services to exceed customer needs and expectations.

Communication: Targets communications for the appropriate audience, clearly articulating and presenting his/her position or decision.

Drives Results: Sets realistic stretch goals for self & others to achieve and exceed defined goals/targets.

Resolves Conflict: Displays sensitivity in interactions and strives to understand others’ views and concerns.

Certifications