Lead - Data Engineering
Indegene
Job Description
Must Have
Lead Data Engineer
You will be responsible for:
Provide technical solution leadership in data engineering team, driving technology decisions, mentoring others, and contributing significantly on an individual level.
Design, develop, test, improve, and maintain new and existing solutions for data integration, data prep, ingestion, cleansing, and transforming raw data into curated datasets for business consumption.
Build frameworks to handle data at high scale using ETL tools like Informatica/DataIKU/Matillion/Spark and data cataloging tools like Apache Hive, AWS Glue on top of a multi-tiered data lake storage
Incorporate deep data management expertise into solutions, including permissions, recovery, security, and monitoring.
Build robust data processing pipelines using AWS Services and integrate with multiple data sources
Identify, troubleshoot, debug, and resolve technical issues.
Work with business and IT groups to understand source systems and enterprise infrastructure offerings.
Work with data from multiple data sources to build integrated views that will drive decisions
Help the Data Engineering team produce high-quality code that allows us to put solutions into production
Help us to shape the next generation of our solutions
Your impact:
About you: (Desired profile)
Must have: (Requirements)
Nice to have:
Experience of Life Sciences and Healthcare domain mandatory
1+ years of experience in atleast one modern programming language like Scala/R/Python/Hive/Powershell
Working knowledge of RESTful APIs and integration will an added advantage
Should have experience in design patterns and scalable architectures
Knowledge of Agile, iterative and other SDLC methodologies
Should have strong analytical skills and good business communication skills to engage technical and Business users
8-10 years of experience in consulting or IT experience supporting Enterprise Data Warehouses & Business Intelligence solutions on cloud
5+ years of experience in designing and developing Cloud based solutions on AWS and Azure
4+ year of experience in AWS Services: RDS, AWS Lambda, AWS Glue, Apache Spark, Kafka, Hive, EC2, SQS, SNS, etc
Experience of Azure Analytics Services - Azure Analysis Services, SQL Data Warehouse, Data Factory, Databricks, AWS
Experience in SQL and NoSQL databases like MySQL, Redshift, Snowflake, Postgres, Elasticsearch
Perks: (Mention if any, otherwise ignore)