Job Description

: A minimum of 10+ years previous experience as a Data Engineer with at least 3+ years of hands on AWS technical/development experience : We intend to hire senior AWS Data Engineers with relevant consulting and communication skills. Role and responsibilities: Build robust, efficient and reliable data pipelines consisting of diverse data sources to ingest and process data Design and develop real time streaming and batch processing pipeline solutions Assemble large, complex data sets to meet functional / non-functional requirements Design, develop and implement data pipelines for data migration and collection, data analytics and other data movement solutions Work with stakeholders and data analyst teams to assist with data-related technical issues and support their data infrastructure needs Collaborate with Architects to define the architecture and technology selection Qualifications and experience: A minimum of 10+ years previous experience as a Data Engineer with at least 3+ years of hands on AWS technical/development experience Previous experience in programming eg. Core Java/Python/Scala programming; Hadoop Stack eg. Hive, Spark (Core and Streaming), Kafka, Flink, NIFI; Experience working on stream transform using Spark Experience working on change data capture in a big data ecosystem; Experience in building real time or batch ingestion and transformation pipelines; Should be able to work on AWS - S3, EMR, Redshift Should be able to implement the Data Vault & start schema Framework using redshift Should be able to work with Devops team on Orchestrating Airflow works Good understanding on iPython and Bigdata and Hands on SQL

foundit