Job Description

Key Responsibilities

• To accelerate the delivery of Polaris strategy outcome including product simplicity, experience focus, multi-channel distribution, hyper-relevant channel interactions, and systematic organizational agility, this role of Manager, Data Engineering, will work with Group Data team to support group data solution delivery as well as supporting local entities to deliver local data initiatives which both group and local data solutions aligning with digital technology operations strategy combining with our infrastructure strategy to standardize our cloud adoption and cloud optimisation.
• Ensure the business, application, data and technology aspects of a solution design is aligned with both the Group Enterprise Architecture strategy and Group Data Architecture strategy.
• Design, develop, document and implement end-to-end data pipelines and data integration processes, both batch and real-time. This include data analysis, data profiling, data cleansing, data lineage, data mapping, data transformation, developing ETL / ELT jobs and workflows, and deployment of data solutions.
• Monitor, recommend, develop and implement ways to improve data quality including reliability, efficiency and cleanliness, and to optimize and fine-tune ETL / ELT processes.
• Recommend, execute and deliver best practices in data management and data lifecycle processes, including modular development of ETL / ELT processes, coding and configuration standards, error handling and notification standards, auditing standards, and data archival standards.
• Prepare test data, assist to create and execute test plans, test cases and test scripts.
• Collaborate with Data Architect, Data Modeler, IT team members, SMEs, vendors and internal business stakeholders, to understand data needs, gather requirements and implement data solutions to deliver business goals.
• BAU support for any data issues and change requests, document all investigations, findings, recommendations and resolutions.

QUALIFICATIONS / EXPERIENCE

• Bachelor in IT, Computer Science or Engineering.
• 3-8 years solid and hands-on experience in real-time event/data streaming experience using Kafka/Spark/Spark Stream/Flink ecological environment development, proficient in basic knowledge and data development technology related to data processing, relevant experience in real-time data processing in large volume environment. Have experience with Kafka and Confluent Platform is highly preferred.
• At least 3+ years of solid hands-on development experience with ETL development to transform complex data structure in multiple data sources environment.
• Strong programming on Python and T-SQL ETL/ELT programming.
• Experience on Azure Databricks for ETL/ELT development and big data analytics programming in Python
• Strong Experience with various of ETL/ELT frameworks, data warehousing concepts, data management framework and data lifecycle processes.

(Both Junior & Senior roles available) Job Types: Full-time, Contract
Contract length: 12 months Salary: RM4,500.00 - RM11,000.00 per month Ability to commute/relocate:

• Kuala Lumpur: Reliably commute or planning to relocate before starting work (Preferred)

Experience:

• Kafka: 3 years (Preferred)
• Spark: 3 years (Preferred)
• Azure Databricks: 3 years (Preferred)
• Python: 3 years (Preferred)