Job Description

Experience

: 10+ years

ONLY MALAYSIANS

Qualification:

Bachelor's degree in IT, computer science, or a relevant field.

- 10+ years of experience in the data engineering/analyst domain.

- Deep understanding of SDLC methodologies (Agile, Scrum, CI/CD).

- Proven experience leading and mentoring junior engineers.

- Able to work independently in a fast-paced and complex environment.

- Good communication skills (both verbal and written) - English is
mandatory.

- Expert knowledge in data, data engineering and other related
subdomains.

- Working knowledge of MS SQL, PostgreSQL, CDC, data Streaming
Application (e.g. Apache Kafka, Qlik Replicate) and other related
systems.

- Experience with Azure DevOps, GitHub tools and methodologies.

- Understanding of cloud security, governance best practices in Azure.

About Role:

- Data Integration and Orchestration: Designs and implements data
pipelines to orchestrate data movement from various sources
(databases, APIs, etc.) into Azure storage solutions (e.g., Azure
Data Lake Storage, Azure Blob Storage).

- Technical Leadership: Drive the full lifecycle of scalable ETL and
data pipeline development. Apply software development best practices
to ensure code is robust, maintainable, and well-tested. Mentor
junior engineers, offering technical guidance and support. Lead code
reviews, testing strategies, deployment processes, and documentation
efforts

- Innovation and Optimization: Continuously refine data workflows and
infrastructure for performance and scalability using Spark within
Azure Databricks. Evaluate and enhance tools and systems to improve
data quality and operational efficiency.

- Data Pipeline CI/CD: Designs, implements, maintains and enhanced
CI/CD pipelines for data pipelines using Azure DevOps services
(e.g., Azure Pipelines, Azure Repos). This includes automating
testing, deployment, and configuration management for data
infrastructure on Azure.

- Azure Infrastructure Management: Provisions, configures, and manages
infrastructure for data pipelines using Azure services (e.g., Azure
Data Factory, Azure Databricks VMs, Azure Functions).

- Version Control and Collaboration: Utilizes GitHub, Azure DevOps
services (e.g., Azure Repos) to manage code changes for data
pipelines and collaborates with other engineers on infrastructure
development.

- Security and Compliance: Ensures secure deployment of data pipelines
on Azure, adhering to data privacy and compliance regulations using
Azure Security Center and other security tools. Implement data
quality and governance processes to ensure the accuracy,
consistency, and reliability of data.

- Monitoring and Troubleshooting: Sets up Azure Monitor to track data
pipeline health and performance and implements alerting systems for
potential issues through Azure Alerts. Perform troubleshooting to
identify and resolve issues with data pipelines and infrastructure.

- Schema and Storage Design
Ability to define and manage schema structures, data lake zones, and
storage containers aligned with data governance and quality
frameworks. This includes setting up exception handling zones and
optimizing storage for performance and cost.

Additional Skill :

Building and Setting Up Azure Infrastructure: To architect, deploy, and
manage scalable Azure infrastructure tailored for enterprise-grade data
engineering solutions

Job Types: Full-time, Contract
Contract length: 12 months

Pay: RM4,503.87 - RM12,000.00 per month

Benefits:

Health insurance Professional development
Work Location: In person