Job Title: Sr. Data Engineer
Location: Remote (Mandatory PST working hours)
Experience Required: 15+ years
Employment Type: Contract
Job Description:
We are seeking a highly experienced Sr. Data Engineer who can expertly balance hands-on development with leadership and architectural responsibilities. The ideal candidate will bring over 15 years of experience in data engineering, coupled with deep expertise in Azure, Databricks, and Spark. This role demands a strategic thinker capable of leading teams and designing enterprise-level data solutions.
Key Responsibilities:
1. Development (Hands-On Engineering):
Data Pipeline Development: Architect, develop, and maintain sophisticated data pipelines and workflows in Azure, ensuring scalability, reliability, and performance.
Azure Mastery: Lead the use of Azure services, including Azure Data Factory (ADF) for orchestration, Azure Data Lake Storage (ADLS) for scalable data storage, and Azure Databricks for advanced analytics and ETL processes.
Advanced Spark Programming: Develop, optimize, and manage complex Spark applications (PySpark, SparkSQL) to handle large-scale data transformations.
Scripting & Automation: Utilize PowerShell and other scripting tools for the automation and efficient management of cloud resources.
SQL Proficiency: Expertly craft, optimize, and troubleshoot complex SQL queries across T-SQL and PostgreSQL environments, ensuring optimal data performance.
Required Skills & Qualifications:
Azure: 15+ years of experience (Expert level) in architecting, deploying, and managing complex Azure-based solutions.
Azure Data Factory (ADF): 10+ years of experience (Expert level) in designing and orchestrating advanced data pipelines.
Databricks: 10+ years of experience (Expert level) in managing and optimizing Databricks environments and complex workflows.
Spark (PySpark, SparkSQL): 10+ years of experience (Expert level) in advanced Spark programming and performance optimization.
Azure DevOps CI/CD: 8+ years of experience in designing and managing sophisticated CI/CD pipelines for data engineering projects.
PowerShell & Scripting: Advanced proficiency in PowerShell and other scripting languages for automation and cloud resource management.
SQL Mastery: Extensive experience with T-SQL, PostgreSQL, and advanced query optimization, indexing, and partitioning techniques.
Data Formats: In-depth experience with Parquet, Delta, and other advanced data storage formats.
Additional Requirements:
Extensive experience leading and mentoring senior engineering teams.
Strong strategic thinking and experience in designing enterprise-level data solutions in large, complex environments.
Proven ability to manage large-scale data processing and distributed computing architectures.
Excellent communication, collaboration, and leadership skills, with the ability to engage effectively with stakeholders at all levels.
Strong problem-solving abilities and the capability to troubleshoot complex technical issues in data engineering.
Preferred Qualifications:
Experience in large enterprise environments with multi-terabyte or petabyte-scale data processing needs.
Relevant Azure certifications, such as Azure Solutions Architect, Azure Data Engineer, or similar.
Deep familiarity with cutting-edge trends in data engineering, cloud technologies, and big data frameworks.