Job Title: Python Software Engineer
Job Location: Dallas, TX(Remote)
Job Duration: 100% Remote
Interview: Virtual
We are seeking a skilled Software Engineer with expertise in AWS Glue and Apache Hudi to join our team. The ideal candidate will have strong proficiency in Python and hands-on experience with Change Data Capture (CDC) methodologies.
Responsibilities:
- Develop and implement ETL processes using AWS Glue to extract, transform, and load data from various sources into data lakes and data warehouses.
- Design and optimize data pipelines to ensure efficient data ingestion, processing, and integration.
- Utilize Apache Hudi for incremental data processing and real-time data updates.
- Implement Change Data Capture (CDC) strategies to capture and propagate data changes across systems in near real-time.
- Collaborate closely with data engineers, data scientists, and business stakeholders to understand data requirements and deliver scalable solutions.
- Ensure data quality and integrity throughout the ETL process and provide troubleshooting and support as needed.
- Document technical specifications, processes, and procedures related to ETL development and deployment.
Required Skills:
- Extensive experience (Senior level) in ETL development using AWS Glue.
- Strong proficiency in Python programming for data processing and automation tasks.
- Intermediate-level proficiency in Apache Hudi with hands-on experience in implementing data ingestion and processing workflows.
- Solid understanding and practical experience with Change Data Capture (CDC) methodologies.
- Experience working in cloud environments, particularly AWS.
- Ability to work independently and as part of a collaborative team.
- Excellent problem-solving skills and attention to detail.
- Effective communication skills to interact with technical and non-technical stakeholders.