JOB DESCRIPTION
One of our largest national retail clients is looking for a Data Quality Engineer to join their growing organization. In this role you will be responsible for the validations and quality assessments of the data pipelines. You will be focused on the accuracy, reliability and integrity of the data. Your role will be spent the majority of the time in Apache Spark and SQL to ensure that the team is reducing time and effort required to manage the data.
Key Responsibilities:
- Develop and execute data validation scripts using Spark and Scala to ensure data accuracy and consistency.
- Design and implement data quality checks and metrics to monitor data integrity across various data sources.
- Collaborate with data engineering teams to integrate data quality processes into ETL pipelines.
- Identify and troubleshoot data quality issues, providing actionable insights and solutions.
- Create and maintain documentation for data quality processes, standards, and best practices.
- Work closely with stakeholders to understand data requirements and ensure alignment with business objectives.
- Participate in code reviews and contribute to the continuous improvement of data quality tools and methodologies.
Qualifications:
- Bachelor’s degree in Computer Science, Information Technology, or a related field.
- Proven experience in data validation and quality assurance, preferably in a data engineering environment.
- Strong proficiency in Apache Spark and Scala for data processing and validation.
- Familiarity with data warehousing concepts and ETL processes.
- Experience with SQL and data modeling techniques.
- Excellent analytical and problem-solving skills.
- Strong communication skills and ability to work collaboratively in a team environment.