Job Description
The Data Engineer will be a critical member of the Agile scrum team, responsible for constructing and maintaining scalable data solutions. Leveraging technologies such as Azure, GraphQL, Databricks, and GitHub, this role will focus on building robust data pipelines and workflows to support Pella's business objectives. The Data Engineer will collaborate closely with Data Architects to ensure data quality, reliability, and accessibility while optimizing performance and efficiency.
Responsibilities
- Design and develop data pipelines to manage how data flows between disparate systems
- Build data pipelines to feed analytics use cases, KPI or enterprise apps.
- Develop data quality metrics and performs QC tests (system and visual) to verify data integrity.
- Interface with architects, product managers/SMEs and product analysts to understand data needs and support the implementation of the business rules into transformation.
- Document the data blending process along with the specifications and workflow/data lineage.
- Perform continuous integration to ensure that every step of the pipeline is testable and automated.
- Communicating project status through Agile process and in Jira for the cross-functional team
- Collaborate with the business to understand backlog and refine use cases related to data management, BI reporting and data science deliverables. Research source system data, architecture and transactions.
- Takes lead to perform detailed design (the Physical Data Model and transformations), based on understanding of the Logical Data Model (the business requirements)
- Create design documents for data integration or data reporting projects
- Develop new and improve existing processes to ensure service levels are being met
- Analyze data integration problems, provide solutions and recommend corrective actions.
- Analyze source system data structures and map them to target data warehouse schemas.
- Must have excellent skills in requirements analysis, logical/physical modeling, data transformation and data modeling and technical governance design concepts.
- Participate in design and code reviews, documentation of design, and implementation of methodologies to ensure high quality deployments
Analyze application and data integration problems, provide solutions and recommend corrective actions.
Education/Experience
QUALIFICATIONS
- BS degree in Computer Science, Data Engineering, Software Engineering, or a related field.
- 3+ years' experience in data engineering / software development
QUALIFICATIONS:
Technical / functional skills (includes computer skills):
- Expertise in Azure cloud technologies specifically Synapse, ADF, Delta Lake, Databricks or comparable technology experience within AWS, Snowflake and/or GCP
- Understanding of architecture of Data Quality, Metadata Management and Master Data Management
- Understanding of dimensional data modeling and design as well as data population techniques for target structures such as Star Schemas.
- Skilled in PYSPARK and SQL to build production-grade data pipelines and tools
- Experience navigating a modern data environment and working between on-prem & cloud technologies
- Strong grasp of CI/CD operating practices
- Experience operating within a Product Scrum Agile team
- Experience in Manufacturing industry preferred
Experience with MS Office, Outlook, Jira