Job Overview
We are on the lookout for a driven Senior Data Engineer with a passion for innovation and a knack for solving complex problems that extend beyond conventional data engineering practices. If you possess a wealth of experience in constructing large-scale data pipelines, optimizing SQL performance, and leveraging cloud infrastructure, we encourage you to connect with us. In this pivotal role, you will play a crucial part in advancing our AI initiatives, architecting high-performance data systems, and seamlessly integrating cutting-edge AI technologies within the daily operations of our global data teams.
Key Responsibilities
- Design and build scalable data pipelines capable of processing PB-scale data and managing over 100K jobs daily.
- Create and maintain cloud-native data infrastructure on AWS, utilizing tools such as Kubernetes and Airflow for optimal resource management and deployment.
- Develop an advanced SQL intelligence ecosystem, incorporating elements such as query optimization, dynamic pipeline generation, and data lineage tracking.
Required Skills
- Over 8 years of experience in data engineering, primarily focusing on constructing scalable data systems and pipelines.
- Proficient in Python and SQL, with significant expertise in SQL query profiling, optimization, and performance tuning, ideally with Snowflake.
- Comprehensive understanding of SQL Abstract Syntax Tree (AST) and experience employing SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs.
- Familiarity with building data pipelines using frameworks like Airflow or dbt.
- [Optional] Solid knowledge of cloud platforms, especially AWS.
- [Optional] Experience with Kubernetes (K8s) for containerized environments.
Qualifications
- A Bachelor’s degree in Computer Science, Engineering, or a related discipline.
- At least 8 years of relevant experience in data engineering, with a focus on data systems and pipelines.
Career Growth Opportunities
This position offers a significant pathway into the rapidly advancing domain of AI-driven data engineering, granting exposure to pioneering technologies and applications of generative AI. Additionally, our organization is committed to providing extensive resources for professional development and continuous learning.
Company Culture And Values
Join a vibrant and collaborative environment filled with talented engineers who prioritize innovation and creativity in problem-solving. We value open-source contributions and foster a culture of teamwork and mutual respect.
Compensation And Benefits
- Competitive salary range: \(60,000 - \)120,000 (based on expertise, location, performance during the interview, and suitability for the position).
- Equity options ranging from 0.0% to 0.4%.
- Opportunities to influence company direction and create a lasting impact within the organization.
Employment Type: Full-Time