Job Overview
We are in search of an enthusiastic Senior Data Engineer who possesses a drive for innovation and a knack for resolving complex challenges that extend beyond conventional data engineering. If you have significant experience in crafting extensive data pipelines, optimizing SQL, and managing cloud infrastructure, we invite you to connect with us. In this pivotal role, you will contribute to advancing our AI capabilities, constructing high-performance data systems, and seamlessly integrating cutting-edge AI into the everyday operations of data teams worldwide.
Key Responsibilities
- Develop and enhance PB scale data pipelines, ensuring capacity to manage over 100K jobs and handle petabyte-scale data on a daily basis.
- Design and implement a robust cloud-native data infrastructure on AWS, utilizing Kubernetes and Airflow for optimal resource management and deployment.
- Create a comprehensive SQL intelligence ecosystem that includes query optimization, dynamic pipeline generation, and data lineage tracking.
Required Skills
- Over 8 years of experience in data engineering with a focus on building scalable data pipelines and systems.
- High proficiency in Python and SQL.
- Extensive knowledge in SQL query profiling, optimization, and performance tuning, particularly with Snowflake.
- In-depth understanding of SQL Abstract Syntax Tree (AST) and experience with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs.
- Experience in constructing data pipelines using tools such as Airflow or dbt.
- [Optional] Strong familiarity with cloud platforms, especially AWS.
- [Optional] Basic understanding of Kubernetes (K8s) for managing containerized deployments.
Qualifications
- A Bachelor's degree in Computer Science, Engineering, or a related discipline.
Career Growth Opportunities
This position serves as a launchpad into the rapidly evolving domain of AI-enhanced data engineering, granting you exposure to state-of-the-art technologies and generative AI applications. You will have access to extensive resources for continuous learning and professional development.
Company Culture And Values
Join a dynamic and intellectually stimulating environment alongside a team of skilled engineers who value collaboration and innovation. We foster a culture that encourages creative problem-solving and recognizes the contributions of every team member.
Networking And Professional Opportunities
Your role will offer significant exposure to open-source initiatives, enabling you to gain recognition within the technology community. You will also have numerous networking opportunities that can enhance your professional journey.
Compensation And Benefits
- Competitive Salary and Equity:
- Base Salary Range: \(60,000 - \)120,000 (dependent on your expertise, location, interview performance, and role fit).
- Equity: 0.0% - 0.4%.
- Professional Development: Comprehensive opportunities and resources for continuous learning.
- Dynamic Work Environment: Collaborate with a talented team of engineers and contribute to shaping the company’s future.
If this exceptional opportunity aligns with your career aspirations, we look forward to your application. Let’s embark on an innovative journey together!
Employment Type: Full-Time