We're seeking a data engineering rockstar with a knack for mentoring, a strategic mind, and strong communication skills. You'll not only design, develop, and maintain robust data pipelines using Databricks and your ETL expertise, but you'll also play a key role in building our Databricks practice.
Here's what sets this role apart:
- Technical Guru and Mentor: You'll be an expert in Databricks, ETL tools (like Informatica), Python, and data modeling concepts. You'll guide junior team members, foster a culture of knowledge sharing, and participate as an expert and learner in team tasks.
- Sales Champion: You'll collaborate with the sales team, attending pre-sales calls to showcase your technical expertise and demonstrate the value of Databricks solutions to potential clients. Your insights will be instrumental in closing deals and expanding our Databricks practice.
- Data Visionary: You'll have a strategic understanding of how Databricks can empower businesses, translating client needs into compelling technical solutions.
Day-to-Day Technical Skills:
- Design, develop, test, deploy, and maintain complex data pipelines using Databricks.
- Write and execute data unit/integration tests for data quality using Python and relevant testing frameworks.
- Build and maintain scalable data pipelines for both streaming and batch requirements using Spark.
- Develop API integrations to support data pipelines and future growth.
- Utilize cloud technologies like Azure and AWS to manage large-scale data.
- Implement data cleansing, transformation, and validation routines using techniques like data profiling and data quality checks.
- Create data models that are clear, concise, and perfectly capture the essence of your client's data.
- Ensure data security by championing secure coding practices and adhering to best practices.
Additional Responsibilities:
- Partner with the sales team to develop compelling Databricks solution demos and presentations for pre-sales calls.
- Articulate the technical benefits and ROI of Databricks to potential clients in a clear and concise manner.
- Help craft technical proposals that address client pain points and showcase the value proposition of Databricks.
- Collaborate with sales leadership to identify new market opportunities and expand our Databricks client base.
- Languages: Python, SQL (likely Spark SQL)
- Technologies: Databricks, Apache Spark, Cloud platforms (Azure, AWS)
- Data Engineering: ETL tools (Informatica), data modeling, data cleansing, data transformation, data quality checks, data pipelines (batch & streaming), API integrations
- Version Control: Git
- Scripting: Shell scripting (likely Bash) may be required
- Other: Data profiling, secure coding practices, Jupyter Notebooks