About this Role
Agility Partners is seeking a qualified Advanced Data Engineer to fill an open position with a leading grocer in the Cincinnati area.
Key Responsibilties
- Design, build, and maintain ETL/ELT pipelines in Databricks to process large-scale structured and unstructured datasets.
- Ensure timely ingestion, transformation, and delivery of data to internal consumers, analytics platforms, and machine learning models.
- Perform large-scale data transformation, cleaning, and aggregation using Spark and Databricks notebooks.
- Optimize data processing workflows for efficiency, scalability, and cost-effectiveness by fine-tuning Spark jobs and cluster configurations.
Benefits and Perks
Work within a collaborative team environment where ideas and creativity are welcomed! Family and Work Life balance are important to this organization and valued for the employees.
- Working for an organization that focuses on company culture, inclusion and diversity
- 50% medical coverage for you and your entire family, short/long term disability and life insurance options
- 401(k)
- Life Insurance
- Disability coverage
The Ideal Candidate
Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a related field. A Master’s degree is preferred.
- 7+ years of experience in data engineering, with at least 4 years leading teams and projects in Databricks and Apache Spark.
- Strong hands-on experience with Databricks, including managing clusters, developing pipelines, and optimizing workloads.
- Proficiency in Python, Scala, or SQL for data processing within Databricks.
- Expertise in cloud platforms such as AWS, Azure, or GCP and their data services (e.g., S3, ADLS, BigQuery).
- Deep understanding of ETL/ELT processes, data modeling, and working with large-scale distributed systems.
- Familiarity with DevOps practices, including CI/CD, containerization, and infrastructure automation (e.g., Jenkins, Docker, Kubernetes).
- Experience with data governance, security, and compliance best practices.
- Excellent problem-solving skills and a proactive approach to identifying and resolving performance bottlenecks.
- Strong communication and interpersonal skills, with experience collaborating with both technical and non-technical stakeholders