Overview
We are seeking a highly skilled Databricks and Data Factory Consultant to join our team in New Jersey for a long-term contract role. The ideal candidate will have extensive experience in building and managing data integration solutions using Databricks and Azure Data Factory, along with a strong background in data engineering and big data technologies.
Key Responsibilities
Face to Face Interview - Local consultants are preferred
- Design, develop, and implement ETL pipelines using Databricks and Azure Data Factory.
- Manage Databricks workspaces, clusters, and notebooks to optimize data processing workflows.
- Collaborate with cross-functional teams to gather requirements and translate them into technical solutions.
- Perform data transformations using Spark SQL and PySpark, and implement Delta Lake for data management.
- Monitor and optimize data pipelines for performance and reliability.
- Integrate with various data sources, ensuring seamless data movement and transformation.
- Implement best practices for data governance, security, and compliance.
Qualifications
- Proven experience with Databricks and Azure Data Factory.
- Strong proficiency in SQL and data management principles.
- Experience with big data technologies (e.g., Hadoop, Spark).
- Proficient in programming languages such as Python or Scala.
- Familiarity with CI/CD practices for data solutions.
- Excellent problem-solving skills and the ability to work collaboratively in a team environment.