Job: Databricks SME(W2)

Location: Atlanta, GA(Hybrid)

Experience: 10+

Duration: Long term

Design and develop scalable Databricks data pipelines for real-time and batch processing using Apache Spark.

Lead the integration and optimization of data flows from various sources into Azure Databricks, including structured and unstructured data.

Optimize Spark jobs for performance, resource utilization, and cost efficiency.

Collaborate with data engineers, scientists, and analysts to implement data transformation, cleaning, and validation processes.

Implement Delta Lake solutions to ensure data reliability and versioning within a Lakehouse architecture.

Build, maintain, and optimize CI/CD pipelines for Databricks environments, ensuring smooth deployment of data solutions.

Manage infrastructure as code using tools like Terraform, ARM templates, or Bicep to provision and configure Databricks clusters and other Azure resources.

Automate deployment and testing processes using Azure Pipelines, GitHub Actions.

Implement robust monitoring, logging, and alerting solutions to ensure the smooth operation of data pipelines and infrastructure.

Collaborate with the security team to enforce best practices in data governance, compliance, and access management using Azure AD and role-based access controls (RBAC).

Work closely with stakeholders across data engineering, analytics, and IT teams to understand requirements and deliver high-quality solutions.

Support DevOps practices, including continuous integration, delivery, and deployment across the data lifecycle.

Provide technical support and mentorship to team members on Azure Databricks, Spark, and DevOps practices.

Kindly share your resume at thanuja@infodatasol.com

Informatik Data Solutions Inc. jobs

Databricks SME