Databricks Engineer
Location: Plano, TX Hybrid ( need candidates within TX)
Experience : 10 + years
Must Have
- Databricks
- Apache Spark
- Python(and/or Scala)
- AWS
- SQL
- Java
- CI/CD Pipeline tools
Minimum Industry experience: 10 years
Job Description
- Bachelor’s degree or equivalent work experience
- Must have experience implementing atleast 2 end to end projects with Databricks which includes components like
- Delta lake
- Unity Catalog
- Databricks Connect
- DataBricks API
- Databricks workflow Orchestration
- Must be well-versed with Databricks Lakehouse concepts and its implementation in large scale enterprise environments
- Must have good understanding to create complex data pipelines and use of data ingestion tools like nifi
- Must be well versed in using Apache Spark framework and strong in SQL and spark-sql
- Must have excellent coding skills in Python(and/or Scala), preferably Python
- Must have strong performance optimization skills to improve efficiency and reduce cost using techniques like parallel processing
- Must have worked on AWS and should have working experience of AWS services like S3, Lambda, DynamoDB, SQS, CloudWatch
- Must be able to work with architects and lead engineers to design solutions to meet functional and non-functional requirements
- Must have strong written and verbal communication skills
- Must have experience in an Agile environment
- Data engineering background is a plus