Role : AWS Databricks Platform administrator
Location : Preferably onsite (Tampa, FL) or Remote
Type Contract
Detail JD
Position Overview:
- The AWS Databricks Platform administrator is a technical role to design, implement and maintain process used to manage the organizations Databricks platform.
- The administrator will be responsible for facilitating the work of data analysts, data engineers and data scientists while maintaining best practices for security and compliance.
- Unity catalog will be used to implement access and identity policies.
- Infrastructure will be maintained using Terraform.
- The administrator will be responsible for assisting with compute issues, monitoring and alerting, and assuring the platform functions smoothly.
- This role requires great organization and communication skills.
Job Responsibilities
- Manage and maintain role-based access to data and features in the Databricks Platform using Unity Catalog
- Implement external access controls for outside teams using service principles, SQL warehouses and Delta Sharing
- Work with platform users to solve problems and facilitate work
- Create infrastructure for AWS and Databricks using Terraform : S3, IAM roles, Instance Profiles, KMS Keys
- Improve processes and systems used to manage infrastructure, users and external access
- Keep track of unused assets for pruning
- Understand and implement best practices for security and compliance
- Implement service principles and access tokens for external users
- Understand how to utilize Databricks APIs to automate administrative tasks
- Create queries and dashboards to monitor critical systems and processes
- Create documentation for users and admins
- Automate common admin tasks using Databricks notebooks
- Manage workflow tags, cluster tags, workflow naming convention enforcement
- Clusters health check and best practices implementation
- Regular back up & recovery
- Privilege reviews on users and resources
Required Qualifications
- 4 or more years of experience administering the Databricks platform
- Strong understanding of data engineering needs
- Experience creating AWS infrastructure with Terraform
- Strong understanding of AWS infrastructure required by Databricks
- Experience with compliance audits/audit documentation
- Experience using Databricks APIs with tools like Postman
- Understanding of the Databricks workspace and development environment
- Experience using GitHub to manage source code
- Strong SQL skills for creating admin related queries and dashboards
- Strong debugging skills
- Ability to do risk analysis for 3rd party integrations with AWS resources.
- Knowledge of Python a plus
- Familiarity with Datadog