Our client is a developer of software products for public organizations. They are currently looking to add a
Data Engineer to join their team.
Salary/Hourly Rate
$125k – $135k
Position Overview
The Data Engineer will create, construct, and oversee data intake pipelines for the Azure Data Factory. They will use Azure Service Bus to manage batch and almost real-time data ingestion. This is a full-time W-2 salary position, full remote on EST hours. There is no possibility of sponsorship now or in the future, so please USC and GC applicants only!
Responsibilities Of The Data Engineer
- Combine data sources that are unstructured, semi-structured, and structured.
- Use formats like Avro, JSON, Parquet, and so on, and control loading into the Bronze layer.
- Use Databricks to create and oversee data engineering pipelines (PySpark, SparkSQL).
- Put into practice data transformation tasks using Spark that go from Bronze to Silver and Gold layers.
- Use Databricks and Data Factory to create and maintain a data quality framework.
- Establish data security by managing permissions on sensitive datasets with RBAC and ACLs.
- Use Databricks Workflows to implement automation frameworks for pipeline execution.
- Use Azure Monitor to monitor pipelines, create alarms, and ensure SLAs are fulfilled.
- Secure credentials using Azure Key Vault and optimize storage in ADLS Gen2.
- Use Azure Cost Management to optimize expenses.
Qualifications For The Data Engineer
- Azure Data Factory, which handles ETL and ELT tasks
- Databricks for data transformations (PySpark, SparkSQL).
- Additionally, Azure Data Lake Storage (ADLS Gen2).
- CRM, ERP, Cosmos DB, PostgreSQL, and SQL Server platforms.
- RBAC, ACLs, and data governance for permission management.
- CI/CD tools like Terraform, GitHub Actions, and Azure DevOps.
- Azure tools for logging, alerting, and monitoring.
Education Requirements
- Bachelor’s degree in Computer Science or a related field is mandatory.
Benefits
- Client-provided benefits available, upon eligibility.
As a woman-owned firm, we value diversity. We are an equal opportunity and affirmative action employer and will consider all applications without regard to race, sex (including gender, pregnancy, sexual orientation and gender identity), age, color, religion or creed, national origin or ancestry, veteran status, disability (physical or mental), genetic information, citizenship or any other characteristic protected by law. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Candidates who apply for roles through the Atrium website will be added to our candidate pool and may be considered for additional roles of a similar title. Please contact us to request an accommodation.
EOE/M/F/D/V/SO
Position ID: 149968