We are looking for an experienced Data Engineer to join our dynamic team. The ideal candidate will possess deep expertise in data engineering fundamentals, hands-on experience with Databricks and Snowflake, strong Python skills, and proficiency in cloud platforms like AWS or Azure. You will design, implement, and optimize data pipelines while ensuring data integrity and governance across the organization.
Responsibilities:
- Design, build, and maintain scalable and efficient data pipelines (ETL processes).
- Work extensively with Databricks and Snowflake for data warehousing and advanced data processing.
- Develop robust data models, ensuring efficient data storage, retrieval, and processing.
- Ensure strong governance practices including data lineage, partitioning, and optimization techniques.
- Utilize Python libraries such as Pandas, NumPy, PySpark, and others to process and analyze large datasets.
- Create, optimize, and maintain SQL scripts, stored procedures, and triggers for relational databases (e.g., MSSQL, MySQL) and NoSQL DBs (e.g., DynamoDB, MongoDB).
- Implement and manage ETL workflows using tools like Apache Airflow, AWS Glue, Azure Data Factory, or Talend.
- Collaborate with cross-functional teams to troubleshoot, identify, and resolve data-related issues.
- Develop and enforce data security, encryption, and compliance standards.
- Maintain code repositories, perform version control, and automate workflows using Git, Jenkins, and Azure DevOps.
- Work within Agile frameworks, using tools like Jira and Confluence for project tracking and collaboration.
Qualifications:
- Education: Bachelor’s degree or higher in Computer Science or a related field, or a combination of relevant education, experience, and training.
Experience:
- 5+ years in Data Engineering.
- 3+ years working with Python, specifically for data processing (proficiency in Pandas, NumPy, PySpark, PyOdbc, PyMsSQL, etc.).
- 2+ years of experience with Data bricks and Snowflake.
- Strong SQL skills and experience with relational databases (MSSQL, MySQL) and NoSQL (DynamoDB, MongoDB, Redis).
- Cloud Expertise: AWS and/or Azure.
- ETL Tools: Experience with one or more tools like Apache Airflow, AWS Glue, Azure Data Factory, Talend, or Alteryx.
- Strong understanding of data engineering fundamentals: ETL, data modeling, data lineage, governance, partitioning, and optimization.
- Hands-on experience with Data bricks-specific skills (Apache Spark, Delta Lake, SQL, Workflows, RBAC, Unity Catalog, Encryption & Compliance).
Nice to Have:
- Experience with data governance and metadata management tools such as Collibra or Hackolade.
- Expertise in data migration strategy, tools, and testing methodologies (e.g., migrating data from Snowflake to Data bricks).
- Familiarity with validation techniques, such as checksums, row counts, and query performance benchmarks.
About Brickred Systems:
Brickred Systems is a global leader in next-generation technology, consulting, and business process service companies. We enable clients to navigate their digital transformation. Brickred Systems delivers a range of consulting services to our clients across multiple industries around the world. Our practices employ highly skilled and experienced individuals with a client-centric passion for innovation and delivery excellence.
With ISO 27001 and ISO 9001 certification and over a decade of experience in managing the systems and workings of global enterprises, we harness the power of cognitive computing hyper-automation, robotics, cloud, analytics, and emerging technologies to help our clients adapt to the digital world and make them successful. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.