We are seeking a skilled and experienced Data Engineer to join our dynamic team. The successful candidate will be responsible for building and optimizing data pipelines, supporting data transformation, and managing data flow across various platforms. This role will focus on leveraging big data technologies, cloud-based data warehouses, and NoSQL databases to deliver scalable and high-performance data solutions.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and systems on big data platforms such as Apache Spark and Databricks.
- Implement data ingestion, transformation, and processing workflows on cloud-based data warehouses, including Amazon Redshift, Snowflake, and Google BigQuery.
- Build and maintain data models, optimize data performance, and ensure high data availability in a distributed environment.
- Develop object-oriented code and scripts using languages such as Java and Python to support data operations and automation.
- Work with NoSQL databases like DynamoDB, Cosmos DB, and MongoDB to build and optimize high-throughput data storage solutions.
- Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and build reliable solutions.
- Optimize the performance of data workflows by monitoring and troubleshooting pipelines, ensuring data accuracy and consistency across systems.
- Implement and manage data security, governance, and compliance across all data platforms and pipelines.
- Evaluate emerging technologies and provide recommendations to enhance data architecture and improve data processes.
Required Qualifications:
- Proven experience working with big data platforms such as Apache Spark and Databricks.
- Strong expertise in cloud data warehouses, including Amazon Redshift, Snowflake, and Google BigQuery.
- Proficient in object-oriented programming using Java and Python.
- Solid experience with NoSQL databases such as DynamoDB, Cosmos DB, and MongoDB.
- Knowledge of data pipeline management, ETL/ELT processes, and best practices for big data environments.
- Experience with cloud platforms like AWS, Azure, or Google Cloud Platform.
- Strong understanding of database design, data warehousing concepts, and performance optimization techniques.
- Excellent problem-solving skills and ability to work in a collaborative, team-oriented environment.
Preferred Qualifications:
- Experience with data governance, data security, and compliance frameworks.
- Familiarity with containerization and orchestration technologies such as Docker and Kubernetes.
- Knowledge of CI/CD practices and tools like Jenkins, GitLab, or AWS CodePipeline.
Education:
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.