Overview
The data engineer plays a critical role in the organization, responsible for designing, building, and maintaining scalable data pipelines which transform and move data to meet the needs of stakeholders.
Key Responsibilities
- Collaborate with data scientists and analysts to understand data needs
- Design and develop data models and schemas
- Build and maintain ETL processes
- Ensure data quality and reliability
- Implement scalable and secure data storage solutions
- Develop and maintain data pipeline orchestration
- Optimize and troubleshoot data infrastructure
- Collaborate with cross-functional teams to integrate data from multiple sources
- Conduct performance tuning of data processes
- Create and maintain documentation for data processes and systems
- Stay current with industry trends and best practices
- Participate in the evaluation and implementation of new technologies
- Support data governance and security initiatives
- Contribute to the development of data architecture strategies
- Train and mentor junior members of the data engineering team
Required Qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field
- Proven experience in data engineering or related field
- Strong understanding of relational and NoSQL database systems
- Proficiency in Python and SQL
- Experience with big data technologies such as Hadoop, Spark, or Kafka
- Knowledge of data warehousing and modeling techniques
- Familiarity with data visualization tools and techniques
- Experience with cloud platforms such as AWS, Azure, or GCP
- Expertise in ETL tools and processes
- Ability to work in a fast-paced, collaborative environment
- Excellent problem-solving and analytical skills
- Strong communication and teamwork abilities
- Experience with data security and compliance principles
- Certifications in relevant technologies are a plus
Skills: python,kafka,etl tools,gcp,etl,hadoop,data modeling,aws,data warehousing,spark,data security,sql,compliance principles,big data,data visualization,azure
