Overview
The Data Engineer plays a crucial role in our organization, responsible for designing, developing, and maintaining scalable data pipelines and infrastructure. This role will have a significant impact on our ability to efficiently and effectively process and analyze large volumes of data, enabling data-driven decision-making across the organization.
Key Responsibilities
- Design, build, and maintain an efficient and reliable ETL pipeline for large-scale data processing.
- Develop and optimize data models for use in data science and analytics applications.
- Implement data quality and validation processes to ensure accuracy and reliability of data.
- Collaborate with data scientists and analysts to understand data requirements and implement solutions.
- Manage and optimize data storage and retrieval systems.
- Create and maintain documentation for data infrastructure and processes.
- Monitor and troubleshoot performance issues with data pipelines.
- Implement security and privacy measures to protect sensitive data.
- Explore and evaluate new data technologies and tools to drive innovation.
- Participate in cross-functional teams to support data-related initiatives.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or related field.
- Proven experience in big data technologies and platforms.
- Proficiency in programming languages like Python, Java, or Scala.
- Strong SQL and database management skills.
- Experience with ETL tools and processes.
- Expertise in data modeling and data warehousing concepts.
- Familiarity with cloud-based data solutions such as AWS or Azure.
- Ability to work with distributed computing and parallel processing.
- Understanding of data governance and compliance requirements.
- Excellent problem-solving and analytical abilities.
- Strong communication and collaboration skills.
- Proven ability to manage multiple projects and priorities effectively.
- Experience with version control systems like Git.
- Knowledge of Agile and DevOps methodologies.
- Ability to thrive in a fast-paced, dynamic environment.
Skills: big data,python,sql,etl,data modeling,spark,cloud