Overview
The GCP Data Engineer plays a crucial role in designing, developing, and maintaining the data architecture on Google Cloud Platform (GCP). They are responsible for implementing data solutions, optimizing data pipelines, and ensuring the reliability and scalability of data infrastructure.
Key Responsibilities
- Design and develop data processing and ETL pipelines on GCP using tools such as Dataflow, Dataprep, and Apache Beam
- Implement big data solutions using GCP services like BigQuery, Bigtable, and Pub/Sub
- Work closely with data scientists and analysts to ensure data availability and enable efficient analysis
- Optimize data storage, retrieval, and processing capabilities in GCP
- Automate and monitor data pipelines to ensure efficient and reliable performance
- Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
- Ensure data security, privacy, and compliance with regulatory standards
- Troubleshoot and resolve data infrastructure issues in a timely manner
- Document data engineering processes, workflows, and best practices
- Stay updated with GCP offerings and industry best practices in data engineering
Required Qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field
- Proven experience in designing and building data solutions on Google Cloud Platform
- Strong proficiency in SQL and database management
- Hands-on experience with GCP services such as Dataflow, BigQuery, Pub/Sub, and Bigtable
- Proficiency in programming languages like Python, Java, or Scala
- Experience with data modeling, schema design, and optimization
- Knowledge of best practices for data governance and data quality
- Ability to work in a fast-paced, dynamic environment with tight deadlines
- Excellent problem-solving and analytical skills
- Strong communication and collaboration abilities
Skills: gcp,data engineering,big data,sql,python