Overview
The GCP Data Engineer plays a crucial role in designing, implementing, and managing data processing systems leveraging Google Cloud Platform (GCP) services. This position is essential for ensuring efficient data management, processing, and analysis to support the organization's data-driven decision-making processes and solutions.
Key Responsibilities
- Design, develop, and deploy GCP-based data processing systems and solutions.
- Implement scalable and reliable data pipelines for ingesting, processing, and storing large volumes of data.
- Optimize data storage and retrieval processes using GCP storage solutions.
- Collaborate with data scientists and analysts to understand data requirements and implement appropriate solutions.
- Ensure data integrity, security, and compliance with regulatory requirements.
- Monitor and troubleshoot data processing systems to ensure optimal performance and reliability.
- Develop and maintain documentation for data engineering processes and systems.
- Implement data governance best practices for data quality, lineage, and metadata management.
- Assist in the evaluation and selection of appropriate GCP services for specific data processing needs.
- Stay updated with GCP developments and recommend innovative solutions to improve data engineering processes.
Required Qualifications
- Bachelor's or master's degree in Computer Science, Data Engineering, or related field.
- Proven experience in designing and implementing data processing systems on Google Cloud Platform.
- Proficiency in programming languages such as Python, Java, or Scala for data processing and ETL (Extract, Transform, Load) tasks.
- Strong understanding of big data technologies and frameworks, including Hadoop, Spark, and Kafka.
- Experience with GCP services such as BigQuery, Dataflow, Pub/Sub, and Dataproc.
- Expertise in SQL and database technologies for data manipulation and querying.
- Ability to troubleshoot and optimize data processing workflows for performance and cost-efficiency.
- Excellent communication skills and the ability to collaborate in cross-functional teams.
- Understanding of data governance principles and best practices.
- Familiarity with machine learning pipelines and model serving on GCP is a plus.
- Certifications in GCP data engineering or related areas are preferred.
Skills: gcp,data engineering,big data,etl,sql