Job Title: Data Engineer / Data Architect
Location: Jersey city,NJ
Experience: 10+ Years Expert level
Job Type: Full-Time contract (W2 preferred)
Job Summary
We are seeking an experienced Data Engineer/ Architect with expertise in Python, PySpark, and Databricks to join our dynamic team. You will be responsible for designing, developing, and optimizing scalable data pipelines and workflows.
Key Responsibilities
- Design, build, and maintain scalable data pipelines using Python, PySpark, and Databricks.
- Collaborate with cross-functional teams to integrate data from various sources into data lakes and warehouses.
- Optimize data workflows for efficiency and performance in large-scale environments.
- Develop and manage ETL processes to ensure smooth data flow across systems.
- Implement data quality checks and monitor data pipelines for accuracy and integrity.
- Manage cloud-based data platforms (AWS, Azure, GCP) for seamless integration and performance tuning.
- Document data models, processes, and systems for handovers and ongoing support.
Skills And Qualifications
- Bachelor's or master’s degree in computer science, Engineering, or a related field.
- Experience converting Legacy Data Warehouse applications to Databricks environment.
- Proficiency in Python and PySpark for data manipulation and distributed computing.
- Hands-on experience with Databricks and its ecosystem (Spark, Delta Lake).
- Experience with cloud platforms like AWS, Azure, or GCP, focusing on data engineering services.
- Strong knowledge of SQL and experience with relational and NoSQL databases.
- Familiarity with data modeling, warehousing, and designing schemas for analytics.
- Understanding of CI/CD pipelines and version control systems (e.g., Git).