Title: Sr. Data Engineer
Role: Fulltime / Permanent
Location: Plymouth county/Lynn/Norwell, MA
Responsibilities:
- Provide thought leadership, strategic insight, and clear communication on Data Engineering and execution for complex problems.
- Design data storage systems that scale, perform, and are economical enough to satisfy the organization’s requirements.
- Develop and implement data integration processes, frequently using Azure Data Factory, to extract, transform, and load (ETL) data from various sources into Azure data warehouses or data lakes.
- Use big data technologies like Apache Spark to handle and analyze enormous volumes of data.
- Collaborate with data scientists to create data sets and engineer features for AI Initiatives
- Implement and optimize data retrieval for Power BI solutions.
- Plan, coordinate, and oversee all technical aspects of our Azure data initiatives. Ensure that project goals align with the company’s objectives and meet the needs of our stakeholders.
- Design and implement data-driven solutions using Microsoft Azure’s tools and services, such as Azure Machine Learning, Azure Databricks, Azure Data Factory and Azure databases or Fabric implementation. Ensure these solutions are scalable, reliable, and secure.
- Lead a talented team of data scientists working in multiple regions. Foster a collaborative environment that encourages innovation and professional growth.
- Work closely with cross-functional teams to understand their needs and challenges. Leverage your expertise to provide data-driven AI solutions that address these issues.
- Leverage large volumes of data from various sources(Hybrid Cloud, OnPrem) to support teams building AI solutions and meaningful insights.
- Drive feedback loop for the models implemented in production using metrics to ensure our whole MLOps pipeline is effective and continuously refreshed with the right data set and algorithms.
- Stay updated with the latest trends and advancements in data science, machine learning, GenAI space and cloud computing.
Qualifications
- A master’s degree in computer science, Statistics, Mathematics, or a related field is required. A bachelor’s degree with significant relevant experience would suffice.
- 5+ years of data engineering experience leveraging technologies such as Azure Data Factory, Azure Synapse Analytics, Azure Stream Analytics, Azure Event Hubs, Azure Data Lake Storage, Azure Databricks
- Experience with data pipelines and engineering infrastructure: You should have experience designing and implementing data pipelines and engineering infrastructure for machine learning systems.
- Solid knowledge of data processing languages, including SQL, Python, Scala.
- Experience with AI implementations and Power BI.
- Experience with Microsoft Azure’s data services: You should have hands-on experience with various data services offered by Microsoft Azure, such as Azure Machine Learning, Azure Databricks, and Azure Data Factory.
- Familiarity with Azure Cognitive Services for integrating AI capabilities (like vision, language, and decision functionalities) directly into applications.
- Ability to work with large datasets and experience with data processing tools like SQL, Pandas, and big data technologies such as Apache Spark.
- Proficiency in Python, .NET C#, API development
- Experience with SQL, NoSQL databases and Data Fabric architectures.
- Strong communication skills to effectively collaborate with team members and stakeholders.
- Strong problem-solving skills and ability to think algorithmically.
- Ability to translate complex findings into a compelling narrative for non-technical stakeholders.
- Certifications like Microsoft Certified: Azure Data Engineer Associate will be preferred.