Lead Data Engineer:
We need some strong Lead data engineer profiles… they need good experience with
Python, SQL, ADF and preferably Azure Databricks experience
Job description:
- Building new data pipelines and optimizing data flows using the Azure cloud stack.
- Building data products from scratch.
- Support Business Analysts and Data Architects with discovery and best practices.
- Self-directed and comfortable supporting the data needs of multiple teams, systems and products.
What’s required ?
- Advanced Azure knowledge and experience working and migrating data products from on-prem to Azure.
- Experience building and optimizing ‘big data' data pipelines, architectures and data sets using Py-spark.
- Experience building real-time data pipelines using Event-hub, storage queues and Azure stream analysis.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
- Cloud Big Data Analytics in Azure Synapse Analytics, Azure Analysis Services
- Data Ingestion and Storage including Azure Data Factory, Azure Databricks, Azure Data Lake, Kafka and Spark Streaming, Azure EventHub/IoT Hub, and Azure Stream Analytics
Experience with Object-oriented/ object function scripting languages: Python preferred