About Us: We are a forward-thinking company leveraging cutting-edge technology to drive innovation and efficiency. Our tech stack is 100% Azure Cloud, and we are looking for a skilled Data Scraper and Data Analytics Engineer to join our dynamic team.
Job Description:
Responsibilities:
· Develop and maintain web scraping scripts to extract data from external websites using tools like Python, Scrapy.
· Capture and process clickstream and temporal data from internal websites.
· Design and implement data pipelines using Azure Data Factory, Azure Databricks, and other Azure services.
· Create ETL scripts using python and/or Scala to clean, preprocess, and store data in Azure Data Lake, Azure SQL Database, and Azure Blob Storage.
· Collaborate with data scientists and engineers to train recommendation engines and generate predictions.
· Ensure data quality, integrity, and security throughout the data lifecycle.
· Monitor and optimize data scraping and processing workflows for performance and cost-efficiency.
· Stay updated with the latest trends and best practices in web scraping, data engineering, and Azure Cloud services.
Requirements:
· Bachelor’s degree in Computer Science, Data Science, or a related field.
· 3+ Years of hands-on working experience as Web Scrapping Specialist or Web Scraping Engineer.
· Proven experience in web scraping using Python, BeautifulSoup, Scrapy, or similar tools.
· Strong knowledge of Azure Cloud services, including Azure Data Factory, Azure Databricks, Azure Data Lake, Azure SQL Database, and Azure Blob Storage.
· Proficiency in SQL and experience with data processing frameworks like Apache Spark.
· Familiarity with machine learning concepts and experience working with recommendation engines.
· Bachelor’s degree in Computer Science, Data Science, or a related field.
· Excellent problem-solving skills and attention to detail.
· Strong communication and collaboration skills.
· Ability to work independently and manage multiple tasks effectively.
Preferred Qualifications:
· Experience with API integration and data extraction from various sources.
· Experiences with Microsoft Fabric and Microsoft Data Platform
· Knowledge of data visualization tools like Power BI or Tableau.
· Understanding of data privacy regulations and best practices.