Job Description:
We are seeking a highly skilled machine learning engineer specializing in NLP and AI solutions to join our innovative team. The ideal candidate will have a strong background in Python and experience with AWS services, specifically in deploying and scaling AI/ML solutions. This role will focus on developing, fine-tuning, and deploying NLP models for various applications such as text classification, summarization, and generation.
Core Responsibilities:
Model Development and Deployment: Design, develop, and deploy NLP models for various applications, including text classification, completion, summarization, and generation. Utilize Python libraries such as LangChain, LangIndex, PyTorch, SageMaker SDK, and psycopg2.
AWS Integration: Leverage AWS services (Bedrock, SageMaker, IAM, Glue, S3, Lambda, CodeCommit, and CodePipeline) to build, deploy, and manage machine learning models.
Containerization and Orchestration: Utilize Docker and AWS ECR to containerize applications and manage deployments efficiently.
Application Development: Create quick applications using frameworks like Streamlit, NodeJS, or similar to showcase AI/ML capabilities.
Vector Embeddings and Databases: Use embedding models to create vector embeddings and work with vector databases for optimized retrieval in AI models.
RAG Architecture Implementation: Understand and implement Retrieval-Augmented Generation (RAG) architectures, optimizing for retrieval and understanding trade-offs in different splitting methods.
Benchmarking and Evaluation: Conduct model evaluations using industry benchmarks and methodologies to determine vector similarity and overall model performance.
Scaling and Distributed Training: Scale machine learning training workloads using distributed training techniques on GPUs and develop microservices for AI/ML/GenAI products.
Data Preprocessing and Analysis: Work with large-scale datasets, preprocess data, and perform in-depth analyses to extract meaningful insights for AI model training.
Preferred Qualifications:
Fine-Tuning Expertise: Real-world experience in fine-tuning models, including understanding various methods and data preprocessing techniques for optimal performance.
AWS Certifications: AWS Solutions Architect and/or AWS Machine Learning Specialist certifications are highly desirable.
Core Skills and Experience Required:
Python Expertise: Proficiency in Python and its libraries, such as LangChain, LangIndex, PyTorch, SageMaker SDK, and psycopg2.
AWS Experience: Strong experience with AWS services including Bedrock, SageMaker, IAM, Glue, S3, Lambda, CodeCommit, and CodePipeline.
Containerization: proficiency with Docker and AWS ECR for application containerization and deployment.
Application Frameworks: Experience creating applications using Streamlit, NodeJS, or other frameworks.
NLP and AI Models: Education and experience in developing NLP models and working with embedding models and vector databases.
Distributed Systems and Microservices: Experience with scaling machine learning workloads and developing microservices for AI/ML products.
Data Handling: Ability to work with large datasets, perform preprocessing, and conduct detailed analysis.