need local candidates Hybrid work environment. 2 - 3 days onsite in Newark, NJ.
No USC/GC candidates
Lead data engineer role with a major Financial.
Qualifications
- Engineering lead to help the design/build of the enterprise data platforms Design, Develop and Implement
- End-to-end data solutions (ingest, storage, integration, processing, access) on AWS.
- Data intake/ request/ onboarding services and service documentation.
- Data ingestion services for batch/ real time data ingest and service documentation.
- Data processing services for batch/ real time (Glue/ Kinesis/ EMR) and service documentation.
- Data storage services for data lake (S3)/ data warehouses (RDS/ Redshift)/ data marts and service documentation.
- Data services layer including Athena, Redshift, RDS, microservices and APIs.
- Pipeline orchestration services including lambda, step functions, MWAA (optional).
- Data security services (IAM/ KMS/ SM/ encryption/ anonymization/ RBAC) and service documentation.
- Data access provisioning services (Accounts, IAM Roles RBAC), processes, documentation and education.
- Data provisioning services for data consumption patterns including microservices, APIs and extracts.
- Metadata capture and catalog services for data lake (S3/ Athena), data warehouses (RDS/ Redshift), Microservices/ APIs.
- Metadata capture and catalog services for pipeline/ log data for monitoring / support.
- Architect and implement CI/ CD strategy for EDP .
- Implement high velocity streaming solutions using Amazon Kinesis, SQS, and SMS.
- Migrate data from traditional relational database systems, file systems, NAS shares to AWS relational databases such as Amazon RDS, Aurora, and Redshift.
- Migrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift.
- Implement cost/ spend monitoring for AWS based data platform.
- Implement audit reporting for access of AWS based data platform.
- Contribute to the implementation of a data strategy to enable LOBs and Corporate Functions with a robust, holistic view of data - driven decision making.
- Partner with immediate engineering team, product owner, IT, partners on EDP agenda.
- Leverage and continuously develop best practices, standards, and frameworks.
- Provide technology thought leadership, consulting, and coaching/ mentoring.
- Work with scrum master to develop and own backlog, stories, epics, sprints.
- System design and Architecture for products/ applications for EDP.
- Work closely with various stakeholders (BI, AIML, MDM and other teams) to understand their use-cases and design optimal solution for them.
- Execute technical design and infrastructure strategy for EDP .
- Implement POCs on any new technology or tools to be implemented on EDP and onboard for real use-case.
- Bachelor's degree in computer science, Software Engineering, MIS or equivalent combination of education and experience.
- Experience implementing, supporting data platforms on AWS for large enterprises.
- Full stack development experience building secure internal facing applications.
- Programming experience with Java, Python/ Scala, Shell scripting.
- Solid experience of AWS services such as CloudFormation, S3, Glue, EMR/ Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.
- Solid experience implementing solutions on AWS based data lakes preferred.
- Experience implementing metadata solutions leveraging AWS non-relational data solutions such as ElastiCache and DynamoDB.
- AWS Solutions Architect or AWS Big Data Certification preferred.
- Experience in AWS data lake, data warehouse and business analytics.
- Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Datasync, CloudHealth, CloudWatch, Lambda, Athena, and Redshift.
- Experience in system analysis, design, development, and implementation of data ingestion pipeline in AWS.
- Experience with DevOps and Continuous Integration/ Delivery (CI/ CD) concepts and tools.
- Experience with business intelligence tools such as Tableau, Power BI or equivalent.
- Knowledge of ETL/ ELT.
- Awareness of Data Management & Governance tools.
- Working experience with Hadoop, HDFS, SQOOP, Hive, Python, and Spark is desired.
- Experience working on Agile projects.