AB Initio Developer / Data Engineer
Job Summary:
The AB Initio Developer will be responsible for designing, developing, and implementing high-performance data processing applications and pipelines using AB Initio tools. The individual will collaborate with various teams to understand data requirements, optimize data workflows, and ensure the scalability and reliability of data systems.
Key Responsibilities:
- Data Pipeline Development: Design, develop, and maintain complex ETL (Extract, Transform, Load) processes using AB Initio tools to handle large volumes of data from various sources.
- Data Integration: Work on integrating data from multiple systems and ensuring high data quality and consistency.
- Performance Tuning: Optimize AB Initio processes for performance, ensuring they can handle large data sets efficiently.
- Collaboration: Work closely with data architects, business analysts, and other stakeholders to understand data requirements and provide solutions.
- Testing and Validation: Develop and execute unit/integration tests to ensure data integrity and quality in production environments.
- Documentation: Create comprehensive documentation for all developed processes, including data flow diagrams, design documents, and technical specifications.
- Troubleshooting: Diagnose, troubleshoot, and resolve production issues in data pipelines and workflows.
- Best Practices: Implement and follow best practices for data engineering, including security, monitoring, and maintenance of data systems.
Required Skills:
- AB Initio Expertise: Strong hands-on experience with AB Initio tools such as GDE (Graphical Development Environment), Co>Operating System, Conduct>It, Express>It, and other components.
- ETL Concepts: Deep knowledge of ETL processes, data integration, data cleansing, and data transformation techniques.
- Database Skills: Proficiency with relational databases (e.g., Oracle, SQL Server, MySQL) and SQL for querying and managing data.
- Big Data Technologies (Optional): Experience with Hadoop, Spark, or other big data platforms is a plus.
- Programming Knowledge: Experience with shell scripting, Perl, Python, or other scripting languages.
- Problem-Solving Skills: Ability to troubleshoot and resolve complex data and performance issues.
- Version Control: Experience with version control systems like Git, SVN, etc.