Role Description
The Sr. Splunk Infrastructure Engineer will be responsible for supporting Splunk Enterprise, including managing Windows and Linux servers’ agents, automating infrastructure, configuration, and day-to-day operations through Ansible, and Performing troubleshooting, root cause analysis, and resolution of complex technical issues related to Splunk deployments.
Responsibilities
- Support onboarding and maintenance of logs to Splunk from Windows, Linux, and cloud-based sources.
- Support platform upgrades including coordinating testing of upgrades with users of the platform.
- Automating manual platform management processes through Ansible or other scripting tools/languages
- Troubleshooting incidents impacting the Splunk platform
- Evaluate the use and integration of third-party add-ons.
- Coordinating and collaborating with users of the platform
- Develop training and documentation materials.
Experience General
- Ability to troubleshoot and diagnose complex issues.
- Able to demonstrate experience supporting technical users and conducting requirements analysis.
- Can work independently with minimal guidance & oversight.
- Experience with IT Service Management and familiarity with Incident & Problem management
- Highly skilled in identifying performance bottlenecks, identifying anomalous system behavior, and resolving the root cause of service issues.
- Demonstrated ability to effectively work across teams and functions to influence design, operations, and deployment of universally available software.
- Knowledge of standard methodologies related to security, performance, and disaster recovery.
Required Technical Expertise
- 3 years’ experience managing and configuring Splunk Enterprise and/or Splunk Cloud
- Experience with Splunk clustered deployment topology
- Experience with Linux and Windows agents for Splunk administration
- Experience in designing, developing, and deploying cloud-based solutions using AWS.
- Experience in onboarding new data, configuration, creating new dashboards, and extracting information through Splunk.
- Experience with writing or modifying custom Splunk add-ons.
- Demonstrated proficiency with scripting and automation (bash, python, other programming languages)
- Familiarity with Splunk rest APIs
- Strong scripting skills (e.g., Python, Bash) for automation and custom development.
- In-depth knowledge of log management, data onboarding, and SIEM principles.
Preferred Technical Experience
- Splunk Certification (Admin or Architect)
- Experience with Ansible tower automation
- Experience using Gitlab.
- Experience with large platform migration efforts
- Experience with AWS OpenSearch
- Experience with Crib
- Expertise in languages such as Java, and Python. Implementation knowledge in data processing pipelines using programming languages like Java and Python to extract, transform, and load (ETL) data.
- Create and maintain data models, ensuring efficient storage, retrieval, and analysis of large datasets.
- Troubleshoot and resolve issues related to data processing, storage, and retrieval.
- 3-5 years’ Experience in designing, developing, and deploying data lakes using AWS native services (S3, Glue (Crawlers, ETL, Catalog), IAM, Terraform, Athena)
- Experience in the development of systems for data extraction, ingestion, and processing of large volumes of data
- Experience with data pipeline orchestration platforms
- Experience in Ansible/Terraform/Cloud Formation scripts and Infrastructure as Code scripting is required.
- Implement version control and CI/CD practices for data engineering workflows to ensure reliable and efficient deployments.
- Proficiency in implementing monitoring, logging, and alerting solutions for data infrastructure (e.g., Prometheus, Grafana)
- Proficiency in distributed Linux environments