Principal, Infra PM&A

• Technology Infrastructure is seeking an enthusiastic and dynamic individual to join the Performance Monitoring and Analytics Team as a Principal with a primary focus on developing complex end to end observability monitors for infrastructure, application performance, data analytics, and URL/WEB/CLOUD monitors
• The candidate should be proactive and passionate about identifying, implementing, and promoting the use of monitoring tools across multi-tier applications at an enterprise level, exploring new technologies and best practices, and collaborate candidly with vendors to realize stable scalable monitoring solutions
• A successful candidate will ensure production operational stability for customers through proactive full stack monitoring of clients systems (applications and infrastructure), continuous optimization, and automation of remediation measures to ensure a flawless customer experience
• The candidate will demonstrate expertise and leadership, through leading by example and open communication in a team and customer focused environment

Major Duties

- Collaborate with teams to craft, implement, and maintain observability solutions that provide deep insights into our applications, infrastructure, and operational processes.
- Develop working relationships with application and infrastructure teams to understand and flush out applicable use cases for monitoring and document them for traceability and auditing.
- Scope and gather technical requirements around the customer monitoring use cases and business KPIs, translate them to tool specifications for Dynatrace, Infrastructure, OS, Synthetics, Real User Monitoring, and Dashboards, and ensure successful implementation and operational success.
- Implement automated scaling mechanisms, performance testing frameworks, and capacity planning strategies to ensure the platform can handle increasing demand while maintaining a high-quality user experience.
- Strategize and implement scalable pipeline ready solutions for continuous monitoring and availability using SNOW tools, CI/CD tools and Automation solutions like Chef/Ansible/Puppet/Terraform.

• Promote automated remediation principals by targeting optimal observability strategies for applications and infrastructure services
• Participate in the management of infrastructure monitoring services through a deep understanding of the primary Monitoring of Monitoring application and its requirements

Leadership Responsibilities:
Provides strategic leadership and roadmaps vision, aligned with department and company goals and objectives.
- Develop and execute a comprehensive observability strategy, including the selection, implementation, and integration of appropriate monitoring, logging, and tracing tools.
- Define key performance indicators (KPIs) and establish monitoring frameworks to proactively identify and resolve issues, ensuring high availability and optimal performance.
- Communicates progress, risks, and outcomes to senior leadership and other stakeholders, providing insights and recommendations for informed decision-making.
- Collaborate with cross-functional teams to identify manual processes, bottlenecks, and pain points, and design and implement scalable automation solutions to increase operational efficiency and reduce human errors.
Mentors junior level technical staff within the functional monitoring area of the IT organization

• Lead troubleshooting, analysis, and solution of unexpected systems behaviors that impact the quality of service
• Analyze monitoring metrics (eg Signal:Noise), objectives, and key results (eg reduction of monitoring gaps) to continuously improve the team's level of service and customer experience
• Drive operational excellence using observability tools across partners, managed service providers, and related stakeholders

- Periodically help drive incident investigations, coordinate with relevant teams, and drive root cause analysis to identify systemic issues and implement preventive measures.
- Champion a culture of continuous improvement and digital transformation by implementing feedback loops, analyzing system metrics, and driving iterative enhancements.

Requirements:
- 12+ years of experience as an Observability Engineer, Site Reliability Engineer, or similar role, with a focus on monitoring, logging, tracing, and alerting.
- Experience working in an Agile delivery environment
- Solid understanding of software development and application architecture principles
- Strong knowledge of observability tools and frameworks such as Dynatrace, Azure App Insights, Elastic, Prometheus
- Experience with Azure Managed Services, Serverless Frameworks.
- Prior experience with Java, JS, Python, Teraform, NodeJS, Spring
- Dynatrace Certification Preferred
- ITIL Foundations Certification is preferred
- CIS in Discovery, Service Mapping, Event Mgmt, Cloud Mgmt
- Experienced in implementation on ServiceNow and Dynatrace Discovery, Service Mapping, Event Mgmt and Orchestration use cases.
- Strong knowledge of incident management processes, including incident response, escalation, and post-incident analysis, root cause, error budget, mean time to detect, mean time to restore metrics.
- Demonstrate a strong understanding of Cloud (Azure) services and standard processes
- Solutioning and Design the SNOW ITOM solution using industry best practices.
- Experience with CMDB design, architecture and implementations with a fair understanding of ServiceNow CMDB model and extensions, including integrations with observability tools and APIs.
- Proven experience engineering and implementing end to end observability tools in a large matrixed organization with a variety of technical debt and legacy platforms and applications

Knowledge / Skills / Experience:

• Bachelors Degree in information technology, computer science, or a related field
• Must have 8 to 10 years of experience in Application Performance Monitoring using enterprise standard tools
• Prior experience must include 4 years of experience working with agile scalable software engineering
• Prior experience must include 6 to 8 years of experience in CICD, automation, and DevOps practices
• Must have knowledge in tool sets like Dynatrace, Elk, Catchpoint, SCOM, Pandora, Moogsoft, ServiceNow ITOM Health, Open Source, and related API monitoring integration, deployment and engineering
• Hands on experience on Event Management and ITIL Foundations (certification preferred)
• Must have knowledge in application architecture, OSI layers, and software design and development methodologies
• Strong Automation & Scripting capabilities (Ansible, Shell, Bash, Perl, PowerShell, etc) to execute monitoring tasks for custom requirements within the capabilities of the suite of monitoring tools
• Proven diagnosis and tuning experience with Application, Middleware, and Infrastructure components
• Prior experience working with business metrics reporting, customer experience monitoring, and optimization for digital products
• Experience in documentation and task management tools like JIRA, SharePoint, MS Office tools, etc
• Experience working in Agile teams and familiarity with agile delivery process and ceremonies
• Advanced skills with Excel, Power BI, and related reporting and analytics tools a plus
• Six Sigma Certification a plus