Observability Services on UQ Statushttps://status.its.uq.edu.au/affected/observability-services/Incident historygithub.com/cstateen2025-09-11T09:00:00+00:002025-09-11T09:00:00+00:00[Resolved] OpsGeni Alerting - Degraded Performancehttps://status.its.uq.edu.au/issues/19775/Thu, 11 Sep 2025 09:00:00 +0000https://status.its.uq.edu.au/issues/19775/2025-09-12 07:30:00The alerting platform Opsgeni is currently experiencing degraded performance leading to delays and missed alerts for services which teams monitor The SaaS provider is aware and have advised they have addressed the root cause and have put mitigations in place. Improvements are expected to flow through soon. ITS will monitor this and provide updates as they become available. Update Latest vendor update - Update - Our team is continuing to investigate with the highest level of urgency in order to restore Jira Service Management and Opsgenie services.<p>The alerting platform Opsgeni is currently experiencing degraded performance leading to delays and missed alerts for services which teams monitor</p> <p>The SaaS provider is aware and have advised they have addressed the root cause and have put mitigations in place. Improvements are expected to flow through soon.</p> <p>ITS will monitor this and provide updates as they become available.</p> <hr> <p><strong>Update</strong> Latest vendor update -</p> <p>Update - Our team is continuing to investigate with the highest level of urgency in order to restore Jira Service Management and Opsgenie services.</p> <p>At this time we are prioritising infrastructure to try and restore new and active alerts to be populated with their alert content as soon as possible, while continuing to deliver existing messages as they are ready.</p> <p>We will continue to provide updates as we progress, and will ensure we have an update posted within the hour. Sep 11, 2025 - 04:50 UTC <span class="faded">(15:12 AEST — Sep 11)</span> </p> <p><strong>Update</strong> Vendor Update -</p> <p>Update - Our infrastructure teams are working diligently to try and ensure all services are restored to Jira Service Management and Opsgenie as soon as possible.</p> <p>New alerts are continuing to notify users, however we are still prioritising infrastructure to restore the content inside these alerts for Web and Mobile UIs.</p> <p>We will continue providing updates when available and will ensure we have further update within the hour. Sep 11, 2025 - 05:51 UTC <span class="faded">(16:30 AEST — Sep 11)</span> </p> <p><strong>Update</strong> Vendor update &amp; resolution-</p> <p>Resolved - Between September 10, 2025, 3:19 PM UTC and September 11, 2025, 2:44 PM UTC, there was degraded performance in web experiences and the REST APIs for some Jira Service Management, Opsgenie, and Compass Cloud customers in the US region. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The issue has been resolved and the service is operating normally. Sep 11, 18:45 UTC Monitoring - We have deployed the fix and all operations are back to normal. We will continue to monitor the operations.</p> <p>We will continue providing specific updates when available. Sep 11, 15:05 UTC <span class="faded">(07:30 AEST — Sep 12)</span> </p> <p><strong>Resolution</strong> The vendor has reported the service disruption is now resolved -</p> <p>&ldquo;Deployed a fix to mitigate the issue and have verified that the services have recovered. The issue has been resolved and the service is operating normally.&rdquo; <span class="faded">(07:30 AEST — Sep 12)</span> </p>