NGDC Serverfarm on UQ Statushttps://status.its.uq.edu.au/affected/ngdc-serverfarm/Incident historygithub.com/cstateen2024-03-10T10:30:00+00:002024-03-10T10:30:00+00:00[Scheduled] Emergency Downgrade of APIC and Gen 2 Leaf switches in Prentice DChttps://status.its.uq.edu.au/issues/19357/Sun, 10 Mar 2024 10:30:00 +0000https://status.its.uq.edu.au/issues/19357/Scheduled for 2024-03-10 10:30:00 - 2024-03-10 15:30:00 Due to failed CR13641 the Prentice ACI Fabric now in a mixed non-supported mode, we need to reverse the change and rollback the 3 APICs and 8 Gen 2 Leaf swicthes to Version 4.2(7s). Network Leaf switches will need to have vlan311 restored to the config for F5 Load balancers. If this is not updated by the fabric after a downgrade, a reload for each network leaf switch is to be done, in a non-disruptive process.<p>Scheduled for 2024-03-10 10:30:00 - 2024-03-10 15:30:00</p> <p>Due to failed CR13641 the Prentice ACI Fabric now in a mixed non-supported mode, we need to reverse the change and rollback the 3 APICs and 8 Gen 2 Leaf swicthes to Version 4.2(7s).</p> <p>Network Leaf switches will need to have vlan311 restored to the config for F5 Load balancers. If this is not updated by the fabric after a downgrade, a reload for each network leaf switch is to be done, in a non-disruptive process.</p> <hr> <p><strong>Resolution</strong> Automatic resolution. <span class="faded">(15:30 AEST — Mar 10)</span> </p>[Scheduled] APIC Upgrade and Leaf Switch Upgrade/Replacement in the Prentice Next Gen DChttps://status.its.uq.edu.au/issues/19355/Sat, 09 Mar 2024 06:00:00 +0000https://status.its.uq.edu.au/issues/19355/Scheduled for 2024-03-09 06:00:00 - 2024-03-10 20:00:00 This is again a multi-stage change over 2 days. a) APIC Software Upgrade will start on 06:00 AM on Sat 9th March and expected to be completed by 11:00 AM b) Gen2 Leaf switch upgrade will commence on 11AM on Sat 9th March. These will be done in two groups of leaf switches, which may likely take 4 hours. c) Four (4) Leaf Switches are to be replaced 1 by one.<p>Scheduled for 2024-03-09 06:00:00 - 2024-03-10 20:00:00</p> <p>This is again a multi-stage change over 2 days.</p> <p>a) APIC Software Upgrade will start on 06:00 AM on Sat 9th March and expected to be completed by 11:00 AM</p> <p>b) Gen2 Leaf switch upgrade will commence on 11AM on Sat 9th March. These will be done in two groups of leaf switches, which may likely take 4 hours.</p> <p>c) Four (4) Leaf Switches are to be replaced 1 by one. No Outages are expected if the servers are dual connected.</p> <p>Timelines: This is a two-day activity. Will target to achieve switch replacement for two racks in each day and if time permits, it will be three rack. Starting at 06:00 AM on 9th March and finishing on 06:00 PM and then again starting on 10th march at 06:00 AM and continue.</p> <hr> <p><strong>Resolution</strong> Automatic resolution. <span class="faded">(20:00 AEST — Mar 10)</span> </p>[Scheduled] PRNDC ACI - CIMC-APIC And Gen2 Leafs Firmware Upgradehttps://status.its.uq.edu.au/issues/19349/Sat, 02 Mar 2024 08:00:00 +0000https://status.its.uq.edu.au/issues/19349/Scheduled for 2024-03-02 08:00:00 - 2024-03-03 20:00:00 This is the first Stage of the PRNDC ACI Upgrade: This includes software update of CIMC and three APIC controllers, followed by software update of Eight Gen2 leaf switches. Timelines: a) CIMC Firmware Upgrade, will start at 8:00 AM on 2nd March, which is likely to continue for 6 hours to complete for all the apic nodes. b) APIC Software Upgrade will start on 08:00 AM on 3rd March and expected to be completed by 3:00 PM<p>Scheduled for 2024-03-02 08:00:00 - 2024-03-03 20:00:00</p> <p>This is the first Stage of the PRNDC ACI Upgrade:</p> <p>This includes software update of CIMC and three APIC controllers, followed by software update of Eight Gen2 leaf switches.</p> <p>Timelines:</p> <p>a) CIMC Firmware Upgrade, will start at 8:00 AM on 2nd March, which is likely to continue for 6 hours to complete for all the apic nodes.</p> <p>b) APIC Software Upgrade will start on 08:00 AM on 3rd March and expected to be completed by 3:00 PM</p> <p>c) Gen2 Leaf switch upgrade will commence on 4:00 PM on 3rd march. These will be done in two groups of leaf switches, which may likely take 4 hours.</p> <hr> <p><strong>Resolution</strong> Automatic resolution. <span class="faded">(20:00 AEST — Mar 3)</span> </p>