20h 59m of combined downtime across 23 outages, affecting 5 customers (4h 46m elapsed). 4 unplanned incidents, the most serious a Sev 2 (partial outage). 5 maintenance windows, 1 of them emergency.
At a glance
The headline figures for the period, and what stood out.
Combined downtime
20h59m
across 23 outages · 4h 46m elapsed · 55m on average
Longest single outage
4h1m
Mercury Radio Arts (Glenn Beck)
GLENNBECK-BX-WEST-01
Combined downtime adds up the time every affected broadcaster was down, so two broadcasters down for the same hour count as two hours. Elapsed time counts that hour once.
Unplanned incidents
4
Events that affected service
Still open
0
Every incident resolved
Customers affected
5
CS Test, Mercury Radio Arts (Glenn Beck), Multimedios and 2 more
Maintenance windows
5
1 of them emergency
Notices, no impact
0
Tracked, but service was unaffected
SLA met
—
Not assessed this period
What went well
The Dish Cheyenne (NY) leg carried traffic throughout the 30 June fibre cut, so the LOR never became a customer outage.
Tata pre-notified both LA maintenance windows and accepted a Zixi go-ahead gate before proceeding.
Points to discuss
1The 6 July LA Cloudlyte reboot was scheduled for an 8-hour window but produced ~4 hours of simultaneous outage across five customer broadcasters — the blast radius was wider than the notification implied.
2Two LA broadcasters sat on the same physical host (la-cloudlyte-sr15), so a single host removed redundancy for two separate customers at once. Ask Tata for host-level anti-affinity.
3ZAASOPS-251's outage table was never closed out: three NJ broadcaster rows still read "Ongoing" with no duration, so this period's impact total is a floor rather than a complete figure.
Service impact
How much disruption customers saw, when it happened and where it landed.
23 outages
Regions
Show
When outages happened
All times UTC
LA
NJ
NY
Mercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-WEST-01RegionLAStart6 Jul 08:59End6 Jul 09:02Duration4mHost serverla-cloudlyte-sr23StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-WEST-01RegionLAStart6 Jul 04:15End6 Jul 08:16Duration4h 1mHost serverla-cloudlyte-sr23StatusResolved4h 5m
Mercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart7 Jul 18:35End7 Jul 18:36Duration1mHost serverwl-cloudlyte-sr03StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart7 Jul 18:10End7 Jul 18:10Duration0mHost serverwl-cloudlyte-sr03StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart10 Jul 04:16End10 Jul 04:30Duration13mHost serverwl-cloudlyte-sr03StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart5 Jul 08:34End5 Jul 08:35Duration<1mHost serverwl-cloudlyte-sr03StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart5 Jul 08:05End5 Jul 08:05Duration<1mHost serverwl-cloudlyte-sr03StatusResolved15m
What happenedTata's planned full-node reboot of the LA Cloudlyte environment overran, taking five customer broadcasters down for close to four hours from 04:14 UTC on 6 July. The NJ region then flapped repeatedly the following afternoon.
Root causeCeph RBD remap remediation required a reboot of every node in the LA Cloudlyte environment; NJ flapping was reported by Tata separately and not attributed.
How it was resolvedAll LA broadcasters recovered by 08:16 UTC. NJ flaps self-cleared within a minute each.
Key points
Five broadcasters across four customers were down simultaneously — the notification implied per-instance impact, not a full-region outage.
BX-DIST-WEST-04 and MULTIMEDIOS-BX-WEST-01 share physical host la-cloudlyte-sr15; ask Tata to confirm anti-affinity for redundant pairs.
Request confirmation that the Ceph RBD remap issue is fully cleared and no further reboots are queued.
Combined downtime
19h 52m
Elapsed time
4h 6m
Started (UTC)
6 Jul 04:14
Customers affected
CS Test, Mercury Radio Arts (Glenn Beck), Multimedios, TCN The Country Network, Zixi ContentXchange - CXE
●Sev 3· Degraded serviceSev 3 — Degraded, no outage✓Resolved
Tata Cloudlyte NJ Maintenance
MaintenanceTata CloudlyteNJ
What happenedThree NJ broadcasters were acknowledged down from 04:15 UTC on 10 July. The tracker was resolved the same morning but its outage table was never closed out, so no end time or duration was ever recorded.
Root causeNot yet established.
How it was resolvedDone
Key points
Duration is unrecorded for all three rows — the impact total for this period is a floor, not a complete figure.
Ask Tata for the actual restoration time so this window can be quantified.
Combined downtime
41m
Elapsed time
14m
Started (UTC)
10 Jul 04:15
Customers affected
CS Test, Mercury Radio Arts (Glenn Beck), Multimedios