Prepared for
Tata Communications
Zixi Operations · all times UTC
Bi-weekly operations review
30 Jun – 14 Jul 2026
20h 59m of broadcaster impact over 23 outage windows across 5 customers, 4h 46m of it wall-clock. 4 unplanned incidents logged, most severe Sev 2. 5 maintenance windows, 1 of them emergency.
← → or space to navigate · f fullscreen · g slide index
Prepared for
Tata Communications
Zixi Operations · all times UTC
Bi-weekly operations review
30 Jun – 14 Jul 2026
20h 59m of broadcaster impact over 23 outage windows across 5 customers, 4h 46m of it wall-clock. 4 unplanned incidents logged, most severe Sev 2. 5 maintenance windows, 1 of them emergency.
Where the fortnight landed
Broadcaster-minutes of impact
20h 59m
4h 46m wall-clock across 23 outage windows
Longest single outage
4h 1m
GLENNBECK-BX-WEST-01 · Mercury Radio Arts (Glenn Beck)
Incidents
4
Still open
0
Customers affected
5
Maintenance windows
5
1 emergency
SLA met
—
none assessed
Wins and concerns
What went well
- The Dish Cheyenne (NY) leg carried traffic throughout the 30 June fibre cut, so the LOR never became a customer outage.
- Tata pre-notified both LA maintenance windows and accepted a Zixi go-ahead gate before proceeding.
Concerns to raise
- The 6 July LA Cloudlyte reboot was scheduled for an 8-hour window but produced ~4 hours of simultaneous outage across five customer broadcasters — the blast radius was wider than the notification implied.
- Two LA broadcasters sat on the same physical host (la-cloudlyte-sr15), so a single host removed redundancy for two separate customers at once. Ask Tata for host-level anti-affinity.
- ZAASOPS-251's outage table was never closed out: three NJ broadcaster rows still read "Ongoing" with no duration, so this period's impact total is a floor rather than a complete figure.
Outage timeline
One lane per broadcaster. Overlapping lanes in the same window are one shared fault, not separate incidents.
- LA
- NJ
- NY
All times UTC. Bar width is the outage window; overlapping lanes in the same window indicate one shared fault rather than separate incidents.
Where the impact landed
Broadcaster-minutes of impact.
By customer
By region
By physical host
Incident mix
By severity
- Sev 22
- Sev 32
By cause
P2: Tata Cloudlyte Flapping Issue in NJ region
Impact
19h 52m
broadcaster-minutes
Wall-clock
4h 6m
Started (UTC)
6 Jul 04:14
Broadcasters hit
8
Customers affected
CS Test · Mercury Radio Arts (Glenn Beck) · Multimedios · TCN The Country Network · Zixi ContentXchange - CXE
Tata's planned full-node reboot of the LA Cloudlyte environment overran, taking five customer broadcasters down for close to four hours from 04:14 UTC on 6 July. The NJ region then flapped repeatedly the following afternoon.
Root cause: Ceph RBD remap remediation required a reboot of every node in the LA Cloudlyte environment; NJ flapping was reported by Tata separately and not attributed.
Resolution: All LA broadcasters recovered by 08:16 UTC. NJ flaps self-cleared within a minute each.
- Five broadcasters across four customers were down simultaneously — the notification implied per-instance impact, not a full-region outage.
- BX-DIST-WEST-04 and MULTIMEDIOS-BX-WEST-01 share physical host la-cloudlyte-sr15; ask Tata to confirm anti-affinity for redundant pairs.
- Request confirmation that the Ceph RBD remap issue is fully cleared and no further reboots are queued.
Tata Cloudlyte NJ Maintenance
Impact
41m
broadcaster-minutes
Wall-clock
14m
Started (UTC)
10 Jul 04:15
Broadcasters hit
3
Customers affected
CS Test · Mercury Radio Arts (Glenn Beck) · Multimedios
Three NJ broadcasters were acknowledged down from 04:15 UTC on 10 July. The tracker was resolved the same morning but its outage table was never closed out, so no end time or duration was ever recorded.
Resolution: Done
- Duration is unrecorded for all three rows — the impact total for this period is a floor, not a complete figure.
- Ask Tata for the actual restoration time so this window can be quantified.
Tata Communications LOR - Dish Gilbert LA local loop down (fiber cut Golden Valley-Searchlight NV) - 30 Jun 2026
Impact
0m
broadcaster-minutes
Wall-clock
0m
Started (UTC)
30 Jun 17:47
Broadcasters hit
—
Resolution: Done
59497 [Tata Communications] Re: Re: P2: CloudLyte LA Down
Impact
0m
broadcaster-minutes
Wall-clock
0m
Started (UTC)
30 Jun 03:48
Broadcasters hit
—
Maintenance activity
5 windows in the fortnight, 1 of them emergency.
Action items
3 open of 3 tracked.
Before the next review
Next steps
- Confirm host-level anti-affinity for redundant broadcaster pairs in LA CloudlyteTata
- Provide restoration time for the 10 July NJ Cloudlyte event (ZAASOPS-251)Tata
- Confirm the Ceph RBD remap remediation is complete and no further LA reboots are queuedTata
Window 2026-06-30 → 2026-07-14 · all times UTC