Service review · Tata Communications · 30 Jun – 14 Jul 2026

1 / 13
Zixi

Bi-weekly service review

30 Jun – 14 Jul 2026

Prepared for Tata Communications by Zixi

20h 59m of combined downtime across 23 outages, affecting 5 customers (4h 46m elapsed). 4 unplanned incidents, the most serious a Sev 2 (partial outage). 5 maintenance windows, 1 of them emergency.

This period at a glance

20h 59m
combined downtime
4
unplanned incidents
5
maintenance windows

All times UTC

← → or space to move between slides · f full screen · g all slides

Zixi

Bi-weekly service review

30 Jun – 14 Jul 2026

Prepared for Tata Communications by Zixi

20h 59m of combined downtime across 23 outages, affecting 5 customers (4h 46m elapsed). 4 unplanned incidents, the most serious a Sev 2 (partial outage). 5 maintenance windows, 1 of them emergency.

This period at a glance

20h 59m
combined downtime
4
unplanned incidents
5
maintenance windows

All times UTC

Zixi

Overview

The period at a glance

Combined downtime

20h 59m

across 23 outages · 4h 46m elapsed

Longest single outage

4h 1m

Mercury Radio Arts (Glenn Beck)

GLENNBECK-BX-WEST-01

4

Unplanned incidents

0

Still open

5

Customers affected

5

Maintenance windows

1 emergency

Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20262 / 13
Zixi

Overview

Highlights and points to discuss

What went well

  • The Dish Cheyenne (NY) leg carried traffic throughout the 30 June fibre cut, so the LOR never became a customer outage.
  • Tata pre-notified both LA maintenance windows and accepted a Zixi go-ahead gate before proceeding.

Points to discuss

  • The 6 July LA Cloudlyte reboot was scheduled for an 8-hour window but produced ~4 hours of simultaneous outage across five customer broadcasters — the blast radius was wider than the notification implied.
  • Two LA broadcasters sat on the same physical host (la-cloudlyte-sr15), so a single host removed redundancy for two separate customers at once. Ask Tata for host-level anti-affinity.
  • ZAASOPS-251's outage table was never closed out: three NJ broadcaster rows still read "Ongoing" with no duration, so this period's impact total is a floor rather than a complete figure.
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20263 / 13
Zixi

Service impact

When outages happened

All times UTC.

  • LA
  • NJ
Mercury Radio Arts (Glenn Beck)GLENNBECK-BX-WEST-014h 5m
MultimediosMULTIMEDIOS-BX-WEST-013h 57m
TCN The Country NetworkTCN-BX-WEST-013h 57m
Zixi ContentXchange - CXEBX-IN-WEST-043h 56m
Zixi ContentXchange - CXEBX-DIST-WEST-043h 56m
CS Testtravis-gpu-telev826m
Mercury Radio Arts (Glenn Beck)GLENNBECK-BX-EAST-0115m
MultimediosMULTIMEDIOS-BX-EAST-0114m
Jun 30Jul 2Jul 4Jul 6Jul 8Jul 10Jul 12Jul 14Total

Each row is one broadcaster, and each bar shows how long an outage lasted. Bars that line up across rows come from the same event. Showing the 8 broadcasters with the most downtime, of 9 affected; the full timeline is on the dashboard.

Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20264 / 13
Zixi

Service impact

Where the impact landed

Combined downtime by customer, region and host server.

By customer

Zixi ContentXchange - CXE8h 5m
Mercury Radio Arts (Glenn Beck)4h 19m
Multimedios4h 11m
TCN The Country Network3h 57m
CS Test26m

By region

LA19h 51m
NJ56m
NY12m

By host server

la-cloudlyte-sr157h 54m
la-cloudlyte-sr234h 4m
la-cloudlyte-sr193h 57m
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20265 / 13
Zixi

Incidents

Incidents at a glance

How serious

22
  • Sev 2 · Partial outage2
  • Sev 3 · Degraded service2

What caused them

Cross-connect / fiber2
Maintenance1
Unstable link1
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20266 / 13
Zixi

Incident 1 of 4

P2: Tata Cloudlyte Flapping Issue in NJ region

Sev 2· Partial outageSev 2 — Partial outageResolvedUnstable linkNJLA

19h 52m

Combined downtime

4h 6m

Elapsed time

6 Jul 04:14

Started (UTC)

8

Broadcasters affected

Customers affected: CS Test · Mercury Radio Arts (Glenn Beck) · Multimedios · TCN The Country Network · Zixi ContentXchange - CXE

Tata's planned full-node reboot of the LA Cloudlyte environment overran, taking five customer broadcasters down for close to four hours from 04:14 UTC on 6 July. The NJ region then flapped repeatedly the following afternoon.

Root cause: Ceph RBD remap remediation required a reboot of every node in the LA Cloudlyte environment; NJ flapping was reported by Tata separately and not attributed.

Resolution: All LA broadcasters recovered by 08:16 UTC. NJ flaps self-cleared within a minute each.

  • Five broadcasters across four customers were down simultaneously — the notification implied per-instance impact, not a full-region outage.
  • BX-DIST-WEST-04 and MULTIMEDIOS-BX-WEST-01 share physical host la-cloudlyte-sr15; ask Tata to confirm anti-affinity for redundant pairs.
  • Request confirmation that the Ceph RBD remap issue is fully cleared and no further reboots are queued.
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20267 / 13
Zixi

Incident 2 of 4

Tata Cloudlyte NJ Maintenance

Sev 3· Degraded serviceSev 3 — Degraded, no outageResolvedMaintenanceNJ

41m

Combined downtime

14m

Elapsed time

10 Jul 04:15

Started (UTC)

3

Broadcasters affected

Customers affected: CS Test · Mercury Radio Arts (Glenn Beck) · Multimedios

Three NJ broadcasters were acknowledged down from 04:15 UTC on 10 July. The tracker was resolved the same morning but its outage table was never closed out, so no end time or duration was ever recorded.

Resolution: Done

  • Duration is unrecorded for all three rows — the impact total for this period is a floor, not a complete figure.
  • Ask Tata for the actual restoration time so this window can be quantified.
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20268 / 13
Zixi

Incident 3 of 4

Tata Communications LOR - Dish Gilbert LA local loop down (fiber cut Golden Valley-Searchlight NV) - 30 Jun 2026

Sev 2· Partial outageSev 2 — Partial outageResolvedCross-connect / fiberLALoss of redundancyTata reference CSP06300626157391

0m

Combined downtime

0m

Elapsed time

30 Jun 17:47

Started (UTC)

—

Broadcasters affected

Resolution: Done

Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 20269 / 13
Zixi

Incident 4 of 4

59497 [Tata Communications] Re: Re: P2: CloudLyte LA Down

Sev 3· Degraded serviceSev 3 — Degraded, no outageClosedCross-connect / fiberLoss of redundancy

0m

Combined downtime

0m

Elapsed time

30 Jun 03:48

Started (UTC)

—

Broadcasters affected

Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 202610 / 13
Zixi

Maintenance

Maintenance activity

5 windows in the period, 1 of them emergency.

PlannedTata Communications Planned Maintenance - Fortinet Firewall, Los Angeles - 2026-07-06Tata Cloudlyte · LA · Planned Maintenance6 Jul 15:30
PlannedTata Cloudlyte Planned Maintenance (full node reboot) - Los Angeles - 2026-07-06Tata Cloudlyte · LA · Planned MaintenanceAffected service6 Jul 03:00
PlannedPlanned Maintenance Activity || CHGP1231403No impactNotified 30 Jun 02:08
Planned60585 [Tata Communications] Re: Planned Maintenance Activity in Los-Angeles || Zixi || 06072026Affected serviceNotified 6 Jul 03:34
EmergencyProactive Ticket Emergency Maintenance: NY Switch Hardware Replacement || Zixi || 08072026 Tata CommunicationsNo impactNotified 6 Jul 10:42
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 202611 / 13
Zixi

Action items

Action items

3 open of 3 tracked.

OpenConfirm host-level anti-affinity for redundant broadcaster pairs in LA CloudlyteTata
OpenProvide restoration time for the 10 July NJ Cloudlyte event (ZAASOPS-251)Tata
OpenConfirm the Ceph RBD remap remediation is complete and no further LA reboots are queuedTata
Zixi · Service review for Tata Communications · 30 Jun – 14 Jul 202612 / 13
Zixi

Before the next review

Next steps

  1. Confirm host-level anti-affinity for redundant broadcaster pairs in LA CloudlyteTata
  2. Provide restoration time for the 10 July NJ Cloudlyte event (ZAASOPS-251)Tata
  3. Confirm the Ceph RBD remap remediation is complete and no further LA reboots are queuedTata

Thank you. Review period 30 Jun – 14 Jul 2026 · all times UTC