Zixi

30 Jun – 14 Jul 2026

Tata Communications

Present

Summary

What the fortnight came to: the figures the meeting opens on, what went well, and what to put to Tata.

20h 59m of broadcaster impact over 23 outage windows across 5 customers, 4h 46m of it wall-clock. 4 unplanned incidents logged, most severe Sev 2. 5 maintenance windows, 1 of them emergency.

Broadcaster-minutes of impact

20h 59m

across 23 outage windows · 4h 46m wall-clock · 55m average

Longest single outage

4h 1m

GLENNBECK-BX-WEST-01 · Mercury Radio Arts (Glenn Beck)

Incidents

4

Unplanned, service-affecting

Still open

0

Customers affected

5

CS Test, Mercury Radio Arts (Glenn Beck), Multimedios

Maintenance windows

5

1 emergency

Notifications, no impact

0

Logged, no Zixi impact

SLA met

—

No SLA verdict recorded this period

What went well

Open the meeting with these.

  • The Dish Cheyenne (NY) leg carried traffic throughout the 30 June fibre cut, so the LOR never became a customer outage.
  • Tata pre-notified both LA maintenance windows and accepted a Zixi go-ahead gate before proceeding.

Concerns to raise

Each one should name the thing you want Tata to do.

  • The 6 July LA Cloudlyte reboot was scheduled for an 8-hour window but produced ~4 hours of simultaneous outage across five customer broadcasters — the blast radius was wider than the notification implied.
  • Two LA broadcasters sat on the same physical host (la-cloudlyte-sr15), so a single host removed redundancy for two separate customers at once. Ask Tata for host-level anti-affinity.
  • ZAASOPS-251's outage table was never closed out: three NJ broadcaster rows still read "Ongoing" with no duration, so this period's impact total is a floor rather than a complete figure.
23 outage windows · avg 55m

Impact

Where the downtime landed. Broadcaster-minutes of impact sum every affected broadcaster; wall-clock minutes count how long there was any impact at all. They are different numbers and are never added together.

Region
Measure

Timeline

one lane per broadcaster · UTC · click a lane to filter
  • LA
  • NJ
  • NY
4h 5m
3h 57m
3h 57m
3h 56m
3h 56m
26m
15m
14m
12m
Jun 30Jul 2Jul 4Jul 6Jul 8Jul 10Jul 12Jul 14total

All times UTC. Bar width is the outage window; overlapping lanes in the same window indicate one shared fault rather than separate incidents.

Per day

Outages per day

Counted per broadcaster, stacked by region. Hover a day for its impact minutes; use the region chips above to isolate one.

Attribution

impact · click a bar to filter

By customer

Sum of every broadcaster outage attributed to each customer.

By region

Which Tata region carried the fault.

By physical host

A host appearing repeatedly points at infrastructure rather than chance.

Every window

23 rows · sortable
Every measured broadcaster downtime window in the period.
2 Jul 15:192 Jul 15:3212mBX-TX-TATA-T4-EAST-01Zixi ContentXchange - CXENYNY SR04Resolved
3 Jul 15:313 Jul 15:332mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr03Resolved
5 Jul 08:045 Jul 08:05<1mMULTIMEDIOS-BX-EAST-01MultimediosNJwl-cloudlyte-sr28Resolved
5 Jul 08:055 Jul 08:05<1mGLENNBECK-BX-EAST-01Mercury Radio Arts (Glenn Beck)NJwl-cloudlyte-sr03Resolved
5 Jul 08:055 Jul 08:05<1mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr03Resolved
5 Jul 08:345 Jul 08:35<1mGLENNBECK-BX-EAST-01Mercury Radio Arts (Glenn Beck)NJwl-cloudlyte-sr03Resolved
5 Jul 08:355 Jul 08:35<1mMULTIMEDIOS-BX-EAST-01MultimediosNJwl-cloudlyte-sr28Resolved
6 Jul 04:146 Jul 08:113h 56mBX-DIST-WEST-04Zixi ContentXchange - CXELAla-cloudlyte-sr15Resolved
6 Jul 04:156 Jul 08:164h 1mGLENNBECK-BX-WEST-01Mercury Radio Arts (Glenn Beck)LAla-cloudlyte-sr23Resolved
6 Jul 04:156 Jul 08:113h 56mBX-IN-WEST-04Zixi ContentXchange - CXELAla-cloudlyte-sr16Resolved
6 Jul 04:156 Jul 08:123h 57mMULTIMEDIOS-BX-WEST-01MultimediosLAla-cloudlyte-sr15Resolved
6 Jul 04:156 Jul 08:123h 57mTCN-BX-WEST-01TCN The Country NetworkLAla-cloudlyte-sr19Resolved
6 Jul 08:596 Jul 09:024mGLENNBECK-BX-WEST-01Mercury Radio Arts (Glenn Beck)LAla-cloudlyte-sr23Resolved
7 Jul 18:107 Jul 18:100mGLENNBECK-BX-EAST-01Mercury Radio Arts (Glenn Beck)NJwl-cloudlyte-sr03Resolved
7 Jul 18:107 Jul 18:100mMULTIMEDIOS-BX-EAST-01MultimediosNJwl-cloudlyte-sr28Resolved
7 Jul 18:107 Jul 18:100mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr03Resolved
7 Jul 18:357 Jul 18:361mGLENNBECK-BX-EAST-01Mercury Radio Arts (Glenn Beck)NJwl-cloudlyte-sr03Resolved
7 Jul 18:367 Jul 18:360mMULTIMEDIOS-BX-EAST-01MultimediosNJwl-cloudlyte-sr28Resolved
7 Jul 18:367 Jul 18:360mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr03Resolved
10 Jul 04:1510 Jul 04:3014mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr02Resolved
10 Jul 04:1610 Jul 04:3013mGLENNBECK-BX-EAST-01Mercury Radio Arts (Glenn Beck)NJwl-cloudlyte-sr03Resolved
10 Jul 04:1610 Jul 04:2913mMULTIMEDIOS-BX-EAST-01MultimediosNJwl-cloudlyte-sr29Resolved
13 Jul 14:4313 Jul 14:5310mtravis-gpu-telev8CS TestNJwl-cloudlyte-sr02Resolved
4 unplanned · most costly first

Incidents

Every unplanned, service-affecting event in the window. The root cause and talking points on each are the part no query can produce, and are edited in place here.

The mix

click a segment or a bar to filter the register

By severity

Zixi severity ladder, as classified on the originating ticket.

By cause

Normalized issue category.

The register

every value readable without hover
  1. Sev 2Sev 2 — Partial outage

    P2: Tata Cloudlyte Flapping Issue in NJ region

    Link flappingTata CloudlyteNJLA
    SummaryTata's planned full-node reboot of the LA Cloudlyte environment overran, taking five customer broadcasters down for close to four hours from 04:14 UTC on 6 July. The NJ region then flapped repeatedly the following afternoon.
    Root causeCeph RBD remap remediation required a reboot of every node in the LA Cloudlyte environment; NJ flapping was reported by Tata separately and not attributed.
    ResolutionAll LA broadcasters recovered by 08:16 UTC. NJ flaps self-cleared within a minute each.
    Talking points
    • Five broadcasters across four customers were down simultaneously — the notification implied per-instance impact, not a full-region outage.
    • BX-DIST-WEST-04 and MULTIMEDIOS-BX-WEST-01 share physical host la-cloudlyte-sr15; ask Tata to confirm anti-affinity for redundant pairs.
    • Request confirmation that the Ceph RBD remap issue is fully cleared and no further reboots are queued.
    Impact
    19h 52m
    Wall-clock
    4h 6m
    Started (UTC)
    6 Jul 04:14
    Status
    Resolved
    Customers
    CS Test, Mercury Radio Arts (Glenn Beck), Multimedios, TCN The Country Network, Zixi ContentXchange - CXE
    References
    ZAASOPS-248
  2. Sev 3Sev 3 — Degraded, no outage

    Tata Cloudlyte NJ Maintenance

    Maintenance notificationTata CloudlyteNJ
    SummaryThree NJ broadcasters were acknowledged down from 04:15 UTC on 10 July. The tracker was resolved the same morning but its outage table was never closed out, so no end time or duration was ever recorded.
    Root causeNot yet established.
    ResolutionDone
    Talking points
    • Duration is unrecorded for all three rows — the impact total for this period is a floor, not a complete figure.
    • Ask Tata for the actual restoration time so this window can be quantified.
    Impact
    41m
    Wall-clock
    14m
    Started (UTC)
    10 Jul 04:15
    Status
    Resolved
    Customers
    CS Test, Mercury Radio Arts (Glenn Beck), Multimedios
    References
    ZAASOPS-251
  3. Sev 2Sev 2 — Partial outage

    Tata Communications LOR - Dish Gilbert LA local loop down (fiber cut Golden Valley-Searchlight NV) - 30 Jun 2026

    Cross-connect / fiberZixi CloudLoss of redundancyLA
    SummaryWhat happened, in a sentence.
    Root causeNot yet established.
    ResolutionDone
    Talking points

    None written.

    Impact
    none measured
    Wall-clock
    —
    Started (UTC)
    30 Jun 17:47
    Status
    Resolved
    Customers
    —
    References
    ZAASOPS-247#27518CSP06300626157391
  4. Sev 3Sev 3 — Degraded, no outage

    59497 [Tata Communications] Re: Re: P2: CloudLyte LA Down

    Cross-connect / fiberZixi CloudLoss of redundancy
    SummaryWhat happened, in a sentence.
    Root causeNot yet established.
    Talking points

    None written.

    Impact
    none measured
    Wall-clock
    —
    Started (UTC)
    30 Jun 03:48
    Status
    Closed
    Customers
    —
    References
    #27511
5 notified · 1 emergency · sortable

Maintenance

Provider maintenance notified in this window. The window column is the work itself, as the bulletin stated it; where a bulletin gave no window, the time we were told is shown instead and labelled as such.

Provider maintenance windows notified in the period.
Refs
Planned

Planned Maintenance Activity || CHGP1231403CHGP1231403

window not stated
notified 30 Jun 02:08		No#27509CHGP1231403
Planned

Tata Cloudlyte Planned Maintenance (full node reboot) - Los Angeles - 2026-07-069Y2JDR-YW7G3

Tata CloudlyteLAPlanned Maintenance
6 Jul 03:00 → 6 Jul 11:00YesZAASOPS-246#275179Y2JDR-YW7G3
Planned

60585 [Tata Communications] Re: Planned Maintenance Activity in Los-Angeles || Zixi || 06072026

window not stated
notified 6 Jul 03:34		Yes#27531
Emergency

Proactive Ticket Emergency Maintenance: NY Switch Hardware Replacement || Zixi || 08072026 Tata Communications

window not stated
notified 6 Jul 10:42		No#27535
Planned

Tata Communications Planned Maintenance - Fortinet Firewall, Los Angeles - 2026-07-06TT#58576

Tata CloudlyteLAPlanned Maintenance
6 Jul 15:30 → 6 Jul 17:30Not assessedZAASOPS-245#27512TT#58576
3 open · 0 closed

Follow-up

What is outstanding and who owns it. Open items carry forward with the date they were first raised, so an item that has been open for three meetings says so.

Refs
Open

Confirm host-level anti-affinity for redundant broadcaster pairs in LA Cloudlyte

BX-DIST-WEST-04 and MULTIMEDIOS-BX-WEST-01 both sat on la-cloudlyte-sr15 during the 6 July outage.

Tata2026-07-14——
Open

Provide restoration time for the 10 July NJ Cloudlyte event (ZAASOPS-251)

Outage table still reads Ongoing with no duration.

Tata2026-07-14—ZAASOPS-251
Open

Confirm the Ceph RBD remap remediation is complete and no further LA reboots are queued

Tata2026-07-14—ZAASOPS-246
each measure on its own scale · click a column to open that review

Trend

How this fortnight sits against the ones before it. Completed reviews only: the live review's window overlaps the last meeting by nearly a fortnight, so plotting it would count the same incidents twice.

Incidents per review

4

30 Jun – 14 Jul 2026 · first recorded period, so there is nothing to compare against yet.

Broadcaster-minutes of impact

20h 59m

30 Jun – 14 Jul 2026 · first recorded period, so there is nothing to compare against yet.

Customers affected

5

30 Jun – 14 Jul 2026 · first recorded period, so there is nothing to compare against yet.