Live · last 14 daysAll times UTCUpdated 21 Nov 19:00
In brief
4d 14h 46m of combined downtime across 69 outages, affecting 24 customers (1d 15h 34m elapsed). 5 unplanned incidents, the most serious a Sev 1 (full outage). 2 maintenance windows, 1 of them emergency. 2 incidents still open.
▲ Showing the figures as they stand. The written summary was composed before the latest outage data arrived, so it is being held back. Refresh to have it rewritten.
At a glance
The headline figures for the period, and what stood out.
Combined downtime
4d14h46m
across 69 outages · 1d 15h 34m elapsed · 1h 36m on average
Longest single outage
21h8m
Anuvu
ANUVU-BX-GPU-WEST-01
Combined downtime adds up the time every affected broadcaster was down, so two broadcasters down for the same hour count as two hours. Elapsed time counts that hour once.
Unplanned incidents
↑2 vs 29 Jul – 11 Aug 2026
5
Events that affected service
Still open
↑2 vs 29 Jul – 11 Aug 2026
2
Incidents still being worked
Customers affected
↑22 vs 29 Jul – 11 Aug 2026
24
Anuvu, Arabsat, Blaze TV and 21 more
Maintenance windows
2
1 of them emergency
Notices, no impact
0
Tracked, but service was unaffected
SLA met
—
Not assessed this period
Changes are against the previous review, 29 Jul – 11 Aug 2026.
What went well
▲ Written before the latest figures arrived.
Tata sent its incident report for the 15 November Los Angeles outage (TT 78554) within twelve hours of service returning.
The 12 December Dish Cheyenne fiber repair (CHGP1294505) was announced almost four weeks ahead.
Points to discuss
▲ Written before the latest figures arrived.
1A failed disk on one Tata KVM host in Los Angeles took both of Anuvu's broadcasters there down for about 21 hours from 3 November, because the VMs had to be rebuilt on another host. Tata's report (TT 78004) names disk failure monitoring as the preventive step; please confirm it is in place, and tell us how quickly VMs on a failed host can be moved in future.
2The 15 November Los Angeles outage came from urgent backbone maintenance we had no notice of. Zixi raised it at 00:12 UTC and Tata's first message came at 00:42 UTC, after service had returned. We would like backbone work that can reach the Los Angeles site announced to us however short the notice, and the emergency change lead time raised in July agreed.
3Charter has seen packet loss on the Los Angeles cross connect (VLAN 1732) since 4 November, and the only loss measured on the path is transmit discards on Tata's switch port. Please confirm the date for moving the circuit to the replacement switch, and tell us when it is done so Charter can confirm the errors have stopped.
4Three Cloudlyte VMs in New Jersey have been down since 05:10 UTC on 17 November (TT 78615). Please give us a restoration time, and an incident report once they are back.
5Please send an incident report for the New York router fault of 13 to 14 November (TT 78483).
6The CHGP1294505 notice names the Dish Cheyenne to New York fiber, but its service string also carries an ID we hold against the Cheyenne internet circuit. Please confirm which circuits will drop on 12 December.
Service impact
How much disruption customers saw, when it happened and where it landed.
69 outages
Regions
Show
When outages happened
All times UTC
FRANKFURT
LA
NJ
NY
AnuvuBroadcasterANUVU-BX-GPU-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:30Duration34mHost serverLA SR07StatusResolvedAnuvuBroadcasterANUVU-BX-GPU-WEST-01RegionLAStart3 Nov 19:48End4 Nov 16:56Duration21h 8mHost serverLA SR22StatusResolved21h 41m
AnuvuBroadcasterANUVU-BX-DIST-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:31Duration34mHost serverLA SR07StatusResolvedAnuvuBroadcasterANUVU-BX-DIST-WEST-01RegionLAStart3 Nov 19:47End4 Nov 16:38Duration20h 50mHost serverLA SR22StatusResolved21h 24m
CS TestBroadcastertravis-gpu-telev8RegionNJStart17 Nov 05:10End17 Nov 14:01Duration8h 51mHost serverwl-cloudlyte-sr02StatusResolved8h 51m
Mercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-EAST-01RegionNJStart17 Nov 05:10End17 Nov 14:01Duration8h 51mHost serverwl-cloudlyte-sr03StatusResolved8h 51m
MultimediosBroadcasterMULTIMEDIOS-BX-EAST-01RegionNJStart17 Nov 05:12End17 Nov 14:00Duration8h 49mHost serverwl-cloudlyte-sr29StatusResolved8h 49m
Zixi ContentXchange - CXEBroadcasterBX-IN-EAST-01RegionNYStart13 Nov 21:40End14 Nov 03:00Duration5h 19mHost serverNY SR06StatusResolvedZixi ContentXchange - CXEBroadcasterBX-IN-EAST-01RegionNYStart11 Nov 05:35End11 Nov 05:45Duration10mHost serverNY SR06StatusResolved5h 29m
Blaze TVBroadcasterBLAZE-BX-EAST-01RegionNYStart13 Nov 21:41End14 Nov 03:00Duration5h 19mHost serverNY SR04StatusResolvedBlaze TVBroadcasterBLAZE-BX-EAST-01RegionNYStart11 Nov 05:35End11 Nov 05:45Duration10mHost serverNY SR04StatusResolved5h 28m
Zixi ContentXchange - CXEBroadcasterBX-IN-EAST-02RegionNYStart13 Nov 21:40End14 Nov 03:00Duration5h 20mHost serverNY SR06StatusResolved5h 20m
AnuvuBroadcasterANUVU-BX-DIST-EAST-01RegionNYStart13 Nov 06:12End13 Nov 08:16Duration2h 4mHost serverNY SR10StatusResolved2h 4m
Zixi ContentXchange - CXEBroadcasterBX-IN-WEST-04RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverla-cloudlyte-sr22StatusResolvedZixi ContentXchange - CXEBroadcasterBX-IN-WEST-04RegionLAStart17 Nov 18:19End17 Nov 19:40Duration1h 22mHost serverla-cloudlyte-sr22StatusResolved1h 56m
Tigertech MediaBroadcasterTIGERTECH-EGRESS-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:31Duration34mHost serverLA SR08StatusResolvedTigertech MediaBroadcasterTIGERTECH-EGRESS-BX-WEST-01RegionLAStart11 Nov 05:33End11 Nov 05:45Duration12mHost serverLA SR08StatusResolved46m
Mercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:31Duration34mHost serverla-cloudlyte-sr12StatusResolvedMercury Radio Arts (Glenn Beck)BroadcasterGLENNBECK-BX-WEST-01RegionLAStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverla-cloudlyte-sr12StatusResolved45m
Estrella TV / Liberman BroadcastingBroadcasterESTRELLA-BX-ZAAS-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:31Duration34mHost serverLA SR10StatusResolvedEstrella TV / Liberman BroadcastingBroadcasterESTRELLA-BX-ZAAS-WEST-01RegionLAStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverLA SR10StatusResolved45m
National Black Television BroadcasterNBT-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverLA SR08StatusResolvedNational Black Television BroadcasterNBT-BX-WEST-01RegionLAStart11 Nov 05:33End11 Nov 05:45Duration11mHost serverLA SR08StatusResolved44m
MultimediosBroadcasterMULTIMEDIOS-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverla-cloudlyte-sr16StatusResolvedMultimediosBroadcasterMULTIMEDIOS-BX-WEST-01RegionLAStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverla-cloudlyte-sr16StatusResolved44m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-WEST-04RegionLAStart15 Nov 23:57End16 Nov 00:30Duration32mHost serverLA SR10StatusResolvedYouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-WEST-04RegionLAStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverLA SR10StatusResolved43m
The Word NetworkBroadcasterZAAS-BX-WEST-02RegionLAStart15 Nov 23:55End16 Nov 00:31Duration35mHost serverLA SR07StatusResolved35m
United TeleportsBroadcasterMVPD-BX-ZAAS-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:31Duration35mHost serverLA SR07StatusResolved35m
Fox NetworksBroadcasterfox-ott-bx-youtube-sec-04RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR10StatusResolved34m
Impact Network IncBroadcasterIMPACT-BX-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR08StatusResolved34m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-WEST-02RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR10StatusResolved34m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR10StatusResolved34m
Fox NetworksBroadcasterfox-ott-bx-youtube-sec-02RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR10StatusResolved34m
Fox NetworksBroadcasterfox-ott-bx-youtube-sec-01RegionLAStart15 Nov 23:57End16 Nov 00:31Duration34mHost serverLA SR10StatusResolved34m
Zixi ContentXchange - CXEBroadcasterBX-DIST-WEST-04RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverla-cloudlyte-sr021StatusResolved34m
Tigertech MediaBroadcasterTIGERTECH-INGRESS-BX-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:30Duration34mHost serverLA SR08StatusResolved34m
TCN The Country NetworkBroadcasterTCN-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:30Duration34mHost serverla-cloudlyte-sr16StatusResolved34m
RCN TelevisionBroadcasterrcn-bx-west-01-lataRegionLAStart15 Nov 23:56End16 Nov 00:30Duration33mHost serverLA SR06StatusResolved33m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-WEST-03RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverLA SR10StatusResolved33m
Family Broadcasting Corp.BroadcasterFBC-BX-WEST-01RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverLA SR10StatusResolved33m
CinedigmBroadcasterCIDM-BX-WEST-05RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverLA SR08StatusResolved33m
Fox NetworksBroadcasterfox-ott-bx-youtube-sec-03RegionLAStart15 Nov 23:57End16 Nov 00:30Duration33mHost serverLA SR07StatusResolved33m
Zixi ContentXchange - CXEBroadcasterBX-IN-WEST-01RegionLAStart15 Nov 23:55End16 Nov 00:28Duration33mHost serverLA SR06StatusResolved33m
Zixi ContentXchange - CXEBroadcasterBX-IN-WEST-03RegionLAStart15 Nov 23:55End16 Nov 00:28Duration33mHost serverLA SR06StatusResolved33m
YouTubeTV [Production]BroadcasterYOUTUBE-BX-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR08StatusResolved32m
Zixi ContentXchange - CXEBroadcasterBX-DIST-WEST-02RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR06StatusResolved32m
TV5 USABroadcasterTV5-BX-WEST-01RegionLAStart15 Nov 23:58End16 Nov 00:30Duration32mHost serverLA SR08StatusResolved32m
Zixi ContentXchange - CXEBroadcasterBX-DIST-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR07StatusResolved32m
SKY LINK TVBroadcasterLA-BX-01RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR08StatusResolved32m
Cannella MediaBroadcasterCANNELLA-BX-WEST-01RegionLAStart15 Nov 23:58End16 Nov 00:30Duration32mHost serverLA SR06StatusResolved32m
Zixi ContentXchange - CXEBroadcasterBX-IN-WEST-02RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR06StatusResolved32m
Blaze TVBroadcasterBLAZE-BX-WEST-01RegionLAStart15 Nov 23:56End16 Nov 00:28Duration32mHost serverLA SR07StatusResolved32m
Zixi ContentXchange - CXEBroadcasterBX-DIST-WEST-03RegionLAStart15 Nov 23:57End16 Nov 00:28Duration31mHost serverLA SR06StatusResolved31m
The Word NetworkBroadcasterZAAS-BX-EAST-01RegionNYStart11 Nov 05:33End11 Nov 05:45Duration12mHost serverNY SR04StatusResolved12m
Tigertech MediaBroadcasterTIGERTECH-INGRESS-BX-EAST-01RegionNYStart11 Nov 05:33End11 Nov 05:45Duration12mHost serverNY SR06StatusResolved12m
National Black Television BroadcasterNBT-BX-EAST-01RegionNYStart11 Nov 05:33End11 Nov 05:45Duration12mHost serverNY SR06StatusResolvedNational Black Television BroadcasterNBT-BX-EAST-01RegionNYStart6 Nov 06:14End6 Nov 06:14Duration<1mHost serverNY SR06StatusResolved12m
Cannella MediaBroadcasterCANNELLA-BX-EAST-01RegionNYStart11 Nov 05:33End11 Nov 05:45Duration12mHost serverNY SR07StatusResolved12m
Fox NetworksBroadcasterfox-ott-bx-youtube-pri-01RegionNYStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverNY SR02StatusResolved11m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-EAST-01RegionNYStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverNY SR02StatusResolved11m
ArabsatBroadcasterArabsat-BX-Frankfurt-01RegionFRANKFURTStart11 Nov 05:34End11 Nov 05:45Duration11mHost serverFRK SR06StatusResolved11m
Family Broadcasting Corp.BroadcasterFBC-BX-EAST-02RegionNYStart11 Nov 05:35End11 Nov 05:45Duration10mHost serverNY SR02StatusResolved10m
AnuvuBroadcasterANUVU-BX-GPU-EAST-01RegionNYStart6 Nov 06:16End6 Nov 06:17Duration<1mHost serverNY SR10StatusResolved<1m
YouTubeTV [Production]BroadcasterYOUTUBE-BX-EAST-01RegionNYStart6 Nov 06:15End6 Nov 06:16Duration<1mHost serverNY SR06StatusResolved<1m
Fox NetworksBroadcasterfox-ott-bx-youtube-pri-02RegionNYStart11 Nov 05:04End11 Nov 05:04Duration<1mHost serverNY SR02StatusResolved<1m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-EAST-03RegionNYStart6 Nov 06:15End6 Nov 06:15Duration<1mHost serverNY SR02StatusResolved<1m
InsightTvBroadcasterZAAS_INSIGHTTV_PRIMARYRegionNYStart6 Nov 06:17End6 Nov 06:17Duration<1mHost serverNY SR04StatusResolved<1m
YouTubeTV [Production]BroadcasterYOUTUBE-FOX-BX-EAST-02RegionNYStart6 Nov 06:15End6 Nov 06:16Duration<1mHost serverNY SR02StatusResolved<1m
Nov 4Nov 6Nov 8Nov 10Nov 12Nov 14Nov 16Total
Each row is one broadcaster, and each bar shows how long an outage lasted. Bars that line up across rows come from the same event.
Outages by day
Where the impact landed
Combined downtime
By customer
Which customers were affected most.
0m16h 40m1d 9h 20m2d 2h
By region
Which Tata region the outages were in.
0m16h 40m1d 9h 20m2d 2h2d 18h 40m
By host server
A server that keeps appearing points to the infrastructure rather than to chance.
0m16h 40m1d 9h 20m2d 2h
All outages
69 outages
Every measured broadcaster downtime window in the period.
3 Nov 19:47
4 Nov 16:38
20h 50m
ANUVU-BX-DIST-WEST-01
Anuvu
LA
LA SR22
Resolved
3 Nov 19:48
4 Nov 16:56
21h 8m
ANUVU-BX-GPU-WEST-01
Anuvu
LA
LA SR22
Resolved
6 Nov 06:14
6 Nov 06:14
<1m
NBT-BX-EAST-01
National Black Television
NY
NY SR06
Resolved
6 Nov 06:15
6 Nov 06:15
<1m
YOUTUBE-FOX-BX-EAST-03
YouTubeTV [Production]
NY
NY SR02
Resolved
6 Nov 06:15
6 Nov 06:16
<1m
YOUTUBE-BX-EAST-01
YouTubeTV [Production]
NY
NY SR06
Resolved
6 Nov 06:15
6 Nov 06:16
<1m
YOUTUBE-FOX-BX-EAST-02
YouTubeTV [Production]
NY
NY SR02
Resolved
6 Nov 06:16
6 Nov 06:17
<1m
ANUVU-BX-GPU-EAST-01
Anuvu
NY
NY SR10
Resolved
6 Nov 06:17
6 Nov 06:17
<1m
ZAAS_INSIGHTTV_PRIMARY
InsightTv
NY
NY SR04
Resolved
11 Nov 05:04
11 Nov 05:04
<1m
fox-ott-bx-youtube-pri-02
Fox Networks
NY
NY SR02
Resolved
11 Nov 05:33
11 Nov 05:45
12m
ZAAS-BX-EAST-01
The Word Network
NY
NY SR04
Resolved
11 Nov 05:33
11 Nov 05:45
12m
TIGERTECH-INGRESS-BX-EAST-01
Tigertech Media
NY
NY SR06
Resolved
11 Nov 05:33
11 Nov 05:45
12m
TIGERTECH-EGRESS-BX-WEST-01
Tigertech Media
LA
LA SR08
Resolved
11 Nov 05:33
11 Nov 05:45
12m
NBT-BX-EAST-01
National Black Television
NY
NY SR06
Resolved
11 Nov 05:33
11 Nov 05:45
12m
CANNELLA-BX-EAST-01
Cannella Media
NY
NY SR07
Resolved
11 Nov 05:33
11 Nov 05:45
11m
NBT-BX-WEST-01
National Black Television
LA
LA SR08
Resolved
11 Nov 05:34
11 Nov 05:45
11m
fox-ott-bx-youtube-pri-01
Fox Networks
NY
NY SR02
Resolved
11 Nov 05:34
11 Nov 05:45
11m
YOUTUBE-FOX-BX-EAST-01
YouTubeTV [Production]
NY
NY SR02
Resolved
11 Nov 05:34
11 Nov 05:45
11m
GLENNBECK-BX-WEST-01
Mercury Radio Arts (Glenn Beck)
LA
la-cloudlyte-sr12
Resolved
11 Nov 05:34
11 Nov 05:45
11m
ESTRELLA-BX-ZAAS-WEST-01
Estrella TV / Liberman Broadcasting
LA
LA SR10
Resolved
11 Nov 05:34
11 Nov 05:45
11m
MULTIMEDIOS-BX-WEST-01
Multimedios
LA
la-cloudlyte-sr16
Resolved
11 Nov 05:34
11 Nov 05:45
11m
YOUTUBE-FOX-BX-WEST-04
YouTubeTV [Production]
LA
LA SR10
Resolved
11 Nov 05:34
11 Nov 05:45
11m
Arabsat-BX-Frankfurt-01
Arabsat
FRANKFURT
FRK SR06
Resolved
11 Nov 05:35
11 Nov 05:45
10m
FBC-BX-EAST-02
Family Broadcasting Corp.
NY
NY SR02
Resolved
11 Nov 05:35
11 Nov 05:45
10m
BX-IN-EAST-01
Zixi ContentXchange - CXE
NY
NY SR06
Resolved
11 Nov 05:35
11 Nov 05:45
10m
BLAZE-BX-EAST-01
Blaze TV
NY
NY SR04
Resolved
13 Nov 06:12
13 Nov 08:16
2h 4m
ANUVU-BX-DIST-EAST-01
Anuvu
NY
NY SR10
Resolved
13 Nov 21:40
14 Nov 03:00
5h 20m
BX-IN-EAST-02
Zixi ContentXchange - CXE
NY
NY SR06
Resolved
13 Nov 21:40
14 Nov 03:00
5h 19m
BX-IN-EAST-01
Zixi ContentXchange - CXE
NY
NY SR06
Resolved
13 Nov 21:41
14 Nov 03:00
5h 19m
BLAZE-BX-EAST-01
Blaze TV
NY
NY SR04
Resolved
15 Nov 23:55
16 Nov 00:28
33m
BX-IN-WEST-01
Zixi ContentXchange - CXE
LA
LA SR06
Resolved
15 Nov 23:55
16 Nov 00:31
35m
ZAAS-BX-WEST-02
The Word Network
LA
LA SR07
Resolved
15 Nov 23:55
16 Nov 00:28
33m
BX-IN-WEST-03
Zixi ContentXchange - CXE
LA
LA SR06
Resolved
15 Nov 23:56
16 Nov 00:28
32m
YOUTUBE-BX-WEST-01
YouTubeTV [Production]
LA
LA SR08
Resolved
15 Nov 23:56
16 Nov 00:30
34m
BX-IN-WEST-04
Zixi ContentXchange - CXE
LA
la-cloudlyte-sr22
Resolved
15 Nov 23:56
16 Nov 00:30
34m
IMPACT-BX-WEST-01
Impact Network Inc
LA
LA SR08
Resolved
15 Nov 23:56
16 Nov 00:31
35m
MVPD-BX-ZAAS-WEST-01
United Teleports
LA
LA SR07
Resolved
15 Nov 23:56
16 Nov 00:28
32m
BX-DIST-WEST-02
Zixi ContentXchange - CXE
LA
LA SR06
Resolved
15 Nov 23:56
16 Nov 00:28
32m
BX-DIST-WEST-01
Zixi ContentXchange - CXE
LA
LA SR07
Resolved
15 Nov 23:56
16 Nov 00:30
34m
fox-ott-bx-youtube-sec-02
Fox Networks
LA
LA SR10
Resolved
15 Nov 23:56
16 Nov 00:30
34m
fox-ott-bx-youtube-sec-04
Fox Networks
LA
LA SR10
Resolved
15 Nov 23:56
16 Nov 00:30
34m
BX-DIST-WEST-04
Zixi ContentXchange - CXE
LA
la-cloudlyte-sr021
Resolved
15 Nov 23:56
16 Nov 00:28
32m
BX-IN-WEST-02
Zixi ContentXchange - CXE
LA
LA SR06
Resolved
15 Nov 23:56
16 Nov 00:30
34m
YOUTUBE-FOX-BX-WEST-02
YouTubeTV [Production]
LA
LA SR10
Resolved
15 Nov 23:56
16 Nov 00:30
34m
YOUTUBE-FOX-BX-WEST-01
YouTubeTV [Production]
LA
LA SR10
Resolved
15 Nov 23:56
16 Nov 00:28
32m
LA-BX-01
SKY LINK TV
LA
LA SR08
Resolved
15 Nov 23:56
16 Nov 00:30
34m
TIGERTECH-INGRESS-BX-WEST-01
Tigertech Media
LA
LA SR08
Resolved
15 Nov 23:56
16 Nov 00:30
33m
rcn-bx-west-01-lata
RCN Television
LA
LA SR06
Resolved
15 Nov 23:56
16 Nov 00:28
32m
BLAZE-BX-WEST-01
Blaze TV
LA
LA SR07
Resolved
15 Nov 23:57
16 Nov 00:28
31m
BX-DIST-WEST-03
Zixi ContentXchange - CXE
LA
LA SR06
Resolved
15 Nov 23:57
16 Nov 00:30
33m
YOUTUBE-FOX-BX-WEST-03
YouTubeTV [Production]
LA
LA SR10
Resolved
15 Nov 23:57
16 Nov 00:30
34m
TCN-BX-WEST-01
TCN The Country Network
LA
la-cloudlyte-sr16
Resolved
15 Nov 23:57
16 Nov 00:30
33m
FBC-BX-WEST-01
Family Broadcasting Corp.
LA
LA SR10
Resolved
15 Nov 23:57
16 Nov 00:30
33m
fox-ott-bx-youtube-sec-03
Fox Networks
LA
LA SR07
Resolved
15 Nov 23:57
16 Nov 00:31
34m
fox-ott-bx-youtube-sec-01
Fox Networks
LA
LA SR10
Resolved
15 Nov 23:57
16 Nov 00:31
34m
TIGERTECH-EGRESS-BX-WEST-01
Tigertech Media
LA
LA SR08
Resolved
15 Nov 23:57
16 Nov 00:30
33m
NBT-BX-WEST-01
National Black Television
LA
LA SR08
Resolved
15 Nov 23:57
16 Nov 00:30
34m
ANUVU-BX-GPU-WEST-01
Anuvu
LA
LA SR07
Resolved
15 Nov 23:57
16 Nov 00:31
34m
GLENNBECK-BX-WEST-01
Mercury Radio Arts (Glenn Beck)
LA
la-cloudlyte-sr12
Resolved
15 Nov 23:57
16 Nov 00:31
34m
ESTRELLA-BX-ZAAS-WEST-01
Estrella TV / Liberman Broadcasting
LA
LA SR10
Resolved
15 Nov 23:57
16 Nov 00:30
33m
MULTIMEDIOS-BX-WEST-01
Multimedios
LA
la-cloudlyte-sr16
Resolved
15 Nov 23:57
16 Nov 00:30
33m
CIDM-BX-WEST-05
Cinedigm
LA
LA SR08
Resolved
15 Nov 23:57
16 Nov 00:30
32m
YOUTUBE-FOX-BX-WEST-04
YouTubeTV [Production]
LA
LA SR10
Resolved
15 Nov 23:57
16 Nov 00:31
34m
ANUVU-BX-DIST-WEST-01
Anuvu
LA
LA SR07
Resolved
15 Nov 23:58
16 Nov 00:30
32m
CANNELLA-BX-WEST-01
Cannella Media
LA
LA SR06
Resolved
15 Nov 23:58
16 Nov 00:30
32m
TV5-BX-WEST-01
TV5 USA
LA
LA SR08
Resolved
17 Nov 05:10
17 Nov 14:01
8h 51m
GLENNBECK-BX-EAST-01
Mercury Radio Arts (Glenn Beck)
NJ
wl-cloudlyte-sr03
Resolved
17 Nov 05:10
17 Nov 14:01
8h 51m
travis-gpu-telev8
CS Test
NJ
wl-cloudlyte-sr02
Resolved
17 Nov 05:12
17 Nov 14:00
8h 49m
MULTIMEDIOS-BX-EAST-01
Multimedios
NJ
wl-cloudlyte-sr29
Resolved
17 Nov 18:19
17 Nov 19:40
1h 22m
BX-IN-WEST-04
Zixi ContentXchange - CXE
LA
la-cloudlyte-sr22
Resolved
Incidents
Unplanned events that affected service, and what we know about each one.
5 incidents
Breakdown
How serious
Incidents by severity level.
What caused them
Incidents by cause.
012
Incident details
Largest impact first
◆Sev 1· Full outageSev 1 — Full outage✓Resolved
P1: Anuvu West Broadcasters
StorageZixi CloudLA
What happenedBoth of Anuvu's Los Angeles broadcasters, on the same Tata KVM host, went down at 19:47 UTC on 3 November when the host failed. They came back at 16:38 and 16:56 UTC on 4 November, once the VMs had been rebuilt on another host and redeployed.
Root causeA disk failure on Tata KVM host LA SR22 stopped the host from booting, taking down both VMs on it (Tata incident report TT 78004).
How it was resolvedTata rebuilt both VMs on host LA SR07 by 13:30 UTC on 4 November; the broadcasters were redeployed on them and back by 16:56 UTC.
Key points
About 21 hours down on each broadcaster, from host failure to redeployment.
Both of the customer's Los Angeles VMs were on the one host, so its failure took out both at once.
Tata's report recommends 1+1 services and disk failure monitoring as the preventive steps.
What happenedEvery Zixi VM in Los Angeles, on both Tata KVM and Cloudlyte, became unreachable from 23:55 UTC on 15 November until 00:31 UTC: 36 broadcasters across 20 customers.
Root causeUrgent maintenance by Tata's backbone network team interrupted connectivity to the Los Angeles site (Tata incident report TT 78554).
How it was resolvedService returned when the maintenance finished. Zixi confirmed every VM back by 00:31 UTC.
Key points
We had no notice of the maintenance. Zixi raised it at 00:12 UTC; Tata's first message came at 00:42 UTC, after service had returned.
Tata's report says Tata did not detect the interruption itself.
Both Tata platforms in Los Angeles went down together, so a backup VM on the other platform in the same city was no protection.
Combined downtime
19h 57m
Elapsed time
36m
Started (UTC)
15 Nov 23:55
Customers affected
Anuvu, Blaze TV, Cannella Media, Cinedigm, Estrella TV / Liberman Broadcasting, Family Broadcasting Corp., Fox Networks, Impact Network Inc, Mercury Radio Arts (Glenn Beck), Multimedios, National Black Television, National Black Television , RCN Television, SKY LINK TV, TCN The Country Network, TV5 USA, The Word Network, Tigertech Media, United Teleports, YouTubeTV [Production], Zixi ContentXchange - CXE
What happenedTwo Content Exchange ingest broadcasters and Blaze TV's broadcaster in New York were unreachable from 21:40 UTC on 13 November until 03:00 UTC on 14 November. Zixi classed it as a loss of redundancy.
Root causeA fault on Tata's New York router. Tata's backbone network team reloaded the router to restore service.
How it was resolvedRouter reloaded; all three broadcasters back by 03:00 UTC on 14 November.
Key points
Five hours and twenty minutes, with no restoration estimate available while it lasted.
●Sev 3· Degraded serviceSev 3 — Degraded, no outage◆Open
P2: CC errors on Charter cross connect, Los Angeles, VLAN 1732, Service ID 001LOSA002037559393
Cross-connect / fiberOtherLA
What happenedSince 4 November Charter has reported continuity errors on feeds it takes from Zixi's Los Angeles VMs over the Tata cross connect on VLAN 1732, among them FETV, Canal Sur and The Word Network. The feeds leave Zixi's VMs clean, and Charter's New York copy of FETV is clean.
Root causeLoss is measured at one point only: 73,652 transmit discards on Tata's switch port toward Charter. Charter's port shows no input discards and no pause frames.
How it was resolvedTata is moving the circuit to a switch from a different vendor; the parts were due on 18 November.
Key points
Open since 4 November.
Tata has ruled out a larger output buffer on the current switch.
Maintenance Tata told us about during the period, and whether it affected service.
2 windows · 1 emergency
Provider maintenance windows notified in the period.
Reference
▲Emergency
Urgent backbone maintenance, Los AngelesTT#78554
Tata KVM and CloudlyteLANot announced. Every Zixi VM in Los Angeles was unreachable.
No advance notice. Tata's incident report (TT 78554) puts the interruption at 23:58 to 00:31 GMT, 33 minutes. Zixi reported it to Tata at 00:12 GMT; Tata's first message came at 00:42 GMT.