Comcast Just Doesn't Get It

Perhaps some of my old colleagues will remember the infamous "slow network" problem we experienced that created such bottlenecks in our applications that it just drove us insane. We'd lose replication packages, have timeouts, file transfers seemed like they were on a 56k connection.

I'm reminded of these good times thanks to Comcast, who, for the past 2 months, just can't seem to see the forest for the trees. I have a whopping 8mbps download (with a tolerable 768k upload -- a bit low with today's Optimum Online / Fios standards), yet, it's almost completely useless during peak times. Why?

To answer that question, let's examine a small portion of trace information I've gathered over the past few months. Excuse the rather large images below, they don't resample in a very readable way.

For those who've never used PingPlotter, it's a wonderful utility. I bought a copy back in the late 90's and it's one of the few I still use. We're looking at a visual trace route to an arbitrary site -- in this case, -- at around 12:30pm on 7/27/05. Everything here looks great -- no latency to my router, and very good response over the Comcast network (the 68.x block). As we enter AT&T, things go up a tad (as expected), but, overall, the board shows green (literally and figuratively). The graph on the bottom is a 48-hour view, and it's fairly obvious to see the pattern of spikes. (You can't tell from this image, but this pattern repeats for weeks.)

In the evenings and during peak, I can expect some additional latency. But let's look more closely at what's happening tonight (and every other night):

Now we're looking a bit later in the day -- at 10:30pm. Here, we can very clearly see the problem. My router is still showing no latency, and the first few hops in the Comcast network are fine. But wait! Hop #6 isn't looking too good. The latency spikes to ~125ms, which is ridiculous given this is still "internal" traffic. This router (or routers) is actually located in Bellevue, WA, and funnels a great deal of Puget Sound data. The problem, as you likely already know if you're bothering to read this, is that your connection is only as good as your weakest (slowest) link. After this hop, the picture doesn't get any better.

There are two problems: one is technical, the other is not. Technically speaking, there's obviously problem router(s) here. I have traces going to other sites, as well, and see similar problems -- there's two or three problem routers that spike the latency. The fact that this occurs between about 9pm and midnight is no coincidence -- those are peak usage times. Has Comcast oversold its bandwidth?

The second problem is the manner in which Comcast tries to resolve the problem. They'll refuse to offer support if you're using a router, they claim any latency under ~200ms is fine and not a problem. They send technicians out to check signal strength when there is no problem locally (meaning, specifically at my house or neighborhood). NO ONE cares to see the data I've collected for two months. They simply say, "We don't need it." They'll then resolve the ticket, failing to notify the customer.

All of these are problems because:

1) All of my tests show the same results with or without the router in place. Looking at the data, we can conclude that is not where the problem is. This rule likely exists because non-technical people perhaps used Comcast support to figure out why their IM client or application didn't work behind a router, or other such complaints. This just doesn't apply here.

2) Saying any latency under ~200ms is acceptable is a ridiculous claim. I very well imagine that during peak, it may take 200ms to connect to a server in China, or perhaps, with enough congestion, New York (from the west coast). But a spike to 125ms before getting off the ground (ie within their own network) is a clear problem. Any time you see that kind of latency spike with no great geographical distance and within the same network, you've got a problem.

3) I don't know who gives the Comcast reps authority to close tickets, but it's a generally accepted industry practice that a service organization (any type of support group, for example) will contact the customer (the person who opened the ticket) to close the ticket. Sometimes tickets are closed because they are unresolved, but the customer is almost always involved. In my case, I'll return the call only to hear, "You signal levels look good, we closed your ticket last week." Gee, thanks for the help.

So how specifically does this affect applications? Overall throughput is obviously reduced, though I'll admit even if I only receive half of my 8mbps, that's not a problem. Any time-sensitive application (VOIP, gaming) requires a fast connection. I use VOIP and the call quality degrades in this situation. Gaming, in particular, is a problem. Check out how many Counter Strike servers are available around 5pm with pings under 150ms:

That's quite a few -- 560. Many are under 50ms, most are under 75ms. Pretty nice -- all certainly playable. Now for the 10pm refresh:

Ack! Down to 39 servers -- that's roughly a 94% reduction, not to mention the top 5 servers are all unplayable (empty, full, or locked) and anything over 100ms quickly becomes unplayable as we escalate towards and beyond 150ms.

One of the Comcast techs they sent out (who declined to look at any of the data I collected) did manage to ask neighbors if they've noticed a problem, and all said no. (This was also at 7:30pm, before the peak load starts.) I'm not doubting this, however, I just believe most people may not perceive a problem. Similarly, how grainy must a television picture become before you complain? Everyone has a different threshold. Many users in Comcast's forum on see the same problem, creating a thread with over 628 posts:

Washington State Speed Problems

So, it's pretty clear the problem is pervasive, reproducible, and isolated. Why is this so difficult to fix?
Comments are closed

My Apps

Dark Skies Astrophotography Journal Vol 1 Explore The Moon
Mars Explorer Moons of Jupiter Messier Object Explorer
Brew Finder Earthquake Explorer Venus Explorer  

My Worldmap

Month List