22nd
December
2009
Yesterday afternoon PDXTC had a report that a website we were hosting was down, but the server administrators said that it was up.
An hour later, we had a second report of another website down, and the admins claimed it was up as well, and that’s where I got involved.
I was working from home, and checking both websites came up blank – page not found – through my Comcast internet connection.
I looked up the users FTP information, and I was unable to login top either account using FTP either, simply getting an “unable to connect” message.
One common connection between the two domains was that they happened to be on the same server, but other than that there was no reason why I shouldn’t be able to see these domains.
I went to a free proxy service and from there I was able to see both domains and verify that they were indeed up and running, so clearly this was a Comcast issue.
Was Comcast actually blocking my access to the web server? Had someone on this shared hosting server done something nefarious, and now Comcast was preventing me from accessing it through their servers?
I phoned Comcast, and of course had to sift through their ridiculous voicemail system, finally arriving at the tier 1 tech support where I had to fight my way past his insistence upon rebooting my router and checking my computer settings.
Finally after getting the guy to understand that the problem belonged to Comcast, he transferred me to the “abuse” section, where he claimed they must be “blocking those websites”.
Blocking those websites?!? – I’d never heard of that before!
I waited my turn in the queue with Abuse department, and when the guy answered, he listened patiently, verified what I said was true, and then had me run a trace route and email him the results.
As you can see, I didn’t get very far -
By that point it was after seven o’clock at night, and he told me he would have to escalate it to his next support level but that I shouldn’t expect it to be “fixed”until some time the next day.
At this point I asked if Comcast could be intentionally blocking my server IP address, and he assured me that no,  Comcast does not block access to web servers in the same way that they block mail servers and ports.
Under certain circumstances he says, where a website is known to be distributing Malware or viruses, they may issue a warning before the users arrival that “the site may be harmful…” , but they do not, as far as he knows, intentionally block access to a Web server. As far as he knows… Huh.
Well, this morning, the sites are back up and I may never know what went wrong, so I guess I’ll just move on.
*** Update ***
I got a phone call back from comcast to tell me that all was resolved, and as it turns out, Comcast HAD intentionally blocked all activity from that web server IP address!
Interestingly, sites hosted on that same web server that had been assigned a static IP address continud to work fine, but the shared hosting domains were all blocked, with no warning message to the user, and FTP, Trace Route, and even pings to the IP were blocked entirely.
The Comcast rep read me the notes he had gotten from engineering, and they sais that the IP was blocked because “Malicious data flows were detected over TCP port 80″.
That’s it – no indication of what domain caused it, nor, was there any reason given WHY they would UNblock it at my request, since presumably the malicious activity is still going on from whatever domain was causing the problem. Bizarre.
This was news to me, and news to the guy in security, who said he’s never heard of them blocking IP addresses like this before.
So, is the moral of the story that a shared hosting account may be risky to your visibility? It looks that way, doesn’t it?
JL says:
The virtual host in question was blocked after an Intrusion Detection System (IDS) noticed repeated attempts from that IP to gain unauthorized access to a Comcast system.
The block was removed as a courtesy to you (our customer) but we continue to monitor that IP address. I recommend you work with your virtual hoster to have them investigate security on this server.
JL
Comcast
Internet Systems Engineering
Scott says:
Thanks for commenting here, JL…
There are well over 50 domains hosted on this particular server, which is located in Gresham, Oregon.
Nearly all of the hosting accounts on it are domains owned by local Portland businesses, and every one of their websites was inaccessible to all Comcast users, which seems a bit harsh.
My admins, (or as you put it, my virtual hosting company) are unable to find anything malicious on that server without something more to go on.
1. “Repeated attempts from that IP to gain unauthorized access to a Comcast system.” – That makes it sound like someone is actually trying to hack into Comcast! Are you just referring to some sort of script attack on a Comcast subscriber?
2. Wouldn’t you have a record of what domain or perhaps even the exact pages may be the problem?
3. Do you not have the technology to block just a domain when it’s on a shared IP, in the same way that Google warns visitors to a domain?
4. Wouldn’t you notify the registered owners of the IP (the hosting company) if you’re effectively removing the entire server and all of the domains from the internet for all of your users?
6. Finally, when did Comcast begin the practice of blocking an entire web server from being accessed by browser, FTP or ping?
This is something I’ve personally never heard of, and neither had Kevin, the support person I spoke with in Comcast’s security department.
Thank you for your time…
JL says:
You have my email addr from the blog post. Please email me and I can get you all the logs.
Neil Patmore says:
Good info and thanks for sharing. Merry Christmas and a happy new year. I’m loving your blog theme by the way
Eric says:
I have found several web sites blocked by Comcast cable internet this week. Unfortunately, it is not blocked by DNS but rather by filtering web content. I am able to ping the web sites, browse the web sites by means of proxy servers, but not browse the web sites directly. When I use a dial-up internet, I can browse the web sites. When I use my next-door neighbor’s AT&T DSL, I can browse the web sites. Same computer, different providers, different results.
Scott says:
Interesting – It sounds like a different type of blocking, since you can ping, but yep, it’s blocked all the same!
Dave says:
This same identical thing is happening to me and others. Benn trying for three days to get it resolved. How can reach someone in Comcast that even cares?
Richard says:
The same thing happened to me. However – it turns out it wasn’t comcast that was actually blocking it – but one of the companies they routed through. In my case Level3 !!
I went through various tech’s at comcast before I got to one that knew what he was doing.
Firstly – get hold of the IP address of the server where your domain is hosted – or the site that is being blocked (you may need to get that from the hosting company)
in a dos prompt – do
tracert xxxxxxx (where xxxxxx is the IP you want to get to)
that will trace how you’re getting out.
once you get to the break point
go to: http://www.iptools.com
and in the ‘whois lookup’ – put in the final IP that was successfully resolved.
this will tell you the name of the company that is blocking that IP.
While Comcast will then say it’s not them – which is partly true – they’re still starting the routing off that gets you to this place. When I used a different ISP it resolved, but strictly it wasn’t Comcast that blocked it.
I told my Hosting company what was going on – and I guess they worked it out with Level3, because it now all resolves…
Scott says:
That’s good that you were able to pinpoint it finally, and knowing how to walk someone through a traceroute could be taught to any tech-
Steve Johnson says:
This Comcast behavior (and presumably other “safe” ISPs) has to be a LOT more widespread. I have a layeredtech server that is totally offline to the entire comcast network. I smell a class-action lawsuit!
robert martin says:
Yep, I am dealing with the same issue right now. They have blocked access to anything to do with lunerpages.net’s oxia server (lunarpages is a massive server farm)which means my ftp will not connect, I cannor connect to my website at http://www.gulfcoastmasons.com and my incomming email is also being blacked at mail.gulfcoastmasons.com (a friend using at&t for an isp can see my sites just fine) This has happened before and is a nightmare to fix. strange as it may be, the tech support person in nashville can connect, but nobody here in mobile alabama (on a differant server) can connect. Will be one the phone again in the morning to raise hell with the local comcast office.
eric says:
Hi Robert, just testing from my Comcast location (Lawrenceville, GA as reported by IP source) and both sites are viewable. I really think Comcast is blocking sites like California has had rolling blackouts. I’ve read about (internet) black holes and this may be just how it is happening. Instead of completely disconnecting you (or me when I’m having troubles), they just cut the lines to some part of the internet. This black hole is more insidious than a line cut however as usually packets go in and nothing comes out. Most things allow for an error to indicate a problem, but I’m now suspecting at least some of these reported black holes are intentional. Good luck with phoning the local Comcast office! Eric
JL says:
Eric – Comcast doesn’t do any of that stuff – what would be the objective? This site is accessible from the Comcast network. Most times when we see these reports there are ACL or bogons lists causing problems at the web hosting site. You can of course investigate this by pinging the site and doing a traceroute.
Also, a local Comcast office is not really equipped to deal with these issues (nor is a comment in a blog post from 2009 either). I recommend you go to http://forums.comcast.net or http://www.dslreports.com/forum/comcast.
Jason
Amanda says:
Our company is having this same problem today. Our networking guy says the problem points to our hosting server, but after finding this post I updated him (he’s talking to a Comcast buddy of his in person) and hopefully we can get this resolved. We’re a busy Real Estate company and the entire office is frustrated at their inability to work. Our website and email is blocked. I can access everything on my laptop because I have CLEAR broadband built-in, but nothing comcast-related is working.
One option we’re considering for the long-term insurance that this won’t happen again is purchasing a dedicated IP service from our hosting provider.
This whole situation is very stressful and confusing. I convinced my office to switch to our current hosting provider because all of my websites at home (I’m the web developer for the company) have been using it for a couple of years with amazing results… Now some people in the office are saying that it’s the host’s fault for not being strict enough about who they host and that we need to change again. My judgement is getting put into question and my credibility is being scrutinized because of comcast’s problem. Not cool.
Scott says:
Yes, I think this is clearly a Comcast issue, because they likely can’t even tell you what they’re doing or why they’re doing it. To blame it on the host is unfair – although i realize that coming from a host it may not mean much to your co-workers – but a host cant possibly be expected to screen and make judgments about who uses their services. Being “strict” beyond spam guidelines would likely get us sued!
If Comcast blocked in a way they feel was legitimate, then they really should be able to tell you why, so the host can fix something if necessary, or help you get to the bottom of it. Your host can only act upon what Comcast recommends – either that, or as in our case, Comcast finally says “whoops, sorry. and will fix it… Good luck!
Amanda says:
The Networking guy is convinced it’s a host issue and is putting my webhost through the ringer in trying to figure out what’s wrong. Meanwhile, we’re still sitting here “in the dark” for all of our potential customers using comcast and our entire office (which has comcast) so I’m the only one getting email (because I have CLEAR) in the entire office other than those using their phones.
Scott says:
It may very well be something at your web host however it is on comcast To communicate what the issue with it. Blocking without notice and no explanation is unacceptable in any environment Come on in my opinon.
Amanda says:
Whatever it was, it’s fixed now. I definitely don’t like the way Comcast handles things. I’m glad I live out of range of their services, but I kind of hate that we’re still stuck with it in the office… We switched to Comcast just last week; I guess it’s a good thing, though, because we otherwise wouldn’t have noticed the problem.
Scott says:
Thanks for the update, and good luck!
Stephen says:
Yeah, one more step towards censorship. I just renewed my subscription with http://www.sunvpn.com/, I usually use it when I travel, but it seems that I also need it back home.
Scott says:
Same is happening to my site. I changed my DNS or my home computer to Google DNS and problem solved . But all my customers looking to spend that Christmas money that have comcast cant get to my site …. It is definatly a Comcast DNS I also pay for a deticated IP address so I am not shared with other sites .