Over the past week or so, many of you, especially readers on campus have probably noticed that articles on dailypennsylvanian.com have just plain not worked.
You’ve likely been seeing some version of this error.
Service Unavailable - DNS failureThe server is temporarily unable to service your request. Please try again later.
Reference #11.c48f748.1206020220.6a1305f
Well, I’ve been seeing it too, and now I’m here to explain it to you. The explanation is long and complicated, but trust me, it’s a rip-roaring story involving shadowy corporations, quarreling lovers and at least one Scandinavian country.
The root cause of the network not working is a dispute between two major internet corporations that few people have probably ever heard of. The two players are Tier 2 Internet Service Providers, Cogent Communications, and TeliaSonera, who for the most part sell internet service to other smaller service providers and governmental and educational institutions. Cogent and Telia both operate large swaths of long distance fiber optic cables. (Cogent’s map. Telia, based in Sweden, is one of Europe’s largest ISP’s but also has a network in the US.)
Large ISP’s like Cogent and Telia connect their networks together through a process called “peering,” connecting their networks together at certain locations where fiber from hundreds of companies come together. Like this one, in Los Angeles, where nearly an entire office tower is taken up by ethernet cables, and switches.
For the most part, internet companies peer voluntarily and without charging the other side because, for the most part, equal amounts of traffic pass from one side to the other. And the system works really well. Breakups like Cogent and Telia’s happen very rarely. Another high profile breakup happened between Cogent and a Tier 1 ISP, Level 3 in 2005.
Depeerings partition the internet, blocking off customers hosted on one ISP from seeing sites hosted on the other and vice versa. They are supposed to be painful, just like a lover’s quarrel to inspire wailing customers as a pressure tactic in negotiations. In this case, Cogent initially cut the plug on Telia, accusing Telia of failing to install the necessary network infrastructure at peering locations to better carry traffic. Cogent CEO Dave Schaffer claims Telia was doing that to degrade the experience for Cogent’s customers. Eventually Cogent and Telia WILL repeer, with new terms to their agreement, but until then you can’t get from one network to the other.
According to GigaOM,
Schaffer says the problems have cropped up because Cogent has expanded into Nordic and Baltic countries in addition to building out its reach in Eastern Europe. Telia and Cogent now compete in these markets. “They are resentful of our expansion in these markets,” he says. Schaffer also says his company remains “willing and anxious for settlement-free peering” and that “Telia needs to meet their contractual obligations.”
But how does this business dispute affect Penn, and the DP’s website in particular?
Naila Machado, from Penn’s Information Systems and Computing Department tells me this.
Yes, Penn is multihomed. It has 2 independent peerings with commercial Internet Service Providers (Cogent and Level-3), and 2 peerings with the MAGPI GigaPoP which provides connectivity to Research & Education networks like Internet2. Because of the multi-homed connectivity, very little was disrupted for members of the Penn community. And no action was needed to be taken.
Unfortunately for The Daily Pennslvanian, we don’t have such a luxury of internet connections. Penn provides internet conncetivity to the DP but we are connected through the MAGPI GigaPop, part of an academic research network called Internet2. According to Machado, MAGPI is only connected to the rest of the internet through a Cogent provided connection.
When you type dailypennsylvanian.com into your web browser, the first thing that happens is your computer send out a request to resolve that domain name into a numeric IP address. This request makes it’s way through the Domain Name System, to The Daily Pennsylvanian’s domain name server, which we host on our own, and therefore is also on the Cogent network.
However, not all of the DP website is on the cogent network. In order to speed up load times, the articles and other pages of dailypennsylvanian.com are distributed on a network of servers scattered across the globe. Called the Akamai network, it maintains servers on many different networks, including the Telia network. When you click on a link to an article on dailypennsylvanian.com, your computer sends a request to the DP’s DNS servers, which in turn look up the IP address of the Akamai server closest to you. Many of these Akamai servers are on the Telia network, so when our DNS server goes to try and find them, they come up empty and return an error page to our browser.
The solution to this problem is to reprogram dailypennsylvanian.com so that content is no longer cached on the Akamai network, and instead all requsts are sent to our main host’s (College Publisher/mtvU) servers. That’s what they’ve been busy with for the past week, and that’s why hopefully this DNS error has been resolved.
So it wasn’t monkeys.
Tags: , cogent, technical problems, telia, website errors
An RSS feed of Inside the Newsroom is available
April 1st, 2008 at 9:33 pm
And why has today’s link to the N.Z. breakin to the Penn server been broken all day?
My experience is that the DP web site has significant sluggishness almost always, and
has a major issue at least once a week. Why don’t you put up a hours and minutes
clock since the last problem it has experienced? Last week was just more of the
same from my perspective.
April 27th, 2008 at 5:47 pm
Interesting stuff. Why does the DP host locally when it is part of the College Publisher network? It seems that allowing CP to host would save a lot of money.