Article Written by Alex Henthorn-Iwane, Vice President of Marketing at ThousandEyes
The recent hijack of Amazon Route 53 DNS services was a particularly galling reminder that the Internet is at best an unpredictable place to conduct business, if not downright dangerous. For MyEtherWallet, the malicious breakage of Internet infrastructure was costly to both their brand and revenue. So let's move past the hyped news cycle and pictures of hackers typing in hoodies because there are important lessons for digital business leaders to be taken away from this incident.
1. The Internet is not a traditional infrastructure.
One of the chief misconceptions that people have about the Internet is that it is in some way like traditional infrastructure systems like roadways, bridges, power and water utilities. Using a roadway analogy to explain what happened in this exploit illustrates why that's not the case. What the attackers did was like taking one of the lanes of a highway headed to Chicago, instantly ripping up hundreds of tons of concrete, steel and asphalt, and in real-time placing it all back down in a different direction so that instead of going to Chicago, it went to Kokomo but none of the drivers knew they had changed course. Then, they examined every car and figured out which drivers were headed to use a banking ATM, faked signage and roadways so they'd exit the freeway, stop in what looked like a Bank of America parking lot and type their credentials into a fake ATM. Oh, and they took all the cars not headed for an ATM and threw them into a bottomless pit. So that's an odd analogy, but the point here is that as a digital business leader if you don't adjust your expectations and understanding of how the Internet works, you're prone to making serious errors in judgment about continuity and security risks.
2. Your digital everything relies on an unregulated chain of trustThe Internet is not one thing. It's a collection of tens of thousands of organizations running networks that choose to connect together. These connections and the ability to communicate across all these networks rely on voluntary participation in two core systems meant to facilitate the exchange of information on how to communicate from any point A to point B. One system is the BGP Internet routing system--which is essentially like a highly computerized game of telephone--where one organization tells others, "Hey, you can get to these Internet addresses through me," and others have to pass it along to everyone else. There are hundreds of thousands of *blocks* of such addresses, encompassing over 340 undecillion (3.4×10^38) individual numbers.
The other system is the Domain Name Service (DNS), which converts human-readable text URLs to a numerical network address. DNS is like a volunteer phone chain that answers questions like "what's the actual network address for
www.MyEtherWallet.com," by calling and asking others who are more in the know until an authoritative answer is tracked down.
The continuity risk for digital business is that both of these systems are based on implicit trust and voluntary actions. There is no regulating body in the Internet that checks to make sure that everything is correct before information is forwarded or answers are given. If obviously wrong information is passed forward in the BGP telephone game, some folks will do the right thing and reject it, but others may not. In essence, that's what happened in this case when hackers were able to compromise the systems of a small but well-connected network organization. They then got access to the routers in that network and inserted a fake message from that network into the Internet telephone game, saying that they knew best how to get to Amazon's DNS servers. Most didn't believe the fake message, but some did and passed it along. In addition, the hackers set up a fake DNS server to provide fake answers to questions about how to get to MyEtherWallet and thereby steered unsuspecting users to a different network address that led to a fake website. Folks typed in their credentials and unwittingly handed their cryptocurrency over to the hackers.
Now security-minded engineers will point out, and rightly so, that multiple attempts have been made to corral all the various network organizations on the Internet to implement better security. For BGP Internet routing, these efforts include BCP-38 and
RPKI. These are truly useful and should be universally applied, but to quote the first six words of the RPKI page at the American Registry for Internet Numbers (ARIN) website, "RPKI is a free, opt-in service." All of the security standards are utterly voluntary, so they are by definition inconsistently applied, which means that they're not functioning systems, they're just noble attempts.
The point here is that you cannot count on the Internet to uphold consistent, secure practices. Your digital business is riding on a totally unregulated chain of implicit trust.
3. It's on you to monitor every dependency that impacts digital experienceSo what do you do, given the above? Let's put aside idealistic notions on regulation for the moment--if we can't agree to simple things in one country, how do you get a network that spans the globe to sign up? This means the first and foremost thing you need to equip yourself with in the face of all these continuity and security risks is sound and timely information. And yes, I used the "monitoring" word, because fundamentally you need to continuously measure the performance and continuity of Internet routing, DNS, and every provider you directly rely on including your CDN and DDoS mitigation providers. Just remember that since all these providers and systems are out on the Internet, you'll have to use different techniques to gather data than how you currently manage servers and networks you own. There are various ways to avail yourself of this kind of intelligence--do yourself a big favor and make a plan to get it.
4. Push your providers to opt into sound security measuresThe second order of business is to push your providers to do the "opt-in" move described by ARIN above. When you re-up your cloud and ISP contracts, research common security measures and put them as requirements, so their engineers have a good business justification to invest in those measures. Ask your peers to do the same, and become champions for Internet hygiene and governance. The vast majority of providers want to do the right thing, but they need customers to ask to help them prioritize those good impulses.
5. There's no such thing as steady state in the cloudThe roadway illustration above shows how mutable the Internet is. Every moment, the pathways across the Internet are in constant flux. It's like those moving staircases in Hogwarts Castle in the Harry Potter series, except times a jillion. Cloud providers are constantly evolving their networks, infrastructure, service offerings and even their network address schemes. Mergers and acquisitions, government regulatory schemes, human error and malicious actions continuously muddy the water. Take an agile lifecycle approach and always assume you need your organization and technology to be readying for the next change.
Succeed in an Unpredictable Digital WorldDigital business is unavoidable. Engage and deliver awesome customer experiences or else. Those customer experiences rely in no small part on unpredictable Internet dependencies. What the Route 53 hijack tells us is that playing the digital business game to win today requires a core competency in understanding and managing those dependencies.
##