The experience of building an international backbone for PEER1, a large hosting company, over the last 11 years exposed me to a number of challenges, most notably the difficulty of dealing with Distributed Denial of Service (DDoS). DDoS’s seen in these environments vary in size and complexity, with each type of attack requiring its own strategy to counter. The only pattern to these DDoS attacks was the increasing rate we would see them each year; by the end of my tenure at PEER1 in early 2012, we were seeing between 5-7 attacks a day!
While constantly revisiting our strategies at PEER1 to deal with the increasing frequency and changing nature of these attacks, there were always three fundamental questions that needed to be answered in drawing up a gameplan:
a) How to assess, in real time, the overall damage being incurred by a DoS attack. Is it only affecting an individual customer? Multiple customers? Is the attack causing stress on the backbone infrastructure or worse still, is it actually targeting the infrastructure? A failure of just a few minutes was the difference between maintaining 100% uptime or having multiple datacenters suffer severe packet loss or compete loss of routing.
b) How to mobilize and install a solution quickly. Depending upon the location and complexity of a solution, 3-5 departments may need to get involved in a moment’s notice to implement a solution. Not an easy feat under the best of circumstances.
c) How to balance staff resources. Dealing with a DOS attack can occupy multiple team members for up to 3-4 days. How do you maintain focus for a customer that is being attacked when you have the rest of business too take care of as well? Quite often, we would review the monthly revenue of the attacked customer before determining what resources we could commit to them.
By constantly circling back on these questions and evolving our strategies over the last decade, we were able to achieve some degree of success, at least in safeguarding our own infrastructure. The time spent struggling with DDoS’s at PEER1 allowed me to come to the following set of observations about DDoS and the hosting industry:
i) There is no such thing as a ‘godbox’ or bullet proof firewall – Pretty much every major vendor in the arena of security champions a product that they say will mitigate everything. Always appealing to an enterprise customer, and always meets expectations until there is an actual attack; then all bets are off. If it’s a zero day attack, expect to be treated like a lab subject when you call for vendor support and be prepared to eat the DDoS while the vendor analyzes the incoming traffic. The most frustrating part of these vendor proclamations is how they double down on a false promise and indicate that their solution is perfectly suited for a service provider environment as well.
ii) Capacity is always a concern in a hosting environment – Over the last few years, large UDP/ICMP packet floods, intended to exhaust network capacity, have grown less frequent and less damaging as hosting network providers have implemented policies and ACL’s to mitigate this type of offensive. Hosting companies have also upgraded their backbone links to multiple 10Gb/s links to better handle these spikes of traffic. However, even the more sophisticated attacks of today can eat up some significant capacity, which can cause saturation on the smaller links within a datacenter closer to a shared customer access switch. These types of attacks usually force many hosting providers to null route the ip for a defined period (24-72 hours) in the interest of minimizing collateral damage.
iii) The Need for Focus – I alluded to this earlier, on how in my past roles we came to decide on how much time was to be devoted to a DDoS attack targeting a customer. The reality is that even if a hosting provider spares enough resources to help mitigate an attack, they won’t be able to keep those resources focused for very long. For the vast majority of network teams, DDoS protection is not in the primary duty set, and falls low on the list of priorities. This can be a problem if a dogged attacker is persistent and selectively times the attacks throughout the course of a day/week/month.
It’s this last point that allured me to the world of DDoS protection and ultimately DOSarrest. As DDoS becomes more prevalent and sophisticated, I came to the realization that it would take a singular focus and vigilance to combat these types of attacks. Constantly innovating, DOSarrest understands this concept in helping keeping it’s customer online, all the time.
Chief Technical Officer, DOSarrest Internet Security