Performance and Security

Many factors can impact the user experience of a website or mobile app including bad code, bad architecture, infrastructure, external factors such as fiber cut, government interference (invisible hand), and CDN and DNS providers issues.

Unfortunately, we now have to add another reason: Security.

Per Google: A Distributed Denial of Service (DDoS) attack is an attempt to make an online service unavailable by overwhelming it with traffic from multiple sources. They target a wide variety of important resources, from banks to news websites, and present a major challenge to making sure people can publish and access important information.

We hear about massive DDoS attacks that are impacting brands, taking down critical services and infrastructure on what seems like a daily basis.

If your business relies on your website or app, it’s important to have a strong understanding of this very real security threat. There are two kinds of DDOS attacks:

External and malicious

Recently one of the properties we monitor suffered from one of these attacks. As is the case in many of these attacks, the pattern of the attack was very interesting: two small incidents occurred to probe before the big one that eventually took the site down for eight hours.

There are many ways of protecting yourself from a DDoS, but it really boils down to having enough capacity to handle the load and ensure that the real users can still access your services. This is why many CDN players have launched web application firewall (WAF) at the edge while protecting the origin.

Other methods include deploying solutions like Prolexic, a leader in this space now acquired by Akamai, which basically routes traffic to their data centers and applies scrubbing science to stop the bad guys.

The challenge with this is making sure that valid traffic is not scrubbed. This also happened this past week with one of our customers; some of our IPs were either whitelisted or blacklisted at different time which seems a bit odd (as shown below).

Traceroute after being blacklisted:

1 12 ms 5 ms 11 ms 77.67.108.222

2 <1 ms <1 ms <1 ms 77.67.108.210

3 <1 ms <1 ms <1 ms 77.67.108.208

4 <1 ms <1 ms 4 ms ae24-166.lon11.ip4.gtt.net[141.136.103.33]

5 1 ms 2 ms 1 ms ae-9.r00.londen10.uk.bb.gin.ntt.net[141.136.96.182]

6 2 ms 2 ms 5 ms ae-15.r02.londen03.uk.bb.gin.ntt.net[129.250.3.30]

7 10 ms 2 ms 2 ms ae-2.r02.londen01.uk.bb.gin.ntt.net[129.250.3.1]

8 2 ms 2 ms 2 ms ae-1.r03.londen01.uk.bb.gin.ntt.net[129.250.3.37]

9 2 ms 2 ms 2 ms 212.119.29.26

10 2 ms 2 ms 2 ms unknown.prolexic.com[72.52.60.202]

11 7 ms 2 ms 2 ms unknown.prolexic.com[72.52.60.205]

12 2 ms 8 ms 13 ms 209.200.181.144

13 * * * Timed Out

14 * * * Timed Out

15 * * * Timed Out

16 * * * Timed Out

It’s extremely important to make sure you deploy solutions to ensure that your DDoS protection systems are not causing reachability problems for real users, since my IP address in our UK office went to the cyber dump at the same time:

1 192.168.64.1 (192.168.64.1) 1.001 ms 4.125 ms 0.836 ms

2 10.43.10.1 (10.43.10.1) 1.245 ms 1.369 ms 2.440 ms

3 195.110.84.178 (195.110.84.178) 1.477 ms 2.005 ms 3.264 ms

4 80.169.168.137 (80.169.168.137) 2.450 ms 3.104 ms 2.575 ms

5 xe0-0-0-pr2.lon.router.colt.net (212.74.85.225) 2.956 ms 4.010 ms 4.095 ms

6 195.66.236.31 (195.66.236.31) 6.260 ms 5.151 ms 4.639 ms

7 unknown.prolexic.com (72.52.60.194) 3.594 ms 3.757 ms

unknown.prolexic.com (72.52.60.202) 3.666 ms

8 unknown.prolexic.com (72.52.60.205) 9.877 ms

unknown.prolexic.com (72.52.60.197) 8.500 ms 16.638 ms

9 unknown.prolexic.com (72.52.20.230) 25.894 ms

209.200.181.144 (209.200.181.144) 20.651 ms 23.919 ms

Internal and self-inflicted via trusted party

With all of the third parties that a site must deploy these days, another DDoS vector is internal. A trusted third party has a bug and does an auto-refresh or auto-reload and creates an infinite loop, thus a self-inflicted DDoS.

This is what happened to a few big sites last summer during Velocity 2015 (bad timing) where a third-party tag was setting a reload to the entire page and causing a loop. The incident lasted about four hours.

We all know that the customer experience defines the product. I have often said that the new marketing mix includes a fifth ‘P,’ for Performance.

The events of last week reminded me that security attacks and the counter measures deployed can also have a huge impact to this customer experience and I will be updating my infamous performance iceberg I have been using since the late ’90s.

1998 Version:

2016 Version:

Some closing thoughts

Back at Doubleclick in 2002, after hiring one of our best CSOs Dr Rey Leclerc, discussions started around performance, monitoring, and security and where  they really fit in an organization. Many organizations lack a single group in charge of performance monitoring at the macro level. We were the few in 1999 to establish a group dedicated to quality of experience; but in 2016, we still have not clearly defined a function that owns performance both horizontally (IT working with marketing, for example) and vertically ( integrating performance with agile dev teams).

If performance is going to be this critical to a company where it defines its brand, its revenue success, and product adoption, is it not time to elevate this function the same way we have CSO, Red Team and Blue Team, and have individuals and teams whose sole mission is to monitor, baseline, benchmark report, and enforce performance policies while reporting to a C-level?

Performance is a journey not a destination; it’s never done the same way as Security!

Mehdi – Catchpoint