A Provider-to-Provider Route Leak Incident Review

This blog continues the topic of border gateway protocol (BGP) route leak incidents. The previous blog in this series about BGP security explained route leaks in-depth. For a full review, read the blog.

In short, a route leak is formally defined as the “propagation of a BGP announcement(s) beyond their intended scope” [RC7908].

The Provider-to-Provider BGP Route Leak

The media widely covered our next event, mainly because it involved famous networks such as Cloudflare and AWS. Additionally, Verizon was blamed for contributing to spread the leak on a global scale.

In the morning of June 26, 2019, ATI propagated thousands of routes (Figure 1) received from its provider DQE communications to its other provider Verizon. This created a provider-to-provider leak, formally defined as a Hairpin Turn with Full Prefix [RFC7908].

Incident Diagram

The Incident Origin

Those routes were generated by routing optimizer software running within DQE network. These routes were not supposed to be announced externally.

In fact, these generated routes were more specific routes toward popular internet destinations, generated to balance the traffic within DQE network.

A more specific route toward a given destination network P is a route toward a prefix Q included in P.

For example, let’s consider the Cloudflare network 104.26.0.0/20. The routes generated by DQE were two routes toward the more specific networks 104.26.0.0/21 and 104.26.8.0/21.

An AS will very likely accept a route not previously installed in its BGP routing table unless some filtering mechanism is in place. An example of a filter would include dropping every route found as invalid during the RPKI validation. The AS will always prefer to forward the traffic to a more specific route in the routing table over the less specific route due to the longest-prefix-matching rule.

Why Some Networks Propegated the Leak (and Some Did Not)

In this specific incident, Verizon accepted the leaked routes received from ATI and propagated them to its neighbors, thereby contributing to spread the leak on a global scale.

These neighbors included thousands of Verizon’s customers and other Tier-1 networks. Some of these networks accepted the leaked routes, while some others (such as AT&T and NTT) dropped them thanks to tight filtering mechanisms they had in place, like RPKI validation and peer-lock.

FRT Peers Announcing to the Collector at Least one Leaked Route

In this route leak incident, the vast majority of FRT peers announced to a route collector at least one route involved in the leak (Figure 2).

FRT Peers Announcing to the Collector at Least one Leaked Route

Drilling down these numbers per Internet region (Figure 3), it is possible to see that the majority of the AS’s not affected were in the ARIN region. Among them, there are those AS’s applying tight filtering mechanisms and their related customers.

Figure 4 and Figure 5 show the distribution of the number of origins and subnets that each peer perceived involved in the leak, respectively.

Number of Origins Involved per FRT Peer

Number of Subnets AS’s Involved per FRT Peer

Like SafeHost case we covered in our last blog, only a small percentage of FRT peers announced routes involving a significant number of origins and subnets.

For example, only 10% of FRT peers announced leaked routes involving more than 100 origins and more than 800 subnets.

Among them, Verizon is the FRT peer to involve the highest number of origins and subnets. Because Verizon is connected to one of the Route Views collectors (route-views2), we can see the potential impact of the leak: more than 4,000 different origins and more than 65,000 subnets were impacted.

The leak lasted almost two hours (Figure 6) and involved more than 4,000 different origins worldwide, including famous networks, banks, and ISPs.

Learn More About BGP Security

Route leaks are not the only BGP security concerns for companies. Read the next blog in the series to learn more about BGP security, route hijacks in particular.

Can’t wait for the next blog? Read our ebook Comprehensive Guide to BGP Monitoring for a deeper look into BGP routing.