Debugging CDNs: A Need for Change

It’s a Saturday afternoon: An email from a customer drove me to start troubleshooting an issue they were experiencing with Amazon S3. They were experiencing random high wait time, usually an indication of a problem with communication back to some sort of an origin system. Should’ve been a piece of cake, right?

Well, it wasn’t. Each CDN out there uses different debug headers and some don’t send anything back, and in this case, S3 was even more difficult to work with because the only thing available was a useless set of headers:

• HTTP/1.1 200 OK
• Cache-Control: max-age=2592000
• Date: Sat, 18 Feb 2017 21:28:55 GMT
• Content-Length: 66153
• Content-Type: image/jpeg
• Last-Modified: Wed, 20 May 2015 21:21:04 GMT
• Accept-Ranges: bytes
• ETag: “6a4e8ea14b77871a29798980aa09cece”
• Server: AmazonS3
• x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
• x-amz-request-id: BB24FC949CCC93CC

My next thought was to just share the request ID with our customer so they could download some sort of log; however, AWS does not provide that information unless you are paying for support.

One can easily argue that S3 is not a CDN. On the S3 performance page, they ask you to use CloudFront for GET-intensive workload, though many people use S3 as an origin.

So, let’s take a look at S3 in comparison with other CDN providers when it comes to helping troubleshoot via the information available in the HTTP headers.

S3:

• HTTP/1.1 200 OK
• Cache-Control: max-age=2592000
• Date: Sat, 18 Feb 2017 21:28:55 GMT
• Content-Length: 66153
• Content-Type: image/jpeg
• Last-Modified: Wed, 20 May 2015 21:21:04 GMT
• Accept-Ranges: bytes
• ETag: “6a4e8ea14b77871a29798980aa09cece”
• Server: AmazonS3
• x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
• x-amz-request-id: BB24FC949CCC93CC

Akamai:

If you pass this Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id

This is what you get:

• HTTP/1.1 200 OK
• Server: Jetty(9.2.4.v20141028)
• X-Request-Id: 238520
• Content-Type: text/html; charset=UTF-8
• Content-Language: en-US
• Set-Edgescape-Cookie: true
• Cache-Control: max-age=9821
• Expires: Sun, 19 Feb 2017 04:40:52 GMT
• Date: Sun, 19 Feb 2017 01:57:11 GMT
• X-Cache: TCP_MEM_HIT from a184-51-206-135.deploy.akamaitechnologies.com (AkamaiGHost/8.2.2.1.1-19192351) (-)
• Connection: keep-alive
• X-Check-Cacheable: YES

Limelight:

• HTTP/1.1 200 OK
• Cache-Control: max-age=4
• Connection: keep-alive
• Date: Fri, 27 Jan 2017 18:47:22 GMT
• Content-Length: 99450
• Content-Type: audio/x-mpegurl
• Expires: Fri, 27 Jan 2017 18:47:26 GMT
• Last-Modified: Sat, 16 Mar 2013 21:41:29 GMT
• Accept-Ranges: bytes
• Server: Apache/2.4.12 (Unix)
• X-Served-By: apache05.live01.hls.gen.med1.prod.mlbam.net
• Access-Control-Allow-Origin: *
• X-Cache: HIT from cds1351.lga.llnw.net c:0,c0:0,r:4,r0:0,s:5
• X-Cache: HIT from cds1232.iad.llnw.net d:2,d0:0,c:10,c0:0,r:22,r0:0,s:22
• X-Cache: MISS from sw.cds1294.iad.llnw.net

Fastly:

• HTTP/1.1 200 OK
• Content-Encoding: gzip
• Content-Language: en
• Content-Type: text/html; charset=utf-8
• Link: ; rel=”canonical”,; rel=”shortlink”
• Surrogate-Control: max-age=86400
• Surrogate-Key: node node/2058
• Via: 1.1 varnish
• X-Content-Type-Options: nosniff
• X-Frame-Options: SAMEORIGIN
• Server: Artisanal bits
• Last-Modified: Sat, 18 Feb 2017 20:48:21 GMT
• Via: 1.1 varnish
• Content-Length: 11329
• Accept-Ranges: bytes
• Date: Sun, 19 Feb 2017 01:58:24 GMT
• Via: 1.1 varnish
• Age: 18602
• Connection: keep-alive
• Fastly-Debug-Path: (D cache-lhr6351-LHR 1487469504) (F cache-lhr6320-LHR 1487450940) (D cache-sjc3128-SJC 1487450940) (F cache-sjc3120-SJC 1487450901)
• Fastly-Debug-TTL: (H cache-lhr6351-LHR 67836.790 2592000.000 18602) (H cache-sjc3128-SJC – – 39)
• Fastly-Debug-Digest: 09a954816c7b2f9614bfd284e92ed6cec21c76c5434c2deabab127826154417a
• X-Served-By: cache-sjc3128-SJC, cache-lhr6351-LHR
• X-Cache: HIT, HIT
• X-Cache-Hits: 1, 5
• X-Timer: S1487469504.425617,VS0,VE0
• Vary: Cookie,fastly-ssl,Accept-Encoding
• X-XSS-Protection: 1; mode=block
• Cache-Control: max-age=0, private, must-revalidate
• Strict-Transport-Security: max-age=31536000

Edgecast / Verizon Digital Media Services

• HTTP/1.1 200 OK
• Content-Encoding: gzip
• Accept-Ranges: bytes
• Alt-Svc: clear
• Cache-Control: max-age=604800
• Content-Type: text/css
• Date: Sun, 19 Feb 2017 02:00:47 GMT
• Expires: Sun, 26 Feb 2017 02:00:47 GMT
• Last-Modified: Thu, 16 Feb 2017 06:52:22 GMT
• Server: ECS (oxr/838B)
• Strict-Transport-Security: max-age=31536000
• Vary: Accept-Encoding
• Via: 1.1 google
• X-Cache: HIT
• X-Served-By: us-central1-c
• Content-Length: 479

CDNetworks:

• HTTP/1.1 200 OK
• Date: Sun, 19 Feb 2017 11:40:15 GMT
• Server: PWS/8.2.0.5
• Transfer-Encoding: chunked
• X-Px: nc h0-s1027.p12-sjc ( origin)
• Cache-Control: private
• Content-Type: text/html; charset=UTF-8
• Content-Encoding: gzip
• Vary: Accept-Encoding
• X-Zen-Fury: d22ebc323ffa4b916cbe2b078d0b36643cd7e07a
• Link: <https://www.cdnetworks.com/wp-json/>; rel=”https://api.w.org/“
• Link: <https://www.cdnetworks.com/>; rel=shortlink
• X-Pingback: https://www.cdnetworks.com/xmlrpc.php
• X-Cache-Status: NOTCACHED
• X-Hello-Human: If you want to work with us, please send e-mail to jobs at cdnetworks dot com or @CDNetworks on Twitter.
• Connection: keep-alive

There is no consistency, and with more and more companies relying on multiple CDNs, it’s impossible to debug and troubleshoot.

In my opinion, there is a list of must-have HTTP headers:

• Which Edge Server handled the request
• Hit / Miss / and any other information (including hierarchical tiering/original shield information)
• Time to Serve
• Time to Fetch from origin
• Transaction ID for further troubleshooting on the CDN side

Please, let’s all urge our CDN friends to adopt a common troubleshooting header structure (by the way, Limelight, could you not use the same x-header to display 3 different things?) and CDNetworks you are wasting bytes with that text to hire people!

Example:

• X-Served-By: ABC
• X-Status: HIT / MISS
• X-Time: in milliseconds
• X-Time-Origin: in millisecond
• X-Transaction-ID

We thank all of the CDNs out there for making the internet faster and the user experience better, but the time has come to make time to troubleshoot faster and easier as well.

Mehdi – Catchpoint