How HTTP Works - Anatomy of an HTTP Transaction

When dealing with web performance monitoring or optimization, it is important to learn the fundamentals of the HTTP protocol. The web as we know is built on top of this protocol, which has a large impact on performance of a webpage. Understanding how HTTP works and what each component in an HTTP transaction means, is key to interpreting the data collected by any monitoring service. This post describes the basics of HTTP and is the first of a series on how to diagnose performance problems.

HTTP data rides above the TCP protocol, which guarantees reliability of delivery, and breaks down large data requests and responses into network-manageable chunks. TCP is a “connection” oriented protocol, which means when a client starts a dialogue with a server the TCP protocol will open a connection, over which the HTTP data will be reliably transferred, and when the dialogue is complete that connection should be closed. All of the data in the HTTP protocol is expressed in human-readable ASCII text.

Simple HTTP Transaction

A simple HTTP transaction is one where the client makes a single request for HTTP content.

1. DNS Lookup: The client tries to resolve the domain name for the request.
2. Client sends DNS Query to local ISP DNS server.
3. DNS server responds with the IP address for hostname.com
4. Connect: Client establishes TCP connection with the IP address of hostname.com
5. Client sends SYN packet.
6. Web server sends SYN-ACK packet.
7. Client answers with ACK packet, concluding the three-way TCP connection establishment.
8. Send: Client sends the HTTP request to the web server.
9. Wait: Client waits for the server to respond to the request.
10. Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content.
11. Load: Client loads the content of the response.
12. Web server sends second TCP segment with the PSH flag set.
13. Client sends ACK. (Client sends ACK every two segments it receives. from the host)
14. Web server sends third TCP segment with HTTP_Continue.
15. Close: Client sends a a FIN packet to close the TCP connection.

Serial HTTP Connection

A Serial HTTP connection occurs when multiple requests are issued sequentially to a server, and each request establishes a new connection. This method rarely occurs today because all modern browsers support parallel connections to a host. However, this may also happen when a browser or server supports only HTTP 1.0, without Keep Alive (or HTTP 1.0 +) and the first request is a blocking request (for example an inline JavaScript Request)

Request I

1. DNS Lookup: Client tries to resolve the domain name for the request.
2. Client sends DNS Query to local ISP DNS server.
3. DNS server responds with the IP address to hostname.com
4. Connect: Client establishes TCP Connection (1) with the IP address of hostname.com
5. Client sends SYN packet.
6. Web server sends SYN-ACK packet.
7. Client sends ACK packet, concluding the three-way TCP connection establishment.
8. Send: Client sends the HTTP request to the web server.
9. Wait: Client waits for the server to respond to the request.
10. Load: Client loads the content of the response.

Request II

1. Connect: Client establishes TCP Connection (2) with the IP address of hostname.com
2. Send: Client sends the HTTP request to the web server
3. Wait: Client waits for the server to respond to the request.
4. Load: Client loads the content of the response.

Persistent HTTP Transactions

Persistent connections allow the browser / HTTP client to utilize the same connection for different object requests to the same hostname. The HTTP 1.1 protocol supports persistent connections natively, and does not require any specific HTTP header information. For HTTP 1.0, persistent connections are controlled via the Keep-Alive HTTP header.

1. DNS Lookup: Client tries to resolve the domain name.
2. Client sends DNS Query to local ISP DNS server.
3. DNS server responds with the IP address to hostname.com
4. Connect: Client establishes TCP Connection (1) with the IP address of hostname.com
5. Client sends SYN packet.
6. Web server sends SYN-ACK packet
7. Client sends ACK packet, concluding three way TCP connection establishment
8. Send: Client sends the HTTP request to the web server.
9. Wait: Client waits for the server to respond to the request.
10. Load: Client loads the content of the response.
11. Send: Client sends the HTTP request to the web server
12. Wait: Client waits for the server to respond to the request.
13. Load: Client loads the content of the response.

Parallel HTTP Transactions

The HTTP 1.1 protocol provides browsers with the ability to open multiple connections and perform HTTP requests in parallel. (This is also supported by HTTP1.0 with Keep-Alive but with a slightly different implementation.)

In this example the browser loads an HTML page from hostname.com that contains two image requests to hostname.com and two requests to adserver.com

Once the first request is completed, the subsequent requests are performed in parallel:

Default Browser Connection Settings

Here is a list of the default maximum number of connections per host that can be open in parallel by various browsers. The limits increase over time as newer browsers are released. The more parallel connections, the faster the webpage renders however studies have shown that there might be limits to it – having it at 10 would not necessarily yield better results.

To learn more about this topic, we highly recommend “HTTP: The Definitive Guide” by David Gourley and Brian Totty. Or, watch this video, Packet Flight.