HAProxy selective TLS termination · Blog

HAProxy ("The Reliable, High Performance TCP/HTTP Load Balancer") is a TCP/HTTP Reverse proxy, that can do TLS termination.

The SSL termination proxy decrypts incoming HTTPS traffic and forwards it to a webservice. — Galgalesh CC BY-SA 4.0

It is very useful as a web-facing frontend, offloading the certificates' handling and TLS termination for "backend" servers. As such, it proxies incoming encrypted traffic to internal (non-encrypted) traffic, allowing for simpler backend services.

HTTP TLS termination

In most HTTPS cases, handling TLS certificates and termination in the reverse proxy makes sense, as the backend servers don't need knowledge of the TLS layer. Let's take a concrete example: one "playground" server we have for Liip is a quite large and powerful machine that is "shared" through team-specific containers in which developers can freely experiment. But as the services are exposed to the Internet, the Internal IT team cares about the external-facing HAProxy instance, and the developement teams focus on providing their services on unencrypted ports (80, or others) on the internal network. With this, the services get state-of-the-art TLS configuration, free Let's Encrypt certificates with automated renewal, monitoring, etc, without the associated concerns.

Given two hostnames (agile.example.com and hola.example.com), pointing to the same HAProxy server, these are the relevant /etc/haproxy/haproxy.conf configuration lines:

frontend ft_https
    mode http
    # HAProxy will take the fitting certificate from the available ones
    bind *:443 ssl crt /etc/haproxy/certs/

    # Spread the requests between backends
    use_backend     bk_agile if {hdr(host) -i agile.example.com}
    use_backend     bk_hola  if {hdr(host) -i hola.example.com}
    default_backend bk_traditional

backend bk_agile
    server agile.internal.example.com:80 check

backend bk_hola
    server hola.internal.example.com:80 check

backend bk_traditional
    server traditional.internal.example.com:80 check

In the frontend configuration, the mode http line configures HAProxy to work at this layer for this frontend, and the bind *:443 ssl crt /etc/haproxy/certs/ line configures it to use the "right" key/certificate pairs from the /etc/haproxy/certs directory. With this, the frontend ft_https terminates the HTTP TLS connection, and passes the unencrypted HTTP stream to the backends.

The two use_backend lines use the Host: information from the decrypted HTTP header (hdr(host)) to determine towards which backend configuration the requests should be proxied.

Finally, the three backend configurations tell HAProxy to which server the (now unencrypted) requests should be proxied. In prose; this configuration tells HAProxy to:

Terminate the HTTP TLS connection in the frontend with the most appropriate available certificates; then:

proxy the agile.example.com requests to agile.internal.example.com (on port 80)

proxy the hola.example.com requests to hola.internal.example.com (on port 80)

proxy unmatched requests to traditional.internal.example.com (on port 80).

TCP pass-through

In some cases, "backend" services need access to their corresponding TLS certificates, because they don't operate purely as TLS-encapsulated HTTP. An example could be an email.example.com email service, that would use the same Let's Encrypt certificates in:

the postfix SMTP(S) daemon (to allow email sending);
the dovecot IMAP(S) daemon (to allow email fetching);
the nginx HTTP(S) daemon (for a web presence of sorts; or a webmail interface).

In this case, HAProxy, when used as HTTPS reverse-proxy, can also directly pass the TCP traffic to the backend server, to let it handle the TLS termination itself. (It's also possible to use HAProxy to proxy SMTP and IMAP protocols, which is a different way to address this specific case.)

An important aspect to take into account is that TLS-encrypted TCP connections can use SNI — Server Name Indication. SNI is a way for TLS clients to tell the servers to which Server Name they want to correspond with, before the connection is wrapped in TLS. (Although it is widely supported nowadays, there are still some network stacks that don't support it.)

This enables the server to select the correct virtual domain early and present the browser with the certificate containing the correct name. Therefore, with clients and servers that implement SNI, a server with a single IP address can serve a group of domain names for which it is impractical to get a common certificate.

Transforming the previous HTTPS example in a pure TCP example gives the following configuration:

frontend ft_tcp
    mode tcp
    bind *:443

    tcp-request content accept if { req_ssl_hello_type 1 }
    # The SNI (Server Name Indication) is not encrypted, so inspect the SSL hello for SNI
    # Spread the requests between backends
    use_backend     bk_agile if {req_ssl_sni -i agile.example.com}
    use_backend     bk_hola  if {req_ssl_sni -i hola.example.com}
    default_backend bk_traditional

backend bk_agile
    mode tcp
    # This backend server will need to terminate TLS for agile.example.com
    server agile.internal.example.com:443 check    

backend bk_hola
    mode tcp
    # This backend server will need to terminate TLS for hola.example.com
    server hola.internal.example.com:443 check

backend bk_traditional
    mode tcp
    # This backend server will need to terminate TLS
    server traditional.internal.example.com:443 check

In the frontend configuration, the mode tcp line configures HAProxy to work at the TCP layer for this frontend, and the bind *:443 configures it to listen to the port usually assigned to HTTPS (without the ssl keyword, HAProxy will not decrypt the traffic). Then, the two use_backend lines use the SNI information from the unencrypted TCP connection to determine towards which backend configuration the requests should be proxied. Finally, the three backend configurations tell HAProxy to which server the TCP connections should be proxied. It is essential for the mode to be set to tcp in both the frontend and the backends.

In prose; this configuration tells HAProxy to:

Look at the SNI header in the TCP connection; then:

proxy the agile.example.com requests to agile.internal.example.com (on port 443)

proxy the hola.example.com requests to hola.internal.example.com (on port 443)

proxy unmatched requests to traditional.internal.example.com (on port 443);
The TLS unwrapping is delegated to the backend servers.

Doing both TCP passthrough and HTTP TLS termination

In the two above cases, one frontend takes hold of the port 443 for exclusive handling by HAProxy, either for HTTPS TLS termination of for TCP passthrough. Let's see how it is possible for a single HAProxy to do both at the same time!

The trick here is to chain the two processing steps. Upon incoming connection on port 443, first process the TCP-level information (SNI), then hand over processing to another HAProxy frontend for HTTP-level processing. Let's see:

frontend ft_tcp
    mode tcp
    bind *:443

    tcp-request content accept if { req_ssl_hello_type 1 }
    # The SNI (Server Name Indication) is not encrypted, so inspect the TLS hello for SNI
    # Spread the requests between backends
    use_backend     bk_agile if {req_ssl_sni -i agile.example.com}
    default_backend bk_tcp_to_https

backend bk_tcp_to_https
    mode tcp
    server haproxy-https 127.0.0.1:8443 check

frontend ft_https
    mode http
    # HAProxy will take the fitting certificate from the available ones
    bind *:8443 ssl crt /etc/haproxy/certs/

    # Spread the requests between backends
    use_backend     bk_hola  if {hdr(host) -i hola.example.com}
    default_backend bk_traditional

backend bk_agile
    mode tcp
    # This backend server will need to terminate TLS for agile.example.com
    server agile.internal.example.com:443 check    

backend bk_hola
    server hola.internal.example.com:80 check

backend bk_traditional
    server traditional.internal.example.com:80 check

Again, in prose; this configuration tells HAProxy to:

In the ft_tcp frontend, listening to port 443, look at the SNI header in the TCP connection; then:

proxy the agile.example.com requests to agile.internal.example.com (on port 443)

proxy to HAProxy's own port 8443
In the ft_https frontend, listening to port 8443, terminate the TCP TLS with the most appropriate available certificates; then:

proxy the hola.example.com requests to hola.internal.example.com (on port 80)

proxy unmatched requests to traditional.internal.example.com (on port 80).

With such a setup, one single HAProxy, listening on port 443, can defer TLS termination to backend servers (for agile.example.com in the above config) as well as terminate TLS for HTTPS connections for other hosts.

Resources and closing words

Now, some references that were much needed to build this solution:

Some useful reads from Wikipedia

For the record, the above configuration snippets are meant to be minimal working examples; but in a real setup, there are plenty more options that are needed for a well-tuned setup. This was tested on a Debian Buster LXC container, with HAProxy 1.18.19-1.

Do you have a question, a comment, or just feeling inspired? Mention us or share this article on Mastodon, Twitter or LinkedIn.