Easy HAProxy

Service discovery for HAProxy

This Docker image will dynamically create the haproxy.cfg based on the labels defined in docker containers or from a simple Yaml.

EasyHAProxy can detect and configure automatically HAProxy on the folowing platforms:

• Docker
• Docker Swarm
• Kubernetes

Features

EasyHAProxy will discover the services based on the Docker Tags of the running containers in a Docker host or Docker Swarm cluster and dynamically set up the haproxy.cfg. Below, EasyHAProxy main features:

• Use Letsencrypt with HAProxy.
• Set your custom SSL certificates
• Balance traffic between multiple replicas
• Set SSL with three different levels of validations and according to the most recent definitions.
• Setup HAProxy to listen to TCP.
• Add redirects.
• Enable/disable Stats on port 1936 with a custom password.
• Enable/disable custom errors.

Also, it is possible to set up HAProxy from a simple Yaml file instead of creating haproxy.cfg file.

Basic Usage

The Easy HAProxy will automatically create the haproxy.cfg file based on the containers or a YAML provided.

The basic command line to run is:

docker run -d \
    --name easy-haproxy-container \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e EASYHAPROXY_DISCOVER="swarm|docker|static" \
    # + Environment Variables \
    # + ports mapped to the host \
    byjg/easy-haproxy

The mapping to /var/run/docker.sock is necessary to discover the docker containers and get the labels;

The environment variables will setup the HAProxy.

Environment Variable	Description	Default
EASYHAPROXY_DISCOVER	How the services will be discovered to create haproxy.cfg: static, docker, swarm or kubernetes	required
EASYHAPROXY_LABEL_PREFIX	(Optional) The key will search for matching resources.	easyhaproxy
EASYHAPROXY_LETSENCRYPT_EMAIL	(Optional) The email will be used to request the certificate to Letsencrypt	empty
EASYHAPROXY_SSL_MODE	(Optional) strict supports only the most recent TLS version; default good SSL integration with recent browsers; loose supports all old SSL protocols for old browsers (not recommended).	default
EASYHAPROXY_REFRESH_CONF	(Optional) Check configuration every N seconds.	10
EASYHAPROXY_LOG_LEVEL	(Optional) The log level for EasyHAproxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL	DEBUG
CERTBOT_LOG_LEVEL	(Optional) The log level for Certbot messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL	DEBUG
HAPROXY_LOG_LEVEL	(Optional) The log level for HAProxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL	DEBUG
HAPROXY_USERNAME	(Optional) The HAProxy username to the statistics.	admin
HAPROXY_PASSWORD	(Optional) The HAProxy password to the statistics. If not set, statistics will be available with no password	empty
HAPROXY_STATS_PORT	(Optional) The HAProxy port to the statistics. If set to false, disable statistics	1936
HAPROXY_CUSTOMERRORS	(Optional) If HAProxy will use custom HTML errors. true/false.	false

The environment variable EASYHAPROXY_DISCOVER will define where is located your containers (see below for more details):

• docker
• swarm
• static

Automatic Discover Services

Easy HAProxy can automatically discover the container services running in the same network of Docker or in a Docker Swarm cluster.

EASYHAPROXY_DISCOVER: docker

This method will use a standard docker installation to discover the containers and configure the HAProxy.

The only requirement is that containers and easy-haproxy must be in the same docker network.

The discovery will occur every minute.

e.g.:

docker create network easyhaproxy

docker run --network easyhaproxy byjg/easyhaproxy

docker run --network easyhaproxy myimage

or, if the container is already created you can join it using the command:

docker network connect easyhaproxy mycontainer

EASYHAPROXY_DISCOVER: swarm

This method requires a functional Docker Swarm Cluster. The system will search for the labels in all containers on all swarm nodes.

The discovery will occur every minute.

Important: easyhaproxy needs to be in the same network of the containers or otherwise will not access.

EASYHAPROXY_DISCOVER: kubernetes (experimental and limited)

This will query all ingress in the kubernetes cluster and check the annotation kubernetes.io/ingress.class: easyhaproxy-ingress.

e.g.:

kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: easyhaproxy-ingress
  name: example-ingress
  namespace: example
spec:
  rules:
  - host: example.org
    http:
      paths:
      - backend:
          service:
            name: example-service
            port:
              number: 8080
        pathType: ImplementationSpecific
  - host: www.example.org
    http:
      paths:
      - backend:
          service:
            name: example-service
            port:
              number: 8080
        pathType: ImplementationSpecific

Caveats:

• At this point, the implementation don't support all ingress properties nor wildcard domains.
• The ingress will publish externally only the ports 80 and 443, plus 1936 if stats is enable.
• The system will read spec.rules[].host and spec.rules[].http.paths[0].port.number and ignore the other parameters.
• Only the first path spec.rules[].http.paths[0] will be parsed.
• There are specific annotations can be added as described bellow.

Kubernetes annotations

annotation	Description	Default	Example
kubernetes.io/ingress.class	(required) Activate EasyHAProxy.	required	easyhaproxy-ingress
easyhaproxy.redirect_ssl	(optional) Boolean. Force redirect all endpoints to https.	false	true or false
easyhaproxy.letsencrypt	(optional) Boolean. It will request letsencript certificates for the ingresses domains.	false	true or false
easyhaproxy.redirect	(optional) Json. Specific a domain and its destination.	empty	{"domain":"redirect_url"}
easyhaproxy.mode	(optional) Set the HTTP mode for that connection.	http	http or tcp

Important: The annotations are per ingress and applied to all hosts in that ingress configuration.

Kubernetes and Letsencrypt:

It is necessary add the annotation easyhaproxy.letsencrypt to the ingress configuration:

kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: easyhaproxy-ingress
    easyhaproxy.letsencrypt: 'true'
  name: example-ingress
  namespace: example
spec:
  ....

Make sure your cluster is accessible both through ports 80 and 443.

Kubernetes and SSL

You need to create a secret with your certificate and key, and associate them in your ingress.

---
apiVersion: v1
kind: Secret
metadata:
  name: host2-tls
  namespace: default
data:
  tls.crt: base64 of your certificate
  tls.key: base64 of your certificate private key
type: kubernetes.io/tls

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: easyhaproxy-ingress
  name: tls-example
  namespace: default
spec:
  tls:
  - hosts:
      - host2.local
    secretName: host2-tls
  rules:
    ...

Container (Docker or Swarm) labels

Tag	Description	Default	Example
easyhaproxy.[definition].host	Host(s) HAProxy is listening. More than one host use comma as delimiter	required	somehost.com OR host1.com,host2.com
easyhaproxy.[definition].mode	(Optional) Is this http or tcp mode in HAProxy.	http	http or tcp
easyhaproxy.[definition].port	(Optional) Port HAProxy will listen for the host.	80	3000
easyhaproxy.[definition].localport	(Optional) Port container is listening.	80	8080
easyhaproxy.[definition].redirect	(Optional) JSON containing key/value pair from host/to URL redirect.	empty	{"foo.com":"https://bla.com", "bar.com":"https://bar.org"}
easyhaproxy.[definition].sslcert	(Optional) Cert PEM Base64 encoded. Do not use this if letsencrypt is enabled.	empty	base64 cert + key
easyhaproxy.[definition].ssl	(Optional) If true you need to provide certificate as a file. See below. Do not use with sslcert.	false	true or false
easyhaproxy.[definition].health-check	(Optional) ssl, enable health check via SSL in mode tcp	empty	ssl
easyhaproxy.[definition].letsencrypt	(Optional) Generate certificate with letsencrypt. Do not use with sslcert parameter.	false	true OR false
easyhaproxy.[definition].redirect_ssl	(Optional) Redirect all requests to https	false	true OR false
easyhaproxy.[definition].clone_to_ssl	(Optional) It copies the configuration to HTTPS(443) and disable SSL from the current config. *Do not use this with ssl or letsencrypt parameters	false	true OR false

Defining the labels in Docker Swarm

if you are deploying a stack in a Docker Swarm cluster, set labels at the deploy level:

services:
 foo:
   deploy:
      labels:
         easyhaproxy.my.host: "www.example.org"
         easyhaproxy.my.localport: 8080
         ...

docker stack deploy --compose-file docker-compose.yml mystack

Single Definition

docker run \
    -l easyhaproxy.webapi.port=80\
    -l easyhaproxy.webapi.host=byjg.com.br \
    ....

Multiples Definitions on the same container

docker run \
    -l easyhaproxy.express.port=80 \
    -l easyhaproxy.express.localport=3000 \
    -l easyhaproxy.express.host=express.byjg.com.br \

    -l easyhaproxy.admin.port=80 \
    -l easyhaproxy.admin.localport=3001 \
    -l easyhaproxy.admin.host=admin.byjg.com.br \
    .... \
    some/myimage

Multiples hosts on the same container

docker run \
    -l easyhaproxy.express.port=80 \
    -l easyhaproxy.express.localport=3000 \
    -l easyhaproxy.express.host=express.byjg.com.br,admin.byjg.com.br \
    .... \
    some/myimage

If you are using docker-compose you can use this way:

version: "3"

services:
  mycontainer:
    image: some/myimage
    labels:
      easyhaproxy.express.port: 80
      easyhaproxy.express.localport: 3000
      easyhaproxy.express.host: >-
        express.byjg.com.br,
        admin.byjg.com.br

TLS passthrough

Used to pass on SSL termination to a backend. Alternatively, you can enable health-check via SSL on the backend with the optional health-check label:

docker run \
    -l easyhaproxy.example.mode=tcp \
    -l easyhaproxy.example.health-check=ssl \
    -l easyhaproxy.example.port=443
    .... \
    some/tcp-service

Redirect Example

docker run \
    -l easyhaproxy.[definition].redirect='{"www.byjg.com.br":"http://byjg.com.br","byjg.com":"http://byjg.com.br"}'

EASYHAPROXY_DISCOVER: static

This method expects a YAML file to setup the haproxy.cfg

Create a YAML file and map to /etc/haproxy/easyconfig.yml

stats:
  username: admin
  password: password
  port: 1936         # Optional (default 1936)

customerrors: true   # Optional (default false)

ssl_mode: default

letsencrypt: {
  "email": "acme@example.org"
}

easymapping:
  - port: 80
    hosts:
      host1.com.br: 
        containers:
          - container:5000
        letsencrypt: true
        redirect_ssl: true
      host2.com.br: 
        containers:
          - other:3000
    redirect:
      www.host1.com.br: http://host1.com.br

  - port: 443
    hosts:
      host1.com.br: 
        containers:
          - container:80
        redirect_ssl: false
      ssl: true

  - port: 8080
    hosts:
      host3.com.br: 
        containers: 
          - domain:8181

Running:

docker run -v /my/config.yml:/etc/haproxy/easyconfig.yml .... byjg/easyhaproxy

Letsencrypt

This HAProxy can issue a letsencrypt certificate. The command is as below:

Run the EasyHAProxy:

docker run \
    -e EASYHAPROXY_LETSENCRYPT_EMAIL=john@doe.com
    .... \
    byjg/easy-haproxy

Run your container:

docker run \
    -l easyhaproxy.express.port=80 \
    -l easyhaproxy.express.localport=3000 \
    -l easyhaproxy.express.host=example.org \
    -l easyhaproxy.express.letsencrypt=true \
    .... \
    some/myimage

Caveats:

• Your container must listen to the port 80. Besides no error, the certificate won't be issued if in a different port.
• The port 2080 is reserved for the certbot and should not be exposed.
• You cannot set the port 443 for the container with the Letsencrypt because EasyHAProxy will handle this automatically once the certificate is issued.
• If you don't run the EasyHAProxy with the parameter EASYHAPROXY_LETSENCRYPT_EMAIL no certificate will be issued.
• Be aware of Letsencrypt issue limits - https://letsencrypt.org/docs/duplicate-certificate-limit/ and https://letsencrypt.org/docs/rate-limits/

Exposing Ports

You must expose some ports on the EasyHAProxy container and in the firewall. However, you don't need to expose the other container ports because EasyHAProxy will handle that.

• The ports 80 and 443.
• If you enable the HAProxy statistics, you must also expose the port defined in HAPROXY_STATS_PORT environment variable (default 1936). Be aware that statististics are enabled by default with no password.
• Every port defined in easyhaproxy.[definitions].port also should be exposed.

e.g.

docker run \
    /* other parameters */
    -p 80:80 \
    -p 443:443 \
    -p 1936:1936 \
    -d byjg/easy-haproxy

Mapping custom .cfg files

You can concatenate valid HAProxy .cfg files to the dynamically generated haproxy.cfg by mapping the folder /etc/haproxy/conf.d.

docker run \
    /* other parameters */
    -v /your/local/conf.d:/etc/haproxy/conf.d \
    -d byjg/easy-haproxy

Mapping SSL certificates volumes

EasyHAProxy stores the certificates inside the folder /certs/haproxy and /certs/letsencrypt.

• If you want to preserve the letsencrypt certificates between reloads, map the folder /certs/letsencrypt to your volume.
• If you want to provide your certificates as a file instead of a Base64 parameter, map the folder /certs/haproxy to your volume, and instead of use easyhaproxy.[definition].sslcert, use easyhaproxy.[definition].ssl: true

docker run \
    /* other parameters */
    -v /your/certs/letsencrypt:/certs/letsencrypt \
    -d byjg/easy-haproxy

Handling SSL

You can attach a valid SSL certificate to the request.

1. First, Create a single PEM file including CA.

cat example.com.crt example.com.key > single.pem

cat single.pem

-----BEGIN CERTIFICATE-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC5ZheHqmBnEJP+
U9r1gxYWKLzdqrMrcxtQN6M1hIH9n0peuJeIrybdcV7sMbStMXI=
-----END CERTIFICATE-----

-----BEGIN PRIVATE KEY-----
MIIEojCCA4qgAwIBAgIUegW2BimwuL4RzRZ2WYkHA6U5nkAwDQYJKoZIhvcNAQEL
3j4wz8/I5fdsk090j4s5KA==
-----END PRIVATE KEY-----

2. Convert it to BASE64 in a single line:

cat single.pem | base64 -w0

3. Use this string to define the label easyhaproxy.[definition].sslcert

Setting Custom Errors

If enabled, map the volume : /etc/haproxy/errors-custom/ to your container and put a file named ERROR_NUMBER.http where ERROR_NUMBER is the HTTP error code (e.g., 503.http)

Build

docker build -t byjg/easy-haproxy .

Limitations

EasyHAProxy has some limitations when there is more than one easy-haproxy container running:

• Replicas can be out-of-sync for a few seconds because each replica will discover the pods separately.
• Each replica will request a Letsencrypt certificate and can fail because the letsencrypt challenge can be directed to the other replica.

---

Open source ByJG