badblocks/docker-easy-haproxy
1
0
Fork 0
Code Activity

Fix SSL on Kubernetes

Browse source
This commit is contained in:
Joao Gilberto Magalhaes 2022-08-26 11:38:02 -05:00
parent 0d16c117cb
commit 68eea939a3
8 changed files with 194 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

135
README.md
View file

@ -50,23 +50,20 @@ The mapping to `/var/run/docker.sock` is necessary to discover the docker contai
The environment variables will setup the HAProxy.
| Environment Variable | Description |
|-------------------------------|---------------------------------------------------------------------------------------------------------------|
| EASYHAPROXY_DISCOVER | How `haproxy.cfg` will be created: `static`, `docker`, `swarm` or `kubernetes` |
| EASYHAPROXY_LABEL_PREFIX | (Optional) The key will search for matching resources. Default: `easyhaproxy`. |
| EASYHAPROXY_LETSENCRYPT_EMAIL | (Optional) The email will be used to request the certificate to Letsencrypt |
| EASYHAPROXY_SSL_MODE | (Optional) `STRICT` supports only the most recent TLS version; `DEFAULT` good SSL integration with recent browsers; `LOOSE` supports all old SSL protocols for old browsers (not recommended). |
| EASYHAPROXY_REFRESH_CONF | (Optional) Check configuration every N seconds. Default: 10 |
| EASYHAPROXY_LOG_LEVEL | (Optional) The log level for EasyHAproxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL Default: TRACE |
| CERTBOT_LOG_LEVEL | (Optional) The log level for Certbot messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL Default: TRACE |
| HAPROXY_LOG_LEVEL | (Optional) The log level for HAProxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL Default: TRACE |
| HAPROXY_USERNAME | (Optional) The HAProxy username to the statistics. Default: `admin` |
| HAPROXY_PASSWORD | (Optional) The HAProxy password to the statistics. If not set, statistics will be available with no password |
| HAPROXY_STATS_PORT | (Optional) The HAProxy port to the statistics. Default: `1936`. If set to `false`, disable statistics |
| HAPROXY_CUSTOMERRORS | (Optional) If HAProxy will use custom HTML errors. true/false. Default: `false` |
| Environment Variable | Description | Default |
|-------------------------------|-------------------------------------------------------------------------------------------------|------------------|
| EASYHAPROXY_DISCOVER | How the services will be discovered to create `haproxy.cfg`: `static`, `docker`, `swarm` or `kubernetes` | **required** |
| EASYHAPROXY_LABEL_PREFIX | (Optional) The key will search for matching resources. | `easyhaproxy` |
| EASYHAPROXY_LETSENCRYPT_EMAIL | (Optional) The email will be used to request the certificate to Letsencrypt | *empty* |
| EASYHAPROXY_SSL_MODE | (Optional) `strict` supports only the most recent TLS version; `default` good SSL integration with recent browsers; `loose` supports all old SSL protocols for old browsers (not recommended). | `default`|
| EASYHAPROXY_REFRESH_CONF | (Optional) Check configuration every N seconds. | 10 |
| EASYHAPROXY_LOG_LEVEL | (Optional) The log level for EasyHAproxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL | DEBUG |
| CERTBOT_LOG_LEVEL | (Optional) The log level for Certbot messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL | DEBUG |
| HAPROXY_LOG_LEVEL | (Optional) The log level for HAProxy messages. Available: TRACE,DEBUG,INFO,WARN,ERROR,FATAL | DEBUG |
| HAPROXY_USERNAME | (Optional) The HAProxy username to the statistics. | `admin` |
| HAPROXY_PASSWORD | (Optional) The HAProxy password to the statistics. If not set, statistics will be available with no password | *empty* |
| HAPROXY_STATS_PORT | (Optional) The HAProxy port to the statistics. If set to `false`, disable statistics | `1936` |
| HAPROXY_CUSTOMERRORS | (Optional) If HAProxy will use custom HTML errors. true/false. | `false` |
The environment variable `EASYHAPROXY_DISCOVER` will define where is located your containers (see below for more details):
@ -98,7 +95,7 @@ docker run --network easyhaproxy myimage
or, if the container is already created you can join it using the command:
```
```bash
docker network connect easyhaproxy mycontainer
```
@ -116,7 +113,7 @@ Important: easyhaproxy needs to be in the same network of the containers or othe
This will query all `ingress` in the kubernetes cluster and check the annotation `kubernetes.io/ingress.class: easyhaproxy-ingress`.
e.g.:
```
```yaml
kind: Ingress
metadata:
annotations:
@ -145,36 +142,92 @@ spec:
pathType: ImplementationSpecific
```
At this point the implementation is very limited and doesn't support all ingress properties nor wildcard domains.
Caveats:
The system will read only `host` and `port.number`
There is no necessary to add labels or annotations.
- At this point, the implementation don't support all ingress properties nor wildcard domains.
- The ingress will publish externally only the ports 80 and 443, plus 1936 if stats is enable.
- The system will read `spec.rules[].host` and `spec.rules[].http.paths[0].port.number` and ignore the other parameters.
- Only the first path `spec.rules[].http.paths[0]` will be parsed.
- There are specific annotations can be added as described bellow.
### Kubernetes annotations:
| annotation | Description | Example |
|-----------------------------|-------------------------------------------------------------------------------------------------|--------------|
| kubernetes.io/ingress.class | (required) Activate EasyHAProxy. | easyhaproxy-ingress
| easyhaproxy.redirect_ssl | (optional) Boolean. Force redirect all endpoints to https. | true/false)
| easyhaproxy.letsencrypt | (optional) Boolean. It will request letsencript certificates for the ingresses domains | true/false
| easyhaproxy.redirect | (optional) Json. Specific a domain and its destination | {"domain":"redirect_url"}
| annotation | Description | Default | Example |
|-----------------------------|-----------------------------------------------------------------------------------------|--------------|--------------|
| kubernetes.io/ingress.class | (required) Activate EasyHAProxy. | **required** | easyhaproxy-ingress
| easyhaproxy.redirect_ssl | (optional) Boolean. Force redirect all endpoints to https. | false | true or false
| easyhaproxy.letsencrypt | (optional) Boolean. It will request letsencript certificates for the ingresses domains. | false | true or false
| easyhaproxy.redirect | (optional) Json. Specific a domain and its destination. | *empty* | {"domain":"redirect_url"}
| easyhaproxy.mode | (optional) Set the HTTP mode for that connection. | http | http or tcp
**Important**: The annotations are per ingress and applied to all hosts in that ingress configuration.
### Kubernetes and Letsencrypt:
It is necessary add the annotation `easyhaproxy.letsencrypt` to the ingress configuration:
```yaml
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: easyhaproxy-ingress
easyhaproxy.letsencrypt: 'true'
name: example-ingress
namespace: example
spec:
....
```
Make sure your cluster is accessible both through ports 80 and 443.
### Kubernetes and SSL:
You need to create a secret with your certificate and key, and associate them in your ingress.
```yaml
---
apiVersion: v1
kind: Secret
metadata:
name: host2-tls
namespace: default
data:
tls.crt: base64 of your certificate
tls.key: base64 of your certificate private key
type: kubernetes.io/tls
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: easyhaproxy-ingress
name: tls-example
namespace: default
spec:
tls:
- hosts:
- host2.local
secretName: host2-tls
rules:
...
```
### Container (Docker or Swarm) labels:
| Tag | Description | Example |
|---------------------------------------|---------------------------------------------------------------------------------------------------------|--------------|
| easyhaproxy.[definition].host | Host(s) HAProxy is listening. More than one host use comma as delimiter | somehost.com OR host1.com,host2.com |
| easyhaproxy.[definition].mode | (Optional) Is this `http` or `tcp` mode in HAProxy. (Defaults to http) | http |
| easyhaproxy.[definition].port | (Optional) Port HAProxy will listen for the host. (Defaults to 80) | 80 |
| easyhaproxy.[definition].localport | (Optional) Port container is listening. (Defaults to 80) | 8080 |
| easyhaproxy.[definition].redirect | (Optional) JSON containing key/value pair from host/to URL redirect. | {"foo.com":"https://bla.com", "bar.com":"https://bar.org"} |
| easyhaproxy.[definition].sslcert | (Optional) Cert PEM Base64 encoded. Do not use this if letsencrypt is enabled. | |
| easyhaproxy.[definition].ssl | (Optional) If `true` you need to provide certificate as a file. See below. Do not use with `sslcert`. | true |
| easyhaproxy.[definition].health-check | (Optional) `ssl`, enable health check via SSL in `mode tcp` (Defaults to "empty") | ssl |
| easyhaproxy.[definition].letsencrypt | (Optional) Generate certificate with letsencrypt. Do not use with `sslcert`. | true OR yes OR false OR no |
| easyhaproxy.[definition].redirect_ssl | (Optional) Redirect all requests to https | true OR yes OR false OR no |
| Tag | Description | Default | Example |
|---------------------------------------|-------------------------------------------------------------------------------------------------------|----------------|--------------|
| easyhaproxy.[definition].host | Host(s) HAProxy is listening. More than one host use comma as delimiter | **required** | somehost.com OR host1.com,host2.com |
| easyhaproxy.[definition].mode | (Optional) Is this `http` or `tcp` mode in HAProxy. | http | http or tcp |
| easyhaproxy.[definition].port | (Optional) Port HAProxy will listen for the host. | 80 | 3000 |
| easyhaproxy.[definition].localport | (Optional) Port container is listening. | 80 | 8080 |
| easyhaproxy.[definition].redirect | (Optional) JSON containing key/value pair from host/to URL redirect. | *empty* | {"foo.com":"https://bla.com", "bar.com":"https://bar.org"} |
| easyhaproxy.[definition].sslcert | (Optional) Cert PEM Base64 encoded. Do not use this if `letsencrypt` is enabled. | *empty* | base64 cert + key |
| easyhaproxy.[definition].ssl | (Optional) If `true` you need to provide certificate as a file. See below. Do not use with `sslcert`. | false | true or false |
| easyhaproxy.[definition].health-check | (Optional) `ssl`, enable health check via SSL in `mode tcp` | *empty* | ssl |
| easyhaproxy.[definition].letsencrypt | (Optional) Generate certificate with letsencrypt. Do not use with `sslcert` parameter. | false | true OR false |
| easyhaproxy.[definition].redirect_ssl | (Optional) Redirect all requests to https | false | true OR false |
| easyhaproxy.[definition].clone_to_ssl | (Optional) It copies the configuration to HTTPS(443) and disable SSL from the current config. **Do not use* this with `ssl` or `letsencrypt` parameters | false | true OR false |
### Defining the labels in Docker Swarm

5
examples/kubernetes/easyhaproxy.yml
View file

@ -19,7 +19,6 @@ spec:
labels:
app: easyhaproxy-ingress
spec:
#serviceAccountName: easyhaproxy-ingress
containers:
- image: byjg/easy-haproxy:test
imagePullPolicy: Always
@ -32,8 +31,8 @@ spec:
containerPort: 443
hostPort: 443
- name: stats
containerPort: 1943
hostPort: 1943
containerPort: 1936
hostPort: 1936
resources:
requests:
cpu: "100m"

11
src/easymapping/__init__.py
View file

@ -110,6 +110,9 @@ class HaproxyConfigGenerator:
self.label.create([definition, "letsencrypt"]),
False
) and self.mapping["letsencrypt"]["email"] != ""
clone_to_ssl = self.label.get_bool(
self.label.create([definition, "clone_to_ssl"])
)
if port not in easymapping:
easymapping[port] = {
@ -147,7 +150,7 @@ class HaproxyConfigGenerator:
self.label.create([definition, "redirect"])
)
if letsencrypt:
if letsencrypt or clone_to_ssl:
if "443" not in easymapping:
easymapping["443"] = {
"mode": "http",
@ -160,17 +163,17 @@ class HaproxyConfigGenerator:
easymapping["443"]["hosts"][hostname]["letsencrypt"] = False
easymapping["443"]["hosts"][hostname]["redirect_ssl"] = False
easymapping["443"]["ssl"] = True
self.letsencrypt_hosts.append(hostname) if hostname not in self.letsencrypt_hosts else self.letsencrypt_hosts
self.letsencrypt_hosts.append(hostname) if letsencrypt and hostname not in self.letsencrypt_hosts else self.letsencrypt_hosts
# handle SSL
ssl_label = self.label.create([definition, "sslcert"])
if self.label.has_label(ssl_label):
filename = "{}.pem".format(d[host_label])
easymapping[port]["ssl"] = True
easymapping[port]["ssl"] = True if not clone_to_ssl else False
self.certs[filename] = base64.b64decode(d[ssl_label]).decode('ascii')
if self.label.get_bool(self.label.create([definition, "ssl"])):
easymapping[port]["ssl"] = True
easymapping[port]["ssl"] = True if not clone_to_ssl else False
return easymapping.values()

4
src/functions/__init__.py
View file

@ -31,8 +31,8 @@ class Functions:
Functions.ERROR: 4,
Functions.FATAL: 5
}
level_required = 0 if level not in level_importance else level_importance[level]
level_asked = 0 if log_level_str.upper() not in level_importance else level_importance[log_level_str.upper()]
level_required = 1 if level not in level_importance else level_importance[level]
level_asked = 1 if log_level_str.upper() not in level_importance else level_importance[log_level_str.upper()]
return level_asked < level_required
@staticmethod

13
src/processor/__init__.py
View file

@ -14,7 +14,7 @@ class ContainerEnv:
def read():
env_vars = {
"customerrors": True if os.getenv("HAPROXY_CUSTOMERRORS") == "true" else False,
"ssl_mode": os.getenv("EASYHAPROXY_SSL_MODE") if os.getenv("EASYHAPROXY_SSL_MODE") else 'default'
"ssl_mode": os.getenv("EASYHAPROXY_SSL_MODE").lower() if os.getenv("EASYHAPROXY_SSL_MODE") else 'default'
}
if os.getenv("HAPROXY_PASSWORD"):
@ -157,6 +157,7 @@ class Kubernetes(ProcessorInterface):
letsencrypt = self._check_annotation(ingress.metadata.annotations, "easyhaproxy.letsencrypt")
redirect_ssl = self._check_annotation(ingress.metadata.annotations, "easyhaproxy.redirect_ssl")
redirect = self._check_annotation(ingress.metadata.annotations, "easyhaproxy.redirect")
mode = self._check_annotation(ingress.metadata.annotations, "easyhaproxy.mode")
data = {}
data["creation_timestamp"] = ingress.metadata.creation_timestamp.strftime("%x %X")
@ -193,15 +194,17 @@ class Kubernetes(ProcessorInterface):
rule_data["%s.port" % (definition)] = "80"
rule_data["%s.localport" % (definition)] = port_number
if rule.host in ssl_hosts:
rule_data["%s.ssl" % (definition)] = 'true'
rule_data["%s.clone_to_ssl" % (definition)] = 'true'
if redirect_ssl is not None:
rule_data["%s.redirect_ssl" % (definition)] = 'true'
rule_data["%s.redirect_ssl" % (definition)] = redirect_ssl
if letsencrypt is not None:
rule_data["%s.letsencrypt" % (definition)] = 'true'
rule_data["%s.letsencrypt" % (definition)] = letsencrypt
if redirect is not None:
rule_data["%s.redirect" % (definition)] = redirect
if mode is not None:
rule_data["%s.mode" % (definition)] = mode
service_name = rule.http.paths[0].backend.service.ingress_name
service_name = rule.http.paths[0].backend.service.name
try:
api_response = self.api_instance.read_namespaced_service(service_name, ingress.metadata.namespace)
cluster_ip = api_response.spec.cluster_ip

2
src/tests/fixtures/services-clone-to-ssl vendored Normal file
View file

@ -0,0 +1,2 @@
{"10.152.183.62": {"creation_timestamp": "08/24/22 02:59:44", "resource_version": "72517156", "namespace": "parking", "easyhaproxy.valida-me_8080.host": "valida.me", "easyhaproxy.valida-me_8080.port": "80", "easyhaproxy.valida-me_8080.localport": 8080, "easyhaproxy.valida-me_8080.redirect": "{\"www.valida.me\": \"https://valida.me\"}", "easyhaproxy.www-valida-me_8080.host": "www.valida.me", "easyhaproxy.www-valida-me_8080.port": "80", "easyhaproxy.www-valida-me_8080.localport": 8080, "easyhaproxy.www-valida-me_8080.redirect": "{\"www.valida.me\": \"https://valida.me\"}"},
"10.152.183.215": {"creation_timestamp": "08/26/22 03:06:01", "resource_version": "72522999", "namespace": "default", "easyhaproxy.host2-local_8080.host": "host2.local", "easyhaproxy.host2-local_8080.port": "80", "easyhaproxy.host2-local_8080.localport": 8080, "easyhaproxy.host2-local_8080.clone_to_ssl": "true"}}

2
src/tests/test_containerenv.py
View file

@ -23,7 +23,7 @@ def test_container_env_customerrors():
os.environ['HAPROXY_CUSTOMERRORS'] = ''
def test_container_env_sslmode():
os.environ['EASYHAPROXY_SSL_MODE'] = 'strict'
os.environ['EASYHAPROXY_SSL_MODE'] = 'STRICT'
try:
assert {
"customerrors": False,

77
src/tests/test_parser.py
View file

@ -423,6 +423,83 @@ def test_parser_ssl_letsencrypt():
assert expected_file.read() == haproxy_config
assert ["test.example.org"] == cfg.letsencrypt_hosts
def test_parser_finds_services_clone_to_ssl_raw():
line_list = load_fixture("services-clone-to-ssl")
result = {
"customerrors": False,
"letsencrypt": {
"email": LETSENCRYPT_EMAIL
},
"stats": {
"port": 0
}
}
if os.path.exists(CERT_FILE):
os.remove(CERT_FILE)
cfg = easymapping.HaproxyConfigGenerator(result)
parsed_object = [
{
"health-check":"",
"hosts":{
"host2.local":{
"containers":[
"10.152.183.215:8080"
],
"letsencrypt": False,
"redirect_ssl": False
},
"valida.me":{
"containers":[
"10.152.183.62:8080"
],
"letsencrypt": False,
"redirect_ssl": False
},
"www.valida.me":{
"containers":[
"10.152.183.62:8080"
],
"letsencrypt": False,
"redirect_ssl": False
}
},
"mode":"http",
"port":"80",
"redirect":{
}
},
{
"health-check":"ssl",
"hosts":{
"host2.local":{
"containers":[
"10.152.183.215:8080"
],
"letsencrypt": False,
"redirect_ssl": False
}
},
"mode":"http",
"port":"443",
"redirect":{
},
"ssl": True
}
]
processed = list(cfg.parse(line_list))
assert parsed_object == processed
assert [] == cfg.letsencrypt_hosts
#test_parser_finds_services_raw()
#test_parser_tcp()
#test_parser_multiple_hosts()