514 lines
16 KiB
Markdown
514 lines
16 KiB
Markdown
---
|
|
sidebar_position: 1
|
|
---
|
|
|
|
# Kubernetes
|
|
|
|
## Setup Kubernetes EasyHAProxy
|
|
|
|
:::info How it works
|
|
EasyHAProxy for Kubernetes operates by querying all ingress definitions with either the
|
|
`spec.ingressClassName: easyhaproxy` field (recommended) or the deprecated annotation
|
|
`kubernetes.io/ingress.class: easyhaproxy-ingress` (for backward compatibility). Upon finding
|
|
a matching ingress class, EasyHAProxy immediately sets up HAProxy and begins serving traffic.
|
|
:::
|
|
|
|
For Kubernetes installations, there are three available installation modes:
|
|
- DaemonSet: This mode exposes ports 80, 443, and 1936.
|
|
- NodePort: Ports 31080, 31443, and 31936 are exposed.
|
|
- ClusterIP: In this mode, no ports are exposed externally, and HAProxy is accessible only
|
|
within the cluster.
|
|
|
|
To install EasyHAProxy in your Kubernetes cluster, follow these steps:
|
|
|
|
|
|
### 1) Identify the node where your EasyHAProxy container will run
|
|
|
|
:::warning Single Node Deployment
|
|
EasyHAProxy will be limited to a single node. To understand why, see the [limitations](limitations.md) page.
|
|
:::
|
|
|
|
```bash title="List available nodes"
|
|
$ kubectl get nodes
|
|
|
|
NAME STATUS ROLES AGE VERSION
|
|
node-01 Ready <none> 561d v1.21.13-3
|
|
node-02 Ready <none> 561d v1.21.13-3
|
|
```
|
|
|
|
Add the EasyHAProxy label to the node.
|
|
|
|
```bash title="Label the node for EasyHAProxy"
|
|
kubectl label nodes node-01 "easyhaproxy/node=master"
|
|
```
|
|
|
|
### 2) Install EasyHAProxy with Kubernetes Manifest
|
|
|
|
```bash title="Install EasyHAProxy"
|
|
kubectl create namespace easyhaproxy
|
|
|
|
kubectl apply -f \
|
|
https://raw.githubusercontent.com/byjg/docker-easy-haproxy/6.0.0/deploy/kubernetes/easyhaproxy-daemonset.yml
|
|
```
|
|
|
|
If necessary, you can configure environment variables. To get a list of the variables, please follow the [environment variable guide](environment-variable.md)
|
|
|
|
## Running containers
|
|
|
|
Your container only requires creating an ingress with the `spec.ingressClassName: easyhaproxy` field pointing to your service.
|
|
|
|
e.g.
|
|
|
|
```yaml
|
|
kind: Ingress
|
|
metadata:
|
|
name: example-ingress
|
|
namespace: example
|
|
spec:
|
|
# Use ingressClassName (recommended)
|
|
ingressClassName: easyhaproxy
|
|
rules:
|
|
- host: example.org
|
|
http:
|
|
paths:
|
|
- backend:
|
|
service:
|
|
name: example-service
|
|
port:
|
|
number: 8080
|
|
pathType: ImplementationSpecific
|
|
```
|
|
|
|
:::note Backward Compatibility
|
|
The deprecated annotation `kubernetes.io/ingress.class: easyhaproxy-ingress` is still supported for backward compatibility, but `spec.ingressClassName` is the recommended approach for new deployments.
|
|
:::
|
|
|
|
Once the container is running, EasyHAProxy will detect automatically and start to redirect all traffic from `example.org:80` to your container at port 8080.
|
|
|
|
You don't need to expose any port in your container.
|
|
|
|
:::note Important Limitations
|
|
- The implementation doesn't support all ingress properties or wildcard domains at this time.
|
|
- The ingress will publish ports 80 and 443, plus 1936 if stats are enabled.
|
|
- EasyHAProxy will read all `spec.rules[].host` specifications, however it will parse only the **first path** `spec.rules[].http.paths[0].port.number` for each rule, and ignore the other paths.
|
|
:::
|
|
|
|
## Kubernetes annotations
|
|
|
|
| annotation | Description | Default | Example |
|
|
|-------------------------------------|-------------------------------------------------------------------------------------|--------------|----------------------------|
|
|
| kubernetes.io/ingress.class | (deprecated) Activate EasyHAProxy. Use `spec.ingressClassName` instead. | *optional* | easyhaproxy-ingress |
|
|
| easyhaproxy.redirect_ssl | (optional) Boolean. Force redirect all endpoints to HTTPS. | false | true or false |
|
|
| easyhaproxy.certbot | (optional) Boolean. It will request certbot certificates for the ingresses domains. | false | true or false |
|
|
| easyhaproxy.redirect | (optional) JSON. Key pair with a domain and its destination. | *empty* | \{"domain":"redirect_url"} |
|
|
| easyhaproxy.mode | (optional) Set the HTTP mode for that connection. | http | http or tcp |
|
|
| easyhaproxy.listen_port | (optional) Override the HTTP listen port created for that ingress | 80 | 8081 |
|
|
| easyhaproxy.plugins | (optional) Comma-separated list of plugins to enable for this ingress | *empty* | cloudflare,deny_pages |
|
|
| easyhaproxy.plugin.`{name}`.`{key}` | (optional) Plugin-specific configuration (see [Using Plugins](plugins.md)) | *varies* | See examples below |
|
|
|
|
**Important**: The annotations are per ingress and applied to all hosts in that ingress configuration.
|
|
|
|
## Using Plugins with Kubernetes
|
|
|
|
Plugins extend HAProxy configuration with additional functionality like JWT validation, IP whitelisting, or Cloudflare IP restoration. For a complete list of available plugins, see the [Using Plugins](plugins.md) guide.
|
|
|
|
### Enabling Plugins for an Ingress
|
|
|
|
Add the `easyhaproxy.plugins` annotation with a comma-separated list of plugin names:
|
|
|
|
```yaml
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "cloudflare,deny_pages"
|
|
name: example-ingress
|
|
namespace: example
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
rules:
|
|
- host: example.org
|
|
http:
|
|
paths:
|
|
- backend:
|
|
service:
|
|
name: example-service
|
|
port:
|
|
number: 8080
|
|
pathType: ImplementationSpecific
|
|
```
|
|
|
|
### Configuring Plugin Options
|
|
|
|
Use `easyhaproxy.plugin.{plugin_name}.{option}` annotations to configure individual plugins:
|
|
|
|
```yaml
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "deny_pages"
|
|
easyhaproxy.plugin.deny_pages.paths: "/admin,/private,/config"
|
|
easyhaproxy.plugin.deny_pages.status_code: "403"
|
|
name: secure-app-ingress
|
|
namespace: production
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
rules:
|
|
- host: myapp.example.com
|
|
http:
|
|
paths:
|
|
- backend:
|
|
service:
|
|
name: myapp-service
|
|
port:
|
|
number: 8080
|
|
pathType: ImplementationSpecific
|
|
```
|
|
|
|
### Common Plugin Examples
|
|
|
|
**Protect API with JWT validation:**
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "jwt_validator"
|
|
easyhaproxy.plugin.jwt_validator.algorithm: "RS256"
|
|
easyhaproxy.plugin.jwt_validator.issuer: "https://auth.example.com/"
|
|
easyhaproxy.plugin.jwt_validator.audience: "https://api.example.com"
|
|
easyhaproxy.plugin.jwt_validator.pubkey_path: "/etc/easyhaproxy/jwt_keys/api_pubkey.pem"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
```
|
|
|
|
**Note:** For JWT validation, you'll need to mount the public key file into the EasyHAProxy pod. See [Using Plugins](plugins.md#protect-api-with-jwt-authentication) for details.
|
|
|
|
**Restrict access to specific IPs:**
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "ip_whitelist"
|
|
easyhaproxy.plugin.ip_whitelist.allowed_ips: "192.168.1.0/24,10.0.0.5"
|
|
easyhaproxy.plugin.ip_whitelist.status_code: "403"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
```
|
|
|
|
**Restore Cloudflare visitor IPs:**
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "cloudflare"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
```
|
|
|
|
**Multiple plugins together:**
|
|
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.plugins: "cloudflare,deny_pages"
|
|
easyhaproxy.plugin.deny_pages.paths: "/wp-admin,/wp-login.php"
|
|
easyhaproxy.plugin.deny_pages.status_code: "404"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
```
|
|
|
|
### Global Plugin Configuration
|
|
|
|
Some plugins (like `cleanup`) are global and execute once per discovery cycle. Configure these via environment variables or YAML configuration:
|
|
|
|
**Using Helm values.yaml:**
|
|
|
|
```yaml
|
|
easyhaproxy:
|
|
plugins:
|
|
enabled: cleanup
|
|
config:
|
|
cleanup:
|
|
max_idle_time: 600
|
|
```
|
|
|
|
**Using environment variables:**
|
|
|
|
```yaml
|
|
env:
|
|
- name: EASYHAPROXY_PLUGINS_ENABLED
|
|
value: "cleanup"
|
|
- name: EASYHAPROXY_PLUGIN_CLEANUP_MAX_IDLE_TIME
|
|
value: "600"
|
|
```
|
|
|
|
For more information on plugin types and available plugins, see the [Using Plugins](plugins.md) guide.
|
|
|
|
## Loading Plugin Configuration from Kubernetes Secrets
|
|
|
|
EasyHAProxy supports loading sensitive plugin configuration values directly from Kubernetes Secrets using the `k8s_secret` pattern. This is a **generic, plugin-agnostic feature** that works with any plugin.
|
|
|
|
### Why Use Kubernetes Secrets?
|
|
|
|
- **Security**: Keep sensitive data (API keys, passwords, certificates) out of annotations
|
|
- **Best practices**: Follows Kubernetes conventions for managing sensitive data
|
|
- **Simplicity**: No need to mount volumes or ConfigMaps for secret data
|
|
- **Encryption**: Secrets are encrypted at rest in etcd
|
|
|
|
### Annotation Format
|
|
|
|
```yaml
|
|
# Auto-detect key (tries common variations):
|
|
easyhaproxy.plugin.{plugin_name}.k8s_secret.{config_key}: "secret_name"
|
|
|
|
# Explicit key (no variations):
|
|
easyhaproxy.plugin.{plugin_name}.k8s_secret.{config_key}: "secret_name/key_name"
|
|
```
|
|
|
|
### How It Works
|
|
|
|
1. **You create** a Kubernetes Secret with your sensitive data
|
|
2. **You reference** the secret in your ingress annotation using the `k8s_secret` pattern
|
|
3. **EasyHAProxy reads** the secret from the same namespace as the ingress
|
|
4. **EasyHAProxy transforms** the annotation to inject the secret value
|
|
5. **The plugin receives** the value as if it was provided directly in the annotation
|
|
|
|
**Example transformation:**
|
|
|
|
```yaml
|
|
# Input annotation:
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "my-jwt-secret"
|
|
|
|
# EasyHAProxy reads the secret and transforms to:
|
|
easyhaproxy.plugin.jwt_validator.pubkey: "<base64-encoded-content>"
|
|
```
|
|
|
|
### Auto-Detect vs Explicit Key
|
|
|
|
#### Auto-Detect Key Format
|
|
|
|
When you use `"secret_name"` (without `/`), EasyHAProxy tries to find the key automatically:
|
|
|
|
```yaml
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "my-jwt-secret"
|
|
```
|
|
|
|
EasyHAProxy will try these keys in order:
|
|
1. Exact match: `pubkey`
|
|
2. Common variations based on the config key name
|
|
|
|
**Auto-detect key variations:**
|
|
|
|
| Config Key | Tries (in order) |
|
|
|------------|---------------------------------------|
|
|
| `pubkey` | `pubkey`, `public-key`, `jwt.pub`, `tls.crt` |
|
|
| `password` | `password`, `pass`, `pwd` |
|
|
| `api_key` | `api_key`, `apikey`, `api-key`, `key` |
|
|
|
|
#### Explicit Key Format
|
|
|
|
When you use `"secret_name/key_name"` (with `/`), EasyHAProxy only tries the exact key name:
|
|
|
|
```yaml
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "my-jwt-secret/rsa-public-key"
|
|
```
|
|
|
|
EasyHAProxy will **only** try: `rsa-public-key` (no variations)
|
|
|
|
**Use explicit key when:**
|
|
- Your secret uses a non-standard key name
|
|
- You want to be explicit and avoid ambiguity
|
|
- Multiple keys exist in the secret
|
|
|
|
### Complete Example
|
|
|
|
```yaml
|
|
---
|
|
# 1. Create a secret with your JWT public key
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: jwt-pubkey-secret
|
|
namespace: production
|
|
type: Opaque
|
|
stringData:
|
|
pubkey: |
|
|
-----BEGIN PUBLIC KEY-----
|
|
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
|
|
-----END PUBLIC KEY-----
|
|
|
|
---
|
|
# 2. Reference it in your ingress
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: api-ingress
|
|
namespace: production
|
|
annotations:
|
|
easyhaproxy.plugins: "jwt_validator"
|
|
easyhaproxy.plugin.jwt_validator.algorithm: "RS256"
|
|
easyhaproxy.plugin.jwt_validator.issuer: "https://auth.example.com/"
|
|
easyhaproxy.plugin.jwt_validator.audience: "https://api.example.com"
|
|
# Load pubkey from Kubernetes secret (auto-detect key)
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "jwt-pubkey-secret"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
rules:
|
|
- host: api.example.com
|
|
http:
|
|
paths:
|
|
- path: /
|
|
pathType: Prefix
|
|
backend:
|
|
service:
|
|
name: api-service
|
|
port:
|
|
number: 8080
|
|
```
|
|
|
|
### Example with Explicit Key Name
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: app-credentials
|
|
namespace: production
|
|
type: Opaque
|
|
stringData:
|
|
# Custom key name
|
|
rsa-public-key: |
|
|
-----BEGIN PUBLIC KEY-----
|
|
...
|
|
-----END PUBLIC KEY-----
|
|
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: api-ingress
|
|
namespace: production
|
|
annotations:
|
|
easyhaproxy.plugins: "jwt_validator"
|
|
# Use explicit key name after the slash
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "app-credentials/rsa-public-key"
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
# ... rest of configuration
|
|
```
|
|
|
|
### Using with Any Plugin
|
|
|
|
The `k8s_secret` pattern works with **any plugin configuration**:
|
|
|
|
```yaml
|
|
# JWT Validator - load public key
|
|
easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "jwt-secret"
|
|
|
|
# Hypothetical API auth plugin - load API key
|
|
easyhaproxy.plugin.api_auth.k8s_secret.api_key: "api-credentials/key"
|
|
|
|
# Hypothetical basic auth plugin - load password
|
|
easyhaproxy.plugin.basic_auth.k8s_secret.password: "auth-secret/pwd"
|
|
```
|
|
|
|
### Priority Order
|
|
|
|
When multiple configuration methods are used, this is the priority (highest to lowest):
|
|
|
|
1. **Explicit annotation** (e.g., `easyhaproxy.plugin.jwt_validator.pubkey: "value"`)
|
|
2. **k8s_secret annotation** (e.g., `easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey: "secret"`)
|
|
|
|
Explicit annotations always take precedence over `k8s_secret` annotations.
|
|
|
|
### Troubleshooting
|
|
|
|
**Secret not found:**
|
|
```
|
|
WARNING: Ingress production/api-ingress - Failed to process k8s_secret annotation
|
|
'easyhaproxy.plugin.jwt_validator.k8s_secret.pubkey' with value 'jwt-secret': ...
|
|
```
|
|
- Verify the secret exists: `kubectl get secret jwt-secret -n production`
|
|
- Check the secret is in the same namespace as the ingress
|
|
|
|
**Key not found in secret:**
|
|
```
|
|
WARNING: Ingress production/api-ingress - Secret 'jwt-secret' found but no matching
|
|
key (tried: pubkey, public-key, jwt.pub, tls.crt)
|
|
```
|
|
- List secret keys: `kubectl get secret jwt-secret -n production -o jsonpath='{.data}'`
|
|
- Use explicit key format: `"jwt-secret/actual-key-name"`
|
|
|
|
**Check EasyHAProxy logs:**
|
|
```bash
|
|
kubectl logs -n easyhaproxy -l app=easyhaproxy --tail=100
|
|
```
|
|
|
|
Look for:
|
|
- `INFO: Loaded 'pubkey' from secret 'jwt-secret'` (success)
|
|
- `WARNING: Secret 'xyz' found but no matching key` (key not found)
|
|
|
|
### Security Considerations
|
|
|
|
- Secrets are read from the **same namespace** as the ingress (no cross-namespace access)
|
|
- EasyHAProxy needs RBAC permissions to read secrets (included in default deployment)
|
|
- Secrets are encrypted at rest in etcd
|
|
- Secret values are base64-encoded by Kubernetes automatically
|
|
- Use Kubernetes RBAC to control which service accounts can read which secrets
|
|
|
|
## Certbot / ACME / Letsencrypt
|
|
|
|
It is necessary to add the annotation `easyhaproxy.certbot` to the ingress configuration:
|
|
|
|
```yaml
|
|
kind: Ingress
|
|
metadata:
|
|
annotations:
|
|
easyhaproxy.certbot: 'true'
|
|
name: example-ingress
|
|
namespace: example
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
....
|
|
```
|
|
|
|
More info [here](acme.md).
|
|
|
|
Make sure your cluster is accessible both through ports 80 and 443.
|
|
|
|
## Custom SSL Certificates
|
|
|
|
Create a secret with your certificate and key and associate them with your ingress.
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: host2-tls
|
|
namespace: default
|
|
data:
|
|
tls.crt: base64 of your certificate
|
|
tls.key: base64 of your certificate private key
|
|
type: kubernetes.io/tls
|
|
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: tls-example
|
|
namespace: default
|
|
spec:
|
|
ingressClassName: easyhaproxy
|
|
tls:
|
|
- hosts:
|
|
- host2.local
|
|
secretName: host2-tls
|
|
rules:
|
|
...
|
|
```
|
|
|
|
----
|
|
[Open source ByJG](http://opensource.byjg.com)
|