feat: Enhance gatekeeper resilience and host handling
This commit significantly improves the gatekeeper system's robustness, monitoring capabilities, and simplifies host header management for backend services.
Key changes include:
**Gatekeeper Health, Management & Resilience:**
- Implemented active health checking for individual gatekeeper containers within the `gatekeeper-manager` service.
- The manager now periodically curls the `/metrics` endpoint of each gatekeeper container.
- Reports health status to a new Gatus `services_gatekeeper` endpoint.
- Automatically attempts to restart the gatekeeper stack if any gatekeeper instance is unhealthy or if the expected number of gatekeepers is not running.
- Refactored the `gatekeeper-manager` shell script for improved state management and signal handling:
- Introduced `STARTED`, `RESTARTING`, `TERMINATING` state flags for more controlled operations.
- Enhanced SIGTERM and SIGHUP handling to gracefully manage gatekeeper lifecycles.
- Added `apk add curl` to ensure `curl` is available in the manager container.
- Renamed the gatekeeper Docker Compose template from `docker-compose_gatekeeper.template.yml` to `gatekeepers.template.yml` and its output to `gatekeepers.yml`.
- Updated `dockergen-gatekeeper` to watch the new template file and notify the correct `gatekeeper-manager` service instance (e.g., `pkmntrade-club-gatekeeper-manager-1`).
- Discover services that should be protected by looking for a `gatekeeper=true` label.
**Host Header Management & `ALLOWED_HOSTS` Simplification:**
- HAProxy configuration (`haproxy.cfg`) now consistently sets the `Host` HTTP header for requests to all backend services (e.g., `pkmntrade.club`, `staging.pkmntrade.club`). This centralizes and standardizes host information.
- Consequently, explicit `ALLOWED_HOSTS` environment variables have been removed from the `web` and `celery` service definitions in `docker-compose_web.yml` and `docker-compose_staging.yml`. Backend Django applications should now rely on the `Host` header set by HAProxy for request validation.
- The `gatekeepers.template.yml` now defines a `TARGET_HOST` environment variable for proxied services (e.g., `web`, `web-staging`). This aligns with the ALLOWED_HOSTS on the target to ensure requests aren't blocked.
**Gatus Monitoring & Configuration Updates:**
- In Gatus configuration (`gatus/config.template.yaml`):
- The "Redis" external service endpoint has been renamed to "Cache" for better clarity and to fit the theme of simple names.
- A new external service endpoint "Gatekeeper" has been added to monitor the overall health reported by the `gatekeeper-manager`.
- Health checks for "Web Worker" endpoints (both main and staging) now include the appropriate `Host` header (e.g., `Host: pkmntrade.club`) to ensure accurate health assessments by Django.
- In `docker-compose_core.yml`, the `curl` commands used by `db-redis-healthcheck` for database and cache health now append `|| true`. This prevents the script from exiting on a curl error (e.g., timeout, connection refused), ensuring that the failure is still reported to Gatus via the `success=false` parameter rather than the script terminating prematurely.
These changes collectively make the gatekeeper system more fault-tolerant, provide better visibility into its status, and streamline the configuration of backend applications by standardizing how they receive host information.
This commit is contained in:
parent
6aa15d1af9
commit
c87d73435b
6 changed files with 123 additions and 26 deletions
|
|
@ -1,37 +0,0 @@
|
|||
services:
|
||||
{{ $all_containers := whereLabelValueMatches . "enable_gatekeeper" "true" }}
|
||||
{{ $all_containers = sortObjectsByKeysAsc $all_containers "Name" }}
|
||||
|
||||
{{ range $container := $all_containers }}
|
||||
{{ $serviceLabel := index $container.Labels "com.docker.compose.service" }}
|
||||
{{ $containerNumber := index $container.Labels "com.docker.compose.container-number" }}
|
||||
{{ $port := "" }}
|
||||
{{ if eq $serviceLabel "web" }}
|
||||
{{ $port = ":8000" }}
|
||||
{{ end }}
|
||||
{{ if eq $serviceLabel "web-staging" }}
|
||||
{{ $port = ":8000" }}
|
||||
{{ end }}
|
||||
{{ if eq $serviceLabel "feedback" }}
|
||||
{{ $port = ":3000" }}
|
||||
{{ end }}
|
||||
{{ if eq $serviceLabel "health" }}
|
||||
{{ $port = ":8080" }}
|
||||
{{ end }}
|
||||
gatekeeper-{{ $serviceLabel }}-{{ $containerNumber }}:
|
||||
image: ghcr.io/techarohq/anubis:latest
|
||||
container_name: pkmntrade-club-gatekeeper-{{ $serviceLabel }}-{{ $containerNumber }}
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- TARGET=http://{{ $container.Name }}{{ $port }}
|
||||
networks:
|
||||
default:
|
||||
aliases:
|
||||
- pkmntrade-club-gatekeeper-{{ $serviceLabel }}
|
||||
- gatekeeper-{{ $serviceLabel }}
|
||||
{{ end }}
|
||||
networks:
|
||||
default:
|
||||
name: pkmntrade-club_default
|
||||
external: true
|
||||
Loading…
Add table
Add a link
Reference in a new issue