initial commit
All checks were successful
Build and Publish TechDocs (Helm Chart Resource) / build-and-publish-helm-chart (push) Successful in 1m25s
All checks were successful
Build and Publish TechDocs (Helm Chart Resource) / build-and-publish-helm-chart (push) Successful in 1m25s
Change-Id: Iedffaa05911f8f1c99119ed53d76af84be0baaee
This commit is contained in:
Scaffolder
495
deploy/README.md
Normal file
495
deploy/README.md
Normal file
@@ -0,0 +1,495 @@
|
||||
# 
|
||||
|
||||
## HAProxy Unified Gateway
|
||||
|
||||
A Kubernetes Gateway API controller powered by HAProxy. HAProxy Unified Gateway (HUG) implements the [Gateway API](https://gateway-api.sigs.k8s.io/) specification to provide advanced traffic management capabilities.
|
||||
|
||||
## Introduction
|
||||
|
||||
This chart bootstraps a HAProxy Unified Gateway deployment/daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Kubernetes 1.26+
|
||||
- Helm 3.6+ (recommended 3.7+)
|
||||
|
||||
## Before you begin
|
||||
|
||||
### Setting up a Kubernetes Cluster
|
||||
|
||||
The quickest way to setup a Kubernetes cluster is with [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/), [AWS Elastic Kubernetes Service](https://aws.amazon.com/eks/) or [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) using their respective quick-start guides.
|
||||
|
||||
For setting up Kubernetes on other cloud platforms or bare-metal servers refer to the Kubernetes [getting started guide](http://kubernetes.io/docs/getting-started-guides/).
|
||||
|
||||
### Install Helm
|
||||
|
||||
Get the latest [Helm release](https://github.com/helm/helm#install).
|
||||
|
||||
### Adding Helm chart repo
|
||||
|
||||
Once you have Helm installed, add the haproxytech Chart Repository as follows:
|
||||
|
||||
```console
|
||||
helm repo add haproxytech https://haproxytech.github.io/helm-charts
|
||||
|
||||
helm repo update
|
||||
```
|
||||
|
||||
## Installing the chart
|
||||
|
||||
To install the chart with Helm v3 as _my-release_ deployment:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway
|
||||
```
|
||||
|
||||
### Installing with unique name
|
||||
|
||||
To auto-generate controller and its resources names when installing, use the following:
|
||||
|
||||
```console
|
||||
helm install haproxytech/haproxy-unified-gateway \
|
||||
--generate-name
|
||||
```
|
||||
|
||||
### Installing from a private registry
|
||||
|
||||
To install the chart using a private registry for controller into a separate namespace _prod_.
|
||||
|
||||
**_NOTE_**: Helm v3 requires namespace to be precreated (eg. with `kubectl create namespace prod`)
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--namespace prod \
|
||||
--set controller.image.tag=SOMETAG \
|
||||
--set controller.imagePullSecrets[0].name=my-pull-secret
|
||||
```
|
||||
|
||||
### Using values from YAML file
|
||||
|
||||
As opposed to using many `--set` invocations, much simpler approach is to define value overrides in a separate YAML file and specify them when invoking Helm:
|
||||
|
||||
_myhug.yaml_:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
kind: DaemonSet
|
||||
service:
|
||||
type: LoadBalancer
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
|
||||
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
|
||||
```
|
||||
|
||||
And invoking Helm becomes:
|
||||
|
||||
```console
|
||||
helm install my-release -f myhug.yaml haproxytech/haproxy-unified-gateway
|
||||
```
|
||||
|
||||
### Installing as DaemonSet
|
||||
|
||||
Default controller mode is [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), but it is possible to use [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) as well:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.kind=DaemonSet
|
||||
```
|
||||
|
||||
### Installing with host networking (DaemonSet)
|
||||
|
||||
When using DaemonSet mode, you can enable host networking and host ports:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.kind=DaemonSet \
|
||||
--set controller.daemonset.useHostNetwork=true \
|
||||
--set controller.dnsPolicy=ClusterFirstWithHostNet
|
||||
```
|
||||
|
||||
### Installing with service annotations
|
||||
|
||||
On some environments like EKS and GKE there might be a need to pass service annotations. Syntax can become a little tedious however:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.service.type=LoadBalancer \
|
||||
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-internal"="0.0.0.0/0" \
|
||||
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-cross-zone-load-balancing-enabled"="true"
|
||||
```
|
||||
|
||||
**_NOTE_**: With helm `--set` it is needed to put quotes and escape dots in the annotation key and commas in the value string.
|
||||
|
||||
### Installing with Horizontal Pod Autoscaler (HPA)
|
||||
|
||||
[HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) automatically scales number of replicas in Deployment and adjusts replica count for the controller:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.autoscaling.enabled=true
|
||||
```
|
||||
|
||||
### Enabling Prometheus monitoring
|
||||
|
||||
HUG exposes two separate metrics endpoints:
|
||||
|
||||
- **`stat`** (port 31024) — HAProxy native metrics (`haproxy_*` prefix): connections, request rates, backend health, latency, error codes
|
||||
- **`metrics`** (port 31060) — HUG controller metrics (`hug_*` prefix): event batch processing, config generation, cert/map operations, HAProxy reloads
|
||||
|
||||
The chart supports both [ServiceMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md) and [PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#podmonitor) for Prometheus Operator integration. These are mutually exclusive — enable only one. By default, both endpoints are scraped.
|
||||
|
||||
**Note:** Requires Prometheus Operator installed in the cluster. The `monitoring.coreos.com/v1` API must be available.
|
||||
|
||||
#### Metrics authentication
|
||||
|
||||
The controller metrics endpoint (`metrics` port) supports three authentication modes via `controller.metricsAuth`:
|
||||
|
||||
| Mode | Default | Protocol | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `kube-rbac` | **yes** | HTTPS | Kubernetes TokenReview/SubjectAccessReview — Prometheus authenticates with its ServiceAccount token |
|
||||
| `none` | | HTTP | No authentication |
|
||||
| `basic` | | HTTPS | HTTP Basic Authentication with username/password |
|
||||
|
||||
#### Default setup (kube-rbac)
|
||||
|
||||
By default the chart uses `kube-rbac` authentication. The controller serves metrics over HTTPS and validates bearer tokens via the Kubernetes API. To set it up:
|
||||
|
||||
**Step 1.** Create a ClusterRole that grants access to the `/metrics` endpoint and bind it to the Prometheus ServiceAccount:
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: hug-metrics-reader
|
||||
rules:
|
||||
- nonResourceURLs: ["/metrics"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: hug-metrics-reader
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: hug-metrics-reader
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: prometheus # adjust to your Prometheus SA name
|
||||
namespace: monitoring # adjust to your Prometheus namespace
|
||||
```
|
||||
|
||||
**Step 2.** Enable the ServiceMonitor (or PodMonitor). The default endpoints are pre-configured for kube-rbac — `stat` uses plain HTTP, `metrics` uses HTTPS with the Prometheus pod's ServiceAccount token:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
extraLabels:
|
||||
release: prometheus # match your Prometheus serviceMonitorSelector
|
||||
```
|
||||
|
||||
That's it. The default `values.yaml` endpoints already include the correct HTTPS + bearer token configuration for the `metrics` port:
|
||||
|
||||
```yaml
|
||||
# Default endpoints (already set in values.yaml):
|
||||
endpoints:
|
||||
- port: stat
|
||||
path: /metrics
|
||||
scheme: http
|
||||
interval: 30s
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
scheme: https
|
||||
interval: 30s
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
```
|
||||
|
||||
#### Using no authentication
|
||||
|
||||
To disable metrics authentication:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
metricsAuth: none
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
endpoints:
|
||||
- port: stat
|
||||
path: /metrics
|
||||
scheme: http
|
||||
interval: 30s
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
scheme: http
|
||||
interval: 30s
|
||||
```
|
||||
|
||||
#### Using basic authentication
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
metricsAuth: basic
|
||||
extraArgs:
|
||||
- --metrics-basic-auth-user=prometheus
|
||||
- --metrics-basic-auth-password=secret
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
endpoints:
|
||||
- port: stat
|
||||
path: /metrics
|
||||
scheme: http
|
||||
interval: 30s
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
scheme: https
|
||||
interval: 30s
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
basicAuth:
|
||||
username:
|
||||
name: hug-metrics-basic-auth
|
||||
key: username
|
||||
password:
|
||||
name: hug-metrics-basic-auth
|
||||
key: password
|
||||
```
|
||||
|
||||
#### Using PodMonitor instead of ServiceMonitor
|
||||
|
||||
PodMonitor scrapes pods directly without creating an extra metrics Service. Replace `serviceMonitor` with `podMonitor` in any of the examples above:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
podMonitor:
|
||||
enabled: true
|
||||
extraLabels:
|
||||
release: prometheus
|
||||
```
|
||||
|
||||
### Configuring HugConf
|
||||
|
||||
The chart creates a HugConf custom resource for controller configuration. You can customize logging, and optionally reference Global and Defaults custom resources:
|
||||
|
||||
```yaml
|
||||
hugconf:
|
||||
logging:
|
||||
defaultLevel: Debug
|
||||
categoryLevelList:
|
||||
- category: "gate"
|
||||
level: "Debug"
|
||||
- category: "k8s"
|
||||
level: "Info"
|
||||
# Reference a Global CR for HAProxy global section customization
|
||||
globalRef:
|
||||
group: gate.v3.haproxy.org
|
||||
kind: Global
|
||||
name: global
|
||||
# Reference a Defaults CR for HAProxy defaults section customization
|
||||
defaultsRef:
|
||||
group: gate.v3.haproxy.org
|
||||
kind: Defaults
|
||||
name: haproxytech
|
||||
```
|
||||
|
||||
The Global and Defaults CRDs are automatically installed by the CRD job. When a `globalRef` or `defaultsRef` is set, the controller uses the referenced CR to configure the HAProxy global/defaults sections. When removed, built-in defaults are restored.
|
||||
|
||||
### Adding extra ports
|
||||
|
||||
By default the chart exposes four container ports: `http` (31080), `https` (31443), `stat` (31024) and `metrics` (31060). Additional ports can be added in two places:
|
||||
|
||||
1. **`controller.containerPort`** — exposes the port on the container (pod spec)
|
||||
2. **`controller.service.extraPorts`** — exposes the port on the Service
|
||||
|
||||
#### Using `--set` flags
|
||||
|
||||
To add a container port only (e.g. for a sidecar or internal use):
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.containerPort.custom=8080
|
||||
```
|
||||
|
||||
To also expose it on the Service:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set controller.containerPort.custom=8080 \
|
||||
--set controller.service.extraPorts[0].name=custom \
|
||||
--set controller.service.extraPorts[0].port=8080 \
|
||||
--set controller.service.extraPorts[0].targetPort=8080 \
|
||||
--set controller.service.extraPorts[0].protocol=TCP
|
||||
```
|
||||
|
||||
#### Using a values file
|
||||
|
||||
For multiple extra ports, a values file is cleaner:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
containerPort:
|
||||
http: 31080
|
||||
https: 31443
|
||||
stat: 31024
|
||||
custom: 8080
|
||||
grpc: 9090
|
||||
service:
|
||||
extraPorts:
|
||||
- name: custom
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
protocol: TCP
|
||||
- name: grpc
|
||||
port: 9090
|
||||
targetPort: 9090
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
#### DaemonSet with host ports
|
||||
|
||||
For DaemonSet mode with host ports, also add matching entries in `controller.daemonset.hostPorts`:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
kind: DaemonSet
|
||||
containerPort:
|
||||
http: 31080
|
||||
https: 31443
|
||||
stat: 31024
|
||||
custom: 8080
|
||||
daemonset:
|
||||
useHostPort: true
|
||||
hostPorts:
|
||||
http: 80
|
||||
https: 443
|
||||
stat: 1024
|
||||
custom: 8080
|
||||
service:
|
||||
extraPorts:
|
||||
- name: custom
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
### Passing extra arguments
|
||||
|
||||
Additional controller flags can be passed via `extraArgs`:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
extraArgs:
|
||||
- --controller-name=gate.haproxy.org/hug
|
||||
- --namespaces=default,production
|
||||
- --leader-election-enabled
|
||||
```
|
||||
|
||||
#### Available controller flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `--controller-name` | `gate.haproxy.org/hug` | `spec.controllerName` GatewayClass selector |
|
||||
| `--namespaces` | | Comma-separated list of namespaces to monitor |
|
||||
| `--ipv4-bind-address` | | IPv4 address to bind to |
|
||||
| `--ipv6-bind-address` | | IPv6 address to bind to |
|
||||
| `--disable-ipv4` | `false` | Disable IPv4 support |
|
||||
| `--disable-ipv6` | `false` | Disable IPv6 support |
|
||||
| `--stats-port` | `1024` | Port for HAProxy stats |
|
||||
| `--controller-port` | `31060` | Port for controller metrics (prometheus) |
|
||||
| `--log-type` | `json` | Log output type (`text` or `json`) |
|
||||
| `--sync-period` | `0` | Period at which the controller computes HAProxy configuration (e.g. `5s`, `1m`) |
|
||||
| `--startup-sync-period` | `0` | Startup period for HAProxy config computation |
|
||||
| `--cache-resync-period` | `0` | Controller-runtime manager cache SyncPeriod (defaults to 10 hours if not set) |
|
||||
| `--leader-election-enabled` | `false` | Enable leader election |
|
||||
| `--add-stats-port` | `true` | Add stats port bind to existing stats frontend |
|
||||
| `--metrics-auth` | `none` | Metrics endpoint auth mode: `none`, `kube-rbac`, `basic` |
|
||||
| `--metrics-basic-auth-user` | | Basic auth username (when `--metrics-auth=basic`) |
|
||||
| `--metrics-basic-auth-password` | | Basic auth password (when `--metrics-auth=basic`) |
|
||||
|
||||
**Note:** The `--hugconf-crd` flag is set automatically by the chart via the `hugconfCrd` helper. The `--job-check-crd` and `--job-gwapi` flags are used internally by the CRD/Gateway API installation jobs.
|
||||
|
||||
### Installing with KEDA autoscaling
|
||||
|
||||
[KEDA](https://keda.sh/) provides event-driven autoscaling. It is mutually exclusive with HPA — when KEDA is enabled, HPA is automatically disabled even if `autoscaling.enabled` is set to `true`.
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
keda:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 20
|
||||
pollingInterval: 30
|
||||
cooldownPeriod: 300
|
||||
restoreToOriginalReplicaCount: false
|
||||
scaledObject:
|
||||
annotations: {}
|
||||
triggers:
|
||||
- type: prometheus
|
||||
metadata:
|
||||
serverAddress: http://<prometheus-host>:9090
|
||||
metricName: haproxy_process_idle_time_percent
|
||||
threshold: '50'
|
||||
query: avg(100-avg_over_time(haproxy_process_idle_time_percent{job="haproxy-unified-gateway"}[2m]))
|
||||
```
|
||||
|
||||
Optional advanced configuration:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
keda:
|
||||
enabled: true
|
||||
# ...triggers, minReplicas, maxReplicas...
|
||||
fallback:
|
||||
failureThreshold: 3
|
||||
replicas: 5
|
||||
horizontalPodAutoscalerConfig:
|
||||
behavior:
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 300
|
||||
```
|
||||
|
||||
### Disabling CRD/Gateway API installation jobs
|
||||
|
||||
By default, the chart includes Helm hook jobs that install HUG CRDs and Gateway API CRDs. To disable them:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--set crdjob.enabled=false \
|
||||
--set gwapijob.enabled=false
|
||||
```
|
||||
|
||||
## Upgrading the chart
|
||||
|
||||
To upgrade the _my-release_ deployment:
|
||||
|
||||
```console
|
||||
helm upgrade my-release haproxytech/haproxy-unified-gateway
|
||||
```
|
||||
|
||||
## Uninstalling the chart
|
||||
|
||||
To uninstall/delete the _my-release_ deployment:
|
||||
|
||||
```console
|
||||
helm delete my-release
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
It is possible to generate a set of YAML files for testing/debugging:
|
||||
|
||||
```console
|
||||
helm install my-release haproxytech/haproxy-unified-gateway \
|
||||
--debug \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome all contributions. Please refer to [guidelines](../CONTRIBUTING.md) on how to make a contribution.
|
||||
Reference in New Issue
Block a user