Helm and GitOps at CERN

Ricardo Rocha
Why
Single source of truth

Reusability

Automation of deployment and upgrades

Multi cluster configuration

And also ...
Transition to kubernetes is not trivial ...
“ How do i retrieve my application’s logs? And
how to log rotate? “
“ How do i access the node running container X ? “

“ How do i install package X on the nodes? “

“ Seems like one of the cluster node’s filesystem went

read-only... “
“ Docker, Kubernetes, Ingress … now Helm … this is
a lot of new stuff! “

Significant change in mindset and a steep learning curve

What is Helm
The Kubernetes package manager

A Chart manages the deployment and configuration of an application

Reusable, shareable units

Includes all required manifests, plus any required libraries for lifecycle

Separate values definitions for instance configuration

 Template

apiVersion: v1
kind: ConfigMap
metadata:
name: eosxd-config
labels:
app: {{ template "eosxd.name" . }}
chart: {{ template "eosxd.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
...
{{- range $area, $mountpoints := .Values.mounts }}
{{- range $mountpoint, $letters := $mountpoints }}

fuse.{{ $mountpoint }}.conf: |+

{"name":"{{ $mountpoint }}","hostport":"eos{{
$mountpoint }}.cern.ch","localmountdir":"/eos/{{
$mountpoint }}/", "remotemountdir":"/eos/{{ $area }}/",
"bind":"{{ $letters }}"}

{{- end }}
{{- end }}
 Template Values

apiVersion: v1 image:
kind: ConfigMap repository: gitlab-registry.../eosd
metadata: tag: 0.4.0
name: eosxd-config pullPolicy: IfNotPresent
labels: mounts:
app: {{ template "eosxd.name" . }} ams:
chart: {{ template "eosxd.chart" . }} atlas:
release: {{ .Release.Name }} cms:
heritage: {{ .Release.Service }} experiment:
... lhcb:
{{- range $area, $mountpoints := .Values.mounts }} project:
{{- range $mountpoint, $letters := $mountpoints }} project-i00: "a e j g v k q y"
project-i01: "l h b p s f w n o"
fuse.{{ $mountpoint }}.conf: |+ project-i02: "d c i r m t u x z"
{"name":"{{ $mountpoint }}","hostport":"eos{{ theory:
$mountpoint }}.cern.ch","localmountdir":"/eos/{{ user:
$mountpoint }}/", "remotemountdir":"/eos/{{ $area }}/", home-i00: "l n t z"
"bind":"{{ $letters }}"} home-i01: "a g j k w"
home-i02: "h o r s y"
{{- end }} home-i03: "b e m v x"
{{- end }} home-i04: "c f i p q"
workspace:
Charts Repository
CERN instance: https://charts.cern.ch

A central catalog: the Helm Hub

Quite recent but already points to most popular charts

Does not host the charts, acts like a catalog

https://hub.helm.sh
Charts Repository
A central instance: the Helm Hub

Quite recent but already hosts the most popular charts

Does not host the charts, acts like a catalog

https://hub.helm.sh
Charts Repository
A central instance: the Helm Hub

Quite recent but already hosts the most popular charts

Does not host the charts, acts like a catalog

https://hub.helm.sh
Umbrella Charts
Charts are reusable deployments units

Most applications have multiple dependencies

Umbrella charts wrap all the required charts into a single deployment unit

With any additional manifests required

$ dependencies:
- name: mysql
version: 5.3
repository: https://kubernetes-charts.storage.googleapis.com/
- name: nginx
version: 1.16.1
repository: https://kubernetes-charts.storage.googleapis.com/
 $ helm install fluxcd/flux \
--namespace flux --name flux --values flux-values.yaml

Flux and GitOps --set git.pollInterval=1m

--set git.url=https://gitlab.cern.ch/.../hub

$ cat flux-values.yaml
rbac:
Our end goal from the start create: true
helmOperator:
create: true
Relying on chart updates only chartsSyncInterval: 5m
configureRepositories:
enable: true
repositories:
- name: jupyterhub
url: https://charts.cern.ch/jupyterhub
Registry
...
docker push

Helm
Meta FluxCD Helm
Release
Chart Operator
CRD
git push git pull
Flux and GitOps |-- charts
|-- hub
Chart.yaml requirements.yaml values.yaml
|-- templates
custom-manifest.yaml
What’s in a Helm Release? |-- namespaces
prod.yaml stg.yaml
|-- releases
apiVersion: flux.weave.works/v1beta1 |-- prod
kind: HelmRelease hub.yaml
metadata: |-- stg
name: hub hub.yaml
namespace: prod |-- secrets
spec: |-- prod
releaseName: hub secrets.yaml
chart: |-- stg
git: https://gitlab.cern.ch/.../hub.git secrets.yaml
path: charts/hub
ref: master
valuesFrom:
- secretKeyRef:
name: hub-secrets
key: values.yaml
This is how we plug our encrypted
values: values data
binderhub:
...
A Barbican Secret Plugin for Helm
Similar interface to futuresimple helm-secrets

Builds on existing identity scheme to

access and manage encryption keys
$ helm --name <release> secrets
view secrets.yaml
edit secrets.yaml
install stable/nginx --values secrets.yaml
upgrade stable/nginx --values secrets.yaml
lint --values secrets.yaml

Similar wrapper for kubectl

https://github.com/cernops/helm-barbican
Deployment Model
1 ⇢ 1: This is currently our most common model

Kubernetes clusters live in the end user’s project

Any service aggregation and consolidation is done at that level

1 ⇢ *: Replicate the same application in multiple clusters

HA, Blast Radius, Blue / Green style

* ⇢ *: Workloads also share the underlying resources

Nice separation between service managers and infrastructure

Stronger requirements on multi-tenancy, quotas at cluster level

Conclusion & Next Steps
Helm and (Argo) Flux give us a familiar toolset for containerized applications

Git as the source of truth

After our own tools, working on dissemination (see Rucio’s talk)

Experimenting with the best model to distribute workloads

Likely a mix in the end

Cattle clusters, Blue / Green, Canary with Service Mesh

? Should we keep a WLCG repository for common reusable charts