diff --git a/examples/operations/helm/Chart.lock b/examples/operations/helm/Chart.lock new file mode 100644 index 000000000..6a9cb9260 --- /dev/null +++ b/examples/operations/helm/Chart.lock @@ -0,0 +1,9 @@ +dependencies: +- name: config + repository: "" + version: 0.1.0 +- name: services + repository: "" + version: 0.1.0 +digest: sha256:7550b9a2ad831bd383e6bf22c51013efd466f8d287df676433766fd59e9aac29 +generated: "2025-04-10T10:54:29.929533+01:00" diff --git a/examples/operations/helm/Chart.yaml b/examples/operations/helm/Chart.yaml new file mode 100644 index 000000000..37aedca23 --- /dev/null +++ b/examples/operations/helm/Chart.yaml @@ -0,0 +1,11 @@ +apiVersion: v2 +name: vespa +version: 0.1.0 +dependencies: + - name: config + alias: config + version: 0.1.0 + - name: services + alias: services + version: 0.1.0 + condition: services.enabled diff --git a/examples/operations/helm/README.md b/examples/operations/helm/README.md new file mode 100644 index 000000000..ec79525a0 --- /dev/null +++ b/examples/operations/helm/README.md @@ -0,0 +1,170 @@ + + + + + + #Vespa + + +# Multinode-HA using Helm +This guide uses the multinode-HA configuration and principles and deploys a Vespa application using Kubernetes and Helm. + +## Overview +This deployment is designed for high availability and uses the Helm chart consisting of two primary modules: +1. `config` - contains and deploys **vespa-configserver**, which must be running successfully before starting other components. +2. `services` - starts **admin**, **content**, **feed**, and **query** clusters, depending on a successful `configserver` startup. + +A key mechanism ensures the correct start order of the modules: `initContainers` in `services` waits for the `configserver` to become ready by repeatedly checking its health. Only after the `configserver` successfully initializes, the `services` module will proceed to start. Here’s the command used in the `initContainers`: + +```bash +until curl -f http://vespa-configserver-0.vespa-internal.vespa.svc.cluster.local:19071/state/v1/health; do + echo "Waiting for Vespa ConfigServer to be ready in namespace $CONFIGSERVER_NAMESPACE..."; + sleep 5; +done +``` + +--- + +## Prerequisites +Make sure the following tools are installed and configured: +* [Helm](https://helm.sh/docs/intro/install/) +* A Kubernetes cluster - either local or hosted (e.g., Azure AKS, AWS EKS, etc.) + +--- + +## Installation +Clone the repository: +```bash +git clone --depth 1 https://github.com/vespa-engine/sample-apps.git +cd sample-apps/examples/operations/multinode-HA/helm +``` + +Prepare your `values.yaml` with the desired configuration. Here's an example: +```yaml +config: + serverReplicas: 3 +services: + content: + replicas: 2 + storage: 25Gi +``` + +Deploy Vespa using Helm: +```bash +helm dependency update helm +helm upgrade --install vespa . -n vespa --create-namespace -f values.yaml +``` + +This will create the namespace `vespa` and deploy all components of the application. + +--- + +## Deploy the application package + +``` +kubectl port-forward -n vespa pod/vespa-configserver-0 19071 +``` + +``` +(cd conf && zip -r - .) | \ + curl --header Content-Type:application/zip \ + --data-binary @- \ + http://localhost:19071/application/v2/tenant/default/prepareandactivate +``` + +--- + +## Module Details + +### Config Module +The `config` module contains the **vespa-configserver**, which is essential for Vespa's operation. This module deploys a StatefulSet with `serverReplicas` to ensure high availability. + +#### ConfigServer Health Check +The `configserver` health is verified by an HTTP curl to its `/state/v1/health` endpoint. The `services` module will not start until all `configserver` replicas are running and reachable. + +Here’s an example of the expected `configserver` health response: +```json +{ + "time" : 1678268549957, + "status" : { + "code" : "up" + }, + "metrics" : { + "snapshot" : { + "from" : 1.678268489718E9, + "to" : 1.678268549718E9 + } + } +} +``` + +### Services Module +The `services` module contains the following components: +- **Admin**: Handles Vespa cluster administration. +- **Feed**: Handles document feeding. +- **Query**: Handles document queries. +- **Content**: Stores indexed data. + +This module is configured to depend on the `config` module startup. The `initContainers` logic ensures that no pods in the `services` module are started until the `configserver` reaches a stable, healthy state. + +Below is the typical `initContainers` logic defined for `services`: +```bash +until curl -f http://vespa-configserver-0.vespa-internal.vespa.svc.cluster.local:19071/state/v1/health; do + echo "Waiting for Vespa ConfigServer to be ready in namespace $CONFIGSERVER_NAMESPACE..."; + sleep 5; +done +``` + +--- + +## Verification +Once the installation completes, you can test the Vespa application by: +1. Feeding a document: + ```bash + curl -X POST http://vespa-query-container-0.vespa.svc.cluster.local/document/v1/my-space/my-doc \ + -d '{"id": "id:my-space:my-doc::1", "fields": {"field1": "value1"}}' + ``` + +2. Querying documents: + ```bash + curl "http://vespa-query-container-0.vespa.svc.cluster.local/search/?query=my-query" + ``` + + or + + ``` + kubectl -n vespa port-forward svc/vespa-query 8080 + ``` + ``` + curl --data-urlencode 'yql=select * from sources * where true' \ + http://localhost:8080/search/ + ``` + +--- + +## Customization and Scaling +Values such as `config.serverReplicas`, `services.content.replicas`, and `services.content.storage` can be adjusted in `values.yaml` to match your requirements for scaling and resource configuration. For example: +```yaml +config: + serverReplicas: 5 +services: + content: + replicas: 4 + storage: 50Gi +``` + +Refer to the official [Vespa documentation](https://docs.vespa.ai/en/) for advanced deployment details and customization options. + +--- + +## Troubleshooting +Check Helm release status to confirm all components deployed successfully: +```bash +helm status vespa -n vespa +``` + +If pods are stuck, ensure that: +1. The `configserver` is running and reachable. +2. Kubernetes networking allows communication between the pods. + +For further troubleshooting details, refer to the [Vespa troubleshooting guide](https://docs.vespa.ai/en/operations.html). \ No newline at end of file diff --git a/examples/operations/helm/charts/config/Chart.yaml b/examples/operations/helm/charts/config/Chart.yaml new file mode 100644 index 000000000..37ec018c2 --- /dev/null +++ b/examples/operations/helm/charts/config/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: config +version: 0.1.0 \ No newline at end of file diff --git a/examples/operations/helm/charts/config/templates/configmap.yml b/examples/operations/helm/charts/config/templates/configmap.yml new file mode 100644 index 000000000..d0efa21af --- /dev/null +++ b/examples/operations/helm/charts/config/templates/configmap.yml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: vespa-config +data: + VESPA_CONFIGSERVERS: >- + {{- $domain := printf "%s.%s.svc.cluster.local" .Values.headlessName .Release.Namespace }} + {{- $replicas := int .Values.serverReplicas }} + {{- $serverList := list }} + {{- range $i := until $replicas }} + {{- $serverList = append $serverList (printf "vespa-configserver-%d.%s" $i $domain) }} + {{- end }} + {{ join "," $serverList }} + VESPA_CONFIGSERVER_JVMARGS: "{{ .Values.serverJvmArgs }}" + VESPA_CONFIGPROXY_JVMARGS: "{{ .Values.proxyJvmArgs }}" \ No newline at end of file diff --git a/examples/operations/helm/charts/config/templates/configserver.yml b/examples/operations/helm/charts/config/templates/configserver.yml new file mode 100644 index 000000000..890049890 --- /dev/null +++ b/examples/operations/helm/charts/config/templates/configserver.yml @@ -0,0 +1,84 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: vespa-configserver +spec: + # Use three servers for proper ZooKeeper quorum: + replicas: {{ .Values.serverReplicas }} + selector: + matchLabels: + app: vespa-configserver + name: {{ .Values.headlessName }} + serviceName: {{ .Values.headlessName }} + template: + metadata: + labels: + app: vespa-configserver + name: {{ .Values.headlessName }} + spec: + initContainers: + - name: chown-var + securityContext: + runAsUser: 0 + image: busybox + command: ["sh", "-c", "chown -R 1000 /opt/vespa/var"] + volumeMounts: + - name: vespa-var + mountPath: /opt/vespa/var + - name: chown-logs + securityContext: + runAsUser: 0 + image: busybox + command: ["sh", "-c", "chown -R 1000 /opt/vespa/logs"] + volumeMounts: + - name: vespa-logs + mountPath: /opt/vespa/logs + containers: + - name: vespa-configserver + image: vespaengine/vespa + args: ["configserver,services"] + imagePullPolicy: Always + securityContext: + runAsUser: 1000 + volumeMounts: + - name: vespa-var + mountPath: /opt/vespa/var + - name: vespa-logs + mountPath: /opt/vespa/logs + - name: vespa-workspace + mountPath: /workspace + envFrom: + - configMapRef: + name: vespa-config + # Note that the below are minimum resources for demo use. + # Use 4G or more for real use cases + resources: + requests: + memory: "1.5G" + limits: + memory: "1.5G" + volumeClaimTemplates: + # The below are tiny volumes for demo purposes. + - metadata: + name: vespa-var + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 5Gi + - metadata: + name: vespa-logs + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 5Gi + - metadata: + name: vespa-workspace + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 1Gi diff --git a/examples/operations/helm/charts/config/templates/headless.yml b/examples/operations/helm/charts/config/templates/headless.yml new file mode 100644 index 000000000..c5c31abd0 --- /dev/null +++ b/examples/operations/helm/charts/config/templates/headless.yml @@ -0,0 +1,12 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: v1 +kind: Service +metadata: + name: {{ .Values.headlessName }} + labels: + name: {{ .Values.headlessName }} +spec: + selector: + name: {{ .Values.headlessName }} + clusterIP: None diff --git a/examples/operations/helm/charts/config/values.yaml b/examples/operations/helm/charts/config/values.yaml new file mode 100644 index 000000000..4dc62351f --- /dev/null +++ b/examples/operations/helm/charts/config/values.yaml @@ -0,0 +1,4 @@ +headlessName: "vespa-internal" +serverReplicas: 3 +serverJvmArgs: "-Xms32M -Xmx128M" +proxyJvmArgs: "-Xms32M -Xmx32M" \ No newline at end of file diff --git a/examples/operations/helm/charts/services/Chart.yaml b/examples/operations/helm/charts/services/Chart.yaml new file mode 100644 index 000000000..0cc9693a8 --- /dev/null +++ b/examples/operations/helm/charts/services/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: services +version: 0.1.0 \ No newline at end of file diff --git a/examples/operations/helm/charts/services/templates/admin.yml b/examples/operations/helm/charts/services/templates/admin.yml new file mode 100644 index 000000000..cddc1378c --- /dev/null +++ b/examples/operations/helm/charts/services/templates/admin.yml @@ -0,0 +1,50 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: vespa-admin +spec: + replicas: 1 + selector: + matchLabels: + app: vespa-admin + name: vespa-internal + serviceName: vespa-internal + template: + metadata: + labels: + app: vespa-admin + name: vespa-internal + spec: + initContainers: + - name: wait-for-configserver + image: curlimages/curl + env: + - name: CONFIGSERVER_NAMESPACE + value: {{ .Release.Namespace }} + - name: CONFIGSERVER_SERVICE + value: vespa-configserver-0.vespa-internal + command: + - /bin/sh + - -c + - | + until curl -f http://vespa-configserver-0.vespa-internal.vespa.svc.cluster.local:19071/state/v1/health; do + echo "Waiting for Vespa ConfigServer to be ready in namespace $CONFIGSERVER_NAMESPACE..."; + sleep 5; + done + containers: + - name: vespa-admin + image: vespaengine/vespa + args: ["services"] + imagePullPolicy: Always + envFrom: + - configMapRef: + name: vespa-config + securityContext: + runAsUser: 1000 + resources: + requests: + memory: "1G" + limits: + memory: "1G" diff --git a/examples/operations/helm/charts/services/templates/content.yml b/examples/operations/helm/charts/services/templates/content.yml new file mode 100644 index 000000000..9644b43b4 --- /dev/null +++ b/examples/operations/helm/charts/services/templates/content.yml @@ -0,0 +1,69 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: vespa-content +spec: + replicas: {{ .Values.content.replicas }} + selector: + matchLabels: + app: vespa-content + name: vespa-internal + serviceName: vespa-internal + template: + metadata: + labels: + app: vespa-content + name: vespa-internal + spec: + initContainers: + - name: wait-for-configserver + image: curlimages/curl + env: + - name: CONFIGSERVER_NAMESPACE + value: {{ .Release.Namespace }} + - name: CONFIGSERVER_SERVICE + value: vespa-configserver-0.vespa-internal + command: + - /bin/sh + - -c + - | + until curl -f http://vespa-configserver-0.vespa-internal.vespa.svc.cluster.local:19071/state/v1/health; do + echo "Waiting for Vespa ConfigServer to be ready in namespace $CONFIGSERVER_NAMESPACE..."; + sleep 5; + done + - name: chown-var + securityContext: + runAsUser: 0 + image: busybox + command: [ "sh", "-c", "chown -R 1000 /opt/vespa/var" ] + volumeMounts: + - name: vespa-var + mountPath: /opt/vespa/var + containers: + - name: vespa-content + image: vespaengine/vespa + args: [ "services" ] + imagePullPolicy: Always + envFrom: + - configMapRef: + name: vespa-config + securityContext: + runAsUser: 1000 + volumeMounts: + - name: vespa-var + mountPath: /opt/vespa/var + resources: + requests: + memory: "1G" + limits: + memory: "1G" + volumeClaimTemplates: + - metadata: + name: vespa-var + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: {{ .Values.content.storage }} diff --git a/examples/operations/helm/charts/services/templates/feed-container.yml b/examples/operations/helm/charts/services/templates/feed-container.yml new file mode 100644 index 000000000..e8161ceef --- /dev/null +++ b/examples/operations/helm/charts/services/templates/feed-container.yml @@ -0,0 +1,34 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: vespa-feed-container +spec: + replicas: 2 + selector: + matchLabels: + app: vespa-feed-container + name: vespa-internal + serviceName: vespa-internal + template: + metadata: + labels: + app: vespa-feed-container + name: vespa-internal + spec: + containers: + - name: vespa-feed-container + image: vespaengine/vespa + args: ["services"] + imagePullPolicy: Always + envFrom: + - configMapRef: + name: vespa-config + securityContext: + runAsUser: 1000 + resources: + requests: + memory: "1.5G" + limits: + memory: "1.5G" diff --git a/examples/operations/helm/charts/services/templates/query-container.yml b/examples/operations/helm/charts/services/templates/query-container.yml new file mode 100644 index 000000000..779a393b1 --- /dev/null +++ b/examples/operations/helm/charts/services/templates/query-container.yml @@ -0,0 +1,34 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: vespa-query-container +spec: + replicas: 2 + selector: + matchLabels: + app: vespa-query-container + name: vespa-internal + serviceName: vespa-internal + template: + metadata: + labels: + app: vespa-query-container + name: vespa-internal + spec: + containers: + - name: vespa-query-container + image: vespaengine/vespa + args: ["services"] + imagePullPolicy: Always + envFrom: + - configMapRef: + name: vespa-config + securityContext: + runAsUser: 1000 + resources: + requests: + memory: "1.5G" + limits: + memory: "1.5G" diff --git a/examples/operations/helm/charts/services/templates/service-feed.yml b/examples/operations/helm/charts/services/templates/service-feed.yml new file mode 100644 index 000000000..9e93b9a04 --- /dev/null +++ b/examples/operations/helm/charts/services/templates/service-feed.yml @@ -0,0 +1,17 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: v1 +kind: Service +metadata: + name: vespa-feed + labels: + app: vespa +spec: + # Set LoadBalancer for an endpoint reachable from the internet + #type: LoadBalancer + selector: + app: vespa-feed-container + ports: + - name: api + port: 8080 + targetPort: 8080 diff --git a/examples/operations/helm/charts/services/templates/service-query.yml b/examples/operations/helm/charts/services/templates/service-query.yml new file mode 100644 index 000000000..9b3f8ee17 --- /dev/null +++ b/examples/operations/helm/charts/services/templates/service-query.yml @@ -0,0 +1,17 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +apiVersion: v1 +kind: Service +metadata: + name: vespa-query + labels: + app: vespa +spec: + # Set LoadBalancer for an endpoint reachable from the internet + #type: LoadBalancer + selector: + app: vespa-query-container + ports: + - name: api + port: 8080 + targetPort: 8080 diff --git a/examples/operations/helm/charts/services/values.yaml b/examples/operations/helm/charts/services/values.yaml new file mode 100644 index 000000000..036a885bb --- /dev/null +++ b/examples/operations/helm/charts/services/values.yaml @@ -0,0 +1,3 @@ +content: + replicas: 2 + storage: 10Gi \ No newline at end of file diff --git a/examples/operations/helm/conf/hosts.xml b/examples/operations/helm/conf/hosts.xml new file mode 100644 index 000000000..4fb1638eb --- /dev/null +++ b/examples/operations/helm/conf/hosts.xml @@ -0,0 +1,33 @@ + + + + node0 + + + node1 + + + node2 + + + node3 + + + node4 + + + node5 + + + node6 + + + node7 + + + node8 + + + node9 + + diff --git a/examples/operations/helm/conf/schemas/music.sd b/examples/operations/helm/conf/schemas/music.sd new file mode 100644 index 000000000..3274c43ec --- /dev/null +++ b/examples/operations/helm/conf/schemas/music.sd @@ -0,0 +1,25 @@ +# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + +schema music { + + document music { + + field artist type string { + indexing: summary | index + } + + field album type string { + indexing: summary | index + } + + field year type int { + indexing: summary | attribute + } + + field category_scores type tensor(cat{}) { + indexing: summary | attribute + } + + } + +} diff --git a/examples/operations/helm/conf/services.xml b/examples/operations/helm/conf/services.xml new file mode 100644 index 000000000..ff355f9a8 --- /dev/null +++ b/examples/operations/helm/conf/services.xml @@ -0,0 +1,55 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 2 + + + + + + + + + + diff --git a/examples/operations/helm/hosts.xml b/examples/operations/helm/hosts.xml new file mode 100644 index 000000000..6ac831381 --- /dev/null +++ b/examples/operations/helm/hosts.xml @@ -0,0 +1,33 @@ + + + + node0 + + + node1 + + + node2 + + + node3 + + + node4 + + + node5 + + + node6 + + + node7 + + + node8 + + + node9 + + diff --git a/examples/operations/helm/values.yaml b/examples/operations/helm/values.yaml new file mode 100644 index 000000000..55d55bbe2 --- /dev/null +++ b/examples/operations/helm/values.yaml @@ -0,0 +1,4 @@ +services: + enabled: true + content: + storage: 20Gi