feat: add scripts for qdrant backup and restore #81
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
bennotieno
changed the title
bennotieno
force-pushed
the
PI-1306
branch
7 times, most recently
from
938ad63 to
8d1fe9f
Compare
| failedJobsHistoryLimit: 1 | ||
| jobTemplate: | ||
| spec: | ||
| backoffLimit: 1 |
There was a problem hiding this comment.
I think it would be better if we know why it failed before re-trying it could be a non-issue at the same time be a critical issue.
There was a problem hiding this comment.
always assume a non proficient operator at customer side
There was a problem hiding this comment.
Okay, i can increase to 3 for know since these values will be overridden incase of a deployment.
|
|
||
| # Job history limits | ||
| successfulJobsHistoryLimit: 1 | ||
| failedJobsHistoryLimit: 1 |
There was a problem hiding this comment.
I have used these configuration to avoid saturation of (completed/failed) jobs in the cluster. At the moment i think the one previous run is what is important incase of debugging or manually tracking progress.
| secretKeyRef: | ||
| name: qdrant-credentials-minio # change this to your qdrant kubernetes secret name | ||
| key: QDRANT_S3_ACCESS_KEY_ID | ||
| - name: QDRANT_S3_SECRET_ACCESS_KEY |
There was a problem hiding this comment.
use AWS_* vars then you can use default toolchains/sdk will work out of the box
There was a problem hiding this comment.
The reason i used custom vars is to avoid instances where the script ends up using or forcing update their user's current configuration. For example they could already have a un-related AWS_* vars that they dont want to change but be able to run the script.
Isolation was the main reason here.
There was a problem hiding this comment.
but these are only mounted to this job?
There was a problem hiding this comment.
The script should also be ran directly on a machine(workstation/personal).
| name: qdrant-credentials | ||
| type: Opaque | ||
| stringData: | ||
| QDRANT_S3_ACCESS_KEY_ID: your-s3-access-key-id |
| Important steps to note: | ||
|
|
||
| - **Create** S3 bucket. | ||
| - Here you can use object storage service providers like StackIt, AWS e.t.c or a self hosted solution like [minio](https://www.min.io/). |
There was a problem hiding this comment.
can we still use minio? I think there was a license change
There was a problem hiding this comment.
Technically yeah, it still has community edition but after the license change releases will be missing pre-built binaries and management console but it would be better to switch to another tool like https://garagehq.deuxfleurs.fr/.
There was a problem hiding this comment.
yeah, sure maybe just add it to the docs since at the moment the client already has an S3 storage setup.
qdrant-backup-restore/k8s/README.md
Outdated
| value: "s3" | ||
| ```` | ||
|
|
||
| (Re)Deploy the Qdrant cluster! This should trigger a rolling update on the qdrant nodes. |
| - name: qdrant-backup-restore-script | ||
| configMap: | ||
| name: qdrant-backup-restore-script | ||
| defaultMode: 0755 |
There was a problem hiding this comment.
this is sensible mode for a multi user target, in containers you can be a bit stricter (f.e. 700) but keep it like this for now
create snapshots based on shard location
update documentation
qdrant_backup_recovery.shfor qdrant backup and restore and its documentation.qdrant-backup-restore/k8s/configmap-script.yamlwhenqdrant-backup-restore/qdrant_backup_recovery.shchanges.