S3 file storage (SeaweedFS)¶

The huddo-cp chart deploys a SeaweedFS S3 service — Kubernetes Service name huddo-s3, default bucket huddo — for user-uploaded files. Its data needs a PersistentVolume.

You have two options:

Dynamic provisioning — set s3.persistence.storageClassName to a storage class in your cluster and blank out s3.persistence.nfs. No further setup needed.
NFS — the chart creates a PersistentVolume bound by name from s3.persistence.nfs.server and s3.persistence.nfs.path. Prepare the export as below.

This volume holds user data

The bucket stores attachment binaries — there is no second copy. The claim is annotated helm.sh/resource-policy: keep by default, so helm uninstall leaves it behind and a reinstall of the same release name into the same namespace picks the bucket up again.

That default matters most on a dynamic provisioner: cloud StorageClasses usually reclaim Delete, so without it an uninstall would destroy the disk along with the release. An NFS-backed install is safe either way, since its PersistentVolume is Retain over an export that outlives the release.

s3.persistence.retainOnDelete: false restores the old behaviour and lets the volume be torn down with the release. Deleting the release is not a backup strategy in either case.

NFS setup¶

Create the folder on the NFS server

sudo mkdir /pv-connections/huddo-s3
sudo chmod 755 /pv-connections/huddo-s3

Ensure each node in your Kubernetes cluster can mount this location.

Modify /etc/exports on your NFS server to include this line:

/pv-connections/huddo-s3 <IP_RANGE_OF_YOUR_SERVERS>/<SUBNET_MASK>(rw,no_root_squash)

For example:

/pv-connections/huddo-s3 192.168.0.0/255.255.0.0(rw,no_root_squash)

Apply the new NFS storage to exports
```
exportfs -ra
```

Credentials¶

Set matching values in two places in your config file — the SeaweedFS identity config and the environment the apps read:

global:
  env:
    S3_ACCESS_KEY: '[S3_ACCESS_KEY]'
    S3_SECRET_KEY: '[S3_SECRET_KEY]'

s3:
  accessKey: '[S3_ACCESS_KEY]'
  secretKey: '[S3_SECRET_KEY]'

Warning

The chart ships development defaults for both. Change them before a production install.

Capacity planning¶

SeaweedFS stores objects in volume files of up to 30000 MB (about 31 GB) each, and one server holds at most s3.volumeMax of them. The chart default, 64, gives a ceiling of about 2 TB in total. The huddo bucket can use about 1.6 TB of it with Boards enabled (about 1.8 TB without), because the other collections hold 7 slots each (see below). Size the NFS export (or s3.persistence.size) for your real dataset as well: the ceiling does not reserve disk, and the smaller of the two is what you get.

Volume slots are not shared evenly between buckets. On the first write to a collection SeaweedFS creates 7 volumes at once, and each of these is a collection of its own:

the filer's internal metadata log, written about a minute after startup
the huddo bucket
the huddo-boards bucket, when Huddo Boards is enabled

So the three need at least 21 slots between them before any volume fills.

Do not set s3.volumeMax below 21

With too few slots, the first collection to write takes 7 and the others get what is left. The metadata log usually writes first, which can leave the huddo bucket a single ~31 GB volume. With Boards enabled, the huddo-boards bucket can get none: uploads fail with an internal error, and the SeaweedFS log shows failed to find writable volumes.

Volumes are created on demand and not preallocated (an empty one is a few bytes), so a generous volumeMax costs nothing until the data arrives. For more than about 1.3 TB of data, raise it to at least 21 + (dataset + headroom) / 30 GB, which keeps one batch of 7 spare:

s3:
  # Quoted: "0" means auto configure from free disk space (on a small
  # export that can be a single slot), and a per-disk list ("64,64") is
  # also accepted.
  volumeMax: "128"

Keep s3.volumeSizeLimitMB at 30000, which is also its maximum: SeaweedFS refuses to start above it.

To see how the slots are used, ask the master:

kubectl exec -n connections deploy/huddo-s3 -c seaweedfs -- sh -c 'echo "volume.list" | weed shell'

The Topology line reports volume:<used>/<max>, and each volume line names its Collection (empty for the metadata log). free:0 means no further volumes can be created, though existing ones keep accepting writes until each is full.

Using an external S3 service¶

To use an existing S3-compatible store (AWS S3, Google Cloud Storage, MinIO, …) instead, disable the bundled service and point the apps at your endpoint:

s3:
  enabled: false

global:
  env:
    S3_ENDPOINT: storage.googleapis.com
    S3_PORT: '443'
    S3_SECURE: 'true'
    S3_USE_SSL: 'true'
    S3_BUCKET: '[YOUR_BUCKET]'
    S3_ACCESS_KEY: '[S3_ACCESS_KEY]'
    S3_SECRET_KEY: '[S3_SECRET_KEY]'

Migrating from an existing S3 store¶

To move existing files into the bundled SeaweedFS (from MinIO, Google Cloud Storage or another S3-compatible store), enable the migration Job. While s3.migration.enabled is true, every helm upgrade runs a Job, huddo-s3-migration-<revision>, that copies the source bucket into the huddo bucket. Each run copies only what is new or changed since the last one.

The Job runs rclone copy --update. It never deletes from SeaweedFS and never replaces an object that is newer there, such as a file the apps changed after cutover. A Job that runs again by mistake after cutover therefore does not lose data.

Keep the apps on the old store (global.env.S3_*) until the copy is complete, and check the capacity first. The Job copies everything in one pass:

s3:
  enabled: true
  migration:
    enabled: true
    source:
      provider: Other          # GCS for Google Cloud Storage
      endpoint: http://[OLD_S3_HOST]:9000
      accessKey: '[OLD_S3_ACCESS_KEY]'
      secretKey: '[OLD_S3_SECRET_KEY]'
      bucket: '[OLD_BUCKET]'

For Google Cloud Storage, use an HMAC key (Cloud Storage → Settings → Interoperability) with provider: GCS and endpoint: https://storage.googleapis.com. With the default Other, GCS rejects every request with SignatureDoesNotMatch.

After each upgrade, read the Job's verification report. The finished Job stays until the next upgrade.

kubectl logs -n connections job/huddo-s3-migration-<revision>

It lists objects missing, size-mismatched or extra on the destination. Only objects missing on the destination raise a WARNING. The report ends with No source object is missing on the destination. when every source object is copied. The other two lists are expected after cutover: files that the apps changed or wrote in SeaweedFS.

A failed migration Job fails the upgrade

helm upgrade --wait counts a running Job as ready, but a Job that has already failed when the other resources become ready fails the upgrade. With --rollback-on-failure (or --atomic), the whole release is rolled back. Fix the source settings, or set s3.migration.enabled: false to unblock upgrades.

Cutover¶

The Job starts while the old app pods are still running, so a file an old pod uploads after the copy starts reaches the old store only. Keep the Job enabled through the cutover and run one more upgrade once the old pods have stopped:

Point global.env.S3_* at the bundled service: S3_ENDPOINT: huddo-s3, S3_PORT: '8333', S3_SECURE and S3_USE_SSL 'false', S3_BUCKET: huddo, and the credentials set on s3. Keep s3.migration.enabled: true, and run the upgrade.
When no old app pod is left, run the same upgrade again. Its Job copies the files the old pods wrote during the switch. --update keeps the files the apps have changed in SeaweedFS since, such as user avatars, which the apps rewrite in place.
Read the verification report of that Job. Make sure it ends with No source object is missing on the destination.
Set s3.migration.enabled: false in the next upgrade.
Keep the old store until you have checked the apps against the new one.

If ArgoCD (or another tool that renders the chart with helm template) deploys the release, the Job always has the name huddo-s3-migration-1, so a sync does not run it again. Instead of step 2, delete the finished Job once the old pods have stopped. ArgoCD creates it again on the next sync:

kubectl delete job -n connections huddo-s3-migration-1

Do not use rclone sync

To copy the last files by hand instead, use rclone copy --update --no-update-modtime (old store first, SeaweedFS second). rclone sync deletes the files the apps wrote to SeaweedFS after the switch.

Huddo Boards¶

If you enable Huddo Boards in the same release (products.boards.enabled: true), Boards writes to a separate bucket, huddo-boards, on the same endpoint. The chart only provisions the huddo bucket — create the second one out of band:

aws s3 mb s3://huddo-boards --endpoint-url http://huddo-s3:8333

SeaweedFS will also create it on first write.