Migration Guide: MinIO to SeaweedFS (huddo-boards-cp)¶

Background¶

MinIO changed its licence from Apache 2.0 to AGPLv3, which introduces compliance concerns for customers deploying Huddo charts. huddo-boards-cp v2.1.0 introduces SeaweedFS (Apache 2.0) as the replacement S3-compatible object store.

SeaweedFS deploys alongside MinIO by default, so existing installations continue to work without interruption. A built-in migration Job uses rclone to copy data from MinIO to SeaweedFS, and the cutover is a simple values change.

What changed in v2.1.0¶

Component	Before	After
S3 store	MinIO (AGPLv3) — always deployed	MinIO (conditional, `minio.enabled: true` by default) + SeaweedFS (always deployed)
S3 endpoint default	`huddo-boards-cp-minio-service:9000`	Unchanged by default — apps still point at MinIO until you switch
S3 bucket default	`kudos-boards`	`huddo-boards` (rebrand)
New subchart	—	`huddo-s3` (SeaweedFS, port 8333) deployed as `huddo-boards-cp-s3`
Migration Job	—	`rclone` Job, opt-in via `s3.migration.enabled: true` (`rclone sync` before 2.5.3, `rclone copy --update` from 2.5.3)

Upgrading to v2.1.0 alone changes nothing for your apps. MinIO remains the active S3 backend. SeaweedFS runs alongside it but receives no traffic until you explicitly redirect the apps.

Bucket rename: v2.1.0 changes the default S3 bucket name from kudos-boards to huddo-boards. Migration copies legacy data from MinIO kudos-boards to SeaweedFS huddo-boards. Existing installs must keep reading the legacy bucket name during Phase 1 (override global.env.S3_BUCKET: kudos-boards) and drop the override at cutover so apps fall back to the new default.

Prerequisites¶

huddo-boards-cp v2.1.0 or later. Use 2.5.3 or later: before 2.5.3 the migration Job deletes data from SeaweedFS if it runs again after cutover (see Phase 1).
Your existing MinIO is healthy and accessible
Enough storage capacity for SeaweedFS to hold a copy of your data during migration, on both counts:
- the underlying disk or NFS export is large enough, and
- s3.volumeMax is high enough. Before chart 2.4.1, the default of 8 slots often leaves the huddo-boards bucket a single ~31 GB volume, because SeaweedFS gives 7 of them to its internal metadata log. Set s3.volumeMax: "64" (or at least 14 + (dataset + headroom) / 30 GB) before migrating, or the sync stops with an error containing no free volumes left. See capacity planning.
If using NFS persistence: a directory on the export owned by uid 1000 — see preparing the S3 storage mount

SeaweedFS runs as user 1000 from chart 2.3.5

As of huddo-boards-cp-2.3.5 the SeaweedFS pod runs as user 1000 (matching the MinIO pods), so root_squash exports now work. Data written by charts 2.1.0–2.3.4 (which ran SeaweedFS as root) is chowned automatically by an init container on the first start after the upgrade — no manual step required. If Pod Security admission on your cluster forbids a root container, set s3.persistence.fixOwnership: false and chown 1000:1000 -R the data directory yourself.

Know your current S3 configuration¶

Check your deployed values for the S3 settings your apps are using:

helm get values <release> -n <namespace> | grep -i s3

The v2.1.0 chart defaults are:

global:
  env:
    S3_ENDPOINT: huddo-boards-cp-minio-service
    S3_ACCESS_KEY: ioueygr4t589
    S3_SECRET_KEY: 7a863d41-2d8f-4143-bc8a-02501edbea6f
    S3_BUCKET: huddo-boards   # was 'kudos-boards' in <2.1.0

If you have overridden any of these, use your actual values in the migration configuration below. Existing installs running pre-2.1.0 will have data in the legacy bucket kudos-boards — see the bucket rename note above.

Migration steps¶

Phase 1 — Deploy SeaweedFS and migrate data¶

Apps continue to read and write to MinIO during this phase. SeaweedFS deploys alongside it and the migration Job copies all objects from MinIO to SeaweedFS.

While s3.migration.enabled: true, every helm upgrade of the release starts a new migration Job. From chart 2.5.3 the Job runs rclone copy --update:

It copies objects that are new or changed in MinIO.
It never deletes from SeaweedFS.
It never replaces an object that is newer in SeaweedFS, for example a file the apps changed after cutover.

A repeat run is therefore safe before and after cutover.

Before chart 2.5.3, the migration Job deletes objects from the destination

Charts before 2.5.3 run rclone sync. This command makes the SeaweedFS bucket identical to the MinIO bucket. It also deletes objects from SeaweedFS that are not in MinIO. A repeat run is safe only before cutover. Use chart 2.5.3 or later. If you cannot, read the warning at the start of Phase 2.

Add the following to your values file:

global:
  env:
    # Existing installs only — pin apps to the legacy MinIO bucket name during migration.
    # Drop this override at cutover (Phase 2) to use the new chart default 'huddo-boards'.
    S3_BUCKET: kudos-boards

s3:
  persistence:
    # NFS (recommended for existing NFS-based installations):
    nfs:
      server: <nfs-server>
      path: /pv-connections/huddo-boards-s3
    # Or for dynamic storage provisioning:
    # storageClassName: fast
  migration:
    enabled: true
    source:
      endpoint: "http://huddo-boards-cp-minio-service:9000"
      accessKey: ioueygr4t589
      secretKey: 7a863d41-2d8f-4143-bc8a-02501edbea6f
      bucket: kudos-boards

The migration job copies minio:kudos-boards → seaweedfs:huddo-boards (the new chart-default bucket name).

Important: s3.persistence is required. Without it SeaweedFS has no durable storage and data will be lost on pod restart.

Note: If your MinIO secret key is stored in a Kubernetes Secret rather than in plain text, use secretKeyRef instead of secretKey:
s3:
  migration:
    enabled: true
    source:
      endpoint: "http://huddo-boards-cp-minio-service:9000"
      accessKey: ioueygr4t589
      secretKeyRef:
        name: my-minio-secret
        key: secret-key
      bucket: kudos-boards

Run the upgrade:

# Log in once with the Quay.io credentials you use for our images
helm registry login quay.io

helm upgrade huddo-boards-cp \
  oci://quay.io/huddo/huddo-boards-cp --version 2.6.0 \
  -n <namespace> \
  -f <your-values.yaml>

Verify migration progress¶

The migration runs as a Kubernetes Job. Find the Job name (it includes the Helm release revision):

kubectl get jobs -n <namespace> | grep migration

Follow the rclone logs:

kubectl logs -n <namespace> job/<job-name-from-above> -f

When complete, the Job will log object counts and total size.

Verify data integrity¶

As of chart 2.3.3, the migration Job runs a verification pass after the copy (on by default via s3.migration.verify: true) and prints the report straight to its own logs — no separate pod needed. From chart 2.5.3 the finished Job stays until the next helm upgrade (before 2.5.3 it was deleted 1 hour after it finished). Read the report with:

kubectl logs -n <namespace> job/huddo-boards-cp-s3-migration-<revision>

The report shows:

Totals — object count and total size for source and destination, side by side.
Missing on destination — source objects that were not copied. Should be (none). This is the only list that raises a WARNING.
Size differs — objects whose size is different on the two sides. Before cutover, the apps changed these in MinIO while the Job ran: run the upgrade again to copy them. After cutover, these are files the apps changed in SeaweedFS, and the Job keeps them on purpose.
Extra on destination — objects on SeaweedFS that are not in MinIO: files deleted from MinIO after they were copied, or (after cutover) files the apps wrote to SeaweedFS. The Job never deletes them. Check this list for a wrong-bucket mistake.

Comparison is size-only, so it is cheap and does not download objects. Chart 2.5.3 prints:

>> Missing on destination (source objects not copied):
  (none)
>> Size differs (info: usually changed by the apps on the destination after cutover, kept on purpose):
  (none)
>> Extra on destination (info: deleted from the source, or written by the apps; never deleted):
  (none)
 No source object is missing on the destination.

Charts 2.3.3 to 2.5.2 label the lists "Size mismatch" and "Extra on destination (orphans)", and warn if any list is not empty.

Interpreting a small difference on a busy system: apps keep writing to MinIO during Phase 1, so the source can legitimately grow past the destination after the copy — a handful of "missing on destination" objects with recent timestamps is expected, not a fault. What matters is which objects differ (the report names them), not the raw count. Empty directory markers, zero-byte placeholders, and incomplete multipart uploads on MinIO also account for small gaps and are safe to ignore. To get a stable zero, run the upgrade once more at a quiet time, just before cutover.

If you disabled s3.migration.verify, are on an older chart, or need to re-check without re-running the copy, run rclone against both endpoints from a throwaway pod:

kubectl run rclone-check --rm -it --image=rclone/rclone:latest -n <namespace> -- \
  check \
  ':s3,provider=Other,access_key_id=<key>,secret_access_key=<secret>,endpoint="http://huddo-boards-cp-minio-service:9000",force_path_style=true:kudos-boards' \
  ':s3,provider=Other,access_key_id=<key>,secret_access_key=<secret>,endpoint="http://huddo-boards-cp-s3:8333",force_path_style=true:huddo-boards' \
  --size-only --one-way --missing-on-dst -

Note: the endpoint value contains a :, so it must be double-quoted inside the connection string (and the whole remote single-quoted so the shell keeps the double quotes) — otherwise rclone truncates the endpoint at the first colon and fails with Custom endpoint `http` was not a valid URI.

Tip for large datasets: If your data is too large for a single Job run (e.g. the Job fails or the pod is evicted), run the upgrade again. Each run copies only the objects that are missing or changed. From chart 2.5.3 a repeat run never deletes from SeaweedFS. Before 2.5.3 the Job uses rclone sync, which also deletes destination objects that are missing in the source: run it again only before cutover.

Phase 2 — Cutover to SeaweedFS¶

Once you have verified the data is fully migrated, redirect the apps to SeaweedFS and disable MinIO.

Set s3.migration.enabled: false at cutover (see the values below). From chart 2.5.3, a migration Job that runs again by mistake does not lose data, but it has two side effects:

It copies back files that the apps deleted from SeaweedFS, if MinIO still has them.
It reads the whole MinIO bucket again on every upgrade.

Before chart 2.5.3: disable the migration Job in every values file

After cutover, the apps write new files to SeaweedFS only. These files do not exist in MinIO. On charts before 2.5.3, if the migration Job runs again, rclone sync deletes these files from SeaweedFS. It also replaces changed files with the old versions from MinIO. The verification report does not show this loss. The report runs after the sync, so it lists no extra objects on the destination.

Set s3.migration.enabled: false in every values file and pipeline that deploys this release. This includes old values files, CI/CD variables, and upgrades that use --reuse-values.
After the next upgrade, make sure that the upgrade did not create a migration Job:
```
kubectl get jobs -n <namespace> | grep migration
```
A Job from before cutover can stay for up to 1 hour after it finishes. Check that no Job is new.

Check how MinIO stores its data¶

Setting minio.enabled: false deletes the MinIO volume claim, huddo-boards-minio-claim. What happens to the MinIO data then depends on how its storage was set up:

NFS (the default): the files stay on the NFS server. Nothing to do.
A StorageClass (minio.storageClassName is set): the disk follows the StorageClass reclaim policy. With Delete, the default on most cloud clusters, the cutover upgrade deletes the disk and all MinIO data. You then cannot roll back to MinIO, and files uploaded to MinIO after the last migration run are lost.

Do these steps before the cutover upgrade:

Get the StorageClass and the volume name of the MinIO claim:
```
kubectl get pvc huddo-boards-minio-claim -n <namespace> -o jsonpath='{.spec.storageClassName}{" "}{.spec.volumeName}{"\n"}'
```
If the output has only one name (no StorageClass), MinIO uses NFS. Go to the values file below.

Get the reclaim policy of that volume:

kubectl get pv <volume-name> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}{"\n"}'

If the policy is Delete, change it to Retain so that the disk stays after cutover:
```
kubectl patch pv <volume-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
```
Run step 2 again and make sure that it shows Retain.

With Retain, the disk and its data stay after the claim is deleted, but they are no longer attached to a claim. To roll back to MinIO later, the disk must be reattached by hand: contact support@huddo.com. To free the disk after the soak period, see Reclaiming MinIO data.

Update your values file:

global:
  env:
    S3_ENDPOINT: huddo-boards-cp-s3
    S3_PORT: '8333'
    S3_USE_SSL: 'false'
    # Drop the Phase 1 'S3_BUCKET: kudos-boards' override here so apps fall back
    # to the chart default 'huddo-boards' (matches the SeaweedFS dest bucket).

minio:
  enabled: false

s3:
  migration:
    enabled: false

Note: If you previously had a non-default S3_SECRET_KEY or used secretKeyRef to supply credentials, ensure the apps get the same credentials you configured for SeaweedFS in Phase 1:

Chart 2.6.0 and later: set global.env.S3_SECRET_KEY, or take it from a Secret with a global.envArray entry named S3_SECRET_KEY. The global.envArray entry replaces the chart default.

Chart 2.5.5 and earlier: set global.env.S3_SECRET_KEY to the plain value. A global.envArray entry of the same name is ignored without a warning, and the apps get the chart default key. Setting global.env.S3_SECRET_KEY: null does not remove the default.

The same applies to S3_ACCESS_KEY. If the keys do not match, every S3 call from the apps fails, but the upgrade still reports success.

Run the upgrade:

helm upgrade huddo-boards-cp \
  oci://quay.io/huddo/huddo-boards-cp --version 2.6.0 \
  -n <namespace> \
  -f <your-values.yaml>

This will: 1. Update the S3_ENDPOINT env var in all app pods, triggering a rolling restart 2. Remove the MinIO Deployment, Service, PVC and PV (all rendered from the conditional dependency) 3. Remove the migration Job

Note: Helm prunes the MinIO PVC and PV when minio.enabled: false because they are rendered from the conditional huddo-minio dependency. With NFS storage, the data on the NFS server is not affected — those files persist independently of the K8s objects, so a rollback (re-enable minio.enabled: true) will rebind to the same NFS path. With a StorageClass, the data survives only if you set the reclaim policy to Retain first (Check how MinIO stores its data). To reclaim the space after the soak period, see Reclaiming MinIO data.

Verification¶

1. Apps are using SeaweedFS¶

Confirm the S3_ENDPOINT env var on any app pod points to huddo-boards-cp-s3:

kubectl get pod -n <namespace> -l app=<release>-app \
  -o jsonpath='{.items[0].spec.containers[0].env}' | grep -A1 S3_ENDPOINT

2. MinIO is removed¶

kubectl get deploy -n <namespace> | grep minio
# Should return nothing

3. SeaweedFS is healthy¶

kubectl get pods -n <namespace> | grep huddo-boards-cp-s3
# Should show 1/1 Running

4. Functional verification¶

Log in to Huddo Boards and confirm the board list loads
Open a board with file attachments and confirm files download correctly
Upload a new file attachment and confirm it persists

Troubleshooting¶

Migration Job stays in `Pending` state¶

The rclone container cannot be scheduled. Check for resource constraints:

kubectl get jobs -n <namespace> | grep migration
kubectl describe job <job-name> -n <namespace>

Migration Job fails with `NoSuchBucket` on destination¶

The bucket creation Job must complete before the migration Job starts. The migration Job polls and waits for SeaweedFS to be ready, but if the bucket setup Job itself failed, check its logs:

kubectl logs -n <namespace> job/huddo-boards-cp-s3-bucket-setup

Then re-run the upgrade.

Apps return errors after cutover¶

If the cutover was premature (data not fully migrated), roll back by pointing apps at MinIO again:

global:
  env:
    S3_ENDPOINT: huddo-boards-cp-minio-service
    S3_BUCKET: kudos-boards
    # Remove S3_PORT and S3_USE_SSL overrides to restore defaults

minio:
  enabled: true

helm upgrade huddo-boards-cp \
  oci://quay.io/huddo/huddo-boards-cp --version 2.6.0 \
  -n <namespace> \
  -f <your-values.yaml>

This immediately restores MinIO as the active backend. Files that the apps uploaded or changed after cutover are only in SeaweedFS, so the apps do not see them after the rollback. To keep them, copy them to MinIO with the manual command, with the two paths swapped, once MinIO is running again.

`rclone` is slow on large datasets¶

The default configuration uses 8 parallel transfers and 16 checkers. For very large datasets, you can exec into the migration pod and run rclone manually with tuned settings, or increase the Job timeout by adjusting spec.activeDeadlineSeconds.

Alternatively, run the migration outside of the Helm Job entirely. The first path is the source (MinIO) and the second path is the destination (SeaweedFS). The flags are the same as the migration Job's from chart 2.5.3: the command never deletes from the destination and never replaces an object that is newer there.

kubectl run rclone-manual --rm -it --image=rclone/rclone:latest -n <namespace> -- \
  copy --update --no-update-modtime \
  ':s3,provider=Other,access_key_id=<key>,secret_access_key=<secret>,endpoint="http://huddo-boards-cp-minio-service:9000",force_path_style=true:kudos-boards' \
  ':s3,provider=Other,access_key_id=<key>,secret_access_key=<secret>,endpoint="http://huddo-boards-cp-s3:8333",force_path_style=true:huddo-boards' \
  --progress --transfers 16 --checkers 32

Danger

Do not use rclone sync here. sync makes the destination identical to the source, so it deletes destination objects that are not in the source. With the paths swapped, it deletes your data from MinIO.

Reclaiming MinIO data after the soak period¶

Helm prunes the MinIO PVC and PV automatically when minio.enabled: false (they come from the conditional huddo-minio dependency). The NFS-backed data is not affected — files in kudos-boards-minio under minio.persistentVolumePath (default pv-connections) on the NFS server remain in place so that re-enabling MinIO rebinds to the same data. Once you are confident the SeaweedFS cutover is stable, delete the data on the NFS server manually:

# On the NFS server. Example with the default minio.persistentVolumePath:
rm -rf /pv-connections/kudos-boards-minio

If MinIO used a StorageClass and you set its volume to Retain before cutover, delete the volume:

kubectl delete pv <volume-name>

With Retain, deleting the volume does not delete the disk behind it. Delete the disk in your storage system or cloud console too.