MinIO Image Pull Failure¶

MinIO has removed its images from Docker Hub (minio/minio) and from Quay.io (quay.io/minio/minio). Kubernetes cannot start a new MinIO pod from those images, so a pod restart or reschedule (or a helm upgrade for Component Pack) fails with ErrImagePull or ImagePullBackOff.

MinIO is discontinued: migrate to SeaweedFS

MinIO no longer publishes community releases or security fixes. Boards now uses SeaweedFS for S3 storage. Huddo keeps a copy of the MinIO image in its private Quay.io registry (quay.io/huddo/minio) only so that MinIO keeps running while you migrate. It is not a long-term fix.

Follow the SeaweedFS migration guide now.

Check if you are affected¶

You are affected if you deploy MinIO with one of these:

The huddo-boards-cp Helm chart before 2.6.0 with MinIO enabled. MinIO is enabled unless your values file sets minio.enabled: false.
The standalone minio.yaml for Kubernetes, if it does not pull quay.io/huddo/minio.

Look for a MinIO pod that is not running:

kubectl get pods --all-namespaces | grep minio
kubectl describe pod <minio-pod-name> --namespace <namespace>

The events show a failed pull of docker.io/minio/minio or quay.io/minio/minio. A pull from Quay.io fails with 401 UNAUTHORIZED.

Note

Our earlier workaround used quay.io/minio/minio. That image is also no longer available. Use the steps on this page instead.

If the pull of quay.io/huddo/minio fails with 401 UNAUTHORIZED

Your Quay.io account cannot read huddo/minio. Email support@huddo.com with the Quay.io account name from your image pull secret (not the password).

Component Pack (`huddo-boards-cp`)¶

Upgrade the chart¶

From huddo-boards-cp 2.6.0, the chart pulls MinIO from quay.io/huddo/minio:RELEASE.2022-10-24T18-35-07Z, with the same image pull secret as your Boards images (global.imagePullSecret). It is the same MinIO version as before, so your data does not change.

Before you upgrade, make sure that you can pull the image. The upgrade stops MinIO before it starts the new MinIO pod. If the new pod cannot pull the image, MinIO stays down.

Then upgrade to the latest chart. Read the Breaking Changes column first: v2.0.0 moves to Traefik.

If your chart is before 2.1.0, set global.env.S3_BUCKET: kudos-boards in your values file before you upgrade, as in Phase 1 of the SeaweedFS migration guide. If you do not, Boards uses a new empty bucket and your existing attachments do not show.
MinIO restarts once during the upgrade, so S3 is unavailable for a short time.
If you applied an earlier workaround (kubectl set image or kubectl patch), the upgrade replaces it. You do not need to run it again.
If you installed the release with Helm 4 and applied a workaround, helm upgrade can fail with conflict with "kubectl-set" or conflict with "kubectl-patch". Add --force-conflicts to the helm upgrade command.
If you pull Huddo images from your own registry (global.repository is not quay.io/huddo), copy quay.io/huddo/minio:RELEASE.2022-10-24T18-35-07Z to <global.repository>/minio:RELEASE.2022-10-24T18-35-07Z before you upgrade.

Make sure that you can pull the image¶

Get the name of the image pull secret that your Boards services use:
```
kubectl get deployments --namespace connections -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.imagePullSecrets[*].name}{"\n"}{end}' | grep huddo-boards-cp
```
Replace connections with your Boards namespace if it is different (see kubectl get pods --all-namespaces | grep minio). Use the same namespace in the next steps.

Start a test pod that pulls the image with that secret. Replace [PULL_SECRET_NAME] with the name from step 1:

kubectl run minio-pull-test --namespace connections --restart=Never --image=quay.io/huddo/minio:RELEASE.2022-10-24T18-35-07Z --overrides='{"spec":{"imagePullSecrets":[{"name":"[PULL_SECRET_NAME]"}]}}' -- --version
kubectl get pod minio-pull-test --namespace connections

If the status is Completed, the pull works. If the status is ErrImagePull or ImagePullBackOff, see the note about 401 UNAUTHORIZED at the top of this page.

Delete the test pod:

kubectl delete pod minio-pull-test --namespace connections

If you cannot upgrade yet¶

Make sure that you can pull the image.
Stop MinIO, so that the old and the new MinIO pod do not use the data volume at the same time:
```
kubectl scale deployment huddo-boards-cp-minio --namespace connections --replicas=0
```

Point the MinIO deployment at the Huddo image and add the image pull secret. Replace [PULL_SECRET_NAME] with the name that you found:

kubectl patch deployment huddo-boards-cp-minio --namespace connections -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"[PULL_SECRET_NAME]"}],"containers":[{"name":"minio","image":"quay.io/huddo/minio:RELEASE.2022-10-24T18-35-07Z"}]}}}}'

Start MinIO and make sure that the pod starts:

kubectl scale deployment huddo-boards-cp-minio --namespace connections --replicas=1
kubectl rollout status deployment/huddo-boards-cp-minio --namespace connections

Do these steps again after every helm upgrade to a chart before 2.6.0

A chart before 2.6.0 sets the image back to Docker Hub. The MinIO pod that is running stays up, but Kubernetes cannot start a replacement pod. Immediately after each such helm upgrade, including the upgrades in the SeaweedFS migration, do the last three steps again: stop MinIO, run kubectl patch, then start MinIO. Do not run kubectl patch alone: the old MinIO pod is still running, so a second MinIO pod would start on the same data volume.

If you installed the release with Helm 4, helm upgrade fails with conflict with "kubectl-patch". Add --force-conflicts to the helm upgrade command, then do the three steps again.

Note

Loading the image onto your nodes yourself does not help. The chart sets imagePullPolicy: Always, so Kubernetes always pulls from the registry.

Migrate to SeaweedFS¶

Follow the SeaweedFS migration guide. The migration needs chart huddo-boards-cp v2.1.0 or later. If your chart is older, upgrade to the latest chart as part of Phase 1 of the migration guide, with the Phase 1 values (including S3_BUCKET: kudos-boards), so that Boards keeps your existing attachments.

After the cutover (minio.enabled: false), MinIO is removed and you do not need the MinIO image.

Standalone Kubernetes (`minio.yaml`)¶

Pull the Huddo image¶

Get the name of the image pull secret that your Boards services use:

kubectl get deployments --namespace boards -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.imagePullSecrets[*].name}{"\n"}{end}'

Replace boards with your Boards namespace if it is different.

In your minio.yaml, add the image pull secret to the MinIO pod and change the image. imagePullSecrets is at the same level as containers. Replace [PULL_SECRET_NAME] with the name from step 1:
```
spec:
  dnsPolicy: ClusterFirst
  imagePullSecrets:
    - name: '[PULL_SECRET_NAME]'
  containers:
    - name: minio
      image: quay.io/huddo/minio:RELEASE.2025-09-07T16-13-09Z
```
Keep the MinIO version that your file already uses:
- If your image ends in :RELEASE.2022-10-24T18-35-07Z, use quay.io/huddo/minio:RELEASE.2022-10-24T18-35-07Z.
- Otherwise (for example minio/minio with no tag), use quay.io/huddo/minio:RELEASE.2025-09-07T16-13-09Z. It is the last MinIO image that was published as latest.
Stop MinIO, so that the old and the new MinIO pod do not use the data volume at the same time. Then apply the file, which starts MinIO again, and make sure that the pod starts:
```
kubectl scale deployment minio --namespace boards --replicas=0
kubectl apply -f ./minio.yaml
kubectl rollout status deployment/minio --namespace boards
```
If the pod cannot pull the image, see the note about 401 UNAUTHORIZED at the top of this page.

Your data is on the MinIO volume (kudos-minio-claim), so it does not change.

Migrate to SeaweedFS¶

Deploy SeaweedFS, then copy your data from MinIO with rclone copy --update, as shown in the manual migration command. Use http://minio-service:9000 as the source endpoint and your SeaweedFS service as the destination. In the command, the first path is the source (MinIO) and the second path is the destination (SeaweedFS). MinIO must be running for the copy.

The command never deletes from the destination and never replaces an object that is newer there, so it is safe to run again.

Danger

Do not use rclone sync. It makes the destination identical to the source, so it deletes destination objects that are not in the source. After the apps write to SeaweedFS, it deletes their new files. With the paths swapped, it deletes your data from MinIO.

Then change S3_ENDPOINT in your boards.yaml to SeaweedFS and upgrade Boards. When the upgrade has finished and no old Boards pods are left, run the copy command once more. This copies the files uploaded to MinIO since the first run, including uploads from old pods during the upgrade.

If you need help, email support@huddo.com.

MinIO Image Pull Failure¶

Check if you are affected¶

Component Pack (huddo-boards-cp)¶

Upgrade the chart¶

Make sure that you can pull the image¶

If you cannot upgrade yet¶

Migrate to SeaweedFS¶

Standalone Kubernetes (minio.yaml)¶

Pull the Huddo image¶

Migrate to SeaweedFS¶

Component Pack (`huddo-boards-cp`)¶

Standalone Kubernetes (`minio.yaml`)¶