> ## Documentation Index
> Fetch the complete documentation index at: https://docs.minimus.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Sync with Hauler for Air-Gapped Kubernetes

> Use Hauler to pull Minimus images into an air-gapped Kubernetes cluster, and avoid a known image-export defect

This guide walks through pulling Minimus container images into an air-gapped Kubernetes cluster using [Hauler](https://hauler.dev). Hauler is commonly used to mirror images (and their attached artifacts) from a connected registry into an air-gapped one. Testing is recommended before running this in production.

Minimus images are signed with [Cosign](https://github.com/sigstore/cosign) and published with attached signature and SBOM artifacts alongside the image itself (`sha256-<digest>.sig` and `sha256-<digest>.sbom` tags). This is standard Cosign tag-based discovery and is how most signed-image registries operate. However, because Minimus images carry these extra signed artifacts they are sensitive to Hauler's export method, as explained below.

## Prerequisites

* Network access from a "jump" or "transfer" host to `reg.mini.dev` (or your Minimus registry endpoint)
* If the image is part of a private/subscription set, a valid Minimus [pull token](/manage/network) for that transfer host to authenticate to the registry (standard Minimus images can be pulled anonymously)
* Hauler **v2.0.3 or later** installed on that transfer host (`hauler version` to check)
* A way to move the resulting Hauler store/bundle into the air-gapped environment (removable media, one-way transfer proxy, etc.)
* `containerd`/`ctr` or registry access on the air-gapped side, depending on which import method you use (see [Import into the air-gapped cluster](#import-into-the-air-gapped-cluster))

<Warning>
  **Before you start: version requirement**

  To avoid a container startup failure, use **Hauler v2.0.3 or later**, and pass **`--exclude-extras`** to `hauler store sync`. For a detailed explanation, see the [known issue](#known-issue-images-fail-with-no-command-specified) below, and steps for [verifying the fix](#verifying-the-fix) to confirm your setup is not affected.
</Warning>

## Building the manifest and syncing images

### Write a Hauler manifest

Create a `hauler-manifest.yaml` listing the Minimus images you need. For Hauler v2, the manifest must use the v2 API version:

```yaml hauler-manifest.yaml theme={null}
apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
  name: minimus-images
spec:
  images:
    - name: reg.mini.dev/postgres:v16.14
```

<Warning>
  Hauler v1's `v1alpha1` manifests are **silently skipped** by Hauler v2 — you will get no error, but nothing will be synced. If you are upgrading an existing manifest from Hauler 1.x, update the `apiVersion` field or your sync will silently fail.
</Warning>

### Pin to an immutable reference (recommended)

Minimus rebuilds tags regularly, so note that the tag is mutable. For a reproducible air-gap bundle, pin to either:

* Image digest (`reg.mini.dev/postgres@sha256:<digest>`)
* Minimus timestamp tag (e.g. `reg.mini.dev/postgres:18.6-202608130124`) [Learn more](/foundations/daily-updates#unique-timestamp-tag)

### Sync the store

```bash theme={null}
hauler store sync --files hauler-manifest.yaml --exclude-extras
```

The `--exclude-extras` flag tells Hauler to skip pulling the cosign `.sig`/`.sbom` artifacts into the store. This flag is required.

## Import into the air-gapped cluster

You have two supported paths. Prefer Option A when possible.

### Option A — `hauler store serve registry` (recommended, safest even with extras)

<Steps>
  <Step title="Move the store">
    Move the resulting store/bundle to the air-gapped node.
  </Step>

  <Step title="Load it">
    ```bash theme={null}
    hauler store load <bundle>.tar.zst
    ```
  </Step>

  <Step title="Serve it locally">
    ```bash theme={null}
    hauler store serve registry
    ```
  </Step>

  <Step title="Point your container runtime at it">
    For example, for RKE2, add a `registries.yaml` mirror entry targeting the address `hauler store serve registry` is listening on.
  </Step>
</Steps>

This approach resolves tags correctly (the served registry keeps signatures and SBOMs under their own `sha256-*.sig` / `.sbom` tags rather than colliding with the image tag), so it works even if `--exclude-extras` was not used during sync.

### Option B — `ctr images import`

<Steps>
  <Step title="Move the store">
    Move the store/bundle to the air-gapped node.
  </Step>

  <Step title="Save the OCI layout">
    ```bash theme={null}
    hauler store save --filename bundle.tar
    ```
  </Step>

  <Step title="Import into containerd">
    ```bash theme={null}
    ctr -n k8s.io images import --no-unpack bundle.tar
    ```
  </Step>
</Steps>

This path is only safe if the store was synced with the flag `--exclude-extras` (see [Known issue](#known-issue-images-fail-with-no-command-specified)). Without it, the imported tag can silently point at the wrong artifact.

## Known issue: images fail with "no command specified"

**Symptom:** A pod using a Minimus image fails to start with kubelet events such as:

```bash wrap  theme={null}
Warning  Failed  ... Error: failed to generate container spec: failed to apply OCI options: no command specified
```

Inspecting the image config on the air-gapped node shows an empty `Entrypoint`/`Cmd`, even though the same tag has the correct entrypoint in the source registry.

**Root cause:** This is a Hauler export defect, not a problem with the Minimus image.

When Hauler exports an OCI layout, it labels the Cosign signature and SBOM artifacts with the same `io.containerd.image.name` annotation as the image itself. `ctr images import` binds a tag on a "last-entry-wins" basis, so if the SBOM or signature artifact is written last in the index, the local tag ends up pointing at that artifact — which has an empty config (`{"config":{}}`) — instead of the actual image manifest. As a result, Kubelet has no `Entrypoint`/`Cmd` to build a container spec from, and the pod fails every time it's created.

A related symptom (mismatched image rootfs and manifest layers on `ctr images import`, without `--no-unpack`) has the same root cause: the SBOM's "layer" is an SPDX JSON document, not an actual filesystem layer, so unpacking it fails.

**Fix:** Use Hauler **≥ v2.0.3** and pass **`--exclude-extras`** to `hauler store sync`. Note that simply upgrading Hauler's version without adding the flag does not fix the issue.

Alternatively, use the `hauler store serve registry` workflow described in Option A above, which works correctly without the additional flag.

**Non-Hauler alternatives:** Tools like `crane copy` or `skopeo copy` copy only the image itself, not the attached signature/SBOM artifacts, so they are not affected by this issue.

## Verifying the fix

After importing on the air-gapped node, confirm the entrypoint survived the transfer before deploying workloads against it:

```bash wrap theme={null}
ctr -n k8s.io images ls | grep postgres        # confirm the tag is present
ctr -n k8s.io content ls                        # sanity check content store
crictl inspecti reg.mini.dev/postgres:v16.14 | grep -A3 Entrypoint
```

You should see the original `Entrypoint`/`Cmd` from the source image (e.g. `/usr/bin/docker-entrypoint.sh postgres`), not an empty array.

## Do not remove signatures/SBOMs to work around this

It may be tempting to ask for an unsigned "air-gap tag" without attached artifacts but this is not recommended. `--exclude-extras` (or `crane copy`/`skopeo copy`) already produces exactly that from the existing tag on demand, without requiring Minimus to stop publishing signatures and SBOMs, which would remove supply-chain verification for every other consumer of the image. If you hit this issue, fix the Hauler workflow.

## References

* [Hauler documentation](https://hauler.dev)
* Hauler `--exclude-extras` flag: [hauler-dev/hauler#541](https://github.com/hauler-dev/hauler/issues/541)


## Related topics

- [Sync to Self-hosted Private Registry](/manage/self-hosted-registry.md)
- [Elevate security with Enterprise Edition](/introduction/enterprise-edition-member.md)
- [Image Tags](/introduction/offering.md)