OCI Sync

Pulls an OCI artifact (manifest + layers) from a registry and emits one event per file. Downstream tasks decide what to do with each file’s content — parse it, store it, transform it.

Works with any OCI-compliant registry: GHCR, ECR, GAR, ACR, Artifactory, Harbor, Docker Hub, Quay, self-hosted.

Each UTF-8 text event contains {path, content, digest, artifact_digest} on event.data. The shape mirrors Git Sync so the same downstream pipeline (buffer → diff → cache write) works with either source. Binary layers (or tar entries) that cannot be safely decoded as UTF-8 arrive as EventData::Bytes instead, with {path, digest, artifact_digest} moved to event.meta so downstream tasks such as object_store::write can still route by key.

Both oras push artifacts and Docker container images are supported. oras-style layers — identified by the presence of the org.opencontainers.image.title annotation — emit one event per layer using the annotation as the path. Docker image layers — layers without a title annotation — are merged in manifest order with overlay-fs semantics: later layers override earlier ones, .wh.<name> markers delete files from lower layers, and .wh..wh..opq markers hide entire subtrees. Only the surviving final state is emitted downstream.

Configuration

- oci_sync:
    name: pull_flows
    artifact: "{{env.OCI_FLOWS_ARTIFACT}}"
    credentials_path: /etc/flowgen/credentials/registry.json

Fields

Example: sync flows from a registry into the NATS KV cache

flow:
  tasks:
    - generate:
        name: trigger
        interval: "30s"

    - oci_sync:
        name: pull_repo
        artifact: "{{env.OCI_FLOWS_ARTIFACT}}"
        credentials_path: /etc/flowgen/credentials/registry.json

    - buffer:
        name: collect_layers
        size: 10000
        timeout: "5s"

    - script:
        name: write_cache_keys
        code: |
          let actions = [];
          for layer in event.data.batch {
              // Strip `flows/` prefix and the file extension so the cache
              // key suffix matches the flow's path-based identity.
              let path = regex_replace(
                  layer.path.sub_string(6),
                  "\.(yaml|yml|json)$",
                  ""
              );
              let key = "flows." + path;
              actions.push(#{
                  action: "put",
                  key: key,
                  content: layer.content,
              });
          }
          actions

    - nats_kv_store:
        name: save_to_kv
        operation: put
        bucket: flowgen_system
        key: "{{event.data.key}}"
        credentials_path: /etc/nats/credentials.json
        url: "{{env.NATS_URL}}"

Output

Format: JSON for UTF-8 files, EventData::Bytes for binary files. Each file emitted produces one event.

For UTF-8 files, event.data contains:

For binary files, event.data carries the raw bytes and the same routing fields (path, digest, artifact_digest) land on event.meta instead. Downstream tasks match EventData::Bytes directly and read event.meta.path to know where the payload belongs.

Layer formats

The layer’s mediaType in the manifest dispatches the extractor:

Inside tar and tar+gzip layers, directories, symlinks, and hardlinks are skipped. Only regular files produce events. Docker whiteout markers (.wh.<name>, .wh..wh..opq) are consumed by the layer-merge pass rather than emitted downstream.

This means both push paths work:

• oras push {{env.OCI_FLOWS_ARTIFACT}} flow.yaml:application/yaml — one raw layer per file, path from the :mediaType annotation.
• docker push {{env.OCI_FLOWS_ARTIFACT}} (from a Dockerfile that COPYs YAML files across multiple stages) — layers are merged into a single overlay-fs snapshot; the final state emits one event per surviving file.

Packaging with Docker

When oras is not available in your build environment, docker build + docker push is the supported alternative. Because every file in the image’s final rootfs is emitted as a downstream event, the final stage of the Dockerfile must be FROM scratch — any base image (e.g. busybox, alpine) drops its own /bin, /lib, /etc binaries into the layer, which then arrive at your bootstrap pipeline as EventData::Bytes and break tasks like buffer that expect JSON.

The recommended layout ships flows and resources together in one artifact, called a workspace. Both top-level directories live side by side in the image, and the bootstrap flow routes each layer by its prefix:

flowgen-workspace image
├── flows/
│   └── ...*.yaml
└── resources/
    └── ...

Dockerfile

FROM scratch
LABEL org.opencontainers.image.title="flowgen-workspace"
COPY flows/     /flows/
COPY resources/ /resources/

Build and push

docker build -t registry.example.com/org/flowgen-workspace:prod .
docker push registry.example.com/org/flowgen-workspace:prod

If your CI mandates a base image for provenance scanning, use a multi-stage build and keep the final stage FROM scratch:

FROM your-registry/base:latest AS source
COPY flows/     /source/flows/
COPY resources/ /source/resources/

FROM scratch
LABEL org.opencontainers.image.title="flowgen-workspace"
COPY --from=source /source/ /

Only the final stage ends up in the pushed image, so the layer stays clean.

Verifying the layer

Before pushing, confirm the image’s layer contains only your YAML and resource files — nothing from a base image:

docker save registry.example.com/org/flowgen-workspace:prod -o flowgen-workspace.tar
mkdir -p /tmp/flowgen-workspace && tar -xf flowgen-workspace.tar -C /tmp/flowgen-workspace
tar -tzf /tmp/flowgen-workspace/blobs/sha256/*   # or /tmp/flowgen-workspace/*/layer.tar on older Docker

Every listed entry should live under flows/ or resources/. Any bin/, lib/, etc/, usr/ entries mean the final stage picked up a base image rootfs — fix the Dockerfile to end in FROM scratch before pushing.

Bootstrap flow

examples/oci/sync_workspace.yaml reconciles an OCI artifact into the system cache end-to-end. One artifact carries both flows/ and resources/; the bootstrap routes each layer by its top-level directory — flows/* are keyed by the layer path with the flows/ prefix and file extension stripped (so flows/demo/reader.yaml becomes cache key suffix demo/reader, matching the flow’s path-based identity), resources/* are keyed by the path with the resources/ prefix stripped, and any layer outside those two prefixes (e.g. a README.md) is dropped. It ticks on an interval, lists existing cache entries under both prefixes, and emits one put per layer and one delete per orphaned key.

The flow skips the rest of its pipeline when the artifact digest has not moved, so the only cost on a no-change tick is a manifest HEAD plus a list_keys round-trip. See Resources for how the runtime ResourceLoader reads back from resources.*.

Change detection

Each tick issues an HTTP HEAD against the artifact’s manifest URL to read Docker-Content-Digest from the response header. The digest is compared against the last successful pull, cached under flow.{flow_name}.oci_digest.{artifact} in the shared cache. On a match, the layer blobs are not fetched and the source emits only the upstream completion signal — one line per tick in the logs:

INFO flowgen_oci::sync::processor: OCI manifest digest unchanged since last pull, skipping layer fetch artifact=… digest=sha256:…

Works for both mutable tags (:prod) and immutable digests (@sha256:…) — the digest is authoritative in either case, so a re-tag of :prod to a new release still triggers a pull.

The cached digest is persisted only after every layer event was sent, so a mid-pull failure causes the next tick to re-emit the full batch.

Set force_pull: true to bypass the cache — use only to re-seed a downstream cache mutated out of band.