Working with CoreOS Assembler

  1. Understanding “config git”
  2. Components of “config git”
    1. manifest.yaml
    2. overlay.d/
    3. image.yaml
  3. Hacking on “config git”
  4. Using overrides
  5. Using cosa build-fast
  6. Using cosa run --bind-ro for even faster iteration
  7. Using host binaries
  8. Using cosa buildinitramfs-fast
  9. Performing an in-place OS update manually
  10. Using different CA certificates
  11. Running CoreOS Assembler in OpenShift on Google Compute Platform
  12. I’m a contributor investigating a CoreOS bug. How can I test my fixes?

Understanding “config git”

Conceptually, coreos-assembler ties together generating OSTree commits with disk images into a single “build schema”. The build target is defined by the “config git” which should be in src/config (relative to the build directory).

We can hack on some local input configs by exporting them in the COREOS_ASSEMBLER_CONFIG_GIT env variable. For example:

$ export COREOS_ASSEMBLER_CONFIG_GIT=/path/to/github.com/coreos/fedora-coreos-config/
$ cosa init --force /dev/null
$ cosa fetch && cosa build

Components of “config git”

manifest.yaml

For generating OSTree commits, cosa uses manifest.yaml: An rpm-ostree “manifest” or “treefile”, which mostly boils down to a list of RPMs and a set of rpm-md repositories they come from. It also supports postprocess to make arbitrary changes. See the rpm-ostree documentation for the Treefile format reference.

overlay.d/

coreos-assembler also supports a generic way to embed architecture-independent configuration and scripts by creating subdirectories in overlay.d/. Each subdirectory of the overlay.d. directory is added to the OSTree commit, in lexicographic order. It’s recommended to name directories with a numeric prefix - e.g. 05core, 10extras. Non-directories are ignored. For example, a good practice is to add a README.md file into your overlay directories describing their structure.

image.yaml

This YAML file configures the output disk images. Supported keys are:

  • size: Size in GB for cloud images (OpenStack, AWS, etc.) Required.
  • extra-kargs: List of kernel arguments.
  • include: path to another YAML file to include. List values are appended. For other type, the values in the sourcing config win.

It’s likely in the future we will extend this to support e.g. a separate /var partition or configuring the filesystem types. If you want to do anything like that today it requires forking the assembler and rebuilding it. See the fedora-coreos-config for an example.

Hacking on “config git”

First you can expand the size of the image; edit src/config/image.yaml and e.g. change 8 to 9. Rerun cosa build, and notice that the OSTree commit didn’t change, but a new image is generated in builds. When you cosa run, you’ll get it.

Another thing to try is editing src/config/manifest.yaml - add or remove entries from packages. You can also add local rpm-md file:/// repositories.

Using overrides

Development speed is closely tied to the “edit-compile-debug” cycle. coreos-assembler supports an overrides/ sub-directory of the coreos-assembler working directory, which allows easily overlaying locally-generated content on top of the base OS content.

There are two subdirectories of overrides/:

  • overrides/rootfs
  • overrides/rpm

Let’s say you want to hack on both ostree and ignition-dracut. See for example this PR which added support for make install DESTDIR= to the latter. In general most upstream build systems support something like this; if they don’t it’s a good idea to add.

Concretely, if /path/to/cosa-workdir is where you ran cosa init, then after doing edits in a project, run a command like this from the source repository for the component:

$ make install DESTDIR=/path/to/cosa-workdir/overrides/rootfs

This would then install files like /path/to/cosa-buildroot/overrides/rootfs/usr/bin/ostree etc.

If you then run cosa build from the cosa workdir, those overrides will be automatically incorporated.

You can also choose to use overrides/rpm - this accepts pre-built binary RPMs. This can be convenient when you want to quickly test a binary RPM built elsewhere, or if you want to go through a more “official” build process. If any RPMs are present here, then coreos-assembler will automatically run createrepo_c and ensure that they are used in the build.

In the future, it’s likely coreos-assembler will also support something like overrides/src which could be a directory of symlinks to local git repositories.

Using cosa build-fast

If you’re working on e.g. the kernel or Ignition (things that go into the initramfs), then you probably need a cosa build workflow (or cosa buildinitramfs-fast, see below). However, let’s say you want to test a change to something that runs purely in the real root, such as e.g. rpm-ostree, podman, or console-login-helper-messages. Rather than doing a full image build each time, a fast way to test out changes is to use cosa build-fast.

This command assumes you have a previous local coreos-assembler build. From the git checkout of the project you want to add:

$ export COSA_DIR=/srv/builds/fcos
$ cosa build-fast
$ cosa run

The cosa build-fast command will run make and inject the resulting binaries on a qcow2 overlay file, which will appear in your project working directory. The cosa run command similarly knows to look for these qcow2 files.

This will not affect the “real” cosa build in /srv/builds/fcos, but will use it as a data source.

Using cosa run --bind-ro for even faster iteration

This workflow can be used alongside cosa build-fast, or separate from it. If you invoke e.g.

$ cosa run --bind-ro ~/src/github/containers/podman,/run/workdir

The target VM will have the source directory bound in /run/workdir; then you can directly copy binaries from your development environment into the VM.

If you are running cosa in a container, you will have to change your current working directory to a parent directory common to both project directories and use relative paths:

$ cd ~
$ cosa run \
      --qemu-image src/fcos/build/latest/x86_64/fedora-coreos-*.x86_64.qcow2 \
      --bind-ro src/github/containers/podman,/run/workdir

Then in the booted VM, /run/workdir will point to the libpod directory on your host, allowing you to directly execute binaries from there. You can also use e.g. rpm-ostree usroverlay and then copy binaries from your host /run/workdir into the VM’s rootfs.

Using host binaries

Another related trick is:

$ cosa run --bind-ro /usr/bin,/run/hostbin

Then in the VM you have e.g. /run/hostbin/strace. (This may fail in some scenarios where your dev container is different than the target).

If you are running cosa in a container, you will only have access to the binary installed in this container. You can install binaries before launching the VM with:

$ cosa shell
$ sudo dnf install ...
$ cosa run --bind-ro /usr/bin,/run/hostbin

Using cosa buildinitramfs-fast

If you’re iterating on changes just to the initramfs, you can also use cosa buildinitramfs-fast. For example, suppose you are working on ignition. Follow these steps:

$ make
$ install -D -m 0755 bin/amd64/ignition /path/to/cosadir/overrides/initramfs/usr/bin/ignition
$ cd /path/to/cosadir
$ cosa buildinitramfs-fast
$ cosa run --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign

(Or instead of cosa run use e.g. cosa kola to run tests, etc.)

If you’re adding new fields to the Ignition experimental spec, -i will silently remove those fields, since the copy of Ignition that’s vendored into mantle won’t know about them yet. Instead, you can specify that kola should pass the config to the machine unmodified:

$ kola qemuexec --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign --ignition-direct

You’ll need to manually configure autologin in the Ignition config, since kola won’t be able to do it for you.

Performing an in-place OS update manually

The output of coreos-assembler is conceptually two things:

  • an ostree container image
  • disk images (ISO, AWS AMI, qemu .qcow2, etc)

In many cases, rather than booting from a new disk image with the new OS, you will want to explicitly test in-place upgrades. This uses an ostree native container, which is in the form of an .ociarchive file generated by cosa build container (as well as the default cosa build, which also generates a qemu disk image).

You will need to make the container image available to your targeted system (VM or physical). One way to do this is to push the container to a public registry such as quay.io:

cosa push-container quay.io/exampleuser/fcos

Performing an in-place update from the cosa build output boils down to invoking a command of the form:

$ rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/exampleuser/fcos
$ systemctl reboot

Using different CA certificates

If you need access to CA certificates on your host (for example, when you need to access a git repo that is not on the public Internet), you can mount in the host certificates using the COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS variable.

NOTE Sharing the /etc/pki/ca-trust directory may be blocked by SELinux so you may have to use a directory with the system_u:object_r:container_file_t:s0 file context.

$ export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS='-v /etc/pki/ca-trust:/etc/pki/ca-trust:ro'
$ cosa init https://github.com/coreos/fedora-coreos-config.git
$ cosa fetch && cosa build

See this Stack Overflow question for additional discussion.

Running CoreOS Assembler in OpenShift on Google Compute Platform

This is a guide to run a COSA pod in an OpenShift 4.6+ cluster in Google Compute Platform (GCP).

Install the KVM device plugin.

Up to this point, you needed to be kubeadmin. From this point on though, best practice is to switch to an “unprivileged” user.

(In fact the steps until this point could be run by a separate team that manages the cluster; other developers could just use it as unprivileged users)

Personally, I added a httpasswd identity provider and logged in with a password.

I also did oc new-project coreos-virt etc.

Schedule a cosa pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: cosa
  name: cosa
spec:
  containers:
  - args:
    - shell
    - sleep
    - infinity
    image: quay.io/coreos-assembler/coreos-assembler:latest
    name: cosa
    resources:
      requests:
        # Today COSA hardcodes 2048 for launching VMs.  We could
        # probably shrink that in the future.
        memory: "3Gi"
        devices.kubevirt.io/kvm: "1"
      limits:
        memory: "3Gi"
        devices.kubevirt.io/kvm: "1"
    volumeMounts:
    - mountPath: /srv
      name: workdir
  volumes:
  - name: workdir
    emptyDir: {}
  restartPolicy: Never

Then oc rsh pods/cosa and you should be able to ls -al /dev/kvm - and cosa build etc!

I’m a contributor investigating a CoreOS bug. How can I test my fixes?

If you’re a contributor unfamiliar with how CoreOS is built or provisioned and you’re looking at fixing a bug in an RPM component (thank you!), here’s the minimum you need to know to get started debugging and iterating on CoreOS. This requires access to podman and /dev/kvm.

  1. First, download the latest QEMU CoreOS image. If you’re handling a bug report from someone on the CoreOS team, they may have asked you to download a specific image. Otherwise, for Fedora, you can get the latest from https://fedoraproject.org/coreos/download or by using coreos-installer:

    # as privileged
podman run --privileged --rm -v .:/srv -w /srv quay.io/coreos/coreos-installer:release \
    download --decompress -p qemu -f qcow2.xz
# as unprivileged, but relabeling the working directory
podman run --rm -v .:/srv:z -w /srv quay.io/coreos/coreos-installer:release \
    download --decompress -p qemu -f qcow2.xz

    For RHCOS, you can get an image from the OpenShift mirrors at https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/. If you are a Red Hat employee, you can also access development builds not yet available on the mirrors from the RHCOS release browser.

  2. Run the VM:

    # as privileged
podman run --privileged --rm -ti -v .:/srv quay.io/coreos-assembler/coreos-assembler \
    run --bind-ro /srv,/var/srv fedora-coreos-40.20240519.3.0-qemu.x86_64.qcow2
# as unprivileged, but relabeling the working directory and passing /dev/kvm
podman run --rm -ti -v .:/srv:z --device /dev/kvm quay.io/coreos-assembler/coreos-assembler \
    run --bind-ro /srv,/var/srv fedora-coreos-40.20240519.3.0-qemu.x86_64.qcow2

    This will mount the working directory at /srv as read-only in the VM so that you can more easily pass data into it.

  3. Modifying the OS:

    Once you’re in the VM, you can test your changes multiple ways.

    (1) One universal way is to build a custom RPM, and put it in the mounted directory, then use rpm-ostree override replace /srv/path/to/my.rpm. You can then either reboot, or rpm-ostree apply-live to have the change apply immediately.

    (2) Alternatively, you can do rpm-ostree usroverlay to make the rootfs writable. Then you can do e.g. rpm -Uvh /srv/my.rpm for example as you would on a traditional system. Or at this point, you can directly rsync over the output from your local project build. E.g. assuming your build system uses a Makefile and it supports DESTDIR=, on your host, you can do make install DESTDIR=/path/to/mounted/dir/subdir, and in the VM, rsync -av /srv/subdir/ /.

    (3) Another newer approach is to build a derived container image with your changes. For example, to customize the rawhide image, you can build a Containerfile like:

    FROM quay.io/fedora/fedora-coreos:rawhide
# using a locally built RPM
COPY my.rpm /
RUN dnf install /my.rpm && dnf clean all
# or using project output directly from a `make install DESTDIR=installtree/
COPY installtree/ /tmp
RUN rsync /tmp/ / && rm -rf /tmp

    You can build this image, then copy it to the VM as an OCI archive:

    podman build -t localhost/my-image .
skopeo copy containers-storage:localhost/my-image oci-archive:/path/to/mounted/dir/my.ociarchive

    And then on the VM:

    rpm-ostree rebase ostree-unverified-image:oci-archive:/srv/my.ociarchive

    (4) That said, sometimes you just have to build FCOS/RHCOS from scratch. In that case, follow the steps in https://coreos.github.io/coreos-assembler/building-fcos/ and see the other sections on this page for ways to modify inputs (e.g. custom RPMs or rootfs content). But roughly, the flow looks something like this:

    # for building FCOS
cosa init https://github.com/coreos/fedora-coreos-config
# for building SCOS
cosa init --variant c9s https://github.com/openshift/os
# copy any modifications in e.g. the overrides/rpm directory
cp /path/to/my.rpm overrides/rpm/
# or overrides/rootfs directory
(cd /path/to/my/project && make install DESTDIR=/path/to/overrides/rootfs)
# now we're ready to build
cosa fetch && cosa build
# to run the freshly built image
cosa run
# to build the live ISO
cosa buildextend-metal && cosa buildextend-live --fast
# to run the live ISO
cosa run -p qemu-iso