Fossil SCM
Worked through some difficulties here in applying the runc method on remote systems, then documented what I learned in the containers doc.
Commit
56f4e2ce2f1a6a60832c8f82c336cc46de8fab973a915bd44ae92761ba3f29eb
Parent
d5695c8ef131ac3…
1 file changed
+58
-23
+58
-23
| --- www/containers.md | ||
| +++ www/containers.md | ||
| @@ -468,10 +468,11 @@ | ||
| 468 | 468 | r=$b/rootfs |
| 469 | 469 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 470 | 470 | |
| 471 | 471 | if [ -d "$t" ] && mkdir -p $r |
| 472 | 472 | then |
| 473 | + docker container start $c | |
| 473 | 474 | docker container export $c | sudo tar -C $r -xf - |
| 474 | 475 | id=$(docker inspect --format="{{.Id}}" $c) |
| 475 | 476 | sudo cat $m/$id/config.json | |
| 476 | 477 | jq '.root.path = "'$r'"' | |
| 477 | 478 | jq '.linux.cgroupsPath = ""' > $b/config.json |
| @@ -480,17 +481,17 @@ | ||
| 480 | 481 | |
| 481 | 482 | ---- |
| 482 | 483 | |
| 483 | 484 | The first several lines list configurables: |
| 484 | 485 | |
| 485 | -* **b**: the path of the exported container, called the “bundle” in OCI | |
| 486 | +* **`b`**: the path of the exported container, called the “bundle” in OCI | |
| 486 | 487 | jargon |
| 487 | -* **c**: the name of the Docker container you’re bundling up for use | |
| 488 | +* **`c`**: the name of the Docker container you’re bundling up for use | |
| 488 | 489 | with `runc` |
| 489 | -* **m**: the [moby] directory, both because it’s long and because it’s | |
| 490 | +* **`m`**: the [moby] directory, both because it’s long and because it’s | |
| 490 | 491 | been known to change from one version of Docker to the next |
| 491 | -* **r**: the path of the directory containing the bundle’s root file | |
| 492 | +* **`r`**: the path of the directory containing the bundle’s root file | |
| 492 | 493 | system. |
| 493 | 494 | |
| 494 | 495 | That last doesn’t have to be called `rootfs/`, and it doesn’t have to |
| 495 | 496 | live in the same directory as `config.json`, but it is conventional. |
| 496 | 497 | Because some OCI tools use those names as defaults, it’s best to follow |
| @@ -510,10 +511,16 @@ | ||
| 510 | 511 | human-readable, in case there are other things you want changed in |
| 511 | 512 | this version of the container. Exposing the `config.json` file like |
| 512 | 513 | this means you don’t have to rebuild the container merely to change |
| 513 | 514 | a value like a mount point, the kernel capability set, and so forth. |
| 514 | 515 | |
| 516 | +<a id="why-sudo"></a> | |
| 517 | +We have to do this transformation of `config.json` as the local root | |
| 518 | +user because it isn’t readable by your normal user. Additionally, that | |
| 519 | +input file is only available while the container is started, which is | |
| 520 | +why we ensure that before exporting the container’s rootfs. | |
| 521 | + | |
| 515 | 522 | With the container exported like this, you can start it as: |
| 516 | 523 | |
| 517 | 524 | ``` |
| 518 | 525 | $ cd /path/to/bundle |
| 519 | 526 | $ c=any-name-you-like |
| @@ -544,47 +551,75 @@ | ||
| 544 | 551 | The remaining commands show shutting the container down and destroying |
| 545 | 552 | it, simply to show how these commands change relative to using the |
| 546 | 553 | Docker Engine commands. It’s “kill,” not “stop,” and it’s “delete,” not |
| 547 | 554 | “rm.” |
| 548 | 555 | |
| 549 | -Beware that if you’re doing this on a remote host, your bundle export | |
| 550 | -directory on the build host might not be the same as where it ended up | |
| 551 | -on the remote host. If so, the shell script above will create a broken | |
| 552 | -bundle because it’s assuming the `mkdir` command should go to the same | |
| 553 | -directory as the “`rootfs`” value it set in the `config.json` value. | |
| 554 | -This is a more realistic shell script for that case: | |
| 556 | +If you want the bundle to run on a remote host, the local and remote | |
| 557 | +bundle directories likely will not match, as the shell script above | |
| 558 | +assumes. This is a more realistic shell script for that case: | |
| 555 | 559 | |
| 556 | 560 | ---- |
| 557 | 561 | |
| 558 | 562 | ```shell |
| 559 | -#!/bin/sh | |
| 563 | +#!/bin/bash -ex | |
| 560 | 564 | c=fossil |
| 561 | 565 | b=/var/lib/machines/$c |
| 566 | +h=my-host.example.com | |
| 562 | 567 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 563 | 568 | t=$(mktemp -d /tmp/$c-bundle.XXXXXX) |
| 564 | -r=$t/rootfs | |
| 565 | 569 | |
| 566 | -if [ -d "$t" ] && mkdir -p $r | |
| 570 | +if [ -d "$t" ] | |
| 567 | 571 | then |
| 568 | - docker container export $c | sudo tar -C $r -xf - | |
| 572 | + docker container start $c | |
| 573 | + docker container export $c > $t/rootfs.tar | |
| 569 | 574 | id=$(docker inspect --format="{{.Id}}" $c) |
| 570 | 575 | sudo cat $m/$id/config.json | |
| 571 | - jq '.root.path = "'$r'"' | | |
| 576 | + jq '.root.path = "'$b/rootfs'"' | | |
| 572 | 577 | jq '.linux.cgroupsPath = ""' > $t/config.json |
| 573 | - rsync -av $t/* remotehost:$b | |
| 574 | - sudo rm -rf $t | |
| 578 | + scp -r $t $h:tmp | |
| 579 | + ssh -t $h "{ | |
| 580 | + mv ./$t/config.json $b && | |
| 581 | + sudo tar -C $b/rootfs -xf ./$t/rootfs.tar && | |
| 582 | + rm -r ./$t | |
| 583 | + }" | |
| 584 | + rm -r $t | |
| 575 | 585 | fi |
| 576 | 586 | ``` |
| 577 | 587 | |
| 578 | 588 | ---- |
| 579 | 589 | |
| 580 | -We’ve introduced the “`t`” variable, a temporary directory we populate | |
| 581 | -locally, then `rsync` across to the remote machine, updating a | |
| 582 | -*different* bundle directory, `$b`. We’re using the convention for | |
| 583 | -systemd based machines here, which will play into the [`nspawn`][sdnsp] | |
| 584 | -alternative below. Even if you aren’t using `nspawn`, it’s a reasonable | |
| 585 | -place to put containers under the [Linux FHS rules][LFHS]. | |
| 590 | +We’ve introduced two new variables: | |
| 591 | + | |
| 592 | +* **`h`**: the remote host name | |
| 593 | +* **`t`**: a temporary bundle directory we populate locally, then | |
| 594 | + `scp` to the remote machine, where it’s unpacked | |
| 595 | + | |
| 596 | +We dropped the **`r`** variable because now we have two different | |
| 597 | +“rootfs” types: the tarball and the unpacked version of that tarball. | |
| 598 | +To avoid confusing ourselves between these cases, we’ve replaced uses of | |
| 599 | +`$r` with explicit paths. | |
| 600 | + | |
| 601 | +You need to be aware that this script uses `sudo` for two different purposes: | |
| 602 | + | |
| 603 | +1. To read the local `config.json` file out of the `containerd` managed | |
| 604 | + directory. ([Details above](#why-sudo).) | |
| 605 | + | |
| 606 | +2. To unpack the bundle onto the remote machine. If you try to get | |
| 607 | + clever and unpack it locally, then `rsync` it to the remote host to | |
| 608 | + avoid re-copying files that haven’t changed since the last update, | |
| 609 | + you’ll find that it fails when it tries to copy device nodes, to | |
| 610 | + create files owned only by the remote root user, and so forth. If the | |
| 611 | + container bundle is small, it’s simpler to re-copy and unpack it | |
| 612 | + fresh each time. | |
| 613 | + | |
| 614 | +I point that out because it might ask for your password twice: once for | |
| 615 | +the local sudo command, and once for the remote. | |
| 616 | + | |
| 617 | +The default for the **`b`** variable is the convention for systemd based | |
| 618 | +machines, which will play into the [`nspawn`][sdnsp] alternative below. | |
| 619 | +Even if you aren’t using `nspawn`, it’s a reasonable place to put | |
| 620 | +containers under the [Linux FHS rules][LFHS]. | |
| 586 | 621 | |
| 587 | 622 | [ctrd]: https://containerd.io/ |
| 588 | 623 | [ecg]: https://github.com/opencontainers/runc/pull/3131 |
| 589 | 624 | [LFHS]: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard |
| 590 | 625 | [jq]: https://stedolan.github.io/jq/ |
| 591 | 626 |
| --- www/containers.md | |
| +++ www/containers.md | |
| @@ -468,10 +468,11 @@ | |
| 468 | r=$b/rootfs |
| 469 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 470 | |
| 471 | if [ -d "$t" ] && mkdir -p $r |
| 472 | then |
| 473 | docker container export $c | sudo tar -C $r -xf - |
| 474 | id=$(docker inspect --format="{{.Id}}" $c) |
| 475 | sudo cat $m/$id/config.json | |
| 476 | jq '.root.path = "'$r'"' | |
| 477 | jq '.linux.cgroupsPath = ""' > $b/config.json |
| @@ -480,17 +481,17 @@ | |
| 480 | |
| 481 | ---- |
| 482 | |
| 483 | The first several lines list configurables: |
| 484 | |
| 485 | * **b**: the path of the exported container, called the “bundle” in OCI |
| 486 | jargon |
| 487 | * **c**: the name of the Docker container you’re bundling up for use |
| 488 | with `runc` |
| 489 | * **m**: the [moby] directory, both because it’s long and because it’s |
| 490 | been known to change from one version of Docker to the next |
| 491 | * **r**: the path of the directory containing the bundle’s root file |
| 492 | system. |
| 493 | |
| 494 | That last doesn’t have to be called `rootfs/`, and it doesn’t have to |
| 495 | live in the same directory as `config.json`, but it is conventional. |
| 496 | Because some OCI tools use those names as defaults, it’s best to follow |
| @@ -510,10 +511,16 @@ | |
| 510 | human-readable, in case there are other things you want changed in |
| 511 | this version of the container. Exposing the `config.json` file like |
| 512 | this means you don’t have to rebuild the container merely to change |
| 513 | a value like a mount point, the kernel capability set, and so forth. |
| 514 | |
| 515 | With the container exported like this, you can start it as: |
| 516 | |
| 517 | ``` |
| 518 | $ cd /path/to/bundle |
| 519 | $ c=any-name-you-like |
| @@ -544,47 +551,75 @@ | |
| 544 | The remaining commands show shutting the container down and destroying |
| 545 | it, simply to show how these commands change relative to using the |
| 546 | Docker Engine commands. It’s “kill,” not “stop,” and it’s “delete,” not |
| 547 | “rm.” |
| 548 | |
| 549 | Beware that if you’re doing this on a remote host, your bundle export |
| 550 | directory on the build host might not be the same as where it ended up |
| 551 | on the remote host. If so, the shell script above will create a broken |
| 552 | bundle because it’s assuming the `mkdir` command should go to the same |
| 553 | directory as the “`rootfs`” value it set in the `config.json` value. |
| 554 | This is a more realistic shell script for that case: |
| 555 | |
| 556 | ---- |
| 557 | |
| 558 | ```shell |
| 559 | #!/bin/sh |
| 560 | c=fossil |
| 561 | b=/var/lib/machines/$c |
| 562 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 563 | t=$(mktemp -d /tmp/$c-bundle.XXXXXX) |
| 564 | r=$t/rootfs |
| 565 | |
| 566 | if [ -d "$t" ] && mkdir -p $r |
| 567 | then |
| 568 | docker container export $c | sudo tar -C $r -xf - |
| 569 | id=$(docker inspect --format="{{.Id}}" $c) |
| 570 | sudo cat $m/$id/config.json | |
| 571 | jq '.root.path = "'$r'"' | |
| 572 | jq '.linux.cgroupsPath = ""' > $t/config.json |
| 573 | rsync -av $t/* remotehost:$b |
| 574 | sudo rm -rf $t |
| 575 | fi |
| 576 | ``` |
| 577 | |
| 578 | ---- |
| 579 | |
| 580 | We’ve introduced the “`t`” variable, a temporary directory we populate |
| 581 | locally, then `rsync` across to the remote machine, updating a |
| 582 | *different* bundle directory, `$b`. We’re using the convention for |
| 583 | systemd based machines here, which will play into the [`nspawn`][sdnsp] |
| 584 | alternative below. Even if you aren’t using `nspawn`, it’s a reasonable |
| 585 | place to put containers under the [Linux FHS rules][LFHS]. |
| 586 | |
| 587 | [ctrd]: https://containerd.io/ |
| 588 | [ecg]: https://github.com/opencontainers/runc/pull/3131 |
| 589 | [LFHS]: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard |
| 590 | [jq]: https://stedolan.github.io/jq/ |
| 591 |
| --- www/containers.md | |
| +++ www/containers.md | |
| @@ -468,10 +468,11 @@ | |
| 468 | r=$b/rootfs |
| 469 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 470 | |
| 471 | if [ -d "$t" ] && mkdir -p $r |
| 472 | then |
| 473 | docker container start $c |
| 474 | docker container export $c | sudo tar -C $r -xf - |
| 475 | id=$(docker inspect --format="{{.Id}}" $c) |
| 476 | sudo cat $m/$id/config.json | |
| 477 | jq '.root.path = "'$r'"' | |
| 478 | jq '.linux.cgroupsPath = ""' > $b/config.json |
| @@ -480,17 +481,17 @@ | |
| 481 | |
| 482 | ---- |
| 483 | |
| 484 | The first several lines list configurables: |
| 485 | |
| 486 | * **`b`**: the path of the exported container, called the “bundle” in OCI |
| 487 | jargon |
| 488 | * **`c`**: the name of the Docker container you’re bundling up for use |
| 489 | with `runc` |
| 490 | * **`m`**: the [moby] directory, both because it’s long and because it’s |
| 491 | been known to change from one version of Docker to the next |
| 492 | * **`r`**: the path of the directory containing the bundle’s root file |
| 493 | system. |
| 494 | |
| 495 | That last doesn’t have to be called `rootfs/`, and it doesn’t have to |
| 496 | live in the same directory as `config.json`, but it is conventional. |
| 497 | Because some OCI tools use those names as defaults, it’s best to follow |
| @@ -510,10 +511,16 @@ | |
| 511 | human-readable, in case there are other things you want changed in |
| 512 | this version of the container. Exposing the `config.json` file like |
| 513 | this means you don’t have to rebuild the container merely to change |
| 514 | a value like a mount point, the kernel capability set, and so forth. |
| 515 | |
| 516 | <a id="why-sudo"></a> |
| 517 | We have to do this transformation of `config.json` as the local root |
| 518 | user because it isn’t readable by your normal user. Additionally, that |
| 519 | input file is only available while the container is started, which is |
| 520 | why we ensure that before exporting the container’s rootfs. |
| 521 | |
| 522 | With the container exported like this, you can start it as: |
| 523 | |
| 524 | ``` |
| 525 | $ cd /path/to/bundle |
| 526 | $ c=any-name-you-like |
| @@ -544,47 +551,75 @@ | |
| 551 | The remaining commands show shutting the container down and destroying |
| 552 | it, simply to show how these commands change relative to using the |
| 553 | Docker Engine commands. It’s “kill,” not “stop,” and it’s “delete,” not |
| 554 | “rm.” |
| 555 | |
| 556 | If you want the bundle to run on a remote host, the local and remote |
| 557 | bundle directories likely will not match, as the shell script above |
| 558 | assumes. This is a more realistic shell script for that case: |
| 559 | |
| 560 | ---- |
| 561 | |
| 562 | ```shell |
| 563 | #!/bin/bash -ex |
| 564 | c=fossil |
| 565 | b=/var/lib/machines/$c |
| 566 | h=my-host.example.com |
| 567 | m=/run/containerd/io.containerd.runtime.v2.task/moby |
| 568 | t=$(mktemp -d /tmp/$c-bundle.XXXXXX) |
| 569 | |
| 570 | if [ -d "$t" ] |
| 571 | then |
| 572 | docker container start $c |
| 573 | docker container export $c > $t/rootfs.tar |
| 574 | id=$(docker inspect --format="{{.Id}}" $c) |
| 575 | sudo cat $m/$id/config.json | |
| 576 | jq '.root.path = "'$b/rootfs'"' | |
| 577 | jq '.linux.cgroupsPath = ""' > $t/config.json |
| 578 | scp -r $t $h:tmp |
| 579 | ssh -t $h "{ |
| 580 | mv ./$t/config.json $b && |
| 581 | sudo tar -C $b/rootfs -xf ./$t/rootfs.tar && |
| 582 | rm -r ./$t |
| 583 | }" |
| 584 | rm -r $t |
| 585 | fi |
| 586 | ``` |
| 587 | |
| 588 | ---- |
| 589 | |
| 590 | We’ve introduced two new variables: |
| 591 | |
| 592 | * **`h`**: the remote host name |
| 593 | * **`t`**: a temporary bundle directory we populate locally, then |
| 594 | `scp` to the remote machine, where it’s unpacked |
| 595 | |
| 596 | We dropped the **`r`** variable because now we have two different |
| 597 | “rootfs” types: the tarball and the unpacked version of that tarball. |
| 598 | To avoid confusing ourselves between these cases, we’ve replaced uses of |
| 599 | `$r` with explicit paths. |
| 600 | |
| 601 | You need to be aware that this script uses `sudo` for two different purposes: |
| 602 | |
| 603 | 1. To read the local `config.json` file out of the `containerd` managed |
| 604 | directory. ([Details above](#why-sudo).) |
| 605 | |
| 606 | 2. To unpack the bundle onto the remote machine. If you try to get |
| 607 | clever and unpack it locally, then `rsync` it to the remote host to |
| 608 | avoid re-copying files that haven’t changed since the last update, |
| 609 | you’ll find that it fails when it tries to copy device nodes, to |
| 610 | create files owned only by the remote root user, and so forth. If the |
| 611 | container bundle is small, it’s simpler to re-copy and unpack it |
| 612 | fresh each time. |
| 613 | |
| 614 | I point that out because it might ask for your password twice: once for |
| 615 | the local sudo command, and once for the remote. |
| 616 | |
| 617 | The default for the **`b`** variable is the convention for systemd based |
| 618 | machines, which will play into the [`nspawn`][sdnsp] alternative below. |
| 619 | Even if you aren’t using `nspawn`, it’s a reasonable place to put |
| 620 | containers under the [Linux FHS rules][LFHS]. |
| 621 | |
| 622 | [ctrd]: https://containerd.io/ |
| 623 | [ecg]: https://github.com/opencontainers/runc/pull/3131 |
| 624 | [LFHS]: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard |
| 625 | [jq]: https://stedolan.github.io/jq/ |
| 626 |