Podman support

[gastaldi]

It would be really nice if TestContainers supported podman.
This is a real game changer to no longer depend on Docker daemons to run.

[bsideup]

Thanks for asking! Since this is not the first time somebody asks for Podman, I would like to post a (non complete, most probably) list of requirements we need to do the Testcontainers' magic:

• Cross-platform support. It should work exactly the same on Linux, Mac and Windows, without having us do the VM management
• Docker networks, ability to start/connect/disconnect containers with/to Networks
• Image management - pulling (including the various authorisation mechanisms), listing, building
• Long running processes that survive the test run
• Access to /var/run/docker.sock from inside the container, ability to do regular Docker commands with it
• Log access/streaming
• Stdin/Stdout/Stderr streaming
• There are many Docker images that require root user, these should work too
• Volumes management

I am not a Podman expert, and if somebody from the community can help clarifying these, I would be happy to learn, and, as long as every point is clarified (the list is not 100% complete and we may need to add more in future) we can consider adding the support for Podman.

[bsideup]

and BTW, "daemonless" here makes it harder for Testcontainers :)

[gastaldi]

From my Podman experience:

• Access to /var/run/docker.sock : I don't think so, since there is no daemon;
• Cross Platform support: Podman only works in Linux. I haven't checked the internals of Testcontainers yet, but Podman offers an alias to replace the docker command, so Testcontainers could use Podman in Linux with the same CLI syntax as used in Docker (if the CLI is used);
• Image Building: is performed by Buildah

I wish @rhatdan could chime in the discussion to clarify or correct my statements 😃

[bsideup]

if the CLI is used

We're not using the CLI, but the REST API.

It looks to me that we should rather focus on waiting for the rootless Docker (an ongoing effort upstream), Podman as it is (plus Buildah, plus whatever else is needed) cannot provide the same functionality we need to do what we're doing in Testcontainers.

[Kidlike]

Dropping this here because I found it useful and relevant to the discussion :)

https://medium.com/nttlabs/cgroup-v2-596d035be4d7

[ppalaga]

We're not using the CLI, but the REST API.

Podman recently started offering a REST API: https://podman.io/blogs/2020/01/17/podman-new-api.html

[bsideup]

@ppalaga nice!

Do you have a compatibility table or something, to see which Docker's API endpoints are supported or not? Thanks!

[crunchtime-ali]

@bsideup
I would help putting that table together. Is there a list of Docker API endpoints (and options) testcontainers uses or a test-suite we can run against podman's Docker compability API.

[bsideup]

@crunchtime-ali I believe running Testcontainers' test suite is the best way to check whether all APIs are supported or not. Our tests should cover all types of Docker features we are using.

You can also start a TCP proxy between Testcontainers and Podman to collect the list of endpoints that are used.

[rocketraman]

Some more information that might be helpful:

https://github.com/containers/libpod/blob/master/docs/source/markdown/podman-system-service.1.md

https://github.com/containers/libpod/blob/master/API.md

The Podman API supposedly has a docker API compatible endpoint.

[rocketraman]

And it can also expose a sock socket file as well, which should meet the requirement of having a /var/run/docker.sock (or equivalent) inside the container.

[rhatdan]

Correct we are under heavy development and are looking at testcontainers to prove that we handle the Docker API correctly. We have an extended version to support advanced features of Podman as well.

[cevich]

I'm also looking at doing the reverse: Running testcontainers tests in podman CI. Solving the challenges here, will help me get to there. (subscribed)

[baude]

how does one run the test suite? hints around how to specify the sock to connect to is a bonus. ill do the work.

[rocketraman]

@baude I'm not a testcontainers dev, but I suspect you can get insight into both of your questions from:

https://github.com/testcontainers/testcontainers-java/blob/master/.circleci/config.yml

and

https://github.com/testcontainers/testcontainers-java/blob/master/core/src/main/java/org/testcontainers/dockerclient/UnixSocketClientProviderStrategy.java

[bsideup]

@baude

Thanks for working on it!

we read the same environment variables as the Docker CLI does (as one of the strategies we support), e.g. DOCKER_HOST

By default, we also search for Docker environment in well-known locations (like /var/run/docker.sock)

Our primary CI is Azure Pipelines:
https://github.com/testcontainers/testcontainers-java/blob/1.12.5/azure-pipelines.yml

[kiview]

@baude @rhatdan
Great to see you get involved and that we can hope for great progress towards feature parity and Podman compatibility in Testcontainers. I know about a lot of Testcontainers users in more strict enterprise environments that would be very happy about Podman support 🙂

If there is any way I can support you, feel free to get in touch or join our Slack, if you want to have more extensive discussions.

[cevich]

Thanks @baude for taking this up, the java-stuff is out of my depth, but poke me on IRC if/when I can help with any automation-things.

[baude]

recent update here ... our compatibility layer requires the use of versioned paths (by api). the testcontainers suite does not appear to deal with that. i wrote a small java app using docker-java, which test containers also uses, and by default the versioned paths are not used there but it does provide the capability to do so with:

.withApiVersion("1.24")

So we need some sort of way to configure that in testcontainers and perhaps things will progress.

[baude]

update! a contributor has fixed ^^ so now just a matter of understanding how to dissect the test failures.

[ruddy32]

Defining DOCKER_HOST with /var/run/user/1000/podman/podman.sock generate this error: java.io.IOException: [111] Connection refused, even if the socket is created by the same Linux user account (running podman api with command systemctl --user start podman).

[bsideup]

Defining DOCKER_HOST with /var/run/user/1000/podman/podman.sock generate this error: java.io.IOException: [111] Connection refused, even if the socket is created by the same Linux user account (running podman api with command systemctl --user start podman).

Podman != Docker.

[Stexxen]

Just I'd mention that I came to this ticket after installing Fedora 32, realising I couldn't get docker-ce to run without lots of kernel/firewal changes docker/for-linux#955

Looks like redhat are pushing podman
https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/

So until/if the docker ticket is fixed, future fedora installations of docker are out of reach for layman users.
However I'm wiping and rebuilding with 31, can't live without testcontainer ;-)

[bsideup]

Looks like redhat are pushing podman

It is so indeed, although Podman does not support all Docker's features and does not work with tools like Docker Compose, Fabric8 Docker Maven Plugin by Red Hat ( fabric8io/docker-maven-plugin#1330 ), Testcontainers and others.

It is a bit sad to see a lot of push with statements like "just replace Docker CLI with Podman CLI and everything will be working" while, in reality, there is a ton of things missing.
There were some attempts at adding Docker compatibility layer to Podman, but last information I heard was that it does not pass Testcontainers' test suite.

However I'm wiping and rebuilding with 31, can't live without testcontainer ;-)

Thank you! FYI @kiview is running Fedora 32 with Docker, he just had to apply a couple of modifications (cgroups v1, firewall settings)

---

Also FYI, I did some experiments with Rootless Docker and apparently it is working! (in PoC mode)
I will explore further because I believe that this is the way to go - good old Docker, just without the root requirement.

[ppalaga]

It is a bit sad to see a lot of push with statements like "just replace Docker CLI with Podman CLI and everything will be working" while, in reality, there is a ton of things missing.

Could you please list which specific features Podman is missing to be able to serve as a replacement for Docker in Testcontainers?

[bsideup]

@ppalaga sorry, but I don't know (I am on Mac and can't use Podman to even test)

perhaps ask @baude:
#2088 (comment)

Also, If Podman serves as a drop-in replacement for Docker (as claimed), it should be trivial to test it - just run Testcontainers' and docker-java's test suites with it.

[rhatdan]

The original goal was Podman as a drop in replacement for Docker CLI. Which it has done a pretty damn good job at. The next level is to have Podman implement the Docker API, which is ongoing and heavily being worked on right now. We call this APIV2. You should be seeing Release candidates for Podman 2.0 right now, where we are beginning the testing on it. If people could test suites like Testcontainers and docker-java's test suite, we would like to see the results.

Potential issues we have seen so far are:

• Test suites like docker-py which rely on Docker-swarm, which we don't intend on implementing.
• Docker-compose has issues since Podman networking and Docker networking are very different.
  we are attempting to work around these issues, to see where we are on these test suites.

[rhatdan]

Bottom line on this, is we need help from the community to help with tests, and fixes to get to the point where we can support the Docker.sock API.

[thedewpoint]

@msausu any success with this? wanting to do the same

[guss77]

Just a status report on minimal testing (running JUnit with testcontainers):

• Using Ubuntu 21.04, installed podman 3.0.1 from Ubuntu
• Podman socket-activated user service is preconfigured and enabled
• I added export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock to ~/.profile
• I created ~/.config/containers/registries.conf to set automatic search in docker.io (its the default for Docker servers, but usually isn't for Podman installations) - otherwise docker pull would fail
• log out and back in to let the profile reload.

Testing works fine - I use testcontainers, mariadb, redis and a few other containers for testing - looks fine so far. Everything is local to my user account and no need for system-wide setups, permissions, etc. The podman command line tool can be used to manage stuff with a bit more fidelity (though you probably want to set up ~/.config/containers.conf file, as detailed below, to help with that).

Files edited

~/.config/containers/registries.conf

Set pulls to auto search in Docker Hub and Quay, as Podman's default Ubuntu installation has no auto-search configured at all.

unqualified-search-registries=["docker.io", "quay.io"]

~/.config/containers/containers.conf

This is to get the podman cli to talk to the user service (1000 is my UID, probably yours as well).

[service_destinations.local]
uri = "unix:///run/user/1000/podman/podman.sock"

Update:

The testcontainers alias was not really needed - what I did need to make Podman behave like Docker enough for Testcontainers was that it will automatically search Docker Hub for non-local images.

Other than that, after a lot more testing I'm very happy with the stability and performance of using Testcontainers with Podman - the only issue I have is that sometimes the resource reaper doesn't manage to stop the containers so testing may leave some around - check your system every now and then to see if you also get this.

[bsantanna]

Following @guss77 findings, another status report and workaround for the issue:

I've managed to have a successful Testcontainers and Podman setup on Mac OS by setting up a ssh tunnel that redirects unix socket from Guest CoreOS VM to Host filesystem.

Herewith recipe I have to run tests atm:

# Bootstrap guest CoreOS VM
podman machine init -m 4096
podman machine start

# use this command to identify ssh port guest CoreOS VM is using:
podman system connection list

# Create ssh tunnel, this will create a unix socket in /tmp/podman.sock
ssh -i ~/.ssh/podman-machine-default -p <port> -L'/tmp/podman.sock:/run/user/1000/podman/podman.sock' -N core@localhost

in another shell session:

export DOCKER_HOST=unix:///tmp/podman.sock
export TESTCONTAINERS_CHECKS_DISABLE=true
export TESTCONTAINERS_RYUK_DISABLED=true

mvn clean install # works for me

[edeandrea]

The workaround proposed by @bsantanna works for some containers but not others. Even ones that work I get this a lot:

ERRO[1205] accept tcp [::]:39779: use of closed network connection 

[sv3ndk]

I understand this issue is closed, although just for the record, here is a workaround I found that worked in my case:

My integration test was letting TestContainer launch docker-compose as follows:

      return new DockerComposeContainer(
          "dynamo-metrics-service-integration-tests",
          new File("./path/to/my/docker-compose.yml"),
          )
          .waitingFor("druid-zookeeper_1", Wait.forLogMessage(".*Started AdminServer.*", 1))
          .waitingFor("druid-postgres_1", Wait.forLogMessage(".*database system is ready to accept connections.*", 1))
          .waitingFor("druid-coordinator_1", Wait.forLogMessage(".*reportin' for duty.*", 1))
          .waitingFor("druid-broker_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-router_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-historical_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-middlemanager_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))

I'm using Podman 3.4.2 and TestContainer 1.6.3 on Linux with the following env variables:

DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
TESTCONTAINERS_CHECKS_DISABLE=true
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=unix:///run/user/1000/podman/podman.sock
TESTCONTAINERS_RYUK_DISABLED=true

The DockerComposeContainer statement above failed with the following error message:

Status 500: {"cause":"incorrect volume format, should be [host-dir:]ctr-dir[:option]","message":"fill out specgen: /path/to/my/code/unix:/run/user/1000/podman/podman.socket:/docker.sock:rw: incorrect volume format, should be [host-dir:]ctr-dir[:option]","response":500}

I solved the issue by telling DockerComposeContainer to work with local compose :

          .withLocalCompose(true);

[holly-cummins]

Here's my recipe. I was able to avoid the need to disable ryuk.

Edit ~/.testcontainers.properties and add the following line

ryuk.container.privileged=true

Then run the following

brew install podman
podman machine init -v $HOME:$HOME

If you're podman 4.1 or higher, you don't need the the -v $HOME:$HOME volume mount.

If you're on mac,

sudo /opt/homebrew/Cellar/podman/4.0.3/bin/podman-mac-helper install

Then (for all oses)

podman machine set --rootful
podman machine start

The rootful part makes ryuk behave ok, based on my testing (see containers/podman#14238). If ryuk is still causing problems or you don't want to run rootful then export TESTCONTAINERS_RYUK_DISABLED=true also does the trick, but containers can sometimes hang around and cause problems.

Finally, if you're on Mac M1, update podman to behave better with images for a different architecture (see https://edofic.com/posts/2021-09-12-podman-m1-amd64/ )

podman machine ssh
sudo -i
rpm-ostree install qemu-user-static
systemctl reboot

Once the virtual machine restarts, you should be good to run testcontainers.

[edeandrea]

This is awesome @holly-cummins. One question - With podman 4.1.1 on apple silicon mac, did you have to downgrade qemu back to version 6.2.0?

[holly-cummins]

This is awesome @holly-cummins. One question - With podman 4.1.1 on apple silicon mac, did you have to downgrade qemu back to version 6.2.0?

Ummm. I don't remember doing that, but I am indeed running 6.2.0:

holly@hcummins-mac ~ % podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 36.20220421.dev.0
[...]

[root@localhost ~]# rpm -q qemu-user-static
qemu-user-static-6.2.0-9.fc36.aarch64

I think I must have been lucky, and done my installation before v7 was released. Is 7 not so good on your machine?

[edeandrea]

When I did a brew update and it updated qemu to v7 it broke podman on my M1 mac. I haven't tried it again since, so maybe its fixed, but the issue below makes it seem like it isn't fixed.

Homebrew/homebrew-core#102242

I had to manually back qemu down to 6.2.0_1 using this procedure:

curl -L -H "Authorization: Bearer QQ==" -o qemu-6.2.0_1.monterey.bottle.tar.gz https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:fcc3b1a8139f70dae57f5449f3856f9b3b67448ee0623e64da1e47dc255b46f6

brew uninstall --ignore-dependencies qemu

brew install -f qemu-6.2.0_1.monterey.bottle.tar.gz

I found that procedure out on some GitHub issue somewhere - I just can't seem to remember where exactly...

[edeandrea]

@holly-cummins There's also this - containers/podman#14303

[holly-cummins]

Oh, sorry, @edeandrea, I've got muddled between the qemu-user-static in the podman machine, and the qemu on my mac machine. My mac has podman 4.1.1 and qemu 7.0.0, but it seems happy. So either the issue is fixed, or some other environmental factor on my machine masks the issue.

[edeandrea]

Good to know @holly-cummins . Maybe I'll give it a try.

[grdryn]

@holly-cummins thanks for the ryuk.container.privileged=true tip. testcontainers seems to work fine with that in the native rootless user podman on Fedora (not running in a "podman machine" VM), without needing the TESTCONTAINERS_RYUK_DISABLED env var.

Just wanted to add that in case the podman machine set --rootful might not be needed. I'm not really familiar with the "podman machine" stuff though.

[mario45211]

Is there known issue with pulling image by testcontainers ? I keep getting below errors when image is not in local repo and testcontainers tries to pull it from remote:

2022-10-10 17:02:59,110 INFO  --- [                     docker-java-stream--] docker.io/postgres:14]  :  Starting to pull image 
2022-10-10 17:02:59,197 ERROR --- [                     docker-java-stream--] c.g.d.api.async.ResultCallbackTemplate  :  Error during callback 
java.lang.NullPointerException: Cannot invoke "String.matches(String)" because the return value of "com.github.dockerjava.api.model.PullResponseItem.getStatus()" is null
	at com.github.dockerjava.api.command.PullImageResultCallback.checkForDockerSwarmResponse(PullImageResultCallback.java:48)
	at com.github.dockerjava.api.command.PullImageResultCallback.onNext(PullImageResultCallback.java:35)
	at org.testcontainers.images.LoggedPullImageResultCallback.onNext(LoggedPullImageResultCallback.java:48)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:73)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:24)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onNext(AbstrAsyncDockerCmdExec.java:41)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:315)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:298)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
	at java.base/java.lang.Thread.run(Thread.java:833)

[TheHaf]

@mario45211 I had the same issue initially. When I tried to pull the image manually, I realized that Podman did not have access to that repo yet where the image "repohost:8080/path/to/image:latest" was located. So I did podman login repohost:8080 in the console first and afterwards I was able to pull the image using testcontainers.

It seems there is a bug in docker-java-api that triggers the NPE in such a case. At the code where the NPE occurrs in the class PullImageResultCallback, it tries to compare the result of item.getStatus() without checking it against null first, but the method is clearly marked with @CheckForNull to indicate that the return value could be null. Why the status is null when the login failed, I don't know. Maybe such a problem should be caught at another place.
I'd have to check if there is an issue in their Guthub repo already relating to that.

[TheHaf]

To follow up, I did not find a tracked issue regarding the NPE. As the problem lies in the docker-java-api code and not with testcontainerst itself, I created an issue in their issue tracker: docker-java/docker-java#1998

[AshwinPrabhuB]

For those landing here hunting for the solution for NPE, here is one workaround until the docker-java bug is fixed:
docker-java/docker-java#1998 (comment)

[joanbm]

As far as I can tell, nowadays Podman works fine with Testcontainers, as long as you enable the Podman socket (podman.socket systemd unit) and set DOCKER_HOST appropriately.

The reason why some need to run Ryuk as a privileged container (or disable it altogether), as far as I can tell, is not because of Podman, but rather because of SELinux. If SELinux is enabled, Ryuk will be unable to connect to the Docker/Podman UNIX socket which is bind-mounted inside the container. See this and this discussion for more detail.

Running Ryuk as a privileged container works around this problem because privileged containers are not isolated through SELinux. Similarly, non-privileged Ryuk also works fine with Podman on systems without SELinux. And also, the same problem with Ryuk reproduces if you run the Docker daemon with SELinux enabled (#7177).

I suspect the reason this issue comes up in discussions with Podman because (1) Podman has SELinux support enabled by default and most adopters are likely on Red Hat distros which ship with SELinux enabled and (2) Docker has SELinux support disabled by default, even on distributions with SELinux enabled.