PG Embedded - the Future (Update: 1.x now released)

[mikebell90]

It is of no surprise to anyone watching this repo that
this library hasn't been updated much in the last 3 years.

We apologize for this, but a series of events occurred that
shifted our time elsewhere. So a lot of stuff has bitrotted, and that sucks.

We have been considering the future of this library, and
are currently planning on rewriting it as a TestContainers dependent
project. That would mean it uses Docker images. The DSL would
remaining similar if possible, albeit requiring setting your docker image.

Here's the reasoning:

The advantage of the old methodology (embedded tarball) was a minimal dependency set,
and it was VERY fast (assuming it was already cached, just untar to a temporary location and run)

The disadvantages have been mounting for a while however:

• There's no longer a universal binary for linux. That in turn lead to the need to cut a different tarball per linux distribution.
• Therefore we didn't update postgres, and that's certainly overdue
• In general the local dependencies for things like windows or mac (various libraries, etc) make it fragile.
• The M1 adds yet another architecture (on the mac) to support.
• Security-wise deploying a tarball from an unknown source (well, us in this case), and executing at your
  local security context is an awful idea.
• Java 17 is yet another issue and this is not the only library we need to worry about using embedded processes.

Contrasting with Docker

• Same image works everywhere (except potentially for architecture issues, and multi-arch or direct access to set the image solves that mostly)
• The docker daemon can be hardened to some extent and is less likely to expose the host to attack.
• Admittedly more "machinery" to worry about - gotta have a docker daemon running somewhere. For some ci/cd doing Docker In Docker is problematic.

We are open to feedback, but our current plan is

• Early next year, with a preview hopefully before that to release a 1.0.0 using TestContainers library
• Then close all PRs and Issues with old library, report as such.

For those of you preferring the old approach, feel free to go google "zonky embedded postgres", and you'll find a fork of this library
which is fairly regularly maintained.

Thanks!

Platform Architecture

[khaledez]

That's the correct approach in my opinion. Docker has become ubiquitously available in most developers' machines, in addition to the security features it provides.

I'm having a problem running pg-embedded on my M1 laptop, and considering migrating to TestContainers, but it's great that you're planning to use it under the hood, that means much fewer changes on our side (hundreds of services depending on pg-embedded).

[pabloa]

I like this project like it is. It works great! And the result is small and does not use so much memory as with docker.
For M1 machines, perhaps the M1 version of Postgres could be included. That would solve the problem.

I would like version 13 of Postgres to be included too.

[mikebell90]

Sorry @pabloa that aint gonna happen. We are nearly ready on the docker version and have closed all extant PRs and issues.

[mikebell90]

FWIW that version will support M1 and Postgres 13

[mikebell90]

We'll try to get a testing version out shortly

[mikebell90]

@dkaukov once you have applied changes abd internal review please make an initial public version as 1.0.0.RC1 from the branch. Assuming we get no blockers, we'll release 1.0.0

[mikebell90]

The 1.0.0.RC1 is out on Maven. See #164 for the code and the README.

[mikebell90]

1.0.0.RC2 has been released (will take 6-24 hours to propagate to Maven). #164 containers to be the code and README (or you can look at the tag)

[stevenschlansker]

I tried out 1.0.0.RC2 and unfortunately it doesn't seem to work with Podman (https://podman.io/), which we rely on to run our containers without Docker's new very restrictive licensing.

[ducttape-0] INFO com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec - Recoverable I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:2375
[main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/Cellar/pyenv-virtualenv/1.1.5/shims, /Users/steven/bin, /Users/steven/.m2/apache-maven/bin, /usr/local/sbin, /usr/local/bin, /Users/steven/.rvm/bin, /Users/steven/bin, /Users/steven/.m2/apache-maven/bin, /usr/local/sbin, /usr/local/bin, /usr/local/bin, /usr/bin, /bin, /usr/sbin, /sbin, /usr/local/MacGPG2/bin, /opt/X11/bin, /Applications/Wireshark.app/Contents/MacOS, /Users/steven/code/wspl-tools/bin, /Users/steven/ws-code/wspl-tools/bin, /Users/steven/go/bin, /bin])
[main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
[main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy -     EnvironmentAndSystemPropertyClientProviderStrategy: failed with exception TimeoutException (Timeout waiting for result with exception). Root cause LastErrorException ([2] No such file or directory)
[main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy -     UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)
[main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue

java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
	at org.jdbi.v3.testing.JdbiRuleTest.migrateWithFlywayCustomLocation(JdbiRuleTest.java:67)

I tried futzing around with custom socket paths, eventually ended up at testcontainers/testcontainers-java#2088, and there's lots more things to try but I didn't get anything working yet.

So switching to Testcontainers may force us to use the Zonky fork or choose some other path forward, unless we can figure this out - we cannot provide a Docker environment everywhere we expect our programs to run.

[mikebell90]

try these @stevenschlansker .

They worked for me, but in genersal podman is 2 years from being production ready (google "podman testcontainerS" to see the ACKNOWLEDGED bugs in their socket api ;)

brew install podman
--
podman machine init (with appropriate settings)
Add symlink sudo ln -s /usr/local/bin/docker $(brew --prefix)/bin/podman
 
Add to .zprofile
podman machine list \| grep "running"if [[ "$?" == "1" ]]; then        podman machine startfiexport DOCKER_HOST=unix:///tmp/podman.sockexport TESTCONTAINERS_CHECKS_DISABLE=trueexport TESTCONTAINERS_RYUK_DISABLED=true
 
Hack 1
Whenever you need to run docker, you must have this running
sshPort=$(ps -ef \| grep -i gvproxy \| grep -v grep \| sed 's/.*ssh-port //')
ssh -i ~/.ssh/podman-machine-default -p $sshPort -L'/tmp/podman.sock:/run/user/1000/podman/podman.sock' -N core@localhost
 
Hack 2
To get podman to run x64 images
 
podman machine ssh
sudo -i
rpm-ostree install qemu-user-static
systemctl reboot

[mikebell90]

This is probably the most important part @stevenschlansker

export DOCKER_HOST=unix:///tmp/podman.sock
export TESTCONTAINERS_CHECKS_DISABLE=true
export TESTCONTAINERS_RYUK_DISABLED=true

[mikebell90]

1.0.0.RC3 is out now

[mikebell90]

1.0.0 will be released in a few minutes. Please feel free to open issues on 1.x. Issues on pre 1.x will be closed, unless people decide they wish to maintain a legacy branch (to which PRs should be targeted)