[BUG] Environment Variable handling is maddeningly inconsistent

[ilsaloving]

Description

According to https://docs.docker.com/compose/compose-file/12-interpolation/:

Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE}. Both $VARIABLE and ${VARIABLE} syntax is supported.

In practice, this is untrue. Variable interpolation is all over the map. Sometimes they work, sometimes they don't.
Different settings handle interpolation differently. (eg: #4249, or ports)
Different files handle interpolation differently. (eg: #11755)

This is causing needless confusion and frustration, and wastes countless hours of people's time with needless trial and error.

Variables should work across the board. If you're reading variables, regardless of source, variable interpolation should occur. If you're applying settings, regardless of the type, variable interpolation should occur. Heck, if you want to have an entire docker-compose yaml stanza pulled from a variable, do variable interpolation.

Steps To Reproduce

No response

Compose Version

2.24.7

Docker Environment

Client:
 Version:    25.0.4-rd
 Context:    rancher-desktop
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     /Users/ilsa.loving/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.7
    Path:     /Users/ilsa.loving/.docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 4e1fe7492b9df85914c389d1f15a3ceedbb280ac
 runc version: 18a0cb0f32bcac2ecc9a10f327d282759c144dab
 init version:
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.14-0-virt
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 5.784GiB
 Name: lima-rancher-desktop
 ID: c8f6b4ee-3832-4f33-85c9-19102fd7f961
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

interpolation works for sure, in most cases confusion comes from setting values based on env file, and how those get loaded with some default paths.

Different settings handle interpolation differently. (eg: #4249, or ports)

actually, this is the opposite. all attribute handle interpolation the same way, and when it comes to list of values to be set, this makes things harder for user as this assumes a fixed number of variables.

Different files handle interpolation differently. (eg: #11755)

This one is (yet another) case where user get confused between interpolation and container environment being set by env_file. We get such issue reported every week, so I've proposed docker/docs#19481 as clarification as there's massive confusion in this area with setting container environment.

If you have a specific bug to report, please give more details. I only can read "I'm frustrated this didn't worked out of the box" which won't bring us anywhere. I acknowledge the docs are bad documented this feature and need some love

[ilsaloving]

How the heck do you expect me to file a "specific bug report" when there is no way of knowing how this thing is supposed to be working in the first place? I did just file a report, as have countless other people, and Dockers' developers response is "You just don't know how to use it. Oh and the docs are crap so HAHA sucks to be you."

The only thing that works reliably is if I directly set, or pull an OS environment variable from the immediate environment, in the docker-compose.yaml file. eg:

services:
  my-service:
    environment:
      MY_VAR: my_val
      MY_VAR2: $MY_VAR2
      MY_VAR3: ${MY_VAR3:-default_val}

With external environment files, the only thing I have been able to get working semi-reliably is MY_VAR=my_val. The other two syntaxes do not work in a reliable/predictable fashion.

And heaven forbid I include: a separate docker-compose.yaml file, docker-compose-override.yaml file, or multiple environment files, cause then only $DEITY knows what if anything is supposed to happen. I have a project where I am including one docker-compose file into another, and the environment files specified by the included docker-compose file are ignored entirely.

I don't know or care if it's a code issue or a documentation issue. It is not only a bug, but a CRITICAL bug.

I appreciate you've attempted to get some progress on this, but clearly none of the other developers consider this a problem because docker/docs#19481 has been stalled for months.

If variable handling /w docker compose is so challenging that it cannot even be documented properly, it should be ripped out of the product entirely. As things stand, given Docker's reach, this has probably cost millions in lost productivity. Maybe Docker should spend less time coming up with ways to squeeze it's user base for money, and more time making a product it's users would be willing to pay money for.

[ndeloof]

The other two syntaxes do not work in a reliable/predictable fashion.

Please provide an illustration example so I can confirm this is a bug or explain / propose docs improvements accordingly

the environment files specified by the included docker-compose file are ignored entirely

same request. Please provide an illustration example if this sounds like a bug

none of the other developers consider this a problem

I can understand some frustration, but please also adopt a respectful tone. The team is working hard to maintain Compose with thousands users and we don't have much bandwidth to answer everyone. If you want some assistance to better understand how Compose works, or get your issue considered and fixed, just consider we also have to manage our priorities and a well reported issue with an example we can test and adopt as a new test case will bring faster answer/fix vs just complaining.