Temporary error (try again later)

[rommik]

I run into this issue when building a docker image on Ubuntu Host.
The Same build on Windows 10 using docker-tools CLI (so technically inside a VirtualBox VM) has no issues and Docker images are built correctly.

I have reinstalled Docker on my Ubuntu completely to have a fresh version just in case.
Any suggestions what else I can do to troubleshoot this issue?

Build command output

Sending build context to Docker daemon  572.4kB
Step 1/15 : FROM alpine:edge
edge: Pulling from library/alpine
cc5efb633992: Pull complete 
Digest: sha256:2b796ae57cb164a11ce4dcc9e62a9ad10b64b38c4cc9748e456b5c11a19dc0f3
Status: Downloaded newer image for alpine:edge
 ---> f96c4363411f
Step 2/15 : RUN apk add --update nodejs
 ---> Running in 12865b082f34
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.066df28d.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.b53994b4.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  nodejs (missing):
    required by: world[nodejs]
The command '/bin/sh -c apk add --update nodejs' returned a non-zero code: 1

My dockerfile

FROM alpine:edge
RUN apk add --update nodejs

#create an app directory
RUN mkdir -p /app
WORKDIR /app

# create some useful folders
RUN mkdir -p /data

# Copy apps build in another folder
COPY main.js /app
COPY package.json /app
COPY version.json /app
COPY dist /app/dist
COPY data /data
COPY app_modules /app/app_modules
COPY node_modules /app/node_modules

#expose the port
EXPOSE 5555

#start the app
#ENV NODE_ENV=development
#ENV debug=true

WORKDIR /app

# comment if need a console in the container
CMD [ "node", "main.js" ]

[andyshinn]

This keeps popping up in other threads and I wonder if there might be a change in the way Fastly is doing DNS. To debug this we really need to see the output of something like curl -v -s http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz > /dev/null to get headers and the HTTP response from within a container. My guess is a temporary issue with a Fastly POP or a change in POP IP address that hasn't yet properly timed out in Docker DNS proxy yet.

[rommik]

@andyshinn I'll see if I can provide you with the output. However, I believe the issue is in Docker Engine, and possibly in DNS caching. My Host Machine was on the corporate network 1, but when we switched it to corporate network 2, the issue was gone.
After speaking to networking people, they weren't able to tell how the difference between networks could have affected the Docker Build. It points to a combination of DNS and Docker Engine. However, we ruled out the DNS, because on both networks we were able to do curl from the host and download the .gz files.

I checked my docker config and its DNS is set to 8.8.8.8 and 8.8.4.4. From both networks these IPs are reachable.

[andyshinn]

I think it is related to #279 which has more history. I think maybe when Fastly changes server IPs they may be redirecting with a 301, which causes apk to choke. But this is why I want to output to see where it is redirecting. I'm not sure if it comes from Fastly or a broken upstream server.

[rlewkowicz]

So I'm having dns issues, but my results are different. My stuff just hangs at getting packages.
https://www.whatsmydns.net/#A/mirrors.aliyun.com

This is the mirror mine tries to use.

curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
*   Trying 124.47.11.88...
* Connected to mirrors.aliyun.com (124.47.11.88) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0

It will just hang here indefinitely in docker (at least for an hour+) and the curl times out. This is from my laptop:

recipes: curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
*   Trying 219.147.157.84...
* TCP_NODELAY set
* Connected to mirrors.aliyun.com (219.147.157.84) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: application/octet-stream
< Content-Length: 732690
< Connection: keep-alive
< Date: Wed, 27 Sep 2017 13:45:47 GMT
< Last-Modified: Thu, 21 Sep 2017 10:13:30 GMT
< ETag: "59c390ca-b2e12"
< Expires: Wed, 27 Sep 2017 13:50:47 GMT
< Cache-Control: max-age=300
< Accept-Ranges: bytes
< Via: cache14.l2nu29[36,304-0,H], cache32.l2nu29[38,0], cache2.cn547[0,200-0,H], cache6.cn547[1,0]
< Age: 18
< X-Cache: HIT TCP_MEM_HIT dirn:3:81977809 mlen:-1
< X-Swift-SaveTime: Wed, 27 Sep 2017 13:45:48 GMT
< X-Swift-CacheTime: 120
< Timing-Allow-Origin: *
< EagleId: db939d4615065199657165411e
<
{ [12630 bytes data]
* Curl_http_done: called premature == 0
* Connection #0 to host mirrors.aliyun.com left intact

@andyshinn this may be a different issue. But it's DNS related, and works one place but not the other. This is from my system, but if you need it from inside docker, I can provide it.

[andyshinn]

Have you tried another mirror? From that debug output, the IP address resolved from Docker and from the laptop are different, so it could definitely be an issue with just that host. Have you actually tried using the same IP on your laptop that Docker resolves to (something like curl -v -H 'Host: mirrors.aliyun.com' http://124.47.11.88/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null)?

[rlewkowicz]

Other mirrors do work. I suppose I have to go dig and see if they hard coded a mirror or something. It's odd though that it breaks apk so hard. I suppose I would have thought it to be more resilient

Its in general, related to DNS/apk fragility, but I think perhaps a different issue then op:

laradock/laradock#749 (comment)

[matkam]

I was seeing this error on my Drone build server, running on Kubernetes. By default it users "Docker-in-Docker" (dind) for builds. The error went away after I disabled dind on the Drone server.

[Zest-nikunj]

@matkam - Could you detail your solution ? We seem to having the same issue with our drone server.

[matkam]

@Zest-nikunj sure. My Drone server was deployed by a Helm chart to a Kubernetes cluster: https://github.com/kubernetes/charts/tree/master/incubator/drone
I just redeployed it, setting dind.enabled to false. I'd imagine if you're not using Helm or Kubernetes, you'd have to mount the host's docker socket on the drone agent's container: https://github.com/kubernetes/charts/blob/master/incubator/drone/templates/deployment-agent.yaml#L52

[Zest-nikunj]

Thanks @matkam. We are not on Kubernetes and the settings seem correct but we continue to have the issue. We will investigate further. If needed, I shall come back to you. Thanks again.

[publicocean0]

same problem ... before i resolved with reboot. Now it is not working

[tdhawan]

Same problem, just kept on re-trying, and 3rd time it went thru. seems like there is a bad server which the load balancer hits.

[rogerluiz]

same error here

[markk12]

same problem here . I tried many many times ... it is not a bad server ... it seams a bad code line , a bug

[ghost]

I got same. I tried to change dns on docker with no luck.
I'm on Ubuntu 18.04, my colleague can run successfully exactly same build in the same network on Arch

[wlechowicz]

Hit the same problem (again) on Windows + Docker and tried many suggested (in #279) solutions, involving changing apk settings in Dockerfile and such. but nothing helped. What did help finally was to restart Docker and then apk just pulled the APKINDEX like nobody's business.

[quisido]

I get this as well on my workstation using Docker for Windows. Using a mirror did not help.

[brunowdev]

I had this error with Gitlab, i just change docker build command to run with --network host option and it works!

[EvgeniGordeev]

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf helped on Ubuntu 16 build agent to make CI jobs build alpine based docker images again as per #279 (comment)

[montao]

@EvgeniGordeev That solution worked for me too, running Windows 10 host + Ubuntu 18.04 with Vagrant and alpinelinux in a docker container in the Ubuntu.

[hezzze]

@wlechowicz restart docker service solved my issue 👍

[jbcurtin]

@andyshinn , RE: #334 (comment)

It was a DNS error for me. By setting /etc/docker/daemon.json with,

{
  "dns": ["8.8.8.8"]
}

and then restarting docker with,

sudo service docker restart

I was able to build images again.

[hackaugusto]

Just for reference, I got a similar error, but in my case it was a problem with nftables:

 ---> Running in 2522cf9dc903
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:

Stopping nftables and restarting docker will give the container internet access.

[gx14ac]

Since there are times when the bridge is disturbing, it is successful if you execute it as host.
docker build -t hoge:latest . --network=host

[When-Jay]

restarting docker service maybe help you

[akarimvnera]

Run docker build command to run with --network=host option and it works

[rvillane]

any idea how to add this workaround using docker-compose ? the --network flag is not available there

[dacoburn]

I just ran into this, everything had been working fine on:

Windows 10 host with Hyper-V
Ubuntu 18.04 Guest
Docker in the Ubuntu 18.04

Suddenly apk and go mod download started failing. The APK issue was the temporary error. Did the fix suggested in this comment #334 (comment) and everything was working again. DNS Was working fine on the guest and the host, only when building the docker containers was I having issues.

[akovalyov]

just for reference, https://dl-cdn.alpinelinux.org/ and https://alpine.global.ssl.fastly.net are still redirecting from https to http:

$ curl -I https://dl-cdn.alpinelinux.org/alpine/v3.13/main 
HTTP/2 301 
server: nginx
content-type: text/html
location: http://dl-cdn.alpinelinux.org/alpine/v3.13/main/

$ curl -I https://alpine.global.ssl.fastly.net/alpine/v3.13/main 
HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 162
Server: nginx
Content-Type: text/html
Location: http://alpine.global.ssl.fastly.net/alpine/v3.13/main/

[adrian5]

I ran into the same apk failure with (rootless) podman. Strangely enough, nslookup worked fine from within the container!

What worked for me was setting the global dns, i.e. /etc/containers/containers.conf:

dns_servers = [ "1.1.1.1", "8.8.8.8" ]

[hlakioui]

@wlechowicz restart docker service solved my issue +1

thank you boss, you save my life

[coderRPN]

For Raspberry Pi:

The answer is this: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0#time64_requirements
(alpine 3.13 isn't compatible with raspberry pi os by default due to some outdated package)

[source]

For Raspberry Pi, you can upgrade libseccomp2 to a supported version from the Debian Repos eg.

wget http://ftp.de.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1_armhf.deb
dpkg -i libseccomp2_2.5.1-1_armhf.deb

[source]

Same issue exists for apline 3.14 and this solution works

[hamedsahebi]

I had the same issue.
Changing the Alpin version solved mine. alpin:3.6

[S474N]

Actual version is 2.5.3-2

For Raspberry Pi, you can upgrade libseccomp2 to a supported version from the Debian Repos eg.

wget http://ftp.de.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.3-2_armhf.deb
dpkg -i libseccomp2_2.5.3-2_armhf.deb

[dawhc]

I used the host network mode to run alpine and the issue was solved:

docker run -it --network host alpine

[codeninja-ru]

this helped me https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0#musl_1.2

there are several solutions

so I had to download default.json, change one line and pass --security-opt=seccomp=default.json to docker run

[lovecoding-git]

@andyshinn , RE: #334 (comment)

It was a DNS error for me. By setting /etc/docker/daemon.json with,

{
  "dns": ["8.8.8.8"]
}

and then restarting docker with,

sudo service docker restart

I was able to build images again.

-In Windows

C:\Users\Administrator\.docker\daemon.json

And add this line.

{
    ...
    "dns": ["8.8.8.8"]
}

[fullbl]

it happens to me if I run
docker compose up,
but it doesn't if I run
docker-compose up

[blockloop]

My fix for drone was setting the repository as Trusted under the repo settings in drone and then adding this to the docker build steps:

steps:
  -
    name: docker
    image: plugins/docker
    network_mode: host   <----- this 

[Startouf]

Similar error happen to me when I try to build docker images from within my kubernetes cluster but my workflow is more complex

1. Locally I build a small docker image with awscli/docker/bash/git

FROM docker:latest
# Update & Upgrade OS
RUN apk update
RUN apk upgrade
# https://github.com/aws/aws-cli/issues/4971#issuecomment-1330633153
COPY --from=devopscorner/aws-cli:latest /usr/local/aws-cli/ /usr/local/aws-cli/
COPY --from=devopscorner/aws-cli:latest /usr/local/bin/ /usr/local/bin/
RUN apk add bash git

2. My jenkins (deployed itself on a kubernetes cluster) has some pipelines that spawn some pods that grab this image

node(POD_LABEL) {

  stage('Clone') {
    git url: 'https://github.com/example/example/', branch: '${build_branch_name}', credentialsId: 'github-app'

    container('aws-dockerizer') {
      stage('Build and deploy') {
        withAWS(credentials: 'aws-credentials', region: 'eu-central-1') {
          sh '''#!/usr/bin/env bash
            git config --global --add safe.directory ${WORKSPACE}
            scripts/build_and_push_docker_image_to_aws.sh
          '''
        }
      }
    }
  }  

}

4. The script runs a docker build step that builds my application image with some dependencies, here is an extract of the Dockerfile being built

FROM ruby:3.1.2-alpine
RUN apk add --update --no-cache   binutils-gold   build-base   curl   file   g++   gcc   git   less   libc-dev   htop   libffi-dev   libgcrypt-dev   libstdc++   libxml2-dev   libxslt-dev   linux-headers   make   netcat-openbsd   nodejs   yarn   openssl   pkgconfig   postgresql-dev   ruby-full   tzdata

I run into the following errors from the jenkins-spawned pod

#6 4.372 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
#6 9.404 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
#6 9.404 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/main: temporary error (try again later)
#6 14.41 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/community: temporary error (try again later)

So I don't really have a sysadmin-like access, my image is being built from a pod connected to docker via a volume mount on the docker socket. It make it hard to run any OS/config patch

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: aws-dockerizer
    image: example/aws-dockerizer:0.1.3
    command: ['cat']
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock

[Startouf]

Since there are times when the bridge is disturbing, it is successful if you execute it as host. docker build -t hoge:latest . --network=host

After struggling for so long with what I though was a kubernetes specific docker-in-docker issue, I ended up adding the --network=host in my scripts commands and it fixed it. I should have tried that first instead of going nuts.

[overbost]

I have the same issue with "slow" internet connection. Is "slow" for Docker, maybe it have low parameter for "temporary error"

[exotexot]

Still happening...

[axisofentropy]

I also had this issue randomly building ingress-nginx images on Docker Desktop.

0.260 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz                  
5.295 WARNING: updating https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)                                                                                               
5.295 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
10.41 WARNING: updating https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)

I added the "dns": ["8.8.8.8"] line to the Docker Desktop config and now it builds consistently.

[dvdn]

Same issue trying to build python:3.11-alpine.
Solved by adding another dns, like '9.9.9.9' (Quad9), in my existing list in /etc/docker/daemon.json :

{
        "dns": ["42.xx.xx.42", "9.9.9.9"]
}

as described in previous comment > #334 (comment)

[build630]

Any recommendations to address it ?

So far, I have tried

1. Enabling UnsafeLegacyRenegotiation in /etc/ssl/openssl.cnf
2. Adding DNS to /etc/docker/daemon.json

[ashikzubair-pge]

Since there are times when the bridge is disturbing, it is successful if you execute it as host. docker build -t hoge:latest . --network=host

After struggling for so long with what I though was a kubernetes specific docker-in-docker issue, I ended up adding the --network=host in my scripts commands and it fixed it. I should have tried that first instead of going nuts.

where did u add the netwrok =true ? is that in the jenkins agent config command?

[Zerwin]

Ignore everything below, it was an IPV6 problem, not related

---

Figured it out what caused it for me. I had enabled IPV6 on my host and that caused the only DNS server in /etc/resolv.conf to also be IPV6. Disabling IPV6 again caused the build to run through. I could pretty much consistently trigger a failed/successful build by switching between an IPV6 DNS server and an IPV4 DNS server.

Some more info:
OS: Debian 12
OCI: Podman
Network changes relevant:
sudo nano /etc/network/interfaces
and then adding

iface eth0 inet6 dhcp
        dhcp6-pd yes

(Replace eth0 with whatever your actual network interface is)
Then just restart networking (sudo service networking restart)

As soon as you get the files once it won't reproduce until you clear all the build caches.

[OreoProMax]

I had this error with Gitlab, i just change docker build command to run with --network host option and it works!

works for me

[ygxxii]

In my case, I got this error when ca-certificates package is not installed and set /etc/apk/repositories to a HTTPS mirror, e.g.:

https://mirrors.aliyun.com/alpine/v3.18/main
https://mirrors.aliyun.com/alpine/v3.18/community

To work around, use HTTP mirror:

RUN sed -e 's/https:\/\/dl-cdn.alpinelinux.org/http:\/\/mirrors.aliyun.com/g' \
        -i /etc/apk/repositories \
    && apk add --no-cache tzdata

[virtualbeck]

I used a borrowed retry function to get past this:

function retry {
  local retries=$1
  shift

  local count=0
  until "$@"; do
    exit=$?
    wait=$((2 ** $count))
    count=$(($count + 1))
    if [ $count -lt $retries ]; then
      echo "Retry $count/$retries exited $exit, retrying in $wait seconds..."
      sleep $wait
    else
      echo "Retry $count/$retries exited $exit, no more retries left."
      return $exit
    fi
  done
  return 0
}

retry 10 apk add --no-cache aws-cli jq