Docker build with img failing on infra.ci with SYNC_USERMAP_ACK: got 255: Invalid argument

[dduportal]

Service

infra.ci.jenkins.io

Summary

All the Docker builds on infra.ci, which are using img , are failing with an error SYNC_USERMAP_ACK: got 255: Invalid argument, as caught by @smerle33.

The complete error is

nsenter: failed to execv: No such file or directory
nsenter: failed to sync with parent: SYNC_USERMAP_ACK: got 255: Invalid argument
nsenter: failed to use newuidmap: Invalid argument

Reproduction steps

• https://infra.ci.jenkins.io/blue/organizations/jenkins/Docker%20Builds%2Fdocker-jenkins-weekly/detail/PR-360/8/pipeline
• https://infra.ci.jenkins.io/blue/organizations/jenkins/Docker%20Builds%2Fdocker-hashicorp-tools/detail/main/7/pipeline

[dduportal]

We already had this error in the early phases of using img:

• img build: newuidmap: write to uid_map failed: Invalid argument genuinetools/img#144
• That's the reason why we were copying the binaries newuidmap and newguidmap https://github.com/jenkins-infra/docker-builder/blob/1.2.8/Dockerfile#L68-L70

[dduportal]

• The error is present with the image jenkinsciinfra/builder:2.0.4, but is absent with the image jenkinsciinfra/builder:1.2.8
• It appeared after a batch of PRs that changed the version of the image on the pipeline-library used to build Docker images:
  • [updatecli] Bump docker-builder version on shared library resources pipeline-library#275
  • [updatecli] Bump docker-builder version on shared library resources pipeline-library#280

[dduportal]

Found it: we changed the based image from Alpine to Debian in jenkins-infra/docker-builder#34 , which breaks img.

[dduportal]

Panel of upcoming solutions:

• Short term: revert back the pipeline library to 2.0.2: only affect docker builds (not the plugin-site)
• Medium term: either find a way to fix this issue under a debian base image or split images and go back to alpine for img
• Long term: either full switch to Docker CE with Allow infra.ci.jenkins.io to build on VMs with Docker Engine #13 or use Docker in Docker pods with sysbox

[timja]

this helpful at all? genuinetools/img#348 (comment)

[dduportal]

this helpful at all? genuinetools/img#348 (comment)

That is extremly helpful ! I missed this one, thanks a lot

[dduportal]

At first sight, it seems feasible:

kubectl -n jenkins-agents exec -ti docker-builds-docker-jenkins-weekly-pr-360-10-d83kz-773jv-rpsc1 -- cat /proc/sys/kernel/unprivileged_userns_clone 
1

(required for rootless https://github.com/moby/buildkit/blob/master/docs/rootless.md)

[timja]

or this? genuinetools/img#340 (comment)

[halkeye]

Or https://github.com/genuinetools/img/blob/master/Dockerfile.dev#L69

[dduportal]

TBH, moving to buildkit sounds the most appealing part, since img is slowly dying (it was released before the buildkit stuff)

[dduportal]

But mobykit is also using alpine: https://github.com/moby/buildkit/blob/master/Dockerfile#L277 :'(

Sounds like @halkeye solution with building the correct binaries is the closest one

[dduportal]

Short term PR: jenkins-infra/pipeline-library#282

[dduportal]

With the version 2.0.4, the problem is that the binaries newuidmap and newgidmap come from the img alpine image, so they cannot work at all under a debian distribution:

$ ldd /usr/bin/newuidmap
        linux-vdso.so.1 (0x00007ffdc2b4c000)
        libc.musl-x86_64.so.1 => not found

Let's try to keep the original debian binaries.

[dduportal]

Since jenkins-infra/pipeline-library#311, you can set the attribute useContainer to true in the argument map of buildDockerAndPublishImage() to build using a valid Docker engine.

Gotta update parallelDockerUpdatecli() to support this attribute and use Docker by default so we would be able to proceed with this.