commit 436e406fba7a982d6c191835681b60f4323d849f
parent c11c10b9f5415a50bca9bb261ed55004925b83eb
Author: Andreas Gruhler <andreas.gruhler@adfinis.com>
Date: Sat, 25 May 2024 22:13:03 +0200
doc(jenkins): document Jenkins use case
Diffstat:
1 file changed, 248 insertions(+), 0 deletions(-)
diff --git a/docker/docker-jenkins-inbound-agent/README b/docker/docker-jenkins-inbound-agent/README
@@ -0,0 +1,248 @@
+= Jenkins Agent Dockerfile =
+
+Based on the upstream docker file for Jenkins inbound agents:
+* https://github.com/jenkinsci/docker-agent
+* https://github.com/jenkinsci/docker-inbound-agents
+
+Change the uid/gid to be unique. Create a unique user jenkins on the Nomad
+agents and assign this user a unique uid/gid that is only used for Jenkins
+Docker builds.
+
+== Jenkins requirements ==
+
+docker-workflow plugin:
+* https://plugins.jenkins.io/docker-workflow
+* https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow
+* https://www.jenkins.io/doc/book/installing/docker
+* https://www.jenkins.io/doc/book/pipeline/docker
+
+Ideally, the Jenkins as code config plugin:
+* https://plugins.jenkins.io/configuration-as-code
+
+Note, that you don't need to install the `docker-plugin` for Jenkins. This
+plugin has a different use. It should be used to spawn Jenkins agents on Docker
+containers. What we do here is spawning new Nomad jobs instead. The Nomad jobs
+(workers with a particular label that signal that they can run Docker workload)
+will then launch the Docker container.
+
+== Nomad requirements ==
+
+Nomad plugin for Jenkins (aka Nomad cloud):
+* https://plugins.jenkins.io/nomad
+* https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57
+* https://github.com/GastroGee/jenkins-nomad
+
+The Nomad job template for new Jenkins agent nodes can be configured as code
+(Jenkins as code configuration plugin):
+* https://code.in0rdr.ch/nomad/file/hcl/default/jenkins/templates/jenkins.yaml.tmpl.html
+
+
+== Mount point requirements ==
+
+Because the Nomad jobs launch the Docker containers, it is required to give
+access to a Docker socket (tcp or unix). In this example, we can share the unix
+socket from the Nomad node with the Nomad job. The YAML template for the new
+Jenkins agent (a Nomad job):
+
+"Tasks": [
+ {
+ "Name": "jenkins-podman-worker",
+ "Driver": "podman",
+ "User": "1312",
+ "Config": {
+ "volumes": [
+ "/run/user/1312/podman/podman.sock:/home/jenkins/agent/podman.sock",
+ "/home/jenkins/workspace:/home/jenkins/workspace"
+ ],
+ "image": "127.0.0.1:5000/jenkins-inbound-agent:latest"
+...
+
+
+This can be configured in the Jenkins configuration as code plugin mentioned in
+the requirements sections above.
+
+Note:
+* The user UID/GID can be anything here, just make sure to build the Jenkins
+ inbound agent with the same UID/GID that are used by Nomad to spawn the
+ Jenkins agent job.
+* The Docker socket shared with the Nomad job (the Jenkins agent) here needs to
+ be activated and run in the background (see enable-linger below)
+
+Because the Jenkins agent sits in between our Podman host (the Nomad agent) and
+the downstream Docker container where we run our app logic, it is essential to
+mount the directory from the Nomad agent node to the Jenkins agent. Otherwise,
+the downstream container where we run our app logic will always see an empty
+directory, because in the end all containers are run (in a flat structure, as
+you so will) on the Nomad agent.
+
+This is also why UID/GID needs to match between the user that runs the Podman
+socket on the Nomad node and the user that spawns the Jenkins agent (the Nomad
+job).
+
+More on this relationship between the container, the Jenkins agent and the
+Nomad node in the graphic below.
+
+== Nomad node requirements ==
+
+Because in the end, the Docker containers are spawned on the Nomad nodes (as
+Podman containers), the Nomad job (the Jenkins agent) needs direct access to
+the filesystem of the Nomad node. Specifically, the Jenkins workspace directory
+needs to be mounted to each Jenkins agent.
+
+ +------------------+
+ | Docker container |- - - - - - +
+ +------------------+ |
+ ^ mounts workspace |
+ | spawns Docker |
+ +---------------------------+ |
+ | Jenkins agent (Nomad job) | | is scheduled
+ +---------------------------+ | on and writes
+ ^ | workspace
+ | /home/jenkins/workspace |
+ | has user with UID/GID |
+ +------------+ |
+ | Nomad node |<- - - - - - - - -+
+ +------------+
+
+The Docker pipeline (docker-workflow) plugin has the default configuration to
+automatically mound a Docker volume in `/home/jenkins/workspace`.
+
+This can be seen in the log of Jenkins when the first Job with a docker
+pipeline configuration starts:
+
+$ docker run -t -d -u 1312:1312 -u root -w /home/jenkins/workspace/tutut -v
+/home/jenkins/workspace/tutut:/home/jenkins/workspace/tutut:rw,z -v
+/home/jenkins/workspace/tutut@tmp:/home/jenkins/workspace/tutut@tmp:rw,z -e
+******** -e ******** -e ******** -e ******** -e ******** -e ******** -e
+******** -e ******** -e ******** -e ******** -e ******** -e ******** -e
+******** -e ******** -e ******** -e ******** -e ******** -e ******** -e
+******** -e ******** -e ******** -e ******** -e ******** -e ******** -e
+******** -e ******** -e ******** docker.io/arm64v8/alpine:latest cat
+
+Here we see that the default workspace directory (-w flag) is allocated in
+`/home/jenkins/workspace` (the directory which is mounted through the Nomad job
+from the Nomad node). Also, the plugin automatically mounts a sub-directory in
+that folder for purposes of running the particular pipeline.
+
+If this mount propagation from Nomad node to job to the final container does
+not work, there will be error similar to this one in the Jenkins pipeline:
+
+ cp: can't stat '/home/jenkins/workspace/dubi@tmp/durable-9195a799/script.sh': No such file or directory
+ sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-log.txt: nonexistent directory
+ sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp: nonexistent directory
+ mv: can't rename '/home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp': No such file or directory
+
+Once again, it's critical to mount the Jenkins workspace(s) as volume inside
+the container because of how the docker-workflow plugin works:
+
+> For inside to work, the Docker server and the Jenkins agent must use the same
+> filesystem, so that the workspace can be mounted. The easiest way to ensure
+> this is for the Docker server to be running on localhost (the same computer
+> as the agent). Currently neither the Jenkins plugin nor the Docker CLI will
+> automatically detect the case that the server is running remotely; a typical
+> symptom would be errors from nested sh commands such as
+>
+> "cannot create /…@tmp/durable-…/pid: Directory nonexistent"
+
+https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow#docker-workflow-sect-inside
+
+The same warning note is also given again here:
+* https://www.jenkins.io/doc/book/pipeline/docker/#using-a-remote-docker-server
+
+Because the `/home/jenkins/workspace` is the default workspace directory of the
+plugin, the cleanest approach is probably to to simply create this dedicated
+Jenkins service user on all Nomad nodes (with a dedicated UID/GID combination),
+then start the Podman socket as systemd user job on the Nomad nodes.
+
+On all the Nomad clients, prepare the Jenkins user and the workspace directory
+(1312 can be any UID/GID combination, it just needs to map with the User in the
+Jenkins cloud plugin configuration where the Nomad job is spawned):
+
+ groupadd -g 1312 jenkins
+ useradd -m -s /bin/bash -u 1312 -g 1312 jenkins
+
+ # keep my podman.socket enabled even if no jenkins user is logged in
+ loginctl enable-linger jenkins
+ #loginctl user-status jenkins | grep -i linger
+
+ # https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#-M
+ systemctl --user -M jenkins@ start podman.socket
+ systemctl --user -M jenkins@ enable podman.socket
+ #systemctl --user -M jenkins@ is-enabled podman.socket
+
+ # create the mountpoint for the workspaces, podman does not create it for us
+ sudo -u jenkins mkdir /home/jenkins/workspace
+
+<TODO: LINK-TO-PACKER-SCRIPT>
+
+If you need to redo the jenkins user configuration (e.g., to change the
+UID/GID), make sure to stop the systemd service for the user. Otherwise, new
+directories will still be created with old GIDs.
+
+ systemctl stop user@1312 # removes /run/user/1312
+ rm -rf /run/user/1312
+
+== Example build process ==
+
+To configure a different UID/GID for the Jenkins user, it is also required to
+rebuild a Jenkins agent image with that particular UID/GID.
+
+To build the Jenkins agent docker container for the purposes of using it in
+Nomad:
+
+ git clone https://github.com/jenkinsci/docker-agent.git docker-agent.git
+ cd docker-agent.git
+
+ # change the uid/gid
+ buildah bud --no-cache --arch arm64/v8 --build-arg=JAVA_VERSION=21.0.3_9 \
+ --build-arg=uid=1312 --build-arg=gid=1312 -f alpine/Dockerfile \
+ -t 127.0.0.1:5000/jenkins-inbound-agent:latest .
+
+ podman push 127.0.0.1:5000/jenkins-inbound-agent:latest
+
+== Jenkins Pipeline examples ==
+
+The declarative pipeline examples need to ensure that the user (`-u` flag) is
+set to the user of the Docker image. Two examples:
+
+ pipeline {
+ agent {
+ docker {
+ image 'docker.io/arm64v8/alpine:latest'
+ // The Nomad cloud spawns Nomad agents with that label
+ label 'podman'
+ // User needs to match the user in the container, otherwise you
+ // encounter errors while accessing "durable" directories.
+ args '-u root'
+ }
+ }
+
+ stages {
+ stage('runInDocker') {
+ steps {
+ sh 'echo "hello world"'
+ }
+ }
+ }
+ }
+
+ pipeline {
+ agent {
+ docker {
+ image 'docker.io/hashicorp/vault:latest'
+ label 'podman'
+ args '-u root'
+ }
+ }
+
+ stages {
+ stage('runInDocker') {
+ steps {
+ sh '''
+ vault version
+ '''
+ }
+ }
+ }
+ }
+
