README (12704B)
1 = Jenkins Agent Dockerfile = 2 3 Based on the upstream docker file for Jenkins inbound agents: 4 * https://github.com/jenkinsci/docker-agent 5 * https://github.com/jenkinsci/docker-inbound-agents 6 7 Change the uid/gid to be unique. Create a unique user jenkins on the Nomad 8 agents and assign this user a unique uid/gid that is only used for Jenkins 9 Docker builds. 10 11 Example (based on the jenkinsci/docker-inbound-agents example above): 12 * https://github.com/jenkinsci/docker-agent/compare/master...in0rdr:docker-agent:debug/podman_x86_64 13 14 == Jenkins requirements == 15 16 docker-workflow plugin: 17 * https://plugins.jenkins.io/docker-workflow 18 * https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow 19 * https://www.jenkins.io/doc/book/installing/docker 20 * https://www.jenkins.io/doc/book/pipeline/docker 21 22 Ideally, the Jenkins as code config plugin: 23 * https://plugins.jenkins.io/configuration-as-code 24 25 Note, that you don't need to install the `docker-plugin` for Jenkins. This 26 plugin has a different use. It should be used to spawn Jenkins agents on Docker 27 containers. What we do here is spawning new Nomad jobs instead. The Nomad jobs 28 (workers with a particular label that signal that they can run Docker workload) 29 will then launch the Docker container. 30 31 == Nomad requirements == 32 33 Nomad plugin for Jenkins (aka Nomad cloud): 34 * https://plugins.jenkins.io/nomad 35 * https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57 36 * https://github.com/GastroGee/jenkins-nomad 37 38 The Nomad job template for new Jenkins agent nodes can be configured as code 39 (Jenkins as code configuration plugin): 40 * https://code.in0rdr.ch/nomad/file/hcl/default/jenkins/templates/jenkins.yaml.tmpl.html 41 42 43 == Mount point requirements == 44 45 Because the Nomad jobs launch the Docker containers, it is required to give 46 access to a Docker socket (tcp or unix). In this example, we can share the unix 47 socket from the Nomad node with the Nomad job. The YAML template for the new 48 Jenkins agent (a Nomad job): 49 50 "Tasks": [ 51 { 52 "Name": "jenkins-podman-worker", 53 "Driver": "podman", 54 "User": "1312", 55 "Config": { 56 "volumes": [ 57 "/run/user/1312/podman/podman.sock:/home/jenkins/agent/podman.sock", 58 "/home/jenkins/workspace:/home/jenkins/workspace" 59 ], 60 "image": "127.0.0.1:5000/jenkins-inbound-agent:latest" 61 ... 62 63 64 This can be configured in the Jenkins configuration as code plugin mentioned in 65 the requirements sections above. 66 67 Note: 68 * The user UID/GID can be anything here, just make sure to build the Jenkins 69 inbound agent with the same UID/GID that are used by Nomad to spawn the 70 Jenkins agent job. 71 * The Docker socket shared with the Nomad job (the Jenkins agent) here needs to 72 be activated and run in the background (see 73 https://code.in0rdr.ch/hashipi/file/nomad.sh.html) 74 75 Because the Jenkins agent sits in between our Podman host (the Nomad agent) and 76 the downstream Docker container where we run our app logic, it is essential to 77 mount the directory from the Nomad agent node to the Jenkins agent. Otherwise, 78 the downstream container where we run our app logic will always see an empty 79 directory, because in the end all containers are run (in a flat structure, as 80 you so will) on the Nomad agent. 81 82 This is also why UID/GID needs to match between the user that runs the Podman 83 socket on the Nomad node and the user that spawns the Jenkins agent (the Nomad 84 job). 85 86 More on this relationship between the container, the Jenkins agent and the 87 Nomad node in the graphic below. 88 89 == Nomad node requirements == 90 91 Because in the end, the Docker containers are spawned on the Nomad nodes (as 92 Podman containers), the Nomad job (the Jenkins agent) needs direct access to 93 the filesystem of the Nomad node. Specifically, the Jenkins workspace directory 94 needs to be mounted to each Jenkins agent. 95 96 +------------------+ 97 | Docker container |- - - - - - + 98 +------------------+ | 99 ^ mounts workspace | 100 | spawns Docker | 101 +---------------------------+ | 102 | Jenkins agent (Nomad job) | | is scheduled 103 +---------------------------+ | on and writes 104 ^ | workspace 105 | /home/jenkins/workspace | 106 | has user with UID/GID | 107 +------------+ | 108 | Nomad node |<- - - - - - - - -+ 109 +------------+ 110 111 The Docker pipeline (docker-workflow) plugin has the default configuration to 112 automatically mount a Docker volume in `/home/jenkins/workspace`. 113 114 This can be seen in the log of Jenkins when the first Job with a docker 115 pipeline configuration starts: 116 117 $ docker run -t -d -u 1312:1312 -u root -w /home/jenkins/workspace/tutut -v 118 /home/jenkins/workspace/tutut:/home/jenkins/workspace/tutut:rw,z -v 119 /home/jenkins/workspace/tutut@tmp:/home/jenkins/workspace/tutut@tmp:rw,z -e 120 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 121 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 122 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 123 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 124 ******** -e ******** -e ******** docker.io/arm64v8/alpine:latest cat 125 126 Here we see that the default workspace directory (-w flag) is allocated in 127 `/home/jenkins/workspace` (the directory which is mounted through the Nomad job 128 from the Nomad node). Also, the plugin automatically mounts a sub-directory in 129 that folder for purposes of running the particular pipeline. 130 131 If this mount propagation from Nomad node to job to the final container does 132 not work, there will be error similar to this one in the Jenkins pipeline: 133 134 cp: can't stat '/home/jenkins/workspace/dubi@tmp/durable-9195a799/script.sh': No such file or directory 135 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-log.txt: nonexistent directory 136 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp: nonexistent directory 137 mv: can't rename '/home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp': No such file or directory 138 139 Once again, it's critical to mount the Jenkins workspace(s) as volume inside 140 the container because of how the docker-workflow plugin works: 141 142 > For inside to work, the Docker server and the Jenkins agent must use the same 143 > filesystem, so that the workspace can be mounted. The easiest way to ensure 144 > this is for the Docker server to be running on localhost (the same computer 145 > as the agent). Currently neither the Jenkins plugin nor the Docker CLI will 146 > automatically detect the case that the server is running remotely; a typical 147 > symptom would be errors from nested sh commands such as 148 > 149 > "cannot create /…@tmp/durable-…/pid: Directory nonexistent" 150 151 https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow#docker-workflow-sect-inside 152 153 The same warning note is also given again here: 154 * https://www.jenkins.io/doc/book/pipeline/docker/#using-a-remote-docker-server 155 156 Because the `/home/jenkins/workspace` is the default workspace directory of the 157 plugin, the cleanest approach is probably to to simply create this dedicated 158 Jenkins service user on all Nomad nodes (with a dedicated UID/GID combination), 159 then start the Podman socket as systemd user job on the Nomad nodes. 160 161 On all the Nomad clients, prepare the Jenkins user and the workspace directory 162 (1312 can be any UID/GID combination, it just needs to map with the User in the 163 Jenkins cloud plugin configuration where the Nomad job is spawned). Example 164 script: https://code.in0rdr.ch/hashipi/file/nomad.sh.html 165 166 If you need to redo the jenkins user configuration (e.g., to change the 167 UID/GID), make sure to stop the systemd service for the user. Otherwise, new 168 directories will still be created with old GIDs. 169 170 systemctl stop user@1312 # removes /run/user/1312 171 rm -rf /run/user/1312 172 173 == /home/jenkins (local) and /var/jenkins_home (NFS) == 174 175 * There exists a truststore `/home/jenkins/nomad-agent-ca.p12` on each Nomad 176 node, the Nomad provisioning script configures this truststore in 177 `/home/jenkins` 178 * This truststore only contains the public CA certificate of the Nomad API 179 (:4646), password is irrelevant here 180 * The Jenkins Nomad server job mounts the p12 truststore from `/home/jenkins` 181 to `/etc/ssl/certs/` in the Jenkins server container 182 * `/var/jenkins_home` is the path of the CSI volume mount in the Jenkins server 183 container, it contains for instance all the plugins of the Jenkins server 184 * The Jenkins workspaces `/home/jenkins/workspace` are not stored on the CSI 185 volume, but mounted directly to the downstream jenkins podman workers 186 187 To summarize: 188 189 $ nomad node status | grep jenkins 190 jenkins service 50 running 2024-07-27T23:08:10+02:00 191 jenkins-podman-122de767400f batch 50 running 2024-07-27T23:17:03+02:00 192 193 The Jenkins server container has the CSI volume mounted as `/var/jenkins_home`. 194 195 The jenkins-podman downstream containers have the `/home/jenkins/workspace` 196 folder mounted at `/home/jenkins/workspace`. 197 198 TODO: Can we move the storage of the jenkins-podman downstream containers to 199 the CSI volume as well? Can we add the volume_mounts section to the json 200 template of the nomad cloud configuration? 201 202 ```json 203 "VolumeMounts": [ 204 { 205 "Volume": "jenkins", 206 "Destination": "/home/jenkins", 207 "ReadOnly": false, 208 "PropagationMode": "private", 209 "SELinuxLabel": "" 210 } 211 ], 212 ``` 213 214 The workspaces would then probably be created on the CSI volume as well. 215 216 == Example build process == 217 218 To configure a different UID/GID for the Jenkins user, it is also required to 219 rebuild a Jenkins agent image with that particular UID/GID. 220 221 To build the Jenkins agent docker container for the purposes of using it in 222 Nomad: 223 224 git clone https://github.com/jenkinsci/docker-agent.git docker-agent.git 225 cd docker-agent.git 226 227 # change the uid/gid 228 buildah bud --no-cache --arch arm64/v8 --build-arg=JAVA_VERSION=21.0.3_9 \ 229 --build-arg=uid=1312 --build-arg=gid=1312 -f alpine/Dockerfile \ 230 -t 127.0.0.1:5000/jenkins-inbound-agent:latest . 231 232 podman push 127.0.0.1:5000/jenkins-inbound-agent:latest 233 234 == Jenkins Pipeline examples == 235 236 Two "declarative" pipeline examples: 237 238 pipeline { 239 agent { 240 docker { 241 image 'docker.io/arm64v8/alpine:latest' 242 // The Nomad cloud spawns Nomad agents with that label 243 label 'podman' 244 args '-u root' 245 } 246 } 247 248 stages { 249 stage('runInDocker') { 250 steps { 251 sh 'echo "hello world"' 252 } 253 } 254 } 255 } 256 257 pipeline { 258 agent { 259 docker { 260 image 'docker.io/hashicorp/vault:latest' 261 label 'podman' 262 args '-u root --entrypoint=""' 263 } 264 } 265 266 stages { 267 stage('runInDocker') { 268 steps { 269 sh ''' 270 vault version 271 ''' 272 } 273 } 274 } 275 } 276 277 The "scripted" pipeline is similar: 278 * https://www.jenkins.io/doc/book/pipeline/docker 279 280 Makes sure to set the label of the node ("podman"). Few examples: 281 282 node('podman') { 283 stage('runInDocker') { 284 docker.image('docker.io/arm64v8/alpine:latest').inside("-u root") { 285 writeFile(file: "readme", text: "was here --in0rdr") 286 sh ''' 287 echo "hello world" 288 cat readme 289 ''' 290 } 291 } 292 } 293 294 node('podman') { 295 stage('runInDocker') { 296 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""') { 297 sh "vault version" 298 } 299 } 300 } 301 302 node('podman') { 303 stage('runInDocker') { 304 docker.image('docker.io/arm64v8/alpine:latest').withRun() { c -> 305 sh 'echo "hi there in the logs"' 306 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""' + 307 ' -e "VAULT_ADDR=https://vault.in0rdr.ch"') { 308 sh "echo VAULT_VERSION: `vault version`" 309 sh "printenv | grep -i vault" 310 } 311 // inspect container before the pipeline exits 312 sh "docker inspect ${c.id}" 313 } 314 } 315 } 316 317 Even more examples (not necessarily docker related): 318 * https://www.jenkins.io/doc/pipeline/examples 319 320 == Integration with Git post-receive hooks == 321 322 There exists the option to nudge Jenkins on every push, see 323 https://plugins.jenkins.io/git/#plugin-content-push-notification-from-repository:: 324 * Create token in security settings of jenkins 325 * Configure post-receive hook, add the curl request (branch optional) 326 327 curl --max-time 5 -s \ 328 "https://jenkins.in0rdr.ch/git/notifyCommit?token=$GIT_TOKEN&url=$REPO_URL&branches=$BRANCH"