README (12818B)
1 = Jenkins Agent Dockerfile = 2 3 Based on the upstream docker file for Jenkins inbound agents: 4 * https://github.com/jenkinsci/docker-agent 5 * https://github.com/jenkinsci/docker-inbound-agents 6 7 Change the uid/gid to be unique. Create a unique user jenkins on the Nomad 8 agents and assign this user a unique uid/gid that is only used for Jenkins 9 Docker builds. 10 11 == Jenkins requirements == 12 13 docker-workflow plugin: 14 * https://plugins.jenkins.io/docker-workflow 15 * https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow 16 * https://www.jenkins.io/doc/book/installing/docker 17 * https://www.jenkins.io/doc/book/pipeline/docker 18 19 Ideally, the Jenkins as code config plugin: 20 * https://plugins.jenkins.io/configuration-as-code 21 22 Note, that you don't need to install the `docker-plugin` for Jenkins. This 23 plugin has a different use. It should be used to spawn Jenkins agents on Docker 24 containers. What we do here is spawning new Nomad jobs instead. The Nomad jobs 25 (workers with a particular label that signal that they can run Docker workload) 26 will then launch the Docker container. 27 28 == Nomad requirements == 29 30 Nomad plugin for Jenkins (aka Nomad cloud): 31 * https://plugins.jenkins.io/nomad 32 * https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57 33 * https://github.com/GastroGee/jenkins-nomad 34 35 The Nomad job template for new Jenkins agent nodes can be configured as code 36 (Jenkins as code configuration plugin): 37 * https://code.in0rdr.ch/nomad/file/hcl/default/jenkins/templates/jenkins.yaml.tmpl.html 38 39 40 == Mount point requirements == 41 42 Because the Nomad jobs launch the Docker containers, it is required to give 43 access to a Docker socket (tcp or unix). In this example, we can share the unix 44 socket from the Nomad node with the Nomad job. The YAML template for the new 45 Jenkins agent (a Nomad job): 46 47 "Tasks": [ 48 { 49 "Name": "jenkins-podman-worker", 50 "Driver": "podman", 51 "User": "1312", 52 "Config": { 53 "volumes": [ 54 "/run/user/1312/podman/podman.sock:/home/jenkins/agent/podman.sock", 55 "/etc/containers/registries.conf:/etc/containers/registries.conf", 56 "/home/jenkins/workspace:/home/jenkins/workspace" 57 ], 58 "image": "127.0.0.1:5000/jenkins-inbound-agent:latest" 59 ... 60 61 62 This can be configured in the Jenkins configuration as code plugin mentioned in 63 the requirements sections above. 64 65 Note: 66 * The user UID/GID can be anything here, just make sure to build the Jenkins 67 inbound agent with the same UID/GID that are used by Nomad to spawn the 68 Jenkins agent job. 69 * The Docker socket shared with the Nomad job (the Jenkins agent) here needs to 70 be activated and run in the background (see 71 https://code.in0rdr.ch/hashipi/file/nomad.sh.html) 72 * The registries.conf from the host is used to connect to insecure (non-https) 73 registries 74 75 Because the Jenkins agent sits in between our Podman host (the Nomad agent) and 76 the downstream Docker container where we run our app logic, it is essential to 77 mount the directory from the Nomad agent node to the Jenkins agent. Otherwise, 78 the downstream container where we run our app logic will always see an empty 79 directory, because in the end all containers are run (in a flat structure, as 80 you so will) on the Nomad agent. 81 82 NOTE: THIS IS NOT DOCKER-IN-DOCKER 83 * https://github.com/jenkinsci/docker-workflow-plugin/tree/docker-workflow-1.12/demo 84 85 > the container only runs the CLI and connects back to the host to start sister 86 > containers 87 88 This is also why UID/GID needs to match between the user that runs the Podman 89 socket on the Nomad node and the user that spawns the Jenkins agent (the Nomad 90 job). 91 92 More on this relationship between the container, the Jenkins agent and the 93 Nomad node in the graphic below. 94 95 == Nomad node requirements == 96 97 Because in the end, the Docker containers are spawned on the Nomad nodes (as 98 Podman containers), the Nomad job (the Jenkins agent) needs direct access to 99 the filesystem of the Nomad node. Specifically, the Jenkins workspace directory 100 needs to be mounted to each Jenkins agent. 101 102 +------------------+ 103 | Docker container |- - - - - - + 104 +------------------+ | 105 ^ mounts workspace | 106 | spawns Docker | 107 +---------------------------+ | 108 | Jenkins agent (Nomad job) | | is scheduled 109 +---------------------------+ | on and writes 110 ^ | workspace 111 | /home/jenkins/workspace | 112 | has user with UID/GID | 113 +------------+ | 114 | Nomad node |<- - - - - - - - -+ 115 +------------+ 116 117 The Docker pipeline (docker-workflow) plugin has the default configuration to 118 automatically mount a Docker volume in `/home/jenkins/workspace`. 119 120 This can be seen in the log of Jenkins when the first Job with a docker 121 pipeline configuration starts: 122 123 $ docker run -t -d -u 1312:1312 -u root -w /home/jenkins/workspace/tutut -v 124 /home/jenkins/workspace/tutut:/home/jenkins/workspace/tutut:rw,z -v 125 /home/jenkins/workspace/tutut@tmp:/home/jenkins/workspace/tutut@tmp:rw,z -e 126 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 127 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 128 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 129 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 130 ******** -e ******** -e ******** docker.io/arm64v8/alpine:latest cat 131 132 Here we see that the default workspace directory (-w flag) is allocated in 133 `/home/jenkins/workspace` (the directory which is mounted through the Nomad job 134 from the Nomad node). Also, the plugin automatically mounts a sub-directory in 135 that folder for purposes of running the particular pipeline. 136 137 If this mount propagation from Nomad node to job to the final container does 138 not work, there will be error similar to this one in the Jenkins pipeline: 139 140 cp: can't stat '/home/jenkins/workspace/dubi@tmp/durable-9195a799/script.sh': No such file or directory 141 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-log.txt: nonexistent directory 142 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp: nonexistent directory 143 mv: can't rename '/home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp': No such file or directory 144 145 Once again, it's critical to mount the Jenkins workspace(s) as volume inside 146 the container because of how the docker-workflow plugin works: 147 148 > For inside to work, the Docker server and the Jenkins agent must use the same 149 > filesystem, so that the workspace can be mounted. The easiest way to ensure 150 > this is for the Docker server to be running on localhost (the same computer 151 > as the agent). Currently neither the Jenkins plugin nor the Docker CLI will 152 > automatically detect the case that the server is running remotely; a typical 153 > symptom would be errors from nested sh commands such as 154 > 155 > "cannot create /…@tmp/durable-…/pid: Directory nonexistent" 156 157 https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow#docker-workflow-sect-inside 158 159 The same warning note is also given again here: 160 * https://www.jenkins.io/doc/book/pipeline/docker/#using-a-remote-docker-server 161 162 Because the `/home/jenkins/workspace` is the default workspace directory of the 163 plugin, the cleanest approach is probably to to simply create this dedicated 164 Jenkins service user on all Nomad nodes (with a dedicated UID/GID combination), 165 then start the Podman socket as systemd user job on the Nomad nodes. 166 167 On all the Nomad clients, prepare the Jenkins user and the workspace directory 168 (1312 can be any UID/GID combination, it just needs to map with the User in the 169 Jenkins cloud plugin configuration where the Nomad job is spawned). Example 170 script: https://code.in0rdr.ch/hashipi/file/nomad.sh.html 171 172 If you need to redo the jenkins user configuration (e.g., to change the 173 UID/GID), make sure to stop the systemd service for the user. Otherwise, new 174 directories will still be created with old GIDs. 175 176 systemctl stop user@1312 # removes /run/user/1312 177 rm -rf /run/user/1312 178 179 == /home/jenkins (local) and /var/jenkins_home (NFS) == 180 181 * There exists a truststore `/home/jenkins/nomad-agent-ca.p12` on each Nomad 182 node, the Nomad provisioning script configures this truststore in 183 `/home/jenkins` 184 * This truststore only contains the public CA certificate of the Nomad API 185 (:4646), password is irrelevant here 186 * The Jenkins Nomad server job mounts the p12 truststore from `/home/jenkins` 187 to `/etc/ssl/certs/` in the Jenkins server container 188 * `/var/jenkins_home` is the path of the CSI volume mount in the Jenkins server 189 container, it contains for instance all the plugins of the Jenkins server 190 * The Jenkins workspaces `/home/jenkins/workspace` are not stored on the CSI 191 volume, but mounted directly to the downstream jenkins podman workers 192 193 To summarize: 194 195 $ nomad node status | grep jenkins 196 jenkins service 50 running 2024-07-27T23:08:10+02:00 197 jenkins-podman-122de767400f batch 50 running 2024-07-27T23:17:03+02:00 198 199 The Jenkins server container has the CSI volume mounted as `/var/jenkins_home`. 200 201 The jenkins-podman downstream containers have the `/home/jenkins/workspace` 202 folder mounted at `/home/jenkins/workspace`. 203 204 TODO: Can we move the storage of the jenkins-podman downstream containers to 205 the CSI volume as well? Can we add the volume_mounts section to the json 206 template of the nomad cloud configuration? 207 208 ```json 209 "VolumeMounts": [ 210 { 211 "Volume": "jenkins", 212 "Destination": "/home/jenkins", 213 "ReadOnly": false, 214 "PropagationMode": "private", 215 "SELinuxLabel": "" 216 } 217 ], 218 ``` 219 220 The workspaces would then probably be created on the CSI volume as well. 221 222 == Example build process == 223 224 To configure a different UID/GID for the Jenkins user, it is also required to 225 rebuild a Jenkins agent image with that particular UID/GID. 226 227 To build the Jenkins agent docker container for the purposes of using it in 228 Nomad: 229 230 # change the uid/gid 231 buildah bud --no-cache --arch arm64/v8 --build-arg=JAVA_VERSION=21.0.3_9 \ 232 --build-arg=uid=1312 --build-arg=gid=1312 -f alpine/Dockerfile \ 233 -t 127.0.0.1:5000/jenkins-inbound-agent:latest . 234 235 podman push 127.0.0.1:5000/jenkins-inbound-agent:latest 236 237 == Jenkins Pipeline examples == 238 239 Two "declarative" pipeline examples: 240 241 pipeline { 242 agent { 243 docker { 244 image 'docker.io/arm64v8/alpine:latest' 245 // The Nomad cloud spawns Nomad agents with that label 246 label 'podman' 247 args '-u root' 248 } 249 } 250 251 stages { 252 stage('runInDocker') { 253 steps { 254 sh 'echo "hello world"' 255 } 256 } 257 } 258 } 259 260 pipeline { 261 agent { 262 docker { 263 image 'docker.io/hashicorp/vault:latest' 264 label 'podman' 265 args '-u root --entrypoint=""' 266 } 267 } 268 269 stages { 270 stage('runInDocker') { 271 steps { 272 sh ''' 273 vault version 274 ''' 275 } 276 } 277 } 278 } 279 280 The "scripted" pipeline is similar: 281 * https://www.jenkins.io/doc/book/pipeline/docker 282 283 Makes sure to set the label of the node ("podman"). Few examples: 284 285 node('podman') { 286 stage('runInDocker') { 287 docker.image('docker.io/arm64v8/alpine:latest').inside("-u root") { 288 writeFile(file: "readme", text: "was here --in0rdr") 289 sh ''' 290 echo "hello world" 291 cat readme 292 ''' 293 } 294 } 295 } 296 297 node('podman') { 298 stage('runInDocker') { 299 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""') { 300 sh "vault version" 301 } 302 } 303 } 304 305 node('podman') { 306 stage('runInDocker') { 307 docker.image('docker.io/arm64v8/alpine:latest').withRun() { c -> 308 sh 'echo "hi there in the logs"' 309 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""' + 310 ' -e "VAULT_ADDR=https://vault.in0rdr.ch"') { 311 sh "echo VAULT_VERSION: `vault version`" 312 sh "printenv | grep -i vault" 313 } 314 // inspect container before the pipeline exits 315 sh "docker inspect ${c.id}" 316 } 317 } 318 } 319 320 Even more examples (not necessarily docker related): 321 * https://www.jenkins.io/doc/pipeline/examples 322 323 == Integration with Git post-receive hooks == 324 325 There exists the option to nudge Jenkins on every push, see 326 https://plugins.jenkins.io/git/#plugin-content-push-notification-from-repository:: 327 * Create token in security settings of jenkins 328 * Configure post-receive hook, add the curl request (branch optional) 329 330 curl --max-time 5 -s \ 331 "https://jenkins.in0rdr.ch/git/notifyCommit?token=$GIT_TOKEN&url=$REPO_URL&branches=$BRANCH"