README (12672B)
1 = Jenkins Agent Dockerfile = 2 3 Based on the upstream docker file for Jenkins inbound agents: 4 * https://github.com/jenkinsci/docker-agent 5 * https://github.com/jenkinsci/docker-inbound-agents 6 7 Change the uid/gid to be unique. Create a unique user jenkins on the Nomad 8 agents and assign this user a unique uid/gid that is only used for Jenkins 9 Docker builds. 10 11 Example (based on the jenkinsci/docker-inbound-agents example above): 12 * https://github.com/jenkinsci/docker-agent/compare/master...in0rdr:docker-agent:debug/podman_x86_64 13 14 == Jenkins requirements == 15 16 docker-workflow plugin: 17 * https://plugins.jenkins.io/docker-workflow 18 * https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow 19 * https://www.jenkins.io/doc/book/installing/docker 20 * https://www.jenkins.io/doc/book/pipeline/docker 21 22 Ideally, the Jenkins as code config plugin: 23 * https://plugins.jenkins.io/configuration-as-code 24 25 Note, that you don't need to install the `docker-plugin` for Jenkins. This 26 plugin has a different use. It should be used to spawn Jenkins agents on Docker 27 containers. What we do here is spawning new Nomad jobs instead. The Nomad jobs 28 (workers with a particular label that signal that they can run Docker workload) 29 will then launch the Docker container. 30 31 == Nomad requirements == 32 33 Nomad plugin for Jenkins (aka Nomad cloud): 34 * https://plugins.jenkins.io/nomad 35 * https://faun.pub/jenkins-build-agents-on-nomad-workers-626b0df4fc57 36 * https://github.com/GastroGee/jenkins-nomad 37 38 The Nomad job template for new Jenkins agent nodes can be configured as code 39 (Jenkins as code configuration plugin): 40 * https://code.in0rdr.ch/nomad/file/hcl/default/jenkins/templates/jenkins.yaml.tmpl.html 41 42 43 == Mount point requirements == 44 45 Because the Nomad jobs launch the Docker containers, it is required to give 46 access to a Docker socket (tcp or unix). In this example, we can share the unix 47 socket from the Nomad node with the Nomad job. The YAML template for the new 48 Jenkins agent (a Nomad job): 49 50 "Tasks": [ 51 { 52 "Name": "jenkins-podman-worker", 53 "Driver": "podman", 54 "User": "1312", 55 "Config": { 56 "volumes": [ 57 "/run/user/1312/podman/podman.sock:/home/jenkins/agent/podman.sock", 58 "/home/jenkins/workspace:/home/jenkins/workspace" 59 ], 60 "image": "127.0.0.1:5000/jenkins-inbound-agent:latest" 61 ... 62 63 64 This can be configured in the Jenkins configuration as code plugin mentioned in 65 the requirements sections above. 66 67 Note: 68 * The user UID/GID can be anything here, just make sure to build the Jenkins 69 inbound agent with the same UID/GID that are used by Nomad to spawn the 70 Jenkins agent job. 71 * The Docker socket shared with the Nomad job (the Jenkins agent) here needs to 72 be activated and run in the background (see enable-linger below) 73 74 Because the Jenkins agent sits in between our Podman host (the Nomad agent) and 75 the downstream Docker container where we run our app logic, it is essential to 76 mount the directory from the Nomad agent node to the Jenkins agent. Otherwise, 77 the downstream container where we run our app logic will always see an empty 78 directory, because in the end all containers are run (in a flat structure, as 79 you so will) on the Nomad agent. 80 81 This is also why UID/GID needs to match between the user that runs the Podman 82 socket on the Nomad node and the user that spawns the Jenkins agent (the Nomad 83 job). 84 85 More on this relationship between the container, the Jenkins agent and the 86 Nomad node in the graphic below. 87 88 == Nomad node requirements == 89 90 Because in the end, the Docker containers are spawned on the Nomad nodes (as 91 Podman containers), the Nomad job (the Jenkins agent) needs direct access to 92 the filesystem of the Nomad node. Specifically, the Jenkins workspace directory 93 needs to be mounted to each Jenkins agent. 94 95 +------------------+ 96 | Docker container |- - - - - - + 97 +------------------+ | 98 ^ mounts workspace | 99 | spawns Docker | 100 +---------------------------+ | 101 | Jenkins agent (Nomad job) | | is scheduled 102 +---------------------------+ | on and writes 103 ^ | workspace 104 | /home/jenkins/workspace | 105 | has user with UID/GID | 106 +------------+ | 107 | Nomad node |<- - - - - - - - -+ 108 +------------+ 109 110 The Docker pipeline (docker-workflow) plugin has the default configuration to 111 automatically mount a Docker volume in `/home/jenkins/workspace`. 112 113 This can be seen in the log of Jenkins when the first Job with a docker 114 pipeline configuration starts: 115 116 $ docker run -t -d -u 1312:1312 -u root -w /home/jenkins/workspace/tutut -v 117 /home/jenkins/workspace/tutut:/home/jenkins/workspace/tutut:rw,z -v 118 /home/jenkins/workspace/tutut@tmp:/home/jenkins/workspace/tutut@tmp:rw,z -e 119 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 120 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 121 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 122 ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e 123 ******** -e ******** -e ******** docker.io/arm64v8/alpine:latest cat 124 125 Here we see that the default workspace directory (-w flag) is allocated in 126 `/home/jenkins/workspace` (the directory which is mounted through the Nomad job 127 from the Nomad node). Also, the plugin automatically mounts a sub-directory in 128 that folder for purposes of running the particular pipeline. 129 130 If this mount propagation from Nomad node to job to the final container does 131 not work, there will be error similar to this one in the Jenkins pipeline: 132 133 cp: can't stat '/home/jenkins/workspace/dubi@tmp/durable-9195a799/script.sh': No such file or directory 134 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-log.txt: nonexistent directory 135 sh: can't create /home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp: nonexistent directory 136 mv: can't rename '/home/jenkins/workspace/dubi@tmp/durable-9195a799/jenkins-result.txt.tmp': No such file or directory 137 138 Once again, it's critical to mount the Jenkins workspace(s) as volume inside 139 the container because of how the docker-workflow plugin works: 140 141 > For inside to work, the Docker server and the Jenkins agent must use the same 142 > filesystem, so that the workspace can be mounted. The easiest way to ensure 143 > this is for the Docker server to be running on localhost (the same computer 144 > as the agent). Currently neither the Jenkins plugin nor the Docker CLI will 145 > automatically detect the case that the server is running remotely; a typical 146 > symptom would be errors from nested sh commands such as 147 > 148 > "cannot create /…@tmp/durable-…/pid: Directory nonexistent" 149 150 https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/docker-workflow#docker-workflow-sect-inside 151 152 The same warning note is also given again here: 153 * https://www.jenkins.io/doc/book/pipeline/docker/#using-a-remote-docker-server 154 155 Because the `/home/jenkins/workspace` is the default workspace directory of the 156 plugin, the cleanest approach is probably to to simply create this dedicated 157 Jenkins service user on all Nomad nodes (with a dedicated UID/GID combination), 158 then start the Podman socket as systemd user job on the Nomad nodes. 159 160 On all the Nomad clients, prepare the Jenkins user and the workspace directory 161 (1312 can be any UID/GID combination, it just needs to map with the User in the 162 Jenkins cloud plugin configuration where the Nomad job is spawned). Example 163 script: https://code.in0rdr.ch/hashipi/file/nomad.sh.html 164 165 If you need to redo the jenkins user configuration (e.g., to change the 166 UID/GID), make sure to stop the systemd service for the user. Otherwise, new 167 directories will still be created with old GIDs. 168 169 systemctl stop user@1312 # removes /run/user/1312 170 rm -rf /run/user/1312 171 172 == /home/jenkins (local) and /var/jenkins_home (NFS) == 173 174 * There exists a truststore `/home/jenkins/nomad-agent-ca.p12` on each Nomad 175 node, the Nomad provisioning script configures this truststore in 176 `/home/jenkins` 177 * This truststore only contains the public CA certificate of the Nomad API 178 (:4646), password is irrelevant here 179 * The Jenkins Nomad server job mounts the p12 truststore from `/home/jenkins` 180 to `/etc/ssl/certs/` in the Jenkins server container 181 * `/var/jenkins_home` is the path of the CSI volume mount in the Jenkins server 182 container, it contains for instance all the plugins of the Jenkins server 183 * The Jenkins workspaces `/home/jenkins/workspace` are not stored on the CSI 184 volume, but mounted directly to the downstream jenkins podman workers 185 186 To summarize: 187 188 $ nomad node status | grep jenkins 189 jenkins service 50 running 2024-07-27T23:08:10+02:00 190 jenkins-podman-122de767400f batch 50 running 2024-07-27T23:17:03+02:00 191 192 The Jenkins server container has the CSI volume mounted as `/var/jenkins_home`. 193 194 The jenkins-podman downstream containers have the `/home/jenkins/workspace` 195 folder mounted at `/home/jenkins/workspace`. 196 197 TODO: Can we move the storage of the jenkins-podman downstream containers to 198 the CSI volume as well? Can we add the volume_mounts section to the json 199 template of the nomad cloud configuration? 200 201 ```json 202 "VolumeMounts": [ 203 { 204 "Volume": "jenkins", 205 "Destination": "/home/jenkins", 206 "ReadOnly": false, 207 "PropagationMode": "private", 208 "SELinuxLabel": "" 209 } 210 ], 211 ``` 212 213 The workspaces would then probably be created on the CSI volume as well. 214 215 == Example build process == 216 217 To configure a different UID/GID for the Jenkins user, it is also required to 218 rebuild a Jenkins agent image with that particular UID/GID. 219 220 To build the Jenkins agent docker container for the purposes of using it in 221 Nomad: 222 223 git clone https://github.com/jenkinsci/docker-agent.git docker-agent.git 224 cd docker-agent.git 225 226 # change the uid/gid 227 buildah bud --no-cache --arch arm64/v8 --build-arg=JAVA_VERSION=21.0.3_9 \ 228 --build-arg=uid=1312 --build-arg=gid=1312 -f alpine/Dockerfile \ 229 -t 127.0.0.1:5000/jenkins-inbound-agent:latest . 230 231 podman push 127.0.0.1:5000/jenkins-inbound-agent:latest 232 233 == Jenkins Pipeline examples == 234 235 Two "declarative" pipeline examples: 236 237 pipeline { 238 agent { 239 docker { 240 image 'docker.io/arm64v8/alpine:latest' 241 // The Nomad cloud spawns Nomad agents with that label 242 label 'podman' 243 args '-u root' 244 } 245 } 246 247 stages { 248 stage('runInDocker') { 249 steps { 250 sh 'echo "hello world"' 251 } 252 } 253 } 254 } 255 256 pipeline { 257 agent { 258 docker { 259 image 'docker.io/hashicorp/vault:latest' 260 label 'podman' 261 args '-u root --entrypoint=""' 262 } 263 } 264 265 stages { 266 stage('runInDocker') { 267 steps { 268 sh ''' 269 vault version 270 ''' 271 } 272 } 273 } 274 } 275 276 The "scripted" pipeline is similar: 277 * https://www.jenkins.io/doc/book/pipeline/docker 278 279 Makes sure to set the label of the node ("podman"). Few examples: 280 281 node('podman') { 282 stage('runInDocker') { 283 docker.image('docker.io/arm64v8/alpine:latest').inside("-u root") { 284 writeFile(file: "readme", text: "was here --in0rdr") 285 sh ''' 286 echo "hello world" 287 cat readme 288 ''' 289 } 290 } 291 } 292 293 node('podman') { 294 stage('runInDocker') { 295 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""') { 296 sh "vault version" 297 } 298 } 299 } 300 301 node('podman') { 302 stage('runInDocker') { 303 docker.image('docker.io/arm64v8/alpine:latest').withRun() { c -> 304 sh 'echo "hi there in the logs"' 305 docker.image('docker.io/hashicorp/vault:latest').inside('-u root --entrypoint=""' + 306 ' -e "VAULT_ADDR=https://vault.in0rdr.ch"') { 307 sh "echo VAULT_VERSION: `vault version`" 308 sh "printenv | grep -i vault" 309 } 310 // inspect container before the pipeline exits 311 sh "docker inspect ${c.id}" 312 } 313 } 314 } 315 316 Even more examples (not necessarily docker related): 317 * https://www.jenkins.io/doc/pipeline/examples 318 319 == Integration with Git post-receive hooks == 320 321 There exists the option to nudge Jenkins on every push, see 322 https://plugins.jenkins.io/git/#plugin-content-push-notification-from-repository:: 323 * Create token in security settings of jenkins 324 * Configure post-receive hook, add the curl request (branch optional) 325 326 curl --max-time 5 -s \ 327 "https://jenkins.in0rdr.ch/git/notifyCommit?token=$GIT_TOKEN&url=$REPO_URL&branches=$BRANCH"