README.md (8598B)
1 # HashiPi 2 A test cluster for HashiCorp Nomad and OpenBao. 3 4 ## Client-Server Architecture 5 The cluster can be built with 3 or 5 nodes. 6 7 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep 8 the Raft traffic minimal. 9 10 Example architecture with 5 RaspberyPi nodes: 11 12 | Node | Arch | RAM | Nomad function | Bao function | 13 | ---- | ----- |-----|----------------|--------------| 14 | 00 | arm64 | 4GB | client/server | server | 15 | 01 | arm64 | 4GB | client | n/a | 16 | 02 | arm64 | 4GB | client/server | server | 17 | 03 | arm64 | 4GB | client | n/a | 18 | 04 | arm64 | 4GB | client/server | server | 19 | 05 | amd64 | 4GB | client/server | server | 20 21 For best [performance on low power 22 devices](https://developer.hashicorp.com/consul/docs/install/performance), the 23 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier) 24 of Nomad servers is set to the 5. 25 26 The nomad clients have a higher [heartbeat 27 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats) 28 set to make sure the low powered devices are not reported down too often: 29 30 ``` 31 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats 32 # Increase heartbeat ttl under unreliable network conditions to prevent 33 # client: error heartbeating. retrying: error="failed to update status: rpc 34 # error: Not ready to serve consistent reads" 35 heartbeat_grace = "30s" 36 min_heartbeat_ttl = "30s" 37 ``` 38 39 ## Packer Builders 40 These Packer files use the following builders: 41 * [Packer builder for ARM (plugin 42 "cross")](https://github.com/michalfita/packer-plugin-cross) 43 * [QEMU](https://developer.hashicorp.com/packer/integrations/hashicorp/qemu) 44 builder 45 46 The packer builder "cross" requires `qemu-img`. 47 48 ## Supported Architectures 49 * ARM >= ARMv7 50 * AMD64 51 52 ## Self-signed TLS Certificates 53 ### OpenBao 54 OpenBao installation uses the self-signed certificates that are initially 55 created in `/opt/openbao/tls` during the OpenBao installation process from the 56 official package. 57 58 ### Nomad 59 The steps to create a set of self-signed certificates for Nomad are not fully 60 automated to have control over the certificate generation process. 61 62 TLS is a prerequisite for workload identities with JWT: 63 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity 64 65 A `nomad` binary on the Packer build host is required to create the initial CA 66 certificate (trust anchor): 67 ```bash 68 # create nomad CA 69 mkdir -p tls/nomad && cd tls/nomad 70 nomad tls ca create 71 ``` 72 73 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config: 74 ```bash 75 # https://plugins.jenkins.io/nomad 76 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh 77 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \ 78 -alias nomad -storetype pkcs12 -storepass 123456 -noprompt 79 ``` 80 81 Then run the script from the projects root directory to create a new set of 82 certificates in the directory `./tls/nomad/certs`: 83 ```bash 84 ./nomad-tls.sh 85 ``` 86 87 ## Nomad workload identity configuration 88 Follow along the tutorial to configure Nomad workload identities with Bao: 89 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl 90 91 ```bash 92 $ cat vault-jwt-config.json 93 { 94 "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json", 95 "jwt_supported_algs": ["RS256", "EdDSA"], 96 "default_role": "nomad-workloads" 97 } 98 99 # reuse the nomad-agent-ca.pem to configure the jwt auth backend 100 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json 101 102 $ cat vault-jwt-role.json 103 { 104 "role_type": "jwt", 105 "bound_audiences": ["vault.in0rdr.ch"], 106 "user_claim": "/nomad_job_id", 107 "user_claim_json_pointer": true, 108 "claim_mappings": { 109 "nomad_namespace": "nomad_namespace", 110 "nomad_job_id": "nomad_job_id", 111 "nomad_task": "nomad_task" 112 }, 113 "token_type": "service", 114 "token_policies": ["nomad-workloads"], 115 "token_period": "30m", 116 "token_explicit_max_ttl": 0 117 } 118 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json 119 120 # keep the bao policy a bit simpler (only job level nesting for kv path) 121 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad 122 $ cat vault-policy-nomad-workloads.hcl 123 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" { 124 capabilities = ["list", "read"] 125 } 126 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl 127 ``` 128 129 ## Nomad ACL 130 Manual setup, see [nomad-acl/README.md](./nomad-acl/README.md) 131 132 ## Authorized Keys 133 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable. 134 135 ## Statically linked QEMU emulator 136 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu: 137 ```bash 138 sudo apt-get install qemu-user-static 139 ``` 140 141 For other distributions, a [recent `qemu-aarch64-static` binary can be 142 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved 143 to the proper location for 144 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up: 145 ```bash 146 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static 147 chmod +x qemu-aarch64-static 148 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static 149 ``` 150 151 Also, make sure to choose the correct "static" binary for the OS architecture 152 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl): 153 ```bash 154 "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", 155 "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" 156 ``` 157 158 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package 159 `qemu-linux-user`): 160 ```bash 161 sudo su 162 # Remove binfmt_misc entry files before register the entry. When same name's file 163 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with 164 # an error message "sh: write error: File exists": 165 # https://github.com/multiarch/qemu-user-static/blob/master/README.md 166 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \; 167 # Example from: 168 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh 169 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin 170 171 # Check 172 cat /proc/sys/fs/binfmt_misc/qemu-aarch64 173 enabled 174 interpreter /usr/bin/qemu-aarch64-static 175 ... 176 ``` 177 178 ## QEMU emulator on NixOS 179 On NixOS, it is sufficient to [enable the binfmt wrapper in the 180 configuration](https://wiki.nixos.org/wiki/NixOS_on_ARM/Building_Images): 181 ``` 182 boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; 183 boot.binfmt.preferStaticEmulators = true; 184 ``` 185 186 The wrapper on NixOS can be checked on this path: 187 ```bash 188 cat /proc/sys/fs/binfmt_misc/aarch64-linux 189 ``` 190 191 On NixOS, the interpreter is on this path (`hashi-pi.pkr.hcl`): 192 ``` 193 "qemu_binary_source_path": "/run/binfmt/aarch64-linux", 194 "qemu_binary_destination_path": "/run/binfmt/aarch64-linux" 195 ``` 196 197 Also, set the chroot env appropriately: 198 ``` 199 image_chroot_env = ["PATH=/run/current-system/sw/bin:/run/current-system/sw/sbin:/usr/bin:/bin"] 200 ``` 201 202 ## Run Packer 203 Initialize required packer plugins: 204 ```bash 205 sudo packer init . 206 ``` 207 208 Run packer with a value file to build an image for one host. Example for arm64 host: 209 ```bash 210 sudo packer build \ 211 -only cross.hashipi \ 212 -var-file=variables.auto.pkrvars.hcl \ 213 -var-file=hosts/pi0.pkrvars.hcl \ 214 hashi-pi.pkr.hcl 215 ``` 216 217 Example for amd64 host: 218 ```bash 219 sudo packer build \ 220 -only qemu.hashiintel \ 221 -var-file=variables.auto.pkrvars.hcl \ 222 -var-file=hosts/intel0.pkrvars.hcl \ 223 hashi-pi.pkr.hcl 224 ``` 225 226 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables. 227 228 ## Testing the amd64/QEMU image 229 The qemu image can be tested locally: 230 ```bash 231 sudo qemu-system-x86_64 \ 232 -cpu host -machine type=q35,accel=kvm -m 2048 \ 233 -drive if=virtio,format=qcow2,file=intel0/intel0.qcow2 234 ``` 235 236 ## Write amd64 image to USB stick 237 Don't write the image from the test above, that might be modified. Use a 238 freshly built image. 239 240 The QEMU qcow2 image needs to be converted to raw disk format: 241 ```bash 242 sudo qemu-img convert -O raw intel0/intel0.qcow2 intel0/intel0.img 243 ``` 244 245 Then written to USB stick with `cp` & `sync`. 246 247 Use a [live iso](https://grml.org) and the USB stick with the raw image to 248 prepare the target disk on the intel machine: 249 ```bash 250 sudo mount /dev/sdc1 /mnt 251 sudo dd if=/mnt/intel0.img of=/dev/sda bs=4m 252 ``` 253 254 ## Write arm64 image to SD Card 255 To [write the resulting image file to the sd 256 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md) 257 with `dd`: 258 ```bash 259 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync 260 ```