README.md (7898B)
1 # HashiPi 2 A test cluster for HashiCorp Nomad and OpenBao. 3 4 ## Client-Server Architecture 5 The cluster can be built with 3 or 5 nodes. 6 7 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep 8 the Raft traffic minimal. 9 10 Example architecture with 5 RaspberyPi nodes: 11 12 | Node | Arch | RAM | Nomad function | Bao function | 13 | ---- | ----- |-----|----------------|--------------| 14 | 00 | arm64 | 4GB | client/server | server | 15 | 01 | arm64 | 4GB | client | n/a | 16 | 02 | arm64 | 4GB | client/server | server | 17 | 03 | arm64 | 4GB | client | n/a | 18 | 04 | arm64 | 4GB | client/server | server | 19 | 05 | amd64 | 4GB | client/server | server | 20 21 For best [performance on low power 22 devices](https://developer.hashicorp.com/consul/docs/install/performance), the 23 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier) 24 of Nomad servers is set to the 5. 25 26 The nomad clients have a higher [heartbeat 27 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats) 28 set to make sure the low powered devices are not reported down too often: 29 30 ``` 31 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats 32 # Increase heartbeat ttl under unreliable network conditions to prevent 33 # client: error heartbeating. retrying: error="failed to update status: rpc 34 # error: Not ready to serve consistent reads" 35 heartbeat_grace = "30s" 36 min_heartbeat_ttl = "30s" 37 ``` 38 39 ## Packer Builders 40 These Packer files use the following builders: 41 * [Packer builder for ARM (plugin 42 "cross")](https://github.com/michalfita/packer-plugin-cross) 43 * [QEMU](https://developer.hashicorp.com/packer/integrations/hashicorp/qemu) 44 builder 45 46 The packer builder "cross" requires `qemu-img`. 47 48 ## Supported Architectures 49 * ARM >= ARMv7 50 * AMD64 51 52 ## Self-signed TLS Certificates 53 ### OpenBao 54 OpenBao installation uses the self-signed certificates that are initially 55 created in `/opt/openbao/tls` during the OpenBao installation process from the 56 official package. 57 58 ### Nomad 59 The steps to create a set of self-signed certificates for Nomad are not fully 60 automated to have control over the certificate generation process. 61 62 TLS is a prerequisite for workload identities with JWT: 63 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity 64 65 A `nomad` binary on the Packer build host is required to create the initial CA 66 certificate (trust anchor): 67 ```bash 68 # create nomad CA 69 mkdir -p tls/nomad && cd tls/nomad 70 nomad tls ca create 71 ``` 72 73 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config: 74 ```bash 75 # https://plugins.jenkins.io/nomad 76 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh 77 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \ 78 -alias nomad -storetype pkcs12 -storepass 123456 -noprompt 79 ``` 80 81 Then run the script from the projects root directory to create a new set of 82 certificates in the directory `./tls/nomad/certs`: 83 ```bash 84 ./nomad-tls.sh 85 ``` 86 87 ## ACL's 88 tbd 89 90 * ACLs for Nomad 91 92 ## Nomad workload identity configuration 93 Follow along the tutorial to configure Nomad workload identities with Bao: 94 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl 95 96 ```bash 97 $ cat vault-jwt-config.json 98 { 99 "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json", 100 "jwt_supported_algs": ["RS256", "EdDSA"], 101 "default_role": "nomad-workloads" 102 } 103 104 # reuse the nomad-agent-ca.pem to configure the jwt auth backend 105 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json 106 107 $ cat vault-jwt-role.json 108 { 109 "role_type": "jwt", 110 "bound_audiences": ["vault.in0rdr.ch"], 111 "user_claim": "/nomad_job_id", 112 "user_claim_json_pointer": true, 113 "claim_mappings": { 114 "nomad_namespace": "nomad_namespace", 115 "nomad_job_id": "nomad_job_id", 116 "nomad_task": "nomad_task" 117 }, 118 "token_type": "service", 119 "token_policies": ["nomad-workloads"], 120 "token_period": "30m", 121 "token_explicit_max_ttl": 0 122 } 123 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json 124 125 # keep the bao policy a bit simpler (only job level nesting for kv path) 126 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad 127 $ cat vault-policy-nomad-workloads.hcl 128 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" { 129 capabilities = ["list", "read"] 130 } 131 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl 132 ``` 133 134 ## Authorized Keys 135 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable. 136 137 ## Statically linked QEMU emulator 138 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu: 139 ```bash 140 sudo apt-get install qemu-user-static 141 ``` 142 143 For other distributions, a [recent `qemu-aarch64-static` binary can be 144 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved 145 to the proper location for 146 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up: 147 ```bash 148 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static 149 chmod +x qemu-aarch64-static 150 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static 151 ``` 152 153 Also, make sure to choose the correct "static" binary for the OS architecture 154 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl): 155 ```bash 156 "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", 157 "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" 158 ``` 159 160 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package 161 `qemu-linux-user`): 162 ```bash 163 sudo su 164 # Remove binfmt_misc entry files before register the entry. When same name's file 165 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with 166 # an error message "sh: write error: File exists": 167 # https://github.com/multiarch/qemu-user-static/blob/master/README.md 168 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \; 169 # Example from: 170 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh 171 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin 172 173 # Check 174 cat /proc/sys/fs/binfmt_misc/qemu-aarch64 175 enabled 176 interpreter /usr/bin/qemu-aarch64-static 177 ... 178 ``` 179 180 ## QEMU emulator on NixOS 181 On NixOS, it is sufficient to [enable the binfmt wrapper in the 182 configuration](https://wiki.nixos.org/wiki/NixOS_on_ARM/Building_Images): 183 ``` 184 boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; 185 boot.binfmt.preferStaticEmulators = true; 186 ``` 187 188 The wrapper on NixOS can be checked on this path: 189 ```bash 190 cat /proc/sys/fs/binfmt_misc/aarch64-linux 191 ``` 192 193 On NixOS, the interpreter is on this path (`hashi-pi.pkr.hcl`): 194 ``` 195 "qemu_binary_source_path": "/run/binfmt/aarch64-linux", 196 "qemu_binary_destination_path": "/run/binfmt/aarch64-linux" 197 ``` 198 199 Also, set the chroot env appropriately: 200 ``` 201 image_chroot_env = ["PATH=/run/current-system/sw/bin:/run/current-system/sw/sbin:/usr/bin:/bin"] 202 ``` 203 204 ## Run Packer 205 Initialize required packer plugins: 206 ```bash 207 sudo packer init . 208 ``` 209 210 Run packer with a value file to build an image for one host. Example for arm64 host: 211 ```bash 212 sudo packer build \ 213 -only cross.hashipi \ 214 -var-file=variables.auto.pkrvars.hcl \ 215 -var-file=hosts/pi0.pkrvars.hcl \ 216 hashi-pi.pkr.hcl 217 ``` 218 219 Example for amd64 host: 220 ```bash 221 sudo packer build \ 222 -only qemu.hashiintel \ 223 -var-file=variables.auto.pkrvars.hcl \ 224 -var-file=hosts/intel0.pkrvars.hcl \ 225 hashi-pi.pkr.hcl 226 ``` 227 228 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables. 229 230 ## Write Image to SD Card 231 232 Convert qcow2 to img: 233 ```bash 234 qemu-img convert intel0/intel0.qcow2 -O raw intel0.img 235 ``` 236 237 To [write the resulting image file to the sd 238 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md) 239 with `dd`: 240 ```bash 241 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync 242 ```