README.md (8360B)
1 # HashiPi 2 A test cluster for HashiCorp Nomad and OpenBao. 3 4 ## Client-Server Architecture 5 The cluster can be built with 3 or 5 nodes. 6 7 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep 8 the Raft traffic minimal. 9 10 Example architecture with 5 RaspberyPi nodes: 11 12 | Node | Arch | RAM | Nomad function | Bao function | 13 | ---- | ----- |-----|----------------|--------------| 14 | 00 | arm64 | 4GB | client/server | server | 15 | 01 | arm64 | 4GB | client | n/a | 16 | 02 | arm64 | 4GB | client/server | server | 17 | 03 | arm64 | 4GB | client | n/a | 18 | 04 | arm64 | 4GB | client/server | server | 19 | 05 | amd64 | 4GB | client/server | server | 20 21 For best [performance on low power 22 devices](https://developer.hashicorp.com/consul/docs/install/performance), the 23 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier) 24 of Nomad servers is set to the 5. 25 26 The nomad clients have a higher [heartbeat 27 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats) 28 set to make sure the low powered devices are not reported down too often: 29 30 ``` 31 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats 32 # Increase heartbeat ttl under unreliable network conditions to prevent 33 # client: error heartbeating. retrying: error="failed to update status: rpc 34 # error: Not ready to serve consistent reads" 35 heartbeat_grace = "30s" 36 min_heartbeat_ttl = "30s" 37 ``` 38 39 ## Packer Builders 40 These Packer files use the following builders: 41 * [Packer builder for ARM (plugin 42 "cross")](https://github.com/michalfita/packer-plugin-cross) 43 * [QEMU](https://developer.hashicorp.com/packer/integrations/hashicorp/qemu) 44 builder 45 46 The packer builder "cross" requires `qemu-img`. 47 48 ## Supported Architectures 49 * ARM >= ARMv7 50 * AMD64 51 52 ## Self-signed TLS Certificates 53 ### OpenBao 54 To create a new self-signed CA certificate for Bao: 55 ```bash 56 # create bao self-signed CA certificate in ./tls/vault/ 57 ./vault-tls.sh 58 ``` 59 60 A new self-signed server certificate is created in the Bao provisioning stage. 61 62 The Packer JSON supports a few arguments for Bao server certificates: 63 ```json 64 "vault_tls_ca_cert": "./tls/vault/ca/vault_ca.pem", 65 "vault_tls_ca_key": "./tls/vault/ca/vault_ca.key", 66 "vault_tls_subj_alt_name": "" 67 ``` 68 69 * `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host, 70 e.g., created with `./vault-tls.sh` 71 * `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g., 72 created with `./vault-tls.sh` 73 * `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names 74 (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com` 75 76 ### Nomad 77 The steps to create a set of self-signed certificates for Nomad are not fully 78 automated to have control over the certificate generation process. 79 80 TLS is a prerequisite for workload identities with JWT: 81 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity 82 83 A `nomad` binary on the Packer build host is required to create the initial CA 84 certificate (trust anchor): 85 ```bash 86 # create nomad CA 87 mkdir -p tls/nomad && cd tls/nomad 88 nomad tls ca create 89 ``` 90 91 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config: 92 ```bash 93 # https://plugins.jenkins.io/nomad 94 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh 95 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \ 96 -alias nomad -storetype pkcs12 -storepass 123456 -noprompt 97 ``` 98 99 Then run the script from the projects root directory to create a new set of 100 certificates in the directory `./tls/nomad/certs`: 101 ```bash 102 ./nomad-tls.sh 103 ``` 104 105 ## ACL's 106 tbd 107 108 * ACLs for Nomad 109 110 ## Nomad workload identity configuration 111 Follow along the tutorial to configure Nomad workload identities with Bao: 112 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl 113 114 ```bash 115 $ cat vault-jwt-config.json 116 { 117 "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json", 118 "jwt_supported_algs": ["RS256", "EdDSA"], 119 "default_role": "nomad-workloads" 120 } 121 122 # reuse the nomad-agent-ca.pem to configure the jwt auth backend 123 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json 124 125 $ cat vault-jwt-role.json 126 { 127 "role_type": "jwt", 128 "bound_audiences": ["vault.in0rdr.ch"], 129 "user_claim": "/nomad_job_id", 130 "user_claim_json_pointer": true, 131 "claim_mappings": { 132 "nomad_namespace": "nomad_namespace", 133 "nomad_job_id": "nomad_job_id", 134 "nomad_task": "nomad_task" 135 }, 136 "token_type": "service", 137 "token_policies": ["nomad-workloads"], 138 "token_period": "30m", 139 "token_explicit_max_ttl": 0 140 } 141 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json 142 143 # keep the bao policy a bit simpler (only job level nesting for kv path) 144 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad 145 $ cat vault-policy-nomad-workloads.hcl 146 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" { 147 capabilities = ["list", "read"] 148 } 149 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl 150 ``` 151 152 ## Authorized Keys 153 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable. 154 155 ## Statically linked QEMU emulator 156 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu: 157 ```bash 158 sudo apt-get install qemu-user-static 159 ``` 160 161 For other distributions, a [recent `qemu-aarch64-static` binary can be 162 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved 163 to the proper location for 164 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up: 165 ```bash 166 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static 167 chmod +x qemu-aarch64-static 168 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static 169 ``` 170 171 Also, make sure to choose the correct "static" binary for the OS architecture 172 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl): 173 ```bash 174 "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", 175 "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" 176 ``` 177 178 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package 179 `qemu-linux-user`): 180 ```bash 181 sudo su 182 # Remove binfmt_misc entry files before register the entry. When same name's file 183 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with 184 # an error message "sh: write error: File exists": 185 # https://github.com/multiarch/qemu-user-static/blob/master/README.md 186 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \; 187 # Example from: 188 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh 189 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin 190 191 # Check 192 cat /proc/sys/fs/binfmt_misc/qemu-aarch64 193 enabled 194 interpreter /usr/bin/qemu-aarch64-static 195 ... 196 ``` 197 198 ## QEMU emulator on NixOS 199 On NixOS, it is sufficient to [enable the binfmt wrapper in the 200 configuration](https://wiki.nixos.org/wiki/NixOS_on_ARM/Building_Images): 201 ``` 202 boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; 203 ``` 204 205 The wrapper on NixOS can be checked on this path: 206 ```bash 207 cat /proc/sys/fs/binfmt_misc/aarch64-linux 208 ``` 209 210 On NixOS, the interpreter is on this path (`hashi-pi.pkr.hcl`): 211 ```bash 212 "qemu_binary_source_path": "/run/binfmt/aarch64-linux", 213 "qemu_binary_destination_path": "/run/binfmt/aarch64-linux" 214 ``` 215 216 ## Run Packer 217 Initialize required packer plugins: 218 ```bash 219 sudo packer init . 220 ``` 221 222 Run packer with a value file to build an image for one host. Example for arm64 host: 223 ```bash 224 sudo packer build \ 225 -only cross.hashipi \ 226 -var-file=variables.auto.pkrvars.hcl \ 227 -var-file=hosts/pi0.pkrvars.hcl \ 228 hashi-pi.pkr.hcl 229 ``` 230 231 Example for amd64 host: 232 ```bash 233 sudo packer build \ 234 -only qemu.hashiintel \ 235 -var-file=variables.auto.pkrvars.hcl \ 236 -var-file=hosts/intel0.pkrvars.hcl \ 237 hashi-pi.pkr.hcl 238 ``` 239 240 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables. 241 242 ## Write Image to SD Card 243 244 Convert qcow2 to img: 245 ```bash 246 qemu-img convert intel0/intel0.qcow2 -O raw intel0.img 247 ``` 248 249 To [write the resulting image file to the sd 250 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md) 251 with `dd`: 252 ```bash 253 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync 254 ```