README.md (8056B)
1 # HashiPi 2 A test cluster for HashiCorp Nomad and OpenBao. 3 4 ## Client-Server Architecture 5 The cluster can be built with 3 or 5 nodes. 6 7 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep 8 the Raft traffic minimal. 9 10 Example architecture with 5 RaspberyPi nodes: 11 12 | Node | Arch | RAM | Nomad function | Bao function | 13 | ---- | ----- |-----|----------------|--------------| 14 | 00 | arm64 | 4GB | client/server | server | 15 | 01 | arm64 | 4GB | client | n/a | 16 | 02 | arm64 | 4GB | client/server | server | 17 | 03 | arm64 | 4GB | client | n/a | 18 | 04 | arm64 | 4GB | client/server | server | 19 20 For best [performance on low power 21 devices](https://developer.hashicorp.com/consul/docs/install/performance), the 22 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier) 23 of Nomad servers is set to the 5. 24 25 The nomad clients have a higher [heartbeat 26 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats) 27 set to make sure the low powered devices are not reported down too often: 28 29 ``` 30 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats 31 # Increase heartbeat ttl under unreliable network conditions to prevent 32 # client: error heartbeating. retrying: error="failed to update status: rpc 33 # error: Not ready to serve consistent reads" 34 heartbeat_grace = "30s" 35 min_heartbeat_ttl = "30s" 36 ``` 37 38 ## Packer Builders 39 These Packer files use the following builders: 40 * [Packer builder for ARM (plugin 41 "cross")](https://github.com/michalfita/packer-plugin-cross) 42 * [QEMU](https://developer.hashicorp.com/packer/integrations/hashicorp/qemu) 43 builder 44 45 The packer builder "cross" requires `qemu-img`. 46 47 ## Supported Architectures 48 * ARM >= ARMv7 49 * AMD64 50 51 ## Self-signed TLS Certificates 52 ### OpenBao 53 To create a new self-signed CA certificate for Bao: 54 ```bash 55 # create bao self-signed CA certificate in ./tls/vault/ 56 ./vault-tls.sh 57 ``` 58 59 A new self-signed server certificate is created in the Bao provisioning stage. 60 61 The Packer JSON supports a few arguments for Bao server certificates: 62 ```json 63 "vault_tls_ca_cert": "./tls/vault/ca/vault_ca.pem", 64 "vault_tls_ca_key": "./tls/vault/ca/vault_ca.key", 65 "vault_tls_subj_alt_name": "" 66 ``` 67 68 * `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host, 69 e.g., created with `./vault-tls.sh` 70 * `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g., 71 created with `./vault-tls.sh` 72 * `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names 73 (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com` 74 75 ### Nomad 76 The steps to create a set of self-signed certificates for Nomad are not fully 77 automated to have control over the certificate generation process. 78 79 TLS is a prerequisite for workload identities with JWT: 80 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity 81 82 A `nomad` binary on the Packer build host is required to create the initial CA 83 certificate (trust anchor): 84 ```bash 85 # create nomad CA 86 mkdir -p tls/nomad && cd tls/nomad 87 nomad tls ca create 88 ``` 89 90 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config: 91 ```bash 92 # https://plugins.jenkins.io/nomad 93 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh 94 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \ 95 -alias nomad -storetype pkcs12 -storepass 123456 -noprompt 96 ``` 97 98 Then run the script from the projects root directory to create a new set of 99 certificates in the directory `./tls/nomad/certs`: 100 ```bash 101 ./nomad-tls.sh 102 ``` 103 104 ## ACL's 105 tbd 106 107 * ACLs for Nomad 108 109 ## Nomad workload identity configuration 110 Follow along the tutorial to configure Nomad workload identities with Bao: 111 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl 112 113 ```bash 114 $ cat vault-jwt-config.json 115 { 116 "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json", 117 "jwt_supported_algs": ["RS256", "EdDSA"], 118 "default_role": "nomad-workloads" 119 } 120 121 # reuse the nomad-agent-ca.pem to configure the jwt auth backend 122 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json 123 124 $ cat vault-jwt-role.json 125 { 126 "role_type": "jwt", 127 "bound_audiences": ["vault.in0rdr.ch"], 128 "user_claim": "/nomad_job_id", 129 "user_claim_json_pointer": true, 130 "claim_mappings": { 131 "nomad_namespace": "nomad_namespace", 132 "nomad_job_id": "nomad_job_id", 133 "nomad_task": "nomad_task" 134 }, 135 "token_type": "service", 136 "token_policies": ["nomad-workloads"], 137 "token_period": "30m", 138 "token_explicit_max_ttl": 0 139 } 140 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json 141 142 # keep the bao policy a bit simpler (only job level nesting for kv path) 143 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad 144 $ cat vault-policy-nomad-workloads.hcl 145 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" { 146 capabilities = ["list", "read"] 147 } 148 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl 149 ``` 150 151 ## Authorized Keys 152 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable. 153 154 ## Statically linked QEMU emulator 155 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu: 156 ```bash 157 sudo apt-get install qemu-user-static 158 ``` 159 160 For other distributions, a [recent `qemu-aarch64-static` binary can be 161 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved 162 to the proper location for 163 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up: 164 ```bash 165 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static 166 chmod +x qemu-aarch64-static 167 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static 168 ``` 169 170 Also, make sure to choose the correct "static" binary for the OS architecture 171 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl): 172 ```bash 173 "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", 174 "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" 175 ``` 176 177 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package 178 `qemu-linux-user`): 179 ```bash 180 sudo su 181 # Remove binfmt_misc entry files before register the entry. When same name's file 182 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with 183 # an error message "sh: write error: File exists": 184 # https://github.com/multiarch/qemu-user-static/blob/master/README.md 185 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \; 186 # Example from: 187 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh 188 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin 189 190 # Check 191 cat /proc/sys/fs/binfmt_misc/qemu-aarch64 192 enabled 193 interpreter /usr/bin/qemu-aarch64-static 194 ... 195 ``` 196 197 ## QEMU emulator on NixOS 198 On NixOS, it is sufficient to [enable the binfmt wrapper in the 199 configuration](https://wiki.nixos.org/wiki/NixOS_on_ARM/Building_Images): 200 ``` 201 boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; 202 ``` 203 204 The wrapper on NixOS can be checked on this path: 205 ```bash 206 cat /proc/sys/fs/binfmt_misc/aarch64-linux 207 ``` 208 209 On NixOS, the interpreter is on this path (`hashi-pi.pkr.hcl`): 210 ```bash 211 "qemu_binary_source_path": "/run/binfmt/aarch64-linux", 212 "qemu_binary_destination_path": "/run/binfmt/aarch64-linux" 213 ``` 214 215 ## Run Packer 216 Initialize required packer plugins: 217 ```bash 218 sudo packer init . 219 ``` 220 221 Run packer with a value file to build an image for one host: 222 ```bash 223 sudo packer build \ 224 -only cross.hashipi \ # change to qemu.hashiintel to build amd images 225 -var-file=variables.auto.pkrvars.hcl \ 226 -var-file=hosts/pi0.pkrvars.hcl \ 227 hashi-pi.pkr.hcl 228 ``` 229 230 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables. 231 232 ## Write Image to SD Card 233 To [write the resulting image file to the sd 234 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md) 235 with `dd`: 236 ```bash 237 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync 238 ```