hashipi

README.md (7648B)

---

 # HashiPi
 
 A RaspberryPi test cluster for HashiCorp Nomad and OpenBao.
 
 ![HashiPi](./img/HashiPi-small.jpg)
 
 ## Client-Server Architecture
 The cluster can be built with 3 or 5 nodes.
 
 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep
 the Raft traffic minimal.
 
 Example architecture with 5 RaspberyPi nodes:
 
 | Node | RAM | Nomad function | Bao function |
 |------|-----|----------------|--------------|
 | 00   | 4GB | client/server  | server       |
 | 01   | 4GB | client         | n/a          |
 | 02   | 4GB | client/server  | server       |
 | 03   | 4GB | client         | n/a          |
 | 04   | 4GB | client/server  | server       |
 
 For best [performance on low power
 devices](https://developer.hashicorp.com/consul/docs/install/performance), the
 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier)
 of Nomad servers is set to the 5.
 
 The nomad clients have a higher [heartbeat
 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats)
 set to make sure the low powered devices are not reported down too often:
 
 ```
 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats
 # Increase heartbeat ttl under unreliable network conditions to prevent
 # client: error heartbeating. retrying: error="failed to update status: rpc
 # error: Not ready to serve consistent reads"
 heartbeat_grace = "30s"
 min_heartbeat_ttl = "30s"
 ```
 
 ## ARM Builder 
 These Packer files use the [Packer builder for
 ARM](https://github.com/mkaczanowski/packer-builder-arm). Follow the
 instructions in the repo to setup the builder and use the provided
 [examples](https://github.com/mkaczanowski/packer-builder-arm/tree/master/boards)
 to modify to your liking.
 
 ## Supported Architectures
 
 ARM >= ARMv7
 
 ## Self-signed TLS Certificates
 ### OpenBao
 
 To create a new self-signed CA certificate for Bao:
 ```bash
 # create bao self-signed CA certificate in ./tls/vault/
 ./vault-tls.sh
 ```
 
 A new self-signed server certificate is created in the Bao provisioning stage.
 
 The Packer JSON supports a few arguments for Bao server certificates:
 ```json
     "vault_tls_ca_cert": "./tls/vault/ca/vault_ca.pem",
     "vault_tls_ca_key": "./tls/vault/ca/vault_ca.key",
     "vault_tls_subj_alt_name": ""
 ```
 
 * `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host,
   e.g., created with `./vault-tls.sh`
 * `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g.,
   created with `./vault-tls.sh`
 * `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names
   (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com`
 
 ### Nomad
 The steps to create a set of self-signed certificates for Nomad are not fully
 automated to have control over the certificate generation process.
 
 TLS is a prerequisite for workload identities with JWT:
 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity
 
 A `nomad` binary on the Packer build host is required to create the initial CA
 certificate (trust anchor):
 ```bash
 # create nomad CA
 mkdir -p tls/nomad && cd tls/nomad
 nomad tls ca create
 ```
 
 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config:
 ```bash
 # https://plugins.jenkins.io/nomad
 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh
 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \
         -alias nomad -storetype pkcs12 -storepass 123456 -noprompt
 ```
 
 Then run the script from the projects root directory to create a new set of
 certificates in the directory `./tls/nomad/certs`:
 ```bash
 ./nomad-tls.sh
 ```
 
 ## ACL's
 tbd
 
 * ACLs for Nomad
 
 ## Nomad workload identity configuration
 
 Follow along the tutorial to configure Nomad workload identities with Bao:
 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl
 
 ```bash
 $ cat vault-jwt-config.json
 {
   "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json",
   "jwt_supported_algs": ["RS256", "EdDSA"],
   "default_role": "nomad-workloads"
 }
 
 # reuse the nomad-agent-ca.pem to configure the jwt auth backend
 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json
 
 $ cat vault-jwt-role.json
 {
   "role_type": "jwt",
   "bound_audiences": ["vault.in0rdr.ch"],
   "user_claim": "/nomad_job_id",
   "user_claim_json_pointer": true,
   "claim_mappings": {
     "nomad_namespace": "nomad_namespace",
     "nomad_job_id": "nomad_job_id",
     "nomad_task": "nomad_task"
   },
   "token_type": "service",
   "token_policies": ["nomad-workloads"],
   "token_period": "30m",
   "token_explicit_max_ttl": 0
 }
 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json
 
 # keep the bao policy a bit simpler (only job level nesting for kv path)
 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad
 $ cat vault-policy-nomad-workloads.hcl
 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" {
   capabilities = ["list", "read"]
 }
 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl
 ```
 
 
 ## Authorized Keys
 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable.
 
 ## Run Packer
 Build [`packer-builder-arm`
 plugin](https://github.com/mkaczanowski/packer-builder-arm) according to
 projects instructions and move it to the working directory:
 ```bash
 git clone git@github.com:mkaczanowski/packer-builder-arm.git arm-builder
 cd arm-builder
 go build
 cp packer-builder-arm .. && cd -
 ```
 
 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu:
 ```bash
 sudo apt-get install qemu-user-static
 ```
 
 For other distributions, a [recent `qemu-aarch64-static` binary can be
 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved
 to the proper location for
 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up:
 ```bash
 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static
 chmod +x qemu-aarch64-static
 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static
 ```
 
 Also, make sure to choose the correct "static" binary for the OS architecture
 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl):
 ```bash
     "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static",
     "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static"
 ```
 
 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package
 `qemu-linux-user`):
 ```bash
 sudo su
 # Remove binfmt_misc entry files before register the entry. When same name's file
 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with
 # an error message "sh: write error: File exists":
 # https://github.com/multiarch/qemu-user-static/blob/master/README.md
 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \;
 # Example from:
 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh
 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin
 
 # Check
 cat /proc/sys/fs/binfmt_misc/qemu-aarch64
 enabled
 interpreter /usr/bin/qemu-aarch64-static
 ...
 ```
 
 Run packer with a value file to build an image for one host:
 ```bash
 sudo /usr/bin/packer build -var-file=variables.auto.pkrvars.hcl -var-file=hosts/pi0.pkrvars.hcl hashi-pi.pkr.hcl
 ```
 
 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables.
 
 ## Write Image to SD Card
 
 To [write the resulting image file to the sd
 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md)
 with `dd`:
 ```bash
 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync
 ```