hashipi

Raspberry Pi Test Cluster for HashiCorp Vault, Nomad and Consul
git clone https://git.in0rdr.ch/hashipi.git
Log | Files | Refs | Pull requests | README

README.md (7648B)


      1 # HashiPi
      2 
      3 A RaspberryPi test cluster for HashiCorp Nomad and OpenBao.
      4 
      5 ![HashiPi](./img/HashiPi-small.jpg)
      6 
      7 ## Client-Server Architecture
      8 The cluster can be built with 3 or 5 nodes.
      9 
     10 For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep
     11 the Raft traffic minimal.
     12 
     13 Example architecture with 5 RaspberyPi nodes:
     14 
     15 | Node | RAM | Nomad function | Bao function |
     16 |------|-----|----------------|--------------|
     17 | 00   | 4GB | client/server  | server       |
     18 | 01   | 4GB | client         | n/a          |
     19 | 02   | 4GB | client/server  | server       |
     20 | 03   | 4GB | client         | n/a          |
     21 | 04   | 4GB | client/server  | server       |
     22 
     23 For best [performance on low power
     24 devices](https://developer.hashicorp.com/consul/docs/install/performance), the
     25 [`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier)
     26 of Nomad servers is set to the 5.
     27 
     28 The nomad clients have a higher [heartbeat
     29 timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats)
     30 set to make sure the low powered devices are not reported down too often:
     31 
     32 ```
     33 # https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats
     34 # Increase heartbeat ttl under unreliable network conditions to prevent
     35 # client: error heartbeating. retrying: error="failed to update status: rpc
     36 # error: Not ready to serve consistent reads"
     37 heartbeat_grace = "30s"
     38 min_heartbeat_ttl = "30s"
     39 ```
     40 
     41 ## ARM Builder 
     42 These Packer files use the [Packer builder for
     43 ARM](https://github.com/mkaczanowski/packer-builder-arm). Follow the
     44 instructions in the repo to setup the builder and use the provided
     45 [examples](https://github.com/mkaczanowski/packer-builder-arm/tree/master/boards)
     46 to modify to your liking.
     47 
     48 ## Supported Architectures
     49 
     50 ARM >= ARMv7
     51 
     52 ## Self-signed TLS Certificates
     53 ### OpenBao
     54 
     55 To create a new self-signed CA certificate for Bao:
     56 ```bash
     57 # create bao self-signed CA certificate in ./tls/vault/
     58 ./vault-tls.sh
     59 ```
     60 
     61 A new self-signed server certificate is created in the Bao provisioning stage.
     62 
     63 The Packer JSON supports a few arguments for Bao server certificates:
     64 ```json
     65     "vault_tls_ca_cert": "./tls/vault/ca/vault_ca.pem",
     66     "vault_tls_ca_key": "./tls/vault/ca/vault_ca.key",
     67     "vault_tls_subj_alt_name": ""
     68 ```
     69 
     70 * `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host,
     71   e.g., created with `./vault-tls.sh`
     72 * `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g.,
     73   created with `./vault-tls.sh`
     74 * `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names
     75   (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com`
     76 
     77 ### Nomad
     78 The steps to create a set of self-signed certificates for Nomad are not fully
     79 automated to have control over the certificate generation process.
     80 
     81 TLS is a prerequisite for workload identities with JWT:
     82 * https://developer.hashicorp.com/nomad/docs/concepts/workload-identity
     83 
     84 A `nomad` binary on the Packer build host is required to create the initial CA
     85 certificate (trust anchor):
     86 ```bash
     87 # create nomad CA
     88 mkdir -p tls/nomad && cd tls/nomad
     89 nomad tls ca create
     90 ```
     91 
     92 Create Java truststore in pkcs12 format for Jenkins Nomad cloud config:
     93 ```bash
     94 # https://plugins.jenkins.io/nomad
     95 # https://github.com/jenkinsci/nomad-plugin/blob/master/src/test/resources/tls/create_certs.sh
     96 keytool -import -file nomad-agent-ca.pem -keystore nomad-agent-ca.p12 \
     97         -alias nomad -storetype pkcs12 -storepass 123456 -noprompt
     98 ```
     99 
    100 Then run the script from the projects root directory to create a new set of
    101 certificates in the directory `./tls/nomad/certs`:
    102 ```bash
    103 ./nomad-tls.sh
    104 ```
    105 
    106 ## ACL's
    107 tbd
    108 
    109 * ACLs for Nomad
    110 
    111 ## Nomad workload identity configuration
    112 
    113 Follow along the tutorial to configure Nomad workload identities with Bao:
    114 * https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-acl
    115 
    116 ```bash
    117 $ cat vault-jwt-config.json
    118 {
    119   "jwks_url": "https://127.0.0.1:4646/.well-known/jwks.json",
    120   "jwt_supported_algs": ["RS256", "EdDSA"],
    121   "default_role": "nomad-workloads"
    122 }
    123 
    124 # reuse the nomad-agent-ca.pem to configure the jwt auth backend
    125 $ bao write auth/jwt-nomad/config jwks_ca_pem=@tls/nomad/nomad-agent-ca.pem @vault-jwt-config.json
    126 
    127 $ cat vault-jwt-role.json
    128 {
    129   "role_type": "jwt",
    130   "bound_audiences": ["vault.in0rdr.ch"],
    131   "user_claim": "/nomad_job_id",
    132   "user_claim_json_pointer": true,
    133   "claim_mappings": {
    134     "nomad_namespace": "nomad_namespace",
    135     "nomad_job_id": "nomad_job_id",
    136     "nomad_task": "nomad_task"
    137   },
    138   "token_type": "service",
    139   "token_policies": ["nomad-workloads"],
    140   "token_period": "30m",
    141   "token_explicit_max_ttl": 0
    142 }
    143 $ bao write auth/jwt-nomad/role/nomad-workloads @vault-jwt-role.json
    144 
    145 # keep the bao policy a bit simpler (only job level nesting for kv path)
    146 # replace AUTH_METHOD_ACCESSOR with the actual accessor of auth/jwt-nomad
    147 $ cat vault-policy-nomad-workloads.hcl
    148 path "kv/+/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}*" {
    149   capabilities = ["list", "read"]
    150 }
    151 $ bao policy write nomad-workloads vault-policy-nomad-workloads.hcl
    152 ```
    153 
    154 
    155 ## Authorized Keys
    156 Copy the contents of an openssh pubkey to `authorized_keys` Packer variable.
    157 
    158 ## Run Packer
    159 Build [`packer-builder-arm`
    160 plugin](https://github.com/mkaczanowski/packer-builder-arm) according to
    161 projects instructions and move it to the working directory:
    162 ```bash
    163 git clone git@github.com:mkaczanowski/packer-builder-arm.git arm-builder
    164 cd arm-builder
    165 go build
    166 cp packer-builder-arm .. && cd -
    167 ```
    168 
    169 Install the statically linked qemu (emulator) binary. On Debian/Ubuntu:
    170 ```bash
    171 sudo apt-get install qemu-user-static
    172 ```
    173 
    174 For other distributions, a [recent `qemu-aarch64-static` binary can be
    175 downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved
    176 to the proper location for
    177 [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up:
    178 ```bash
    179 curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static
    180 chmod +x qemu-aarch64-static
    181 sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static
    182 ```
    183 
    184 Also, make sure to choose the correct "static" binary for the OS architecture
    185 in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl):
    186 ```bash
    187     "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static",
    188     "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static"
    189 ```
    190 
    191 Configure `binfmt_misc` to use the static binaries (requires OpenSuse package
    192 `qemu-linux-user`):
    193 ```bash
    194 sudo su
    195 # Remove binfmt_misc entry files before register the entry. When same name's file
    196 # /proc/sys/fs/binfmt_misc/qemu-$arch exists, the register command is failed with
    197 # an error message "sh: write error: File exists":
    198 # https://github.com/multiarch/qemu-user-static/blob/master/README.md
    199 find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \;
    200 # Example from:
    201 # https://github.com/multiarch/qemu-user-static/blob/master/containers/latest/register.sh
    202 qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path /usr/bin
    203 
    204 # Check
    205 cat /proc/sys/fs/binfmt_misc/qemu-aarch64
    206 enabled
    207 interpreter /usr/bin/qemu-aarch64-static
    208 ...
    209 ```
    210 
    211 Run packer with a value file to build an image for one host:
    212 ```bash
    213 sudo /usr/bin/packer build -var-file=variables.auto.pkrvars.hcl -var-file=hosts/pi0.pkrvars.hcl hashi-pi.pkr.hcl
    214 ```
    215 
    216 The `variable.auto.pkrvars.hcl` contains all sensitive packer variables.
    217 
    218 ## Write Image to SD Card
    219 
    220 To [write the resulting image file to the sd
    221 card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md)
    222 with `dd`:
    223 ```bash
    224 sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync
    225 ```