commit 4c81785e284aec46e8062009113aa634057385b2
parent 3e0fe47847031156866721df1f33fbc442a01d3d
Author: Andreas Gruhler <andreas.gruhler@adfinis.com>
Date: Sun, 21 Jul 2024 17:45:02 +0200
feat: remove consul
Diffstat:
12 files changed, 52 insertions(+), 279 deletions(-)
diff --git a/README.md b/README.md
@@ -1,33 +1,33 @@
# HashiPi
-A RaspberryPi test cluster for HashiCorp Nomad with Consul and OpenBao.
+A RaspberryPi test cluster for HashiCorp Nomad and OpenBao.
## Client-Server Architecture
The cluster can be built with 3 or 5 nodes.
-For 5 nodes, it is recommended to only have 3 server nodes for Nomad and Consul to keep the Raft traffic minimal.
+For 5 nodes, it is recommended to only have 3 server nodes for Nomad to keep
+the Raft traffic minimal.
Example architecture with 5 RaspberyPi nodes:
-| Node | RAM | Nomad function | Consul function | Bao function |
-|------|-----|----------------|-----------------|--------------|
-| 00 | 4GB | client/server | client/server | server |
-| 01 | 4GB | client | client | n/a |
-| 02 | 4GB | client/server | client/server | server |
-| 03 | 4GB | client | client | n/a |
-| 04 | 4GB | client/server | client/server | server |
+| Node | RAM | Nomad function | Bao function |
+|------|-----|----------------|--------------|
+| 00 | 4GB | client/server | server |
+| 01 | 4GB | client | n/a |
+| 02 | 4GB | client/server | server |
+| 03 | 4GB | client | n/a |
+| 04 | 4GB | client/server | server |
-For best performance on low power devices, the [`raft_multiplier`](https://developer.hashicorp.com/consul/docs/install/performance) is set to the default value 5 (or higher):
+For best [performance on low power
+devices](https://developer.hashicorp.com/consul/docs/install/performance), the
+[`raft_multiplier`](https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier)
+of Nomad servers is set to the 5.
-```
-performance {
- raft_multiplier = 5
-}
-```
-
-The nomad clients have a higher [heartbeat timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats) set to make sure the low powered devices are not reported down too often:
+The nomad clients have a higher [heartbeat
+timeout](https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats)
+set to make sure the low powered devices are not reported down too often:
```
# https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats
@@ -39,31 +39,17 @@ min_heartbeat_ttl = "30s"
```
## ARM Builder
-These Packer files use the [Packer builder for ARM](https://github.com/mkaczanowski/packer-builder-arm). Follow the instructions in the repo to setup the builder and use the provided [examples](https://github.com/mkaczanowski/packer-builder-arm/tree/master/boards) to modify to your liking.
-
+These Packer files use the [Packer builder for
+ARM](https://github.com/mkaczanowski/packer-builder-arm). Follow the
+instructions in the repo to setup the builder and use the provided
+[examples](https://github.com/mkaczanowski/packer-builder-arm/tree/master/boards)
+to modify to your liking.
## Supported Architectures
ARM >= ARMv7
## Self-signed TLS Certificates
-### Consul
-The steps to create a set of self-signed certificates for Consul are not fully automated to have control over the certificate generation process. It would be a minor modification to add the necessary Packer provisioning steps to the JSON file.
-
-A `consul` on the Packer build host is required to create the initial CA certificate (trust anchor):
-```bash
-# create consul CA
-mkdir -p tls/consul && cd tls/consul
-consul tls ca create
-```
-
-Then run the script from the projects root directory to create a new set of certificates in the directory `./tls/consul/certs`:
-```bash
-./consul-tls.sh
-```
-
-The script can be run after each flashed Raspberry Pi image to create a fresh set of certificates for "server", "client" and "cli" usage.
-
### OpenBao
To create a new self-signed CA certificate for Bao:
@@ -81,32 +67,26 @@ The Packer JSON supports a few arguments for Bao server certificates:
"vault_tls_subj_alt_name": ""
```
-* `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host, e.g., created with `./vault-tls.sh`
-* `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g., created with `./vault-tls.sh`
-* `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com`
-
+* `vault_tls_ca_cert`: The path of the CA certificate on the Packer build host,
+ e.g., created with `./vault-tls.sh`
+* `vault_tls_ca_key`: The path of the CA key on the Packer build host, e.g.,
+ created with `./vault-tls.sh`
+* `vault_tls_subj_alt_name`: Comma seperated list of Subject Alternative Names
+ (SAN) for the self-signed certificates, e.g., `DNS:vault.example.com`
## ACL's
-
tbd
-* ACLs for Consul
* ACLs for Nomad
* TLS and Gossip encryption for Nomad
-## Consul Encryption
-
-Generate a new consul encryption key and set as `consul_encrypt` Packer variable:
-```bash
-consul keygen
-```
-
## Authorized Keys
Copy the contents of an openssh pubkey to `authorized_keys` Packer variable.
## Run Packer
-
-Build [`packer-builder-arm` plugin](https://github.com/mkaczanowski/packer-builder-arm) according to projects instructions and move it to the working directory:
+Build [`packer-builder-arm`
+plugin](https://github.com/mkaczanowski/packer-builder-arm) according to
+projects instructions and move it to the working directory:
```bash
git clone git@github.com:mkaczanowski/packer-builder-arm.git arm-builder
cd arm-builder
@@ -119,20 +99,25 @@ Install the statically linked qemu (emulator) binary. On Debian/Ubuntu:
sudo apt-get install qemu-user-static
```
-For other distributions, a [recent `qemu-aarch64-static` binary can be downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved to the proper location for [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up:
+For other distributions, a [recent `qemu-aarch64-static` binary can be
+downloaded](https://github.com/multiarch/qemu-user-static/releases) and moved
+to the proper location for
+[`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) to pick it up:
```bash
curl -LO https://github.com/multiarch/qemu-user-static/releases/download/v7.2.0-1/qemu-aarch64-static
chmod +x qemu-aarch64-static
sudo mv qemu-aarch64-static /usr/bin/qemu-aarch64-static
```
-Also, make sure to choose the correct "static" binary for the OS architecture in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl):
+Also, make sure to choose the correct "static" binary for the OS architecture
+in [`hashi-pi.pkr.hcl`](./hashi-pi.pkr.hcl):
```bash
"qemu_binary_source_path": "/usr/bin/qemu-aarch64-static",
"qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static"
```
-Configure `binfmt_misc` to use the static binaries (requires OpenSuse package `qemu-linux-user`):
+Configure `binfmt_misc` to use the static binaries (requires OpenSuse package
+`qemu-linux-user`):
```bash
sudo su
# Remove binfmt_misc entry files before register the entry. When same name's file
@@ -151,19 +136,18 @@ interpreter /usr/bin/qemu-aarch64-static
...
```
-Don't forget to create a new set of [Consul TLS certificates](#Consul) before
-each run. Run packer with a value file to build an image for one host:
+Run packer with a value file to build an image for one host:
```bash
-./consul-tls.sh && sudo /usr/bin/packer build -var-file=variables.auto.pkrvars.hcl -var-file=hosts/pi0.pkrvars.hcl hashi-pi.pkr.hcl
+sudo /usr/bin/packer build -var-file=variables.auto.pkrvars.hcl -var-file=hosts/pi0.pkrvars.hcl hashi-pi.pkr.hcl
```
The `variable.auto.pkrvars.hcl` contains all sensitive packer variables.
## Write Image to SD Card
-To [write the resulting image file to the sd card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md) with `dd`:
-
+To [write the resulting image file to the sd
+card](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md)
+with `dd`:
```bash
sudo dd bs=4M if=HashiPi-pi0.img of=/dev/sdb status=progress conv=fsync
```
-
diff --git a/bootstrap.sh b/bootstrap.sh
@@ -34,7 +34,7 @@ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://
# Install packages
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y jq podman cloud-init \
- "consul=${CONSUL_VERSION}-1" "nomad=${NOMAD_VERSION}-1"
+ "nomad=${NOMAD_VERSION}-1"
curl -L -o openbao.deb "https://github.com/openbao/openbao/releases/download/v${BAO_VERSION}/bao_${BAO_VERSION}_linux_arm64.deb"
apt install ./openbao.deb
diff --git a/consul-tls.sh b/consul-tls.sh
@@ -1,45 +0,0 @@
-#!/usr/bin/env bash
-#
-# Creates a set of certificates for use with HashiCorp Consul
-# https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide
-
-# set -o errexit
-# set -o nounset
-# set -o xtrace
-
-# https://www.shellhacks.com/yes-no-bash-script-prompt-confirmation/
-read -p "Do you want to generate a new set of Consul certicates in the directory \"./tls/consul/\" [y/N]? " -n 1 -r
-echo # (optional) move to a new line
-if [[ ! $REPLY =~ ^[Yy]$ ]]
-then
- exit 1
-fi
-
-# Set working dir
-CONSUL_TLS_BASE_PATH="${CONSUL_TLS_BASE_PATH:-./tls/consul/}"
-mkdir -p "$CONSUL_TLS_BASE_PATH"
-cd "$CONSUL_TLS_BASE_PATH"
-
-# Cleanup previously generated certificates
-rm -rf certs
-
-# Define cert types
-crt_types=("server" "cli" "client")
-
-for type in "${crt_types[@]}"; do
- # Create certificate
- consul tls cert create -${type}
- cert="dc1-$type-consul-0.pem"
- key="dc1-$type-consul-0-key.pem"
-
- # Show fingerprint
- openssl x509 -in $cert -fingerprint -noout
-
- # Drop index
- mv $cert "dc1-${type}-consul.pem"
- mv $key "dc1-${type}-consul-key.pem"
-done;
-
-# Move to certs folder
-mkdir certs
-mv dc1-* certs/
-\ No newline at end of file
diff --git a/consul.sh b/consul.sh
@@ -1,98 +0,0 @@
-#!/usr/bin/env bash
-#
-# Packer shell provisioner for HashiCorp Consul on Raspberry Pi
-# https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide
-
-# set -o errexit
-# set -o nounset
-set -o xtrace
-
-cd "/home/${USERNAME}"
-
-# Move uploaded tls files
-mkdir -p /opt/consul/tls
-mv /tmp/tls/* /opt/consul/tls/
-
-# Change ownership and permissions for tls certs
-chown consul: /opt/consul/tls/*.pem
-chmod 640 /opt/consul/tls/*.pem
-chmod 644 /opt/consul/tls/dc1-{cli,client}*
-chmod 644 /opt/consul/tls/consul-agent-ca.pem
-
-# Create Consul config files
-mkdir -p /etc/consul.d
-rm -rf /etc/consul.d/*
-
-cat << EOF > /etc/consul.d/consul.hcl
-datacenter = "dc1"
-data_dir = "/opt/consul"
-encrypt = "$CONSUL_ENCRYPT"
-
-retry_join = [
- $(printf '%s\n' $CONSUL_RETRY_JOIN)
-]
-
-ports {
- server = 8300
- serf_lan = 8301
- serf_wan = -1
- http = -1
- https = 8501
- dns = 8600
-}
-
-addresses {
- dns = "{{GetPrivateInterfaces | exclude \"type\" \"IPv6\" | include \"name\" \"eth0\" | attr \"address\" }}"
-}
-
-# The address that should be bound to for internal cluster communications.
-# If there are multiple private IPv4 addresses available, Consul will exit with an error at startup:
-# - https://www.consul.io/docs/troubleshoot/common-errors
-# - https://www.consul.io/docs/agent/options
-bind_addr = "{{GetPrivateInterfaces | exclude \"type\" \"IPv6\" | include \"name\" \"eth0\" | attr \"address\" }}"
-
-performance {
- raft_multiplier = 5
-}
-
-ui_config {
- enabled = true
-}
-
-tls {
- defaults {
- verify_incoming = false
- verify_outgoing = true
- ca_file = "/opt/consul/tls/consul-agent-ca.pem"
- cert_file = "/opt/consul/tls/dc1-server-consul.pem"
- key_file = "/opt/consul/tls/dc1-server-consul-key.pem"
- }
-
- # Auto-encrypt RPC
- # "verify rpc only", because ui enabled
- internal_rpc {
- verify_incoming = true
- verify_server_hostname = true
- }
-}
-EOF
-
-systemctl enable consul
-
-# Configure .bashrc
-cat << EOF >> .bashrc
-
-complete -C /usr/bin/consul consul
-
-export CONSUL_HTTP_ADDR="https://127.0.0.1:8501"
-export CONSUL_CACERT="/opt/consul/tls/consul-agent-ca.pem"
-export CONSUL_CLIENT_CERT="/opt/consul/tls/dc1-cli-consul.pem"
-export CONSUL_CLIENT_KEY="/opt/consul/tls/dc1-cli-consul-key.pem"
-EOF
-
-if [[ "$CONSUL_SERVER" = true ]]; then
-cat << EOF > /etc/consul.d/server.hcl
-server = true
-bootstrap_expect = 3
-EOF
-fi # endif CONSUL_SERVER
diff --git a/hashi-pi.pkr.hcl b/hashi-pi.pkr.hcl
@@ -3,37 +3,6 @@ variable "authorized_keys" {
default = ""
}
-variable "consul_encrypt" {
- type = string
- default = ""
- sensitive = true
-}
-
-variable "consul_retry_join" {
- type = string
- default = "\"pi0.lan\", \"pi1.lan\", \"pi2.lan\", \"pi3.lan\", \"pi4.lan\""
-}
-
-variable "consul_server" {
- type = string
- default = "true"
-}
-
-variable "consul_tls_ca" {
- type = string
- default = "./tls/consul/consul-agent-ca.pem"
-}
-
-variable "consul_tls_certs" {
- type = string
- default = "./tls/consul/certs/"
-}
-
-variable "consul_version" {
- type = string
- default = "1.18.2"
-}
-
variable "flash_device_path" {
type = string
default = "/dev/sda"
@@ -197,7 +166,6 @@ build {
"USERNAME=${var.username}",
"AUTHORIZED_KEYS=${var.authorized_keys}",
"NOMAD_VERSION=${var.nomad_version}",
- "CONSUL_VERSION=${var.consul_version}",
"BAO_VERSION=${var.bao_version}"
]
}
@@ -206,27 +174,6 @@ build {
inline = ["mkdir /tmp/tls"]
}
- provisioner "file" {
- destination = "/tmp/tls/consul-agent-ca.pem"
- source = "${var.consul_tls_ca}"
- }
-
- provisioner "file" {
- destination = "/tmp/tls/"
- source = "${var.consul_tls_certs}"
- }
-
- provisioner "shell" {
- script = "consul.sh"
- remote_folder = "/home/${var.username}"
- environment_vars = [
- "USERNAME=${var.username}",
- "CONSUL_SERVER=${var.consul_server}",
- "CONSUL_ENCRYPT=${var.consul_encrypt}",
- "CONSUL_RETRY_JOIN=${var.consul_retry_join}"
- ]
- }
-
provisioner "shell" {
script = "nomad.sh"
remote_folder = "/home/${var.username}"
diff --git a/hosts/pi0.pkrvars.hcl b/hosts/pi0.pkrvars.hcl
@@ -1,5 +1,4 @@
hostname = "pi0"
img_name = "HashiPi-pi0.img"
-consul_server = true
nomad_server = true
nomad_client = true
diff --git a/hosts/pi1.pkrvars.hcl b/hosts/pi1.pkrvars.hcl
@@ -1,5 +1,4 @@
hostname = "pi1"
img_name = "HashiPi-pi1.img"
-consul_server = false
nomad_server = false
nomad_client = true
diff --git a/hosts/pi2.pkrvars.hcl b/hosts/pi2.pkrvars.hcl
@@ -1,5 +1,4 @@
hostname = "pi2"
img_name = "HashiPi-pi2.img"
-consul_server = true
nomad_server = true
nomad_client = true
diff --git a/hosts/pi3.pkrvars.hcl b/hosts/pi3.pkrvars.hcl
@@ -1,5 +1,4 @@
hostname = "pi3"
img_name = "HashiPi-pi3.img"
-consul_server = false
nomad_server = false
nomad_client = true
diff --git a/hosts/pi4.pkrvars.hcl b/hosts/pi4.pkrvars.hcl
@@ -1,5 +1,4 @@
hostname = "pi4"
img_name = "HashiPi-pi4.img"
-consul_server = true
nomad_server = true
nomad_client = true
diff --git a/nomad.sh b/nomad.sh
@@ -27,14 +27,6 @@ cat << EOF > /etc/nomad.d/nomad.hcl
datacenter = "dc1"
data_dir = "/opt/nomad"
-consul {
- address = "127.0.0.1:8501"
- ssl = true
- ca_file = "/opt/consul/tls/consul-agent-ca.pem"
- cert_file = "/opt/consul/tls/dc1-server-consul.pem"
- key_file = "/opt/consul/tls/dc1-server-consul-key.pem"
-}
-
vault {
# https://www.nomadproject.io/docs/configuration/vault
# Nomad servers require a Vault Token set in /etc/nomad.d/nomad.env
@@ -67,6 +59,9 @@ cat << EOF > /etc/nomad.d/server.hcl
server {
enabled = true
bootstrap_expect = 3
+ # https://developer.hashicorp.com/nomad/docs/configuration/server#raft_multiplier
+ # https://developer.hashicorp.com/consul/docs/install/performance
+ raft_multiplier = 5
# https://developer.hashicorp.com/nomad/docs/configuration/server#client-heartbeats
# Increase heartbeat ttl under unreliable network conditions to prevent
# client: error heartbeating. retrying: error="failed to update status: rpc
@@ -169,8 +164,8 @@ ip6table_raw
ip6table_filter
EOF
-# Install CNI plugin for Consul connect integration for Consul snapshot tasks
-# - https://www.nomadproject.io/docs/integrations/consul-connect
+# Install CNI plugins
+# - https://developer.hashicorp.com/nomad/docs/networking/cni
curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.5.0/cni-plugins-linux-arm64-v1.5.0.tgz"
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -xzf cni-plugins.tgz
diff --git a/openbao.sh b/openbao.sh
@@ -66,12 +66,8 @@ listener "tcp" {
# https://openbao.org/docs/configuration/#high-availability-parameters
# https://openbao.org/docs/concepts/ha#client-redirection
#
-# This uses a go-sockaddr template to fetch the actual ip for an interface.
-# This is the address registered in the Consul service. If this is not an ip
-# here, requests to the Consul DNS will also not return an A record (only
-# CNAME). This leads to problems with Haproxy server-template.
-#
-# API_ADDR for client redirection (fallback, if request forwarding is disabled)
+# API_ADDR for client redirection (fallback, if request forwarding is
+# disabled). Uses go-sockaddr template to fetch the actual ip for an interface
api_addr = "https://{{GetPrivateInterfaces | exclude \"type\" \"IPv6\" | include \"name\" \"eth0\" | attr \"address\" }}:8200"
# CLUSTER_ADDR: Vault listens for server-to-server cluster requests
cluster_addr = "https://{{GetPrivateInterfaces | exclude \"type\" \"IPv6\" | include \"name\" \"eth0\" | attr \"address\" }}:8201"
