Readme.md (6856B)
1 # Terraform/Ansible Workflow for Libvirt 2 3 This repository describes a workflow which helps me to (re)create multiple similar VMs for testing purposes. 4 5 ## 1 Preparation 6 7 ### 1.1 Prerequisites 8 ``` 9 systemctl start libvirtd 10 ``` 11 12 Install [`cdrkit`](https://en.wikipedia.org/wiki/Cdrkit) for `mkisofs` (`genisoimage`) dependency: 13 * https://github.com/dmacvicar/terraform-provider-libvirt/commit/c1ed2ab5631e2c4971a4e207cb9e9294693463d3 14 * https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html 15 16 This is required to build the cloud-init image. 17 18 ### 1.1 Environment Varibles 19 To debug Terraform runs, use: 20 ``` 21 #export TF_LOG=DEBUG 22 ``` 23 24 todo: `LIBVIRT_DEFAULT_URI` does not work as expected? 25 26 ## 2 Run Terraform 27 28 Run Terraform in several "stages". 29 30 Update the local cloud init config file `./cloud_init.cfg`: 31 ```bash 32 # prepare cloud-init config file with ssh key 33 terraform apply -target=null_resource.update_cloudinit -auto-approve 34 ``` 35 36 Have a look at the planned resources: 37 ```bash 38 terraform plan 39 ``` 40 41 Create the planned resources: 42 ```bash 43 # create the remaining resources (e.g., the cloud-init image from the config file prepared above) 44 terraform apply -auto-approve 45 ``` 46 47 * If the cloud-init config file is not prepared before starting the other resources, the cloud-init image will not contain the correct ssh public key. 48 * Terraform automatically recreates the Ansible inventory and the mapping of Qemu VM id to hostnames (see next section), whenever one of the hosts is added or removed (i.e., the Terraform `id` is changed). 49 * Terraform writes the SSH private key into the file `./ssh/id_rsa`. 50 51 ### 2.1 Refresh SSH Keys 52 ``` 53 # create an active ssh (or console/vnc/spice) session 54 ssh root@$HOST -i ../ssh/id_rsa 55 56 # create new ssh key in the './terraform' directory on the local machine 57 terraform taint 'tls_private_key.id_rsa' 58 terraform apply -auto-approve 59 60 # update cloud-init image 61 terraform apply -target=libvirt_cloudinit_disk.commoninit -auto-approve 62 63 # reset cloud-init data on the remote $HOST 64 rm -rf /var/lib/cloud/ && poweroff 65 66 # re init cloud data 67 #sudo virsh shutdown $HOSTNAME 68 sudo virsh start $HOSTNAME 69 70 # known-host changed, remove old key 71 ssh-keygen -R $HOST_IP 72 ``` 73 74 Alternatively, use the [Ansible](#4-ansible) playbook and udpated the [`ssh_key`](./ansible/defaults/all.yml) and [`authorized_key`](./ansible/defaults/all.yml) to skip the re-init of cloud data (no reboot). 75 76 ## 3 Terraform Outputs 77 78 If the Ansible inventory or the mapping of Qemu VM id to hostname needs to be updated manually, the values can be re 79 trieved from the Terraform output any time: 80 ``` 81 terraform output -raw inventory > ../ansible/inventory 82 terraform output -raw qemu_config > ../ansible/qemu-config.yml 83 ``` 84 85 Inspect the name of the ssh key file: 86 ``` 87 terraform output ssh_private_keyfile 88 ``` 89 90 ## 4 Ansible 91 92 ### 4.1 Preconditions and Preparations 93 Ansible depends on the following files written by Terraform, see section "2 Run Terraform" and "3 Terraform Outputs": 94 1. `./ansible/inventory`: The Ansible inventory containing a local connection and one group of remote hosts 95 2. `./ansible/qemu-config.yml`: The mapping of Qemu VM ids to hostnames 96 97 Adjust variables in `./ansible/group_vars/all.yml` (use/cp `./ansible/defaults/all.yml` as template): 98 * `ssh_identity_file`: Relative path name to the SSH privat key (output of `terraform output ssh_private_keyfile`) 99 * Set `ssh_proxy_jump` and `ansible_user` if necessary 100 * Define `additional_users` as needed 101 102 ### 4.2 Run Ansible to Build the SSH Config 103 104 The Ansible playbook runst the following tasks: 105 1. Build the `./ssh/ssh_config` based on the information in the file `./ansible/qemu-config.yml` created with Terraform 106 2. Build an optional `./dnsmasq.conf` to enable static ip in dnsmasqs built-in dhcp subsystem 107 3. Add additional users `additional_users` 108 109 The playbook will set the hostname and restart networking inside the VMs, such that the hostnames are published to the DNS server and all hosts are known/addressable by name. 110 111 Run the playbook: 112 ``` 113 # build ssh config from qemu-config.yml 114 ansible-playbook playbook.yml -i inventory -l local 115 116 # if required, build a dnsmasq snippet for static ip allocation (dhcp) 117 # the snippet is written to the file './dnsmasq.conf' 118 ansible-playbook dhcp-static-hosts.yml -i inventory -l local 119 120 # set hostname, restart networking, modify users and keys 121 ansible-playbook playbook.yml -i inventory -l qemu 122 ``` 123 124 If you choose an unprivileged `ansible_user` to reach the VMs, you may need to specfiy a privileged user to run some of the tasks. Specifically, restarting the network and modifying users requires more privileges. Further, you might not want to touch all hosts and limit the execution to a specific host: 125 ``` 126 # set hostname, restart networking, modify users and keys as privileged user on "myhost" 127 ansible-playbook playbook.yml -i inventory -l myhost -e ansible_user=root 128 ``` 129 130 ### 4.3 Update Known Hosts 131 132 To prepare the local host for consecutive SSH connections, you might want to update you local `./ssh/known_hosts` file as follows: 133 ``` 134 # update known hosts locally and confirm 135 ansible-playbook update-known-hosts.yml -i inventory 136 ``` 137 138 Alternatively, use the following commands: 139 ``` 140 cd ansible 141 142 # remove previous hosts from known hosts 143 for ip in $(cat qemu-config.yml | grep ip4 | awk '{print $2}'); do ssh-keygen -R $ip; done 144 145 # update known hosts and confirm 146 for host in $(cat qemu-config.yml | grep fqdn | awk '{print $3}'); do ssh -F ../ssh/config $host exit; done 147 ``` 148 149 ## 5 Troubleshooting, Tips & Tricks 150 151 ### 5.1 Delete and Recreate Hosts 152 You can either taint the Terraform resources or delete them with `virsh`. To taint resource "cka01": 153 ``` 154 terraform taint 'libvirt_domain.host["cka01"]' 155 ``` 156 157 Afterwards, re-apply to restore the Terraform state: 158 ``` 159 terraform apply 160 ``` 161 162 To delete with `virsh` (this will mess up the Terraform state): 163 ``` 164 sudo virsh destroy cka01 165 ``` 166 167 To only recreate parts of the infrastructure, choose an appropriate Terraform `target`: 168 ``` 169 terraform apply -target="libvirt_domain.host[\"cka01\"]" -target="libvirt_domain.host[\"cka02\"]" 170 ``` 171 172 173 ### 5.2 Retrive private key without running Terraform 174 If needed, retrieve the SSH key (again) without re-applying changes: 175 ``` 176 terraform output -raw ssh_private_key > ../ssh/id_rsa 177 ``` 178 179 Terraform takes care of writing this private key file the first time you run `terraform apply`, however, you might want to retrieve the key again without re-running Terraform. 180 181 ### 5.3 Start Libvirt Domains 182 The plan will fail if you don't have the domains started (problem with `qemu-config.yml.tpl`): 183 ``` 184 Call to function "templatefile" failed: 185 ./templates/qemu-config.yml.tpl:6,48-51: Invalid index; The given key does not 186 identify an element in this collection value.. 187 ``` 188 189 --- 190 ## Dependencies 191 * Terraform provider for Libvirt: https://github.com/dmacvicar/terraform-provider-libvirt