commit 0b1244e79c849586c77a522adaca663465fdb2f8 parent d3cb739c09a49c126db0c19cb4efb1a4bfff49d8 Author: Andreas Gruhler <andreas.gruhler@adfinis-sygroup.ch> Date: Fri, 20 Mar 2020 12:42:40 +0100 add libvirt workflow Diffstat:
38 files changed, 634 insertions(+), 152 deletions(-)
diff --git a/Readme.md b/Readme.md @@ -1,154 +1,3 @@ -# Terraform/Ansible Workflow on Proxmox VE (PVE) +# Terraform/Ansible Workflow for Proxmox VE (PVE) and Libvirt This repository describes a workflow which helps me to (re)create multiple similar VMs for testing purposes. - -## 1 Preparation -### 1.1 PVE Varibles -Define authentication variables for the [PVE API](https://github.com/Telmate/proxmox-api-go). The Terraform provider relies on the PVE API which requires you to defined the following environment variables: -``` -export PM_USER=test@pve -export PM_PASS="secret" -export PM_API_URL="https://pve.cloud.org/api2/json" -#export TF_LOG=DEBUG -``` - -### 1.2 Unicast MAC Addresses and Terraform Variables -Generate a unicast MAC foreach VM. For instance, use a bash script to do so: -``` -#!/bin/sh -# source: https://serverfault.com/a/299563 -function macaddr() { - echo $RANDOM | md5sum | sed 's/^\(..\)\(..\)\(..\)\(..\)\(..\).*$/02:\1:\2:\3:\4:\5/' -} -``` - -Use environment variables or create a new file `./terraform/terraform.tfvars` to specify the details for the VMs (see also `./terraform/variables.tf`). Among other variables, insert the mac addresses from above: -``` -# terraform.tfvars -hosts = ["host0"] -macaddr = { - host0 = "02:a2:1d:38:31:1c" -} -``` - -## 2 Run Terraform - -Run Terraform: -``` -terraform plan -terraform apply -``` - -* Terraform automatically recreates the Ansible inventory and the mapping of Qemu VM id to hostnames (see next section), whenever one of the hosts is added or removed (i.e., the Terraform `id` is changed). -* Terraform writes the SSH private key into the file `./ssh/id_rsa`. - -## 3 Terraform Outputs - -If the Ansible inventory or the mapping of Qemu VM id to hostname needs to be updated manually, the values can be retrieved from the Terraform output any time: -``` -terraform output inventory > ../ansible/inventory -terraform output qemu_config > ../ansible/qemu-config.yml - -# inspect the name of the key file, see instructions below -terraform output ssh_private_keyfile -``` - -## 4 Ansible - -### 4.1 Preconditions and Preparations -Ansible depends on the following files written by Terraform, see section "2 Run Terraform" and "3 Terraform Outputs": -1. `./ansible/inventory`: The Ansible inventory containing a local connection and one group of remote hosts -2. `./ansible/qemu-config.yml`: The mapping of Qemu VM ids to hostnames - -Adjust variables in `./ansible/group_vars/all.yml`: -* `ssh_identity_file`: Relative path name to the SSH privat key (output of `terraform output ssh_private_keyfile`) -* Set `ssh_proxy_jump` and `ansible_user` if necessary -* Ensure `pve_api` points to your compiled PVE API binary -* Define `additional_users` as needed - -### 4.2 Run Ansible to Build the SSH Config - -The Ansible playbook runst the following tasks: -1. Retrieve the IP of the VMs via Qemu guest agent -2. Write the IP to the file `./ansible/qemu-config.yml` -3. Build the `./ssh/ssh_config` based on the information in the previous step -4. Build an optional `./dnsmasq.conf` to enable static ip in dnsmasqs built-in dhcp subsystem -5. Add additional users `additional_users` - -It is necessary to run ansible, because the IP address of the hosts cannot be retrieved by Terraform (the PVE provider is not mature enough yet). Therefore, we need to retrieve the IP addresses of the hosts via the Qemu guest agents running in the VMs. This process is automated and it will amend the IPs to the file `./ansible/qemu-config.yml`. Furthermore, the playbook will set the hostname and restart networking inside the VMs, such that the hostnames are published to the DNS server and all hosts are known/addressable by name. - -Run the playbook: -``` -# build ssh config from qemu-config.yml -ansible-playbook playbook.yml -i inventory -l local -# if required, build a dnsmasq snippet for static ip allocation (dhcp) -# the snippet is written to the file './dnsmasq.conf' -ansible-playbook dhcp-static-hosts.yml -i inventory -l local -# set hostname, restart networking, modify users and keys -ansible-playbook playbook.yml -i inventory -l qemu -``` - -If you choose an unprivileged `ansible_user` to reach the VMs, you may need to specfiy a privileged user to run some of the tasks. Specifically, restarting the network and modifying users requires more privileges. Further, you might not want to touch all hosts and limit the execution to a specific host: -``` -# set hostname, restart networking, modify users and keys as privileged user on "myhost" -ansible-playbook playbook.yml -i inventory -l myhost -e ansible_user=root -``` - -### 4.3 Update Known Hosts - -To prepare the local host for consecutive SSH connections, you might want to update you local `./ssh/known_hosts` file as follows: -``` -# update known hosts locally and confirm -ansible-playbook update-known-hosts.yml -i inventory -``` - -Alternatively, use the following commands: -``` -cd ansible - -# remove previous hosts from known hosts -for ip in $(cat qemu-config.yml | grep ip4 | awk '{print $2}'); do ssh-keygen -R $ip; done - -# update known hosts and confirm -for host in $(cat qemu-config.yml | grep fqdn | awk '{print $3}'); do ssh -F ../ssh/config $host exit; done -``` - -## 5 Troubleshooting, Tips & Tricks - -### 5.1 Delete and Recreate Hosts -You can either taint the Terraform resources or delete them via PVE API. To taint resources: -``` -terraform taint proxmox_vm_qemu.host[\"mysql0\"] -Resource instance proxmox_vm_qemu.host["mysql0"] has been marked as tainted. -``` - -Afterwards, re-apply to restore the Terraform state: -``` -terraform apply -``` - -To delete some of the the infrastructure directly via API (this might confuse your Terraform state): -``` -~/gocode/src/github.com/Telmate/proxmox-api-go/proxmox-api-go destroy 116 -terraform refresh -``` - -To only recreate parts of the infrastructure, choose an appropriate Terraform `target`: -``` -terraform apply -target="proxmox_vm_qemu.host[\"mysql0\"]" -target="proxmox_vm_qemu.host[\"mysql1\"]" -``` - - -### 5.2 Retrive private key without running Terraform -If needed, retrieve the SSH key (again) without re-applying changes: -``` -terraform output ssh_private_key > ../ssh/id_rsa -``` - -Terraform takes care of writing this private key file the first time you run `terraform apply`, however, you might want to retrieve the key again without re-running Terraform. - ---- -## Dependencies -* PVE API: https://github.com/Telmate/proxmox-api-go -* Terraform provider for Proxmox: https://github.com/Telmate/terraform-provider-proxmox - diff --git a/libvirt/Readme.md b/libvirt/Readme.md @@ -0,0 +1,186 @@ +# Terraform/Ansible Workflow for Libvirt + +This repository describes a workflow which helps me to (re)create multiple similar VMs for testing purposes. + +## 1 Preparation + +### 1.1 Prerequisites +``` +systemc start libvirtd +``` + +Install [`cdrkit`](https://en.wikipedia.org/wiki/Cdrkit) for `mkisofs` (`genisoimage`) dependency: +* https://github.com/dmacvicar/terraform-provider-libvirt/commit/c1ed2ab5631e2c4971a4e207cb9e9294693463d3 +* https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html + +This is required to build the cloud-init image. + +### 1.1 Environment Varibles +To debug Terraform runs, use: +``` +#export TF_LOG=DEBUG +``` + +todo: `LIBVIRT_DEFAULT_URI` does not work as expected? + +## 2 Run Terraform + +Plan: +``` +terraform plan +``` + +Run Terraform in two "stages": +``` +# prepare cloud-init config file with ssh key +terraform apply -target=null_resource.update_cloudinit -auto-approve + +# create the remaining resources (e.g., the cloud-init image from the config file prepared above) +terraform apply -auto-approve +``` + +* If the cloud-init config file is not prepared before starting the other resources, the cloud-init image will not contain the correct ssh public key. +* Terraform automatically recreates the Ansible inventory and the mapping of Qemu VM id to hostnames (see next section), whenever one of the hosts is added or removed (i.e., the Terraform `id` is changed). +* Terraform writes the SSH private key into the file `./ssh/id_rsa`. + +### 2.1 Refresh SSH Keys +``` +# create an active ssh (or console/vnc/spice) session +ssh root@$HOST -i ../ssh/id_rsa + +# create new ssh key in the './terraform' directory on the local machine +terraform taint 'tls_private_key.id_rsa' +terraform apply -auto-approve + +# update cloud-init image +terraform apply -target=libvirt_cloudinit_disk.commoninit -auto-approve + +# reset cloud-init data on the remote $HOST +rm -rf /var/lib/cloud/ && poweroff + +# re init cloud data +#sudo virsh shutdown $HOSTNAME +sudo virsh start $HOSTNAME + +# known-host changed, remove old key +ssh-keygen -R $HOST_IP +``` + +Alternatively, use the [Ansible](#4-ansible) playbook and udpated the [`ssh_key`](./ansible/defaults/all.yml) and [`authorized_key`](./ansible/defaults/all.yml) to skip the re-init of cloud data (no reboot). + +## 3 Terraform Outputs + +If the Ansible inventory or the mapping of Qemu VM id to hostname needs to be updated manually, the values can be re +trieved from the Terraform output any time: +``` +terraform output inventory > ../ansible/inventory +terraform output qemu_config > ../ansible/qemu-config.yml +``` + +Inspect the name of the ssh key file: +``` +terraform output ssh_private_keyfile +``` + +## 4 Ansible + +### 4.1 Preconditions and Preparations +Ansible depends on the following files written by Terraform, see section "2 Run Terraform" and "3 Terraform Outputs": +1. `./ansible/inventory`: The Ansible inventory containing a local connection and one group of remote hosts +2. `./ansible/qemu-config.yml`: The mapping of Qemu VM ids to hostnames + +Adjust variables in `./ansible/group_vars/all.yml` (use/cp `./ansible/defaults/all.yml` as template): +* `ssh_identity_file`: Relative path name to the SSH privat key (output of `terraform output ssh_private_keyfile`) +* Set `ssh_proxy_jump` and `ansible_user` if necessary +* Define `additional_users` as needed + +### 4.2 Run Ansible to Build the SSH Config + +The Ansible playbook runst the following tasks: +1. Build the `./ssh/ssh_config` based on the information in the file `./ansible/qemu-config.yml` created with Terraform +2. Build an optional `./dnsmasq.conf` to enable static ip in dnsmasqs built-in dhcp subsystem +3. Add additional users `additional_users` + +The playbook will set the hostname and restart networking inside the VMs, such that the hostnames are published to the DNS server and all hosts are known/addressable by name. + +Run the playbook: +``` +# build ssh config from qemu-config.yml +ansible-playbook playbook.yml -i inventory -l local + +# if required, build a dnsmasq snippet for static ip allocation (dhcp) +# the snippet is written to the file './dnsmasq.conf' +ansible-playbook dhcp-static-hosts.yml -i inventory -l local + +# set hostname, restart networking, modify users and keys +ansible-playbook playbook.yml -i inventory -l qemu +``` + +If you choose an unprivileged `ansible_user` to reach the VMs, you may need to specfiy a privileged user to run some of the tasks. Specifically, restarting the network and modifying users requires more privileges. Further, you might not want to touch all hosts and limit the execution to a specific host: +``` +# set hostname, restart networking, modify users and keys as privileged user on "myhost" +ansible-playbook playbook.yml -i inventory -l myhost -e ansible_user=root +``` + +### 4.3 Update Known Hosts + +To prepare the local host for consecutive SSH connections, you might want to update you local `./ssh/known_hosts` file as follows: +``` +# update known hosts locally and confirm +ansible-playbook update-known-hosts.yml -i inventory +``` + +Alternatively, use the following commands: +``` +cd ansible + +# remove previous hosts from known hosts +for ip in $(cat qemu-config.yml | grep ip4 | awk '{print $2}'); do ssh-keygen -R $ip; done + +# update known hosts and confirm +for host in $(cat qemu-config.yml | grep fqdn | awk '{print $3}'); do ssh -F ../ssh/config $host exit; done +``` + +## 5 Troubleshooting, Tips & Tricks + +### 5.1 Delete and Recreate Hosts +You can either taint the Terraform resources or delete them with `virsh`. To taint resource "cka01": +``` +terraform taint 'libvirt_domain.host["cka01"]' +``` + +Afterwards, re-apply to restore the Terraform state: +``` +terraform apply +``` + +To delete with `virsh` (this will mess up the Terraform state): +``` +sudo virsh destroy cka01 +``` + +To only recreate parts of the infrastructure, choose an appropriate Terraform `target`: +``` +terraform apply -target="libvirt_domain.host[\"cka01\"]" -target="libvirt_domain.host[\"cka02\"]" +``` + + +### 5.2 Retrive private key without running Terraform +If needed, retrieve the SSH key (again) without re-applying changes: +``` +terraform output ssh_private_key > ../ssh/id_rsa +``` + +Terraform takes care of writing this private key file the first time you run `terraform apply`, however, you might want to retrieve the key again without re-running Terraform. + +### 5.3 Start Libvirt Domains +The plan will fail if you don't have the domains started (problem with `qemu-config.yml.tpl`): +``` +Call to function "templatefile" failed: +./templates/qemu-config.yml.tpl:6,48-51: Invalid index; The given key does not +identify an element in this collection value.. +``` + +--- +## Dependencies +* Terraform provider for Libvirt: https://github.com/dmacvicar/terraform-provider-libvirt +\ No newline at end of file diff --git a/libvirt/ansible/defaults/all.yml b/libvirt/ansible/defaults/all.yml @@ -0,0 +1,35 @@ +--- + +# ssh configuraiton to reach the VMs +# this is only an example, each line +# needs to enabled explicitly in group_vars +ansible_user: root +ssh_identity_file: '../ssh/id_rsa' +ssh_proxy_jump: proxyhost +ssh_include_config: '~/.ssh/config' + +# allow sudo/wheel users to execute any command without password +ssh_passwordless_login: no +# example of adding additional users +# additional_users: +# - name: user1 +# # comma seperated list of additional groups +# additional_groups: 'wheel' +# # mutually exclusive with reuse_ssh_key below +# generate_ssh_key: yes +# # mutually exclusive with generate_ssh_key above +# # lets you reuse an existing ssh key +# #ssh_key: '{{ ssh_identity_file }}' +# # adds this key as authorized key +# #authorized_key: '~/.ssh/id_rsa.pub' + +# The cloud-init config template has manage_etc_hosts enabled by default. +# This will overwrite the state of /etc/hosts at each reboot. The default +# cloud-init configuratin resolves {{ ansible_hostname }} to 127.0.0.1. +# This is not desirable in all cases, since sometimes you want it to resolve +# the public IP (not localhost). This option will disable the line that +# resolves {{ ansible_hostname }} to 127.0.0.1 in all relevant files. The file +# /etc/hosts will still be overwritten at each reboot, but without the line resolving +# {{ ansible_hostname }} to 127.0.0.1. Choose 'yes' to enable this temporary fix +# (better configure this at an earlier stage, with Terraform and cloud-init). +cloud_init_disable_localhost_resolver: no diff --git a/ansible/dhcp-static-hosts.yml b/libvirt/ansible/dhcp-static-hosts.yml diff --git a/libvirt/ansible/playbook.yml b/libvirt/ansible/playbook.yml @@ -0,0 +1,128 @@ +--- + +# Local tasks to generate ssh config +# Input/requires: './qemu-config.yml' +- hosts: local + vars: + qemu_config: "{{ lookup('file', 'qemu-config.yml') | from_yaml }}" + tasks: + - name: create ssh config + template: + src: 'templates/config.j2' + dest: '../ssh/config' + +# remote tasks to set hostname, add users and keys +- hosts: qemu + tasks: + - name: include os specific vars + include_vars: '{{ item }}' + with_first_found: + - '{{ ansible_distribution }}_{{ ansible_distribution_major_version }}.yml' + - '{{ ansible_os_family }}.yml' + + - name: remote user information message + debug: + msg: 'Running tasks on remote host as user "{{ ansible_user }}"' + + - name: set hostname + command: 'hostnamectl set-hostname {{ inventory_hostname }}' + register: hostname_update + become: yes + + - name: restart network to register hostname with dns server + service: + name: network + state: restarted + when: hostname_update.changed + ignore_errors: yes + become: yes + + - name: restart NetworkManager to register hostname with dns server + service: + name: NetworkManager + state: restarted + when: hostname_update.changed + ignore_errors: yes + become: yes + + - name: set ssh private key + copy: + src: '{{ ssh_identity_file }}' + dest: '{{ ansible_env.HOME }}/.ssh/id_rsa' + owner: '{{ ansible_user }}' + group: '{{ ansible_user }}' + mode: '0600' + + - block: + - name: add additional users + user: + name: '{{ item.name }}' + shell: /bin/bash + groups: '{{ item.additional_groups }}' + append: yes + loop: '{{ additional_users }}' + become: yes + + - name: generate additional users ssh keys + user: + name: '{{ item.name }}' + generate_ssh_key: '{{ item.generate_ssh_key }}' + loop: '{{ additional_users }}' + when: item.generate_ssh_key | default(false, true) and not item.ssh_key | default(false, true) + become: yes + + - name: ensure ssh directory for additional users exists + file: + path: '/home/{{ item.name }}/.ssh' + state: directory + mode: '0700' + loop: '{{ additional_users }}' + become: yes + + - name: set additional users ssh keys from existing key + copy: + src: '{{ item.ssh_key }}' + dest: '/home/{{ item.name }}/.ssh/id_rsa' + owner: '{{ item.name }}' + group: '{{ item.name }}' + mode: '0600' + loop: '{{ additional_users }}' + when: item.ssh_key | default(false, true) and not item.generate_ssh_key | default(false, true) + become: yes + + - name: set authorized key for user + authorized_key: + user: '{{ item.name }}' + state: present + key: '{{ lookup("file", item.authorized_key) }}' + loop: '{{ additional_users }}' + when: item.authorized_key | default(false, true) + become: yes + when: additional_users | default(false, true) + # endblock add additional users + + - name: set passwordless login + lineinfile: + path: /etc/sudoers + state: present + regexp: '^%{{ sudo_group }}' + line: '%{{ sudo_group }} ALL=(ALL) NOPASSWD: ALL' + validate: 'visudo -cf %s' + when: ssh_passwordless_login | default(false, true) + become: yes + + - block: + - name: disable ipv4 localhost resolver + replace: + path: '{{ item.file }}' + regexp: '^127\.0\.0\.1 {{ item.fqdn }}(.*)$' + replace: '#127.0.0.1 {{ item.fqdn }}\1' + loop: '{{ host_files }}' + + - name: disable ipv6 localhost resolver + replace: + path: '{{ item.file }}' + regexp: '^::1 {{ item.fqdn }}(.*)$' + replace: '#::1 {{ item.fqdn }}\1' + loop: '{{ host_files }}' + when: cloud_init_disable_localhost_resolver diff --git a/ansible/templates/config.j2 b/libvirt/ansible/templates/config.j2 diff --git a/libvirt/ansible/templates/dnsmasq.j2 b/libvirt/ansible/templates/dnsmasq.j2 @@ -0,0 +1,5 @@ +# Ansible generated dnsmasq.conf snippet +{% for host in qemu_config %} +# libvirt host {{ host.id }} +dhcp-host={{ host.macaddr }},{{ host.ip4 }} +{% endfor %} diff --git a/ansible/update-known-hosts.yml b/libvirt/ansible/update-known-hosts.yml diff --git a/ansible/vars/Debian.yml b/libvirt/ansible/vars/Debian.yml diff --git a/ansible/vars/RedHat.yml b/libvirt/ansible/vars/RedHat.yml diff --git a/ansible/vars/Suse.yml b/libvirt/ansible/vars/Suse.yml diff --git a/libvirt/terraform/cloud_init.cfg b/libvirt/terraform/cloud_init.cfg @@ -0,0 +1,23 @@ +#cloud-config +# vim: syntax=yaml +# +# *********************** +# ---- for more examples look at: ------ +# ---> https://cloudinit.readthedocs.io/en/latest/topics/examples.html +# ****************************** +# +# This is the configuration syntax that the write_files module +# will know how to understand. encoding can be given b64 or gzip or (gz+b64). +# The content will be decoded accordingly and then written to the path that is +# provided. +# +# Note: Content strings here are truncated for example purposes. +ssh_pwauth: True +disable_root: False +chpasswd: + list: + root:root + expire: False +ssh_authorized_keys: + - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDirTd1BI4qwFHZc+H3loOGpx4Xfb0XUkDiZIYM48/BF/qhVBgkCrHAP3940Qb4DVjydH7vxML6ZfGnZ0rgFuZlQDoNcM2xS/dcWbHev0vv3JT6YMlUVJvknlGJXTgfkgafm8XYItm7CfE63OC1lJyYMKnnZFCTUn+oAnohhHeMIbZGID0P2XencSIEU+OA+9h8uLsdM/LaI89UeoKTYThH5EhWygL727nnUB/DjngazjIW+lS1HFPKUYexBaHAP3Cfic80aDvouJQ4zBe4ZsTV+x8J/pVEM4ys5TYaHMGGQlHLJoSDRJx75H5eu1CpFRZEhivZ53fCqJlXyfTnVnBp + diff --git a/libvirt/terraform/commoninit.iso b/libvirt/terraform/commoninit.iso Binary files differ. diff --git a/libvirt/terraform/network_config.cfg b/libvirt/terraform/network_config.cfg @@ -0,0 +1,4 @@ +version: 2 +ethernets: + ens3: + dhcp4: true +\ No newline at end of file diff --git a/libvirt/terraform/outputs.tf b/libvirt/terraform/outputs.tf @@ -0,0 +1,25 @@ +output "inventory" { + value = "${templatefile("${path.module}/templates/inventory.tpl", { hosts = libvirt_domain.host })}" +} + +output "qemu_config" { + value = "${templatefile("${path.module}/templates/qemu-config.yml.tpl", { hosts = libvirt_domain.host })}" +} + +output "ssh_private_key" { + value = "${tls_private_key.id_rsa.private_key_pem}" + sensitive = true +} + +output "ssh_private_keyfile" { + value = "${local_file.ssh_private_key.filename}" +} + +output "ssh_public_key" { + value = "${tls_private_key.id_rsa.public_key_openssh}" + sensitive = true +} + +output "ssh_public_keyfile" { + value = "${local_file.ssh_public_key.filename}" +} diff --git a/libvirt/terraform/templates/cloud_init.cfg.tpl b/libvirt/terraform/templates/cloud_init.cfg.tpl @@ -0,0 +1,22 @@ +#cloud-config +# vim: syntax=yaml +# +# *********************** +# ---- for more examples look at: ------ +# ---> https://cloudinit.readthedocs.io/en/latest/topics/examples.html +# ****************************** +# +# This is the configuration syntax that the write_files module +# will know how to understand. encoding can be given b64 or gzip or (gz+b64). +# The content will be decoded accordingly and then written to the path that is +# provided. +# +# Note: Content strings here are truncated for example purposes. +ssh_pwauth: True +disable_root: False +chpasswd: + list: + root:root + expire: False +ssh_authorized_keys: + - ${public_key} +\ No newline at end of file diff --git a/terraform/templates/inventory.tpl b/libvirt/terraform/templates/inventory.tpl diff --git a/terraform/templates/qemu-config.yml.tpl b/libvirt/terraform/templates/qemu-config.yml.tpl diff --git a/libvirt/terraform/terraform.tfvars.example b/libvirt/terraform/terraform.tfvars.example @@ -0,0 +1,6 @@ +project = "myCluster" +hosts = ["myCluster01", "myCluster02"] +# vcpu = 1 +# memory = 1024 +# disk = 13 +# baseimage = "https://cloud-images.ubuntu.com/releases/xenial/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img" +\ No newline at end of file diff --git a/libvirt/terraform/variables.tf b/libvirt/terraform/variables.tf @@ -0,0 +1,54 @@ +variable "project" { + type = string + default = "projectName" +} + +variable "hosts" { + type = list + default = ["node0", "node1"] +} + +variable "vcpu" { + type = number + default = 1 +} + +variable "memory" { + type = number + default = 1048 +} + +variable "disk" { + type = number + default = 12 +} + +variable "cloudinit_iso" { + type = string + default = "commoninit.iso" + description = "Destination of the cloud-init iso image" +} + +variable "cloudinit_userdata" { + type = string + default = "cloud_init.cfg" + description = "cloud-init userdata config" +} + +variable "cloudinit_networkconfig" { + type = string + default = "network_config.cfg" + description = "cloud-init network config file" +} + +variable "baseimage" { + type = string + default = "https://cloud-images.ubuntu.com/releases/xenial/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img" + description = "URL to a qcow2 image used as backing image for all VMs" +} + +variable "baseimage_format" { + type = string + default = "qcow2" + description = "Format of the baseimage used as backing image for all VMs" +} +\ No newline at end of file diff --git a/libvirt/terraform/vms.tf b/libvirt/terraform/vms.tf @@ -0,0 +1,139 @@ +# based on the example from: +# https://github.com/dmacvicar/terraform-provider-libvirt/tree/master/examples/v0.12/ubuntu + +# todo: LIBVIRT_DEFAULT_URI does not work? +provider "libvirt" { + uri = "qemu:///system" +} + +# libvirt pool for the images in the path of the tf module +resource "libvirt_pool" "pool" { + name = var.project + type = "dir" + path = abspath("${path.module}/libvirt-pool-${var.project}") +} + +# backing image to save disk space for multiple vms +# https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html +resource "libvirt_volume" "base_volume" { +# resource "libvirt_volume" "volume" { + name = "${var.project}-base" + pool = var.project + source = var.baseimage + format = var.baseimage_format + + # todo: required for "terraform destroy" + depends_on = [libvirt_pool.pool] +} + +resource "libvirt_volume" "volume" { + # unique image (based on backing file) for each host + for_each = toset(var.hosts) + + name = "${var.project}-cow-${each.value}" + pool = var.project + base_volume_id = libvirt_volume.base_volume.id +} + + +# ssh private key +resource "tls_private_key" "id_rsa" { + algorithm = "RSA" +} +resource "local_file" "ssh_private_key" { + sensitive_content = tls_private_key.id_rsa.private_key_pem + filename = "${path.module}/../ssh/id_rsa" + provisioner "local-exec" { + command = "chmod 600 ${path.module}/../ssh/id_rsa" + } +} +resource "local_file" "ssh_public_key" { + sensitive_content = tls_private_key.id_rsa.public_key_openssh + filename = "${path.module}/../ssh/id_rsa.pub" +} + + +# cloud init config files +data "template_file" "user_data" { + template = file("${path.module}/${var.cloudinit_userdata}") +} +data "template_file" "network_config" { + template = file("${path.module}/${var.cloudinit_networkconfig}") +} +# for more info about paramater check this out +# https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/cloudinit.html.markdown +# Use CloudInit to add our ssh-key to the instance +# you can add also meta_data field +resource "libvirt_cloudinit_disk" "commoninit" { + name = var.cloudinit_iso + user_data = data.template_file.user_data.rendered + network_config = data.template_file.network_config.rendered + pool = var.project +} + +resource "libvirt_domain" "host" { + # create each host + for_each = toset(var.hosts) + + name = each.value + memory = var.memory + vcpu = var.vcpu + + cloudinit = libvirt_cloudinit_disk.commoninit.id + qemu_agent = true + + network_interface { + network_name = "default" + } + + # IMPORTANT: this is a known bug on cloud images, since they expect a console + # we need to pass it + # https://bugs.launchpad.net/cloud-images/+bug/1573095 + console { + type = "pty" + target_port = "0" + target_type = "serial" + } + + console { + type = "pty" + target_type = "virtio" + target_port = "1" + } + + disk { + volume_id = libvirt_volume.volume[each.value].id + } + + graphics { + type = "spice" + listen_type = "address" + autoport = true + } +} + + +resource "null_resource" "update_cloudinit" { + triggers = { + # when the ssh key in the local cloudinit file changes + key_id = local_file.ssh_public_key.id + } + provisioner "local-exec" { + # recreate cloudinit config + command = "echo '${templatefile("${path.module}/templates/cloud_init.cfg.tpl", { public_key = tls_private_key.id_rsa.public_key_openssh })}' > ./cloud_init.cfg" + } +} +resource "null_resource" "update_inventory" { + triggers = { + # when a host id changes + host_ids = "${join(" ", values(libvirt_domain.host)[*].id)}" + } + provisioner "local-exec" { + # recreate ansible inventory + command = "echo '${templatefile("${path.module}/templates/inventory.tpl", { hosts = libvirt_domain.host })}' > ../ansible/inventory" + } + provisioner "local-exec" { + # recreate mapping of qemu VM id to hostnames + command = "echo '${templatefile("${path.module}/templates/qemu-config.yml.tpl", { hosts = libvirt_domain.host })}' > ../ansible/qemu-config.yml" + } +} +\ No newline at end of file diff --git a/Readme.md b/proxmox/Readme.md diff --git a/ansible/defaults/all.yml b/proxmox/ansible/defaults/all.yml diff --git a/ansible/dhcp-static-hosts.yml b/proxmox/ansible/dhcp-static-hosts.yml diff --git a/ansible/get-ips.yml b/proxmox/ansible/get-ips.yml diff --git a/ansible/playbook.yml b/proxmox/ansible/playbook.yml diff --git a/ansible/templates/config.j2 b/proxmox/ansible/templates/config.j2 diff --git a/ansible/templates/dnsmasq.j2 b/proxmox/ansible/templates/dnsmasq.j2 diff --git a/ansible/update-known-hosts.yml b/proxmox/ansible/update-known-hosts.yml diff --git a/ansible/vars/Debian.yml b/proxmox/ansible/vars/Debian.yml diff --git a/ansible/vars/RedHat.yml b/proxmox/ansible/vars/RedHat.yml diff --git a/ansible/vars/Suse.yml b/proxmox/ansible/vars/Suse.yml diff --git a/terraform/outputs.tf b/proxmox/terraform/outputs.tf diff --git a/terraform/templates/inventory.tpl b/proxmox/terraform/templates/inventory.tpl diff --git a/terraform/templates/qemu-config.yml.tpl b/proxmox/terraform/templates/qemu-config.yml.tpl diff --git a/terraform/terraform.tfvars.example b/proxmox/terraform/terraform.tfvars.example diff --git a/terraform/variables.tf b/proxmox/terraform/variables.tf diff --git a/terraform/vms.tf b/proxmox/terraform/vms.tf