I use the docker-compose.yml file from Portainer's doc to installing Portainer with Traefik. On top of it, I have a few extra things:

    # For Traefik container
    command:
      - --api=true # Enable the dashboard
      - --api.dashboard=true
      - --api.insecure=true
      - --providers.file.filename=/etc/traefik/dynamic.yml # Add a dynamic config file
      - --providers.file.watch=true
    volumes:
      # The dynamic config file sits in the same folder with the docker-compose.yaml file
      - "./dynamic.yml:/etc/traefik/dynamic.yml:ro"

For container workloads, I tend to run them as a docker-compose stack. I add the same labels as the Portainer's docker compose file. Specifically:

    networks:
      - xxxx # Remember to put the container in the same network as Traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.servicename.rule=Host(`sub.domain.notcom`)"
      - "traefik.http.routers.servicename.entrypoints=websecure"
      - "traefik.http.routers.servicename.service=servicename"
      - "traefik.http.routers.servicename.tls.certresolver=leresolver"
      - "traefik.http.services.servicename.loadbalancer.server.port=xxxx" # The port on the container, NOT the mapped port on the host

I run Kasm Workspaces on its own VM, its web UI uses a self-signed certificate, I need to tell Traefik to not verify that certificate in my dynamic config file:

http:
  routers:
    kasm-route:
      rule: "Host(`sub.domain.notcom`)"
      service: kasm
      entryPoints:
        - websecure
      tls:
        certresolver: "leresolver"

  services:
    kasm:
      loadBalancer:
        servers:
          - url: "https://10.10.10.10"
        serversTransport: kasmTransport

  serversTransports:
    kasmTransport:
      insecureSkipVerify: true # Internal Server Error if self-signed certificate is not skipped

I use CloudFlare Access to protect some of the services. In this particular setup, I add GitHub to my login methods, I won't get into the details of the setup, as of now it's under "Zero Trust" -> "Settings" -> "Authentication" -> "Login methods" -> "Add new" -> select "GitHub". CloudFlare shows detailed steps of setting it up.

Then, go back to "Zero Trust" -> "Access" -> "Applications", setup a self-hosted application on the domain. For the access policy, it can be as simple as a single Include rule: Selector: Emails. Value: The email address of the GitHub account you are going to login with. We wouldn't want to let anyone who has a GitHub account in, would we?

Because Traefik automatically sets up HTTPS on the server with Let's Encrypt, you can set CloudFlare to full strict mode, rather than full - which does encrypt the traffic between CloudFlare and your server but does not verify the server's certificate.

That's it!

Other implementations might behave slightly differently but I haven't tried. This particular implementation has a weird behavior: it tries to recreate the disk for a virtual machine when running terraform apply - which then fails on the server's side. Fortunately it doesn't stop me from creating more resources e.g. when I add a new LXC to the file, it will still create the LXC just fine, so it kinda "works". A new disk volume will be created and scheduled to replace a VM's disk if it's running (since it doesn't hot swap), which needs to be manually reverted and deleted.

Compared to other services that support IaC (e.g. an AWS EC2 instance), I'm not sure if it's this particular implementation, or if it's the API on Proxmox's side, the whole experience does not feel smooth. It wants to do some "updates" when I haven't changed a single letter, which then doesn't actually seem to change anything, again it mostly "works", and it's better than manually clicking through a few steps each time I want to create a new VM/LXC through the web UI, so it's good enough to me.

My Terraform file looks like this, hopefully it's helpful to you if you are trying to do something similar:

terraform {
  required_providers {
    proxmox = {
      source  = "loeken/proxmox"
      version = ">=2.9.0"
    }
  }
  required_version = ">= 0.14"
}

provider "proxmox" {
  # Default HTTPS port of Proxmox is 8006, yours might be different
  pm_api_url = "https://xxxx:8006/api2/json"
  # I thought this was required as my Proxmox server uses a self-signed certificate. But apparently it works without this set to true anyway
  #pm_tls_insecure = true
  # Obviously not the best practice, you should use environment variables PM_USER and PM_PASS instead. I only do this because I am in a home lab environment
  pm_user     = "root@pam"
  pm_password = "secureultrapluspromax"
}

# https://registry.terraform.io/providers/loeken/proxmox/latest/docs/resources/vm_qemu
resource "proxmox_vm_qemu" "resource_name_here" {
  name        = "VM name here"
  target_node = "Name of the node (under the Datacenter node)"
  vmid        = 200
  cores       = 16
  memory      = 65536 # MiB
  os_type     = "ubuntu"
  # The volume name under the target node "local", then the storage type "ISO Images", then the image's name
  iso         = "local:iso/ubuntu-24.10-live-server-amd64.iso"

  disk {
    type    = "scsi"
    size    = "50G"
    storage = "local-lvm"
  }

  network {
    bridge    = "vmbr0"
    firewall  = false
    link_down = false
    model     = "e1000"
  }
}

# https://registry.terraform.io/providers/loeken/proxmox/latest/docs/resources/lxc
resource "proxmox_lxc" "resource_name_here" {
  target_node = "Name of the node (under the Datacenter node)"
  hostname     = "hostname"
  # The volume name under the target node "local", then the storage type "CT Templates", then the template's name
  ostemplate   = "local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst"
  # Root password
  password     = "secureultrapluspromax"
  unprivileged = true
  vmid         = 201
  cores        = 4
  memory       = 4096 # MiB

  // Terraform will crash without rootfs defined
  rootfs {
    storage = "local-lvm"
    size    = "2G"
  }

  features {
    mount = "nfs"
  }

  # Comments from the provider's doc, I keep it here to remind myself of this weird bug
  # // NFS share mounted on host
  # // Without 'volume' defined, Proxmox will try to create a volume with
  # // the value of 'storage' + : + 'size' (without the trailing G) - e.g.
  # // "/srv/host/bind-mount-point:256".
  # // This behaviour looks to be caused by a bug in the provider.
  mountpoint {
    key     = "0"
    slot    = 0
    storage = "/mnt/mountpoint"
    volume  = "/mnt/mountpoint"
    mp      = "/mnt/data"
    # Unintuitively (and if I remember correctly), this does not work without specifying a size
    size    = "1T"
  }

  network {
    name   = "eth0"
    bridge = "vmbr0"
    ip     = "dhcp"
  }
}

Now, I have been a fan of IaC and I wanted to learn Ansible for a while. I'd prefer Terraform but the Terraform module for Proxmox did not work very well when I tried it a while ago, so it's a good opportunity for me to try Ansible!

This blog series is my learning notes for using Ansible to provision LXCs (and a few VMs) and configure the software/service running in said LXCs.

Or so I thought - the series is no more, but I'm keeping the "Part 1" in the title just for the lols, here is what happened:

Step 1, I create a snippet to quickly get an LXC up, it looks like this:

- name: test playbook
  hosts: localhost

  tasks:
  - name: Create new container with minimal options
    community.general.proxmox:
      vmid: 200
      node: my-proxmox-node
      api_user: ansible-usr@pve
      api_password: 123
      api_host: 192.168.x.x
      hostname: test-lxc
      password: 456
      ostemplate: 'local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst'
      storage: local-lvm

Then I run ansible-playbook playbook.yml to create the LXC. So far, so good.

Step 2, the LXC is up and running but the config is quite loose. I'd like to tighten it up a bit. I add disk_volume block to the template, replacing storage. As per example in the documentation

- name: test playbook
  hosts: localhost

  tasks:
  - name: Create new container with minimal options
    community.general.proxmox:
      vmid: 200
      node: my-proxmox-node
      api_user: ansible-usr@pve
      api_password: 123
      api_host: 192.168.x.x
      hostname: test-lxc
      password: 456
      ostemplate: 'local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst'
      disk_volume:
        storage: local-lvm
        size: 20

What could possibly go wrong?

Step 3, Ansible complains: "disk_volume is not a valid parameter". Huh? But the doc contains the definition and examples, how could...oh, "added in community.general 9.2.0", I see. Let me upgrade my local Ansible version.

Step 3.5 I'll spare you (and myself!) the details of me trying to figure out my local module's version, things are already bad enough.

Step 4, now the error message is something like "400 Bad Request, disk_volume is not defined in schema". OK, maybe I need to upgrade and reboot my Proxmox installation as well. Too bad though, next step please!

Step 5, OK I've given up on trying to make disk_volume work. Let me revert that property and move on to the next thing, adding cores property to limit the CPU resource.

- name: test playbook
  hosts: localhost

  tasks:
  - name: Create new container with minimal options
    community.general.proxmox:
      vmid: 200
      node: my-proxmox-node
      api_user: ansible-usr@pve
      api_password: 123
      api_host: 192.168.x.x
      hostname: test-lxc
      password: 456
      ostemplate: 'local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst'
      storage: local-lvm
      cores: 4

Works like a charm. If we just ignore the whole disk_volume thing for a second...

Step 6, I want to try updating the LXC through Ansible. I change cores from 4 to 6, save, rerun, surely this should update my LXC's CPU to 6, right? Right?

PLAY RECAP **********************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Step 7, I think it's time to revisit Terraform. Wish me luck!