hetzner-devops/incus.md

15 KiB

Incus Configuration Documentation

System Information

  • Date: 2025-07-16
  • Incus Version: 6.14
  • Host: nsntr.ai
  • OS: Ubuntu 24.04
  • Architecture: x86_64

Global Configuration

Server Config

config:
  core.https_address: 0.0.0.0:8443

Certificate Info

Certificate Fingerprint: 7ca55f8f4e8224855eae368bf53ec42e7cfff38409fcfebfd85db9f3697a4287
Auth Method: unix
Auth User: root

Storage Pools

Pool List

NAME        DRIVER  SIZE    USED      STATE
backup      zfs     199GB   684KB     CREATED
default     btrfs   30GB    1.35GB    CREATED
development zfs     298GB   620KB     CREATED
production  zfs     796GB   639KB     CREATED
services    zfs     199GB   632KB     CREATED

ZFS Pool Configuration

Services Pool

name: services
driver: zfs
size: 200GiB
config:
  compression: lz4
  recordsize: 64K
  atime: off
  sync: standard
  primarycache: all
  com.sun:auto-snapshot: true

Development Pool

name: development
driver: zfs
size: 300GiB
config:
  compression: lz4
  recordsize: 128K
  atime: off
  sync: disabled
  primarycache: all
  com.sun:auto-snapshot: false

Production Pool

name: production
driver: zfs
size: 800GiB
config:
  compression: lz4
  recordsize: 32K
  atime: off
  sync: always
  primarycache: all
  com.sun:auto-snapshot: true

Backup Pool

name: backup
driver: zfs
size: 200GiB
config:
  compression: gzip-6
  recordsize: 1M
  atime: off
  sync: standard
  primarycache: metadata
  com.sun:auto-snapshot: false

ZFS System Settings

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=33554432000    # 32GB max
options zfs zfs_arc_min=4294967296     # 4GB min
options zfs zfs_prefetch_disable=0     # Prefetch enabled
options zfs zfs_txg_timeout=5          # 5 second timeout

Projects Configuration

Project List

NAME        IMAGES  PROFILES  STORAGE_VOLUMES  NETWORKS  USED_BY
default     YES     YES       YES              YES       4
development YES     YES       YES              NO        1
production  YES     YES       YES              NO        1
services    YES     YES       YES              NO        1

Services Project

name: services
config:
  features.images: true
  features.profiles: true
  features.storage.buckets: true
  features.storage.volumes: true
  limits.cpu: 8
  limits.memory: 24GiB
  limits.instances: 10
  limits.disk.pool.services: 200GiB
  restricted.networks.access: services-net

Development Project

name: development
config:
  features.images: true
  features.profiles: true
  features.storage.buckets: true
  features.storage.volumes: true
  limits.cpu: 8
  limits.memory: 32GiB
  limits.instances: 20
  limits.disk.pool.development: 300GiB
  restricted.networks.access: development-net

Production Project

name: production
config:
  features.images: true
  features.profiles: true
  features.storage.buckets: true
  features.storage.volumes: true
  limits.cpu: 12
  limits.memory: 60GiB
  limits.instances: 50
  limits.disk.pool.production: 800GiB
  restricted.networks.access: production-net

Network Configuration

Network List

NAME              TYPE     MANAGED  IPV4           IPV6  STATE
development-net   bridge   YES      10.20.20.1/24  none  CREATED
incusbr0          bridge   YES      10.94.230.1/24 auto  CREATED
management-net    bridge   YES      10.40.40.1/24  none  CREATED
production-net    bridge   YES      10.30.30.1/24  none  CREATED
services-net      bridge   YES      10.10.10.1/24  none  CREATED

Services Network

name: services-net
type: bridge
config:
  ipv4.address: 10.10.10.1/24
  ipv4.nat: true
  ipv4.dhcp: true
  ipv4.dhcp.ranges: 10.10.10.50-10.10.10.199
  ipv6.address: none
  ipv6.nat: true

Development Network

name: development-net
type: bridge
config:
  ipv4.address: 10.20.20.1/24
  ipv4.nat: true
  ipv4.dhcp: true
  ipv4.dhcp.ranges: 10.20.20.50-10.20.20.199
  ipv6.address: none
  ipv6.nat: true

Production Network

name: production-net
type: bridge
config:
  ipv4.address: 10.30.30.1/24
  ipv4.nat: true
  ipv4.dhcp: true
  ipv4.dhcp.ranges: 10.30.30.50-10.30.30.199
  ipv6.address: none
  ipv6.nat: true

Management Network

name: management-net
type: bridge
config:
  ipv4.address: 10.40.40.1/24
  ipv4.nat: true
  ipv4.dhcp: true
  ipv4.dhcp.ranges: 10.40.40.50-10.40.40.199
  ipv6.address: none
  ipv6.nat: true

Profiles Configuration

Default Profile (Services Project)

name: default
project: services
config: {}
description: Default profile for services
devices:
  root:
    type: disk
    path: /
    pool: services
  eth0:
    type: nic
    network: services-net
    name: eth0

Default Profile (Development Project)

name: default
project: development
config: {}
description: Default profile for development
devices:
  root:
    type: disk
    path: /
    pool: development
  eth0:
    type: nic
    network: development-net
    name: eth0

Default Profile (Production Project)

name: default
project: production
config: {}
description: Default profile for production
devices:
  root:
    type: disk
    path: /
    pool: production
  eth0:
    type: nic
    network: production-net
    name: eth0

IP Address Allocation

Static IP Ranges (Reserved)

Network           Range               Purpose
services-net      10.10.10.10-49     Static services
development-net   10.20.20.10-49     Static dev services
production-net    10.30.30.10-49     Static prod services
management-net    10.40.40.10-49     Static management

DHCP Ranges

Network           Range               Purpose
services-net      10.10.10.50-199    Dynamic allocation
development-net   10.20.20.50-199    Dynamic allocation
production-net    10.30.30.50-199    Dynamic allocation
management-net    10.40.40.50-199    Dynamic allocation

Planned Static Assignments

Service           IP Address         Network
Traefik           10.10.10.10        services-net
Gitea             10.10.10.20        services-net
Drone CI          10.10.10.30        services-net
Monitoring        10.40.40.10        management-net
Backup Services   10.40.40.20        management-net

Resource Limits Summary

Total System Resources

CPU: 32 cores (AMD Ryzen 9 7950X3D)
RAM: 124GB
Storage: 1.7TB (RAID1 NVMe)

Project Resource Allocation

PROJECT      CPU    MEMORY    STORAGE    INSTANCES
services     8      24GB      200GB      10
development  8      32GB      300GB      20
production   12     60GB      800GB      50
system       4      8GB       -          -
TOTAL        32     124GB     1.5TB      80

Backup Configuration

ZFS Snapshots

# Auto-snapshot enabled for:
- services pool
- production pool

# Manual snapshots for:
- development pool
- backup pool

Snapshot Retention (Planned)

Pool          Frequency    Retention
services      daily        30 days
production    daily        90 days
development   manual       7 days
backup        manual       365 days

Monitoring & Logs

System Logs

# Incus logs
journalctl -u incus

# ZFS events
zpool events

# Network status
ip route show

Performance Monitoring

# ZFS ARC stats
cat /proc/spl/kstat/zfs/arcstats

# Pool I/O stats
zpool iostat -v

# Network stats
incus network list

Maintenance Commands

Regular Maintenance

# Check pool health
zpool status

# Scrub pools (monthly)
zpool scrub services
zpool scrub development
zpool scrub production
zpool scrub backup

# Update container images
incus image list
incus image refresh

# Clean old snapshots
incus snapshot list

Troubleshooting Commands

# Check resource usage
incus info
incus project show <project>

# Network diagnostics
incus network info <network>
incus exec <container> -- ip addr show

# Storage diagnostics
incus storage info <pool>
zfs list -t all

Security Configuration

Network Security

  • Networks isolated by project
  • NAT enabled for internet access
  • No direct inter-project communication
  • Firewall rules per network (planned)

Storage Security

  • ZFS encryption (not enabled yet)
  • Separate pools per environment
  • Quota limits per project
  • Snapshot-based backups

Access Control

  • TLS certificate authentication
  • Unix socket authentication
  • Project-based isolation
  • Resource quotas

Recovery Procedures

Storage Recovery

# Import pools after reboot
zpool import -f <pool>

# Restore from snapshot
zfs rollback <pool>@<snapshot>

# Clone from snapshot
zfs clone <pool>@<snapshot> <new-dataset>

Network Recovery

# Restart network
incus network restart <network>

# Recreate network
incus network delete <network>
incus network create <network>

Container Recovery

# List snapshots
incus snapshot list <container>

# Restore from snapshot
incus snapshot restore <container> <snapshot>

# Backup container
incus export <container> <backup-file>

Generated: 2025-07-16 02:38:24 UTC Status: Infrastructure configured and ready Next: Service container deployment

Current System Status (Live Data)

ZFS Pool Status

pool: backup state: ONLINE config:

NAME                               STATE     READ WRITE CKSUM
backup                             ONLINE       0     0     0
  /var/lib/incus/disks/backup.img  ONLINE       0     0     0

errors: No known data errors

pool: development state: ONLINE config:

NAME                                    STATE     READ WRITE CKSUM
development                             ONLINE       0     0     0
  /var/lib/incus/disks/development.img  ONLINE       0     0     0

errors: No known data errors

pool: production state: ONLINE config:

NAME                                   STATE     READ WRITE CKSUM
production                             ONLINE       0     0     0
  /var/lib/incus/disks/production.img  ONLINE       0     0     0

errors: No known data errors

pool: services state: ONLINE config:

NAME                                 STATE     READ WRITE CKSUM
services                             ONLINE       0     0     0
  /var/lib/incus/disks/services.img  ONLINE       0     0     0

errors: No known data errors

Current Instances

+---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+ | PROJECT | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+ | default | ubuntu01 | RUNNING | 10.94.230.45 (eth0) | fd42:14d8:bd01:cc0a:1266:6aff:fe00:bd62 (eth0) | CONTAINER | 0 | +---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+

ZFS Datasets

NAME USED AVAIL REFER MOUNTPOINT backup 648K 193G 24K legacy backup/buckets 24K 193G 24K legacy backup/containers 24K 193G 24K legacy backup/custom 24K 193G 24K legacy backup/deleted 144K 193G 24K legacy backup/deleted/buckets 24K 193G 24K legacy backup/deleted/containers 24K 193G 24K legacy backup/deleted/custom 24K 193G 24K legacy backup/deleted/images 24K 193G 24K legacy backup/deleted/virtual-machines 24K 193G 24K legacy backup/images 24K 193G 24K legacy backup/virtual-machines 24K 193G 24K legacy development 648K 289G 24K legacy development/buckets 24K 289G 24K legacy development/containers 24K 289G 24K legacy development/custom 24K 289G 24K legacy development/deleted 144K 289G 24K legacy development/deleted/buckets 24K 289G 24K legacy development/deleted/containers 24K 289G 24K legacy development/deleted/custom 24K 289G 24K legacy development/deleted/images 24K 289G 24K legacy development/deleted/virtual-machines 24K 289G 24K legacy development/images 24K 289G 24K legacy development/virtual-machines 24K 289G 24K legacy production 668K 771G 24K legacy production/buckets 24K 771G 24K legacy production/containers 24K 771G 24K legacy production/custom 24K 771G 24K legacy production/deleted 144K 771G 24K legacy production/deleted/buckets 24K 771G 24K legacy production/deleted/containers 24K 771G 24K legacy production/deleted/custom 24K 771G 24K legacy production/deleted/images 24K 771G 24K legacy production/deleted/virtual-machines 24K 771G 24K legacy production/images 24K 771G 24K legacy production/virtual-machines 24K 771G 24K legacy services 652K 193G 24K legacy services/buckets 24K 193G 24K legacy services/containers 24K 193G 24K legacy services/custom 24K 193G 24K legacy services/deleted 144K 193G 24K legacy services/deleted/buckets 24K 193G 24K legacy services/deleted/containers 24K 193G 24K legacy services/deleted/custom 24K 193G 24K legacy services/deleted/images 24K 193G 24K legacy services/deleted/virtual-machines 24K 193G 24K legacy services/images 24K 193G 24K legacy services/virtual-machines 24K 193G 24K legacy

Network Routes

10.10.10.0/24 dev services-net proto kernel scope link src 10.10.10.1 linkdown 10.20.20.0/24 dev development-net proto kernel scope link src 10.20.20.1 linkdown 10.30.30.0/24 dev production-net proto kernel scope link src 10.30.30.1 linkdown 10.40.40.0/24 dev management-net proto kernel scope link src 10.40.40.1 linkdown

System Resource Usage

           total        used        free      shared  buff/cache   available

Mem: 124Gi 2.1Gi 120Gi 1.5Mi 3.6Gi 122Gi Swap: 23Gi 0B 23Gi

Storage Usage

Filesystem Size Used Avail Use% Mounted on /dev/md2 1.7T 5.2G 1.7T 1% / /dev/md1 988M 103M 818M 12% /boot /dev/loop0 30G 1.4G 29G 5% /var/lib/incus/storage-pools/default tmpfs 100K 0 100K 0% /var/lib/incus/shmounts tmpfs 100K 0 100K 0% /var/lib/incus/guestapi


Last Updated: Wed Jul 16 02:39:50 CEST 2025 Configuration Status: Complete and Active Ready for: Service container deployment