hetzner-devops/incus.md

595 lines
15 KiB
Markdown

# Incus Configuration Documentation
## System Information
- **Date**: 2025-07-16
- **Incus Version**: 6.14
- **Host**: nsntr.ai
- **OS**: Ubuntu 24.04
- **Architecture**: x86_64
## Global Configuration
### Server Config
```yaml
config:
core.https_address: 0.0.0.0:8443
```
### Certificate Info
```
Certificate Fingerprint: 7ca55f8f4e8224855eae368bf53ec42e7cfff38409fcfebfd85db9f3697a4287
Auth Method: unix
Auth User: root
```
## Storage Pools
### Pool List
```
NAME DRIVER SIZE USED STATE
backup zfs 199GB 684KB CREATED
default btrfs 30GB 1.35GB CREATED
development zfs 298GB 620KB CREATED
production zfs 796GB 639KB CREATED
services zfs 199GB 632KB CREATED
```
### ZFS Pool Configuration
#### Services Pool
```yaml
name: services
driver: zfs
size: 200GiB
config:
compression: lz4
recordsize: 64K
atime: off
sync: standard
primarycache: all
com.sun:auto-snapshot: true
```
#### Development Pool
```yaml
name: development
driver: zfs
size: 300GiB
config:
compression: lz4
recordsize: 128K
atime: off
sync: disabled
primarycache: all
com.sun:auto-snapshot: false
```
#### Production Pool
```yaml
name: production
driver: zfs
size: 800GiB
config:
compression: lz4
recordsize: 32K
atime: off
sync: always
primarycache: all
com.sun:auto-snapshot: true
```
#### Backup Pool
```yaml
name: backup
driver: zfs
size: 200GiB
config:
compression: gzip-6
recordsize: 1M
atime: off
sync: standard
primarycache: metadata
com.sun:auto-snapshot: false
```
### ZFS System Settings
```bash
# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=33554432000 # 32GB max
options zfs zfs_arc_min=4294967296 # 4GB min
options zfs zfs_prefetch_disable=0 # Prefetch enabled
options zfs zfs_txg_timeout=5 # 5 second timeout
```
## Projects Configuration
### Project List
```
NAME IMAGES PROFILES STORAGE_VOLUMES NETWORKS USED_BY
default YES YES YES YES 4
development YES YES YES NO 1
production YES YES YES NO 1
services YES YES YES NO 1
```
### Services Project
```yaml
name: services
config:
features.images: true
features.profiles: true
features.storage.buckets: true
features.storage.volumes: true
limits.cpu: 8
limits.memory: 24GiB
limits.instances: 10
limits.disk.pool.services: 200GiB
restricted.networks.access: services-net
```
### Development Project
```yaml
name: development
config:
features.images: true
features.profiles: true
features.storage.buckets: true
features.storage.volumes: true
limits.cpu: 8
limits.memory: 32GiB
limits.instances: 20
limits.disk.pool.development: 300GiB
restricted.networks.access: development-net
```
### Production Project
```yaml
name: production
config:
features.images: true
features.profiles: true
features.storage.buckets: true
features.storage.volumes: true
limits.cpu: 12
limits.memory: 60GiB
limits.instances: 50
limits.disk.pool.production: 800GiB
restricted.networks.access: production-net
```
## Network Configuration
### Network List
```
NAME TYPE MANAGED IPV4 IPV6 STATE
development-net bridge YES 10.20.20.1/24 none CREATED
incusbr0 bridge YES 10.94.230.1/24 auto CREATED
management-net bridge YES 10.40.40.1/24 none CREATED
production-net bridge YES 10.30.30.1/24 none CREATED
services-net bridge YES 10.10.10.1/24 none CREATED
```
### Services Network
```yaml
name: services-net
type: bridge
config:
ipv4.address: 10.10.10.1/24
ipv4.nat: true
ipv4.dhcp: true
ipv4.dhcp.ranges: 10.10.10.50-10.10.10.199
ipv6.address: none
ipv6.nat: true
```
### Development Network
```yaml
name: development-net
type: bridge
config:
ipv4.address: 10.20.20.1/24
ipv4.nat: true
ipv4.dhcp: true
ipv4.dhcp.ranges: 10.20.20.50-10.20.20.199
ipv6.address: none
ipv6.nat: true
```
### Production Network
```yaml
name: production-net
type: bridge
config:
ipv4.address: 10.30.30.1/24
ipv4.nat: true
ipv4.dhcp: true
ipv4.dhcp.ranges: 10.30.30.50-10.30.30.199
ipv6.address: none
ipv6.nat: true
```
### Management Network
```yaml
name: management-net
type: bridge
config:
ipv4.address: 10.40.40.1/24
ipv4.nat: true
ipv4.dhcp: true
ipv4.dhcp.ranges: 10.40.40.50-10.40.40.199
ipv6.address: none
ipv6.nat: true
```
## Profiles Configuration
### Default Profile (Services Project)
```yaml
name: default
project: services
config: {}
description: Default profile for services
devices:
root:
type: disk
path: /
pool: services
eth0:
type: nic
network: services-net
name: eth0
```
### Default Profile (Development Project)
```yaml
name: default
project: development
config: {}
description: Default profile for development
devices:
root:
type: disk
path: /
pool: development
eth0:
type: nic
network: development-net
name: eth0
```
### Default Profile (Production Project)
```yaml
name: default
project: production
config: {}
description: Default profile for production
devices:
root:
type: disk
path: /
pool: production
eth0:
type: nic
network: production-net
name: eth0
```
## IP Address Allocation
### Static IP Ranges (Reserved)
```
Network Range Purpose
services-net 10.10.10.10-49 Static services
development-net 10.20.20.10-49 Static dev services
production-net 10.30.30.10-49 Static prod services
management-net 10.40.40.10-49 Static management
```
### DHCP Ranges
```
Network Range Purpose
services-net 10.10.10.50-199 Dynamic allocation
development-net 10.20.20.50-199 Dynamic allocation
production-net 10.30.30.50-199 Dynamic allocation
management-net 10.40.40.50-199 Dynamic allocation
```
### Planned Static Assignments
```
Service IP Address Network
Traefik 10.10.10.10 services-net
Gitea 10.10.10.20 services-net
Drone CI 10.10.10.30 services-net
Monitoring 10.40.40.10 management-net
Backup Services 10.40.40.20 management-net
```
## Resource Limits Summary
### Total System Resources
```
CPU: 32 cores (AMD Ryzen 9 7950X3D)
RAM: 124GB
Storage: 1.7TB (RAID1 NVMe)
```
### Project Resource Allocation
```
PROJECT CPU MEMORY STORAGE INSTANCES
services 8 24GB 200GB 10
development 8 32GB 300GB 20
production 12 60GB 800GB 50
system 4 8GB - -
TOTAL 32 124GB 1.5TB 80
```
## Backup Configuration
### ZFS Snapshots
```bash
# Auto-snapshot enabled for:
- services pool
- production pool
# Manual snapshots for:
- development pool
- backup pool
```
### Snapshot Retention (Planned)
```
Pool Frequency Retention
services daily 30 days
production daily 90 days
development manual 7 days
backup manual 365 days
```
## Monitoring & Logs
### System Logs
```bash
# Incus logs
journalctl -u incus
# ZFS events
zpool events
# Network status
ip route show
```
### Performance Monitoring
```bash
# ZFS ARC stats
cat /proc/spl/kstat/zfs/arcstats
# Pool I/O stats
zpool iostat -v
# Network stats
incus network list
```
## Maintenance Commands
### Regular Maintenance
```bash
# Check pool health
zpool status
# Scrub pools (monthly)
zpool scrub services
zpool scrub development
zpool scrub production
zpool scrub backup
# Update container images
incus image list
incus image refresh
# Clean old snapshots
incus snapshot list
```
### Troubleshooting Commands
```bash
# Check resource usage
incus info
incus project show <project>
# Network diagnostics
incus network info <network>
incus exec <container> -- ip addr show
# Storage diagnostics
incus storage info <pool>
zfs list -t all
```
## Security Configuration
### Network Security
- Networks isolated by project
- NAT enabled for internet access
- No direct inter-project communication
- Firewall rules per network (planned)
### Storage Security
- ZFS encryption (not enabled yet)
- Separate pools per environment
- Quota limits per project
- Snapshot-based backups
### Access Control
- TLS certificate authentication
- Unix socket authentication
- Project-based isolation
- Resource quotas
## Recovery Procedures
### Storage Recovery
```bash
# Import pools after reboot
zpool import -f <pool>
# Restore from snapshot
zfs rollback <pool>@<snapshot>
# Clone from snapshot
zfs clone <pool>@<snapshot> <new-dataset>
```
### Network Recovery
```bash
# Restart network
incus network restart <network>
# Recreate network
incus network delete <network>
incus network create <network>
```
### Container Recovery
```bash
# List snapshots
incus snapshot list <container>
# Restore from snapshot
incus snapshot restore <container> <snapshot>
# Backup container
incus export <container> <backup-file>
```
---
**Generated**: 2025-07-16 02:38:24 UTC
**Status**: Infrastructure configured and ready
**Next**: Service container deployment
## Current System Status (Live Data)
### ZFS Pool Status
pool: backup
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
/var/lib/incus/disks/backup.img ONLINE 0 0 0
errors: No known data errors
pool: development
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
development ONLINE 0 0 0
/var/lib/incus/disks/development.img ONLINE 0 0 0
errors: No known data errors
pool: production
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
production ONLINE 0 0 0
/var/lib/incus/disks/production.img ONLINE 0 0 0
errors: No known data errors
pool: services
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
services ONLINE 0 0 0
/var/lib/incus/disks/services.img ONLINE 0 0 0
errors: No known data errors
### Current Instances
+---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+
| PROJECT | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+
| default | ubuntu01 | RUNNING | 10.94.230.45 (eth0) | fd42:14d8:bd01:cc0a:1266:6aff:fe00:bd62 (eth0) | CONTAINER | 0 |
+---------+----------+---------+---------------------+------------------------------------------------+-----------+-----------+
### ZFS Datasets
NAME USED AVAIL REFER MOUNTPOINT
backup 648K 193G 24K legacy
backup/buckets 24K 193G 24K legacy
backup/containers 24K 193G 24K legacy
backup/custom 24K 193G 24K legacy
backup/deleted 144K 193G 24K legacy
backup/deleted/buckets 24K 193G 24K legacy
backup/deleted/containers 24K 193G 24K legacy
backup/deleted/custom 24K 193G 24K legacy
backup/deleted/images 24K 193G 24K legacy
backup/deleted/virtual-machines 24K 193G 24K legacy
backup/images 24K 193G 24K legacy
backup/virtual-machines 24K 193G 24K legacy
development 648K 289G 24K legacy
development/buckets 24K 289G 24K legacy
development/containers 24K 289G 24K legacy
development/custom 24K 289G 24K legacy
development/deleted 144K 289G 24K legacy
development/deleted/buckets 24K 289G 24K legacy
development/deleted/containers 24K 289G 24K legacy
development/deleted/custom 24K 289G 24K legacy
development/deleted/images 24K 289G 24K legacy
development/deleted/virtual-machines 24K 289G 24K legacy
development/images 24K 289G 24K legacy
development/virtual-machines 24K 289G 24K legacy
production 668K 771G 24K legacy
production/buckets 24K 771G 24K legacy
production/containers 24K 771G 24K legacy
production/custom 24K 771G 24K legacy
production/deleted 144K 771G 24K legacy
production/deleted/buckets 24K 771G 24K legacy
production/deleted/containers 24K 771G 24K legacy
production/deleted/custom 24K 771G 24K legacy
production/deleted/images 24K 771G 24K legacy
production/deleted/virtual-machines 24K 771G 24K legacy
production/images 24K 771G 24K legacy
production/virtual-machines 24K 771G 24K legacy
services 652K 193G 24K legacy
services/buckets 24K 193G 24K legacy
services/containers 24K 193G 24K legacy
services/custom 24K 193G 24K legacy
services/deleted 144K 193G 24K legacy
services/deleted/buckets 24K 193G 24K legacy
services/deleted/containers 24K 193G 24K legacy
services/deleted/custom 24K 193G 24K legacy
services/deleted/images 24K 193G 24K legacy
services/deleted/virtual-machines 24K 193G 24K legacy
services/images 24K 193G 24K legacy
services/virtual-machines 24K 193G 24K legacy
### Network Routes
10.10.10.0/24 dev services-net proto kernel scope link src 10.10.10.1 linkdown
10.20.20.0/24 dev development-net proto kernel scope link src 10.20.20.1 linkdown
10.30.30.0/24 dev production-net proto kernel scope link src 10.30.30.1 linkdown
10.40.40.0/24 dev management-net proto kernel scope link src 10.40.40.1 linkdown
### System Resource Usage
total used free shared buff/cache available
Mem: 124Gi 2.1Gi 120Gi 1.5Mi 3.6Gi 122Gi
Swap: 23Gi 0B 23Gi
### Storage Usage
Filesystem Size Used Avail Use% Mounted on
/dev/md2 1.7T 5.2G 1.7T 1% /
/dev/md1 988M 103M 818M 12% /boot
/dev/loop0 30G 1.4G 29G 5% /var/lib/incus/storage-pools/default
tmpfs 100K 0 100K 0% /var/lib/incus/shmounts
tmpfs 100K 0 100K 0% /var/lib/incus/guestapi
---
**Last Updated**: Wed Jul 16 02:39:50 CEST 2025
**Configuration Status**: Complete and Active
**Ready for**: Service container deployment