Storage Health: ZFS Pools and Disk Checks

Why Storage Deserves Its Own Lesson

Of all the infrastructure a homelab runs, storage is the one where silent failures are the most dangerous. A VM that crashes is obvious. A ZFS pool with a degraded drive or a dataset that is quietly approaching its quota is easy to miss until you need that data. An agent that checks storage health regularly and explains what it finds is genuinely useful -- not just a novelty.

This lesson covers three layers of storage health: pool-level ZFS status, dataset-level space usage, and physical drive health via SMART. You do not need all three every day, but you should know how to ask for each.

ZFS Pool Status

The most important storage command in a ZFS homelab is:

zpool status

With the SSH terminal backend pointed at your TrueNAS or Proxmox+ZFS node, ask your agent:

> check the ZFS pool status and tell me if anything needs attention

The agent will run zpool status and parse the output. Key things it should flag:

• DEGRADED -- pool is running with a failed or removed drive. Data is intact but redundancy is gone. Replace the drive.
• FAULTED -- pool is offline or a drive has failed critically. Immediate attention required.
• ONLINE with errors -- pool is healthy but has accumulated checksum, read, or write errors. Worth investigating even if the pool is functional.
• scan: scrub repaired N errors or scrub in progress -- scrub results tell you whether ZFS found and fixed data errors.

For a quick machine-readable health check without the full tree output:

zpool list -o name,health,capacity,free

This gives you pool name, ONLINE/DEGRADED/FAULTED, percentage used, and free space -- good for a morning summary.

Dataset Space Usage

A healthy pool can still have datasets that are approaching their quota or refquota limits. Check with:

zfs list -o name,used,avail,refer,mountpoint

Ask your agent:

> list all ZFS datasets with their used and available space, and flag any that are more than 80% full

The agent can compare used and avail values to calculate utilization and filter the output to only the datasets that matter. This is much more readable than scanning a long zfs list output yourself.

If you use snapshots heavily, also check snapshot space consumption:

zfs list -t snapshot -o name,used,refer | sort -k2 -h | tail -20

This shows the 20 largest snapshots by space used -- a common source of unexpected storage consumption in homelabs.

Physical Drive Health with SMART

ZFS tells you about pool and dataset health, but it does not tell you about the physical health of the drives underneath. For that you need SMART data. Install smartmontools if it is not already present:

apt install smartmontools  # Debian/Ubuntu

To get a quick health summary for a drive:

smartctl -H /dev/sda

The last line of output is either SMART overall-health self-assessment test result: PASSED or FAILED. Ask your agent to run this for every drive in the system and summarize the results:

> check SMART health for all drives (sda, sdb, sdc...) and tell me if any are failing

For more detail on a specific drive (temperature, reallocated sectors, pending sectors -- the indicators that predict imminent failure):

smartctl -A /dev/sda

The agent can interpret the key attributes: Reallocated_Sector_Ct (non-zero is a warning), Current_Pending_Sector (sectors waiting to be reallocated -- bad), and Uncorrectable_Sector (data loss risk).

TrueNAS REST API

If you are running TrueNAS SCALE, you can query pool and disk status through the REST API instead of SSH shell commands. TrueNAS exposes its management surface at http://truenas-ip/api/v2.0/. Create an API key from TrueNAS UI → Credentials → API Keys (not your root password) and store it in ~/.hermes/.env as TRUENAS_API_KEY. Key endpoints:

# Pool status
curl -H "Authorization: Bearer $TRUENAS_API_KEY" http://truenas/api/v2.0/pool

# Pool dataset usage
curl -H "Authorization: Bearer $TRUENAS_API_KEY" "http://truenas/api/v2.0/pool/dataset?extra.retrieve_children=false"

# Disk list with SMART status
curl -H "Authorization: Bearer $TRUENAS_API_KEY" http://truenas/api/v2.0/disk

Pass TRUENAS_API_KEY via terminal.env_passthrough in config.yaml so the agent can use it as an environment variable without the key appearing in any file it can read.

What to Check and How Often

Not all checks need to run at the same cadence. A sensible schedule for homelab storage monitoring:

• Daily (morning briefing): zpool list health and capacity, top 5 datasets by usage
• Weekly: Full zpool status including scrub results, SMART health for all drives
• After any pool event: Full zpool status -v with error counts, smartctl -A on the affected drive

Lesson 7 shows how to wire the daily check into a scheduled morning briefing cron job.