ZFS Performance Tuning

Understanding ZFS Caching

ARC (Adaptive Replacement Cache)

The ARC is ZFS's primary read cache in RAM.

# View ARC statistics
sysctl kstat.zfs.misc.arcstats

# Key metrics
sysctl kstat.zfs.misc.arcstats.size  # Current size
sysctl kstat.zfs.misc.arcstats.hits  # Cache hits
sysctl kstat.zfs.misc.arcstats.misses  # Cache misses

# ARC hit ratio
# hits / (hits + misses) * 100 = hit ratio %

Tuning ARC Size

# View current ARC limits
sysctl vfs.zfs.arc_min
sysctl vfs.zfs.arc_max

# Set ARC max (e.g., 8GB)
# In /boot/loader.conf:
vfs.zfs.arc_max="8G"

# For systems with limited RAM, reduce ARC
vfs.zfs.arc_max="1G"

# Minimum ARC size
vfs.zfs.arc_min="512M"

L2ARC (Level 2 ARC)

Secondary read cache on fast storage (SSD/NVMe).

# Add L2ARC device
zpool add tank cache /dev/nvd0

# View L2ARC stats
sysctl kstat.zfs.misc.arcstats.l2_hits
sysctl kstat.zfs.misc.arcstats.l2_misses

# Tune L2ARC write speed
# /boot/loader.conf:
vfs.zfs.l2arc_write_max="67108864"  # 64MB/s
vfs.zfs.l2arc_write_boost="134217728"  # 128MB/s initial

SLOG (ZIL - ZFS Intent Log)

Write cache for synchronous writes.

# Add dedicated SLOG (mirrored recommended)
zpool add tank log mirror /dev/nvd0 /dev/nvd1

# Check ZIL stats
sysctl kstat.zfs.misc.zil

# When to use SLOG:
# - Database servers
# - NFS servers with sync exports
# - VMs with sync I/O

Record Size Optimization

# Default recordsize is 128K

# Large files (media, backups)
zfs set recordsize=1M tank/media

# Databases
zfs set recordsize=16K tank/mysql
zfs set recordsize=32K tank/postgresql

# General purpose
zfs set recordsize=128K tank/data

# Small files (VMs)
zfs set recordsize=64K tank/vms

Compression

# LZ4 - Fast, good compression (recommended default)
zfs set compression=lz4 tank

# ZSTD - Better compression, more CPU
zfs set compression=zstd tank/archive
zfs set compression=zstd-3 tank/data  # Level 1-19

# GZIP - Maximum compression, slow
zfs set compression=gzip-9 tank/cold-storage

# Check compression ratio
zfs get compressratio tank

Deduplication (Use Carefully!)

# WARNING: Dedup requires ~5GB RAM per 1TB of data

# Enable dedup
zfs set dedup=on tank/backup

# Check dedup ratio
zfs get dedupratio tank

# Verify DDT (dedup table) size
zpool status -D tank

Loader Tuning (/boot/loader.conf)

# ARC size
vfs.zfs.arc_max="8G"
vfs.zfs.arc_min="1G"

# Prefetch tuning
vfs.zfs.prefetch_disable="0"

# TXG (transaction group) timeout
vfs.zfs.txg.timeout="5"

# Scrub/resilver speed
vfs.zfs.scrub_delay="0"
vfs.zfs.resilver_delay="0"

# ARC metadata limit
vfs.zfs.arc_meta_limit="2G"

# Async writes
vfs.zfs.vdev.async_write_active_min_dirty_percent="30"

Pool Layout Best Practices

# RAID-Z stripe width
# - RAID-Z1: 3-5 disks per vdev
# - RAID-Z2: 4-8 disks per vdev
# - RAID-Z3: 5-9 disks per vdev

# Multiple vdevs for performance
zpool create tank \
    raidz2 da0 da1 da2 da3 da4 da5 \
    raidz2 da6 da7 da8 da9 da10 da11

# Use ashift=12 for 4K sector drives
zpool create -o ashift=12 tank mirror da0 da1

Monitoring Performance

# Real-time I/O stats
zpool iostat -v 1

# Latency histograms
zpool iostat -lq 1

# Detailed ARC stats
sysctl kstat.zfs.misc.arcstats | less

# Per-dataset I/O
zpool iostat -v tank 1

Performance Checklist

1. ARC sized appropriately for available RAM
2. L2ARC on SSD if ARC insufficient
3. SLOG for sync-heavy workloads
4. Compression enabled (lz4 default)
5. atime=off unless needed
6. Correct recordsize for workload
7. Multiple vdevs for parallel I/O
8. Avoid dedup unless you understand requirements