홈랩 스토리지 성능 측정
홈랩에 구축한 TrueNAS Scale 기반 공유 스토리지 성능 기록용 메모
환경
물리 서버: Dell PowerEdge R7525 (AMD EPYC 7352 * 2)
-> 위 1대 서버에서 Proxmox VE 8.4.1 버전 Hypervisor OS 설치 후 모든 호스트는 VM으로 관리
TrueNAS Scale: 12vCPU + 128GB RAM (버전: ElectricEel-24.10.2.1)
KCD61LUL7T68 U.2 NVMe 7.68TB SSD 5장 RAIDZ1 구성
ZFS Pool 구성: LZ4 압축, 128K Record Size
k8s 워커: 24vCPU + 64GB RAM
측정
k8s 클러스터에서 fio 를 사용해서 성능 측정
https://gitlab.bellsoft.net/devops/volume-benchmark
직접 말아놓은 이미지 사용
과연 iSCSI 로 연결된 PVC를 사용할 때 실제 성능은 어떻게 측정될까?
fio 측정 결과
iops_4k_rand_read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
iops_4k_rand_write: (g=1): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
bw_128k_seq_read: (g=2): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
...
bw_128k_seq_write: (g=3): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
...
db_4k_rand_read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
db_4k_rand_write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fs_8k_rand_rw_50_50: (g=6): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
...
fs_8k_rand_rw_70_30: (g=7): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
...
hp_128k_rand_rw_70_30: (g=8): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=128
...
backup_seq_read: (g=9): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128
...
backup_seq_write: (g=10): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128
...
latency_4k_rand_read: (g=11): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.36
Starting 101 processes
iops_4k_rand_read: Laying out IO file (1 file / 10240MiB)
iops_4k_rand_read: (groupid=0, jobs=4): err= 0: pid=11: Sat Jul 12 14:02:46 2025
read: IOPS=28.2k, BW=110MiB/s (116MB/s)(32.3GiB/300007msec)
slat (usec): min=2, max=80239, avg=17.91, stdev=100.27
clat (usec): min=48, max=191887, avg=4512.45, stdev=3147.47
lat (usec): min=266, max=191898, avg=4530.36, stdev=3149.72
clat percentiles (usec):
| 1.00th=[ 1004], 5.00th=[ 1467], 10.00th=[ 1844], 20.00th=[ 2409],
| 30.00th=[ 2966], 40.00th=[ 3523], 50.00th=[ 4113], 60.00th=[ 4686],
| 70.00th=[ 5276], 80.00th=[ 5932], 90.00th=[ 6980], 95.00th=[ 8455],
| 99.00th=[16057], 99.50th=[20841], 99.90th=[34866], 99.95th=[44303],
| 99.99th=[74974]
bw ( KiB/s): min=39277, max=146528, per=100.00%, avg=113163.03, stdev=4743.59, samples=2388
iops : min= 9818, max=36632, avg=28289.93, stdev=1185.93, samples=2388
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.04%, 750=0.17%
lat (usec) : 1000=0.78%
lat (msec) : 2=11.87%, 4=35.31%, 10=48.76%, 20=2.50%, 50=0.52%
lat (msec) : 100=0.03%, 250=0.01%
cpu : usr=1.91%, sys=14.37%, ctx=2904004, majf=0, minf=146
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=8473914,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
iops_4k_rand_write: (groupid=1, jobs=4): err= 0: pid=15: Sat Jul 12 14:02:46 2025
write: IOPS=20.7k, BW=81.0MiB/s (85.0MB/s)(23.7GiB/300006msec); 0 zone resets
slat (usec): min=2, max=51763, avg=15.03, stdev=110.35
clat (usec): min=69, max=378031, avg=6154.40, stdev=8755.84
lat (usec): min=323, max=378047, avg=6169.43, stdev=8757.45
clat percentiles (usec):
| 1.00th=[ 1156], 5.00th=[ 1565], 10.00th=[ 1926], 20.00th=[ 2573],
| 30.00th=[ 3195], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5080],
| 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 9241], 95.00th=[ 15795],
| 99.00th=[ 47973], 99.50th=[ 65274], 99.90th=[104334], 99.95th=[121111],
| 99.99th=[168821]
bw ( KiB/s): min= 9388, max=143251, per=100.00%, avg=83124.22, stdev=9130.57, samples=2392
iops : min= 2346, max=35810, avg=20780.30, stdev=2282.65, samples=2392
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.26%
lat (msec) : 2=10.97%, 4=31.64%, 10=48.06%, 20=5.44%, 50=2.70%
lat (msec) : 100=0.79%, 250=0.12%, 500=0.01%
cpu : usr=1.81%, sys=8.87%, ctx=2513302, majf=0, minf=145
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6222882,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
bw_128k_seq_read: (groupid=2, jobs=4): err= 0: pid=19: Sat Jul 12 14:02:46 2025
read: IOPS=6900, BW=863MiB/s (904MB/s)(253GiB/300015msec)
slat (usec): min=3, max=109058, avg=22.10, stdev=222.77
clat (usec): min=189, max=195076, avg=18525.78, stdev=6699.53
lat (usec): min=677, max=200014, avg=18547.88, stdev=6705.21
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 18],
| 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 23], 95.00th=[ 27],
| 99.00th=[ 47], 99.50th=[ 59], 99.90th=[ 92], 99.95th=[ 108],
| 99.99th=[ 155]
bw ( KiB/s): min=232688, max=1136350, per=100.00%, avg=884437.81, stdev=36212.27, samples=2392
iops : min= 1815, max= 8876, avg=6908.53, stdev=282.93, samples=2392
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.31%, 20=79.62%, 50=19.23%
lat (msec) : 100=0.74%, 250=0.07%
cpu : usr=0.52%, sys=3.83%, ctx=515679, majf=0, minf=146
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=2070161,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
bw_128k_seq_write: (groupid=3, jobs=4): err= 0: pid=23: Sat Jul 12 14:02:46 2025
write: IOPS=4640, BW=580MiB/s (608MB/s)(170GiB/300015msec); 0 zone resets
slat (usec): min=6, max=108209, avg=29.41, stdev=308.94
clat (usec): min=673, max=340391, avg=27548.73, stdev=22009.50
lat (usec): min=959, max=340412, avg=27578.14, stdev=22012.96
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18],
| 30.00th=[ 19], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23],
| 70.00th=[ 25], 80.00th=[ 29], 90.00th=[ 45], 95.00th=[ 73],
| 99.00th=[ 131], 99.50th=[ 150], 99.90th=[ 192], 99.95th=[ 211],
| 99.99th=[ 243]
bw ( KiB/s): min=112790, max=995284, per=100.00%, avg=594974.10, stdev=55799.30, samples=2392
iops : min= 881, max= 7774, avg=4647.21, stdev=435.94, samples=2392
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.14%, 20=45.17%, 50=46.01%
lat (msec) : 100=6.15%, 250=2.52%, 500=0.01%
cpu : usr=1.35%, sys=2.41%, ctx=333096, majf=0, minf=145
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1392314,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
db_4k_rand_read: (groupid=4, jobs=16): err= 0: pid=27: Sat Jul 12 14:02:46 2025
read: IOPS=26.9k, BW=105MiB/s (110MB/s)(30.8GiB/300009msec)
slat (usec): min=2, max=250571, avg=589.35, stdev=1240.20
clat (usec): min=99, max=424892, avg=37508.59, stdev=14847.38
lat (usec): min=407, max=425378, avg=38097.94, stdev=15012.71
clat percentiles (msec):
| 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 29],
| 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 38],
| 70.00th=[ 41], 80.00th=[ 44], 90.00th=[ 51], 95.00th=[ 60],
| 99.00th=[ 96], 99.50th=[ 117], 99.90th=[ 171], 99.95th=[ 197],
| 99.99th=[ 262]
bw ( KiB/s): min=36653, max=159724, per=100.00%, avg=107662.27, stdev=1135.08, samples=9552
iops : min= 9158, max=39927, avg=26912.09, stdev=283.79, samples=9552
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%, 10=0.21%, 20=2.11%, 50=86.86%
lat (msec) : 100=9.87%, 250=0.86%, 500=0.01%
cpu : usr=0.88%, sys=7.69%, ctx=7826974, majf=0, minf=584
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=8062728,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
db_4k_rand_write: (groupid=5, jobs=16): err= 0: pid=43: Sat Jul 12 14:02:46 2025
write: IOPS=19.9k, BW=77.9MiB/s (81.7MB/s)(22.8GiB/300007msec); 0 zone resets
slat (usec): min=2, max=247487, avg=795.56, stdev=2431.42
clat (usec): min=149, max=899521, avg=50532.86, stdev=52536.06
lat (usec): min=441, max=900780, avg=51328.38, stdev=53212.13
clat percentiles (msec):
| 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 26], 20.00th=[ 29],
| 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 36], 60.00th=[ 39],
| 70.00th=[ 43], 80.00th=[ 50], 90.00th=[ 82], 95.00th=[ 136],
| 99.00th=[ 321], 99.50th=[ 372], 99.90th=[ 472], 99.95th=[ 510],
| 99.99th=[ 600]
bw ( KiB/s): min= 7156, max=143354, per=100.00%, avg=79799.01, stdev=2330.27, samples=9552
iops : min= 1787, max=35835, avg=19946.39, stdev=582.57, samples=9552
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%, 10=0.18%, 20=1.55%, 50=78.72%
lat (msec) : 100=11.86%, 250=5.56%, 500=1.99%, 750=0.06%, 1000=0.01%
cpu : usr=0.79%, sys=5.72%, ctx=5926808, majf=0, minf=585
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,5984198,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
fs_8k_rand_rw_50_50: (groupid=6, jobs=8): err= 0: pid=59: Sat Jul 12 14:02:46 2025
read: IOPS=10.5k, BW=81.8MiB/s (85.8MB/s)(24.0GiB/300005msec)
slat (usec): min=2, max=131610, avg=24.48, stdev=280.39
clat (usec): min=3, max=179705, avg=2802.58, stdev=3954.69
lat (usec): min=297, max=188317, avg=2827.06, stdev=3973.16
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 1123], 10.00th=[ 1270], 20.00th=[ 1500],
| 30.00th=[ 1696], 40.00th=[ 1893], 50.00th=[ 2089], 60.00th=[ 2311],
| 70.00th=[ 2606], 80.00th=[ 3032], 90.00th=[ 4047], 95.00th=[ 5735],
| 99.00th=[ 17695], 99.50th=[ 28443], 99.90th=[ 55313], 99.95th=[ 67634],
| 99.99th=[100140]
bw ( KiB/s): min= 9239, max=127791, per=100.00%, avg=83852.95, stdev=3734.54, samples=4776
iops : min= 1153, max=15973, avg=10479.75, stdev=466.86, samples=4776
write: IOPS=10.5k, BW=81.7MiB/s (85.7MB/s)(23.9GiB/300005msec); 0 zone resets
slat (usec): min=2, max=86800, avg=25.89, stdev=253.27
clat (usec): min=58, max=525143, avg=9375.68, stdev=11050.66
lat (usec): min=273, max=525161, avg=9401.57, stdev=11056.46
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 2057], 10.00th=[ 2704], 20.00th=[ 3949],
| 30.00th=[ 5211], 40.00th=[ 6390], 50.00th=[ 7635], 60.00th=[ 8848],
| 70.00th=[ 10159], 80.00th=[ 11600], 90.00th=[ 14222], 95.00th=[ 20317],
| 99.00th=[ 55313], 99.50th=[ 81265], 99.90th=[137364], 99.95th=[158335],
| 99.99th=[244319]
bw ( KiB/s): min= 9845, max=123872, per=100.00%, avg=83780.10, stdev=3705.21, samples=4776
iops : min= 1229, max=15484, avg=10470.69, stdev=463.20, samples=4776
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.06%, 750=0.31%, 1000=1.00%
lat (msec) : 2=23.72%, 4=30.00%, 10=28.30%, 20=13.61%, 50=2.33%
lat (msec) : 100=0.50%, 250=0.15%, 500=0.01%, 750=0.01%
cpu : usr=1.27%, sys=7.19%, ctx=3972241, majf=0, minf=293
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3141192,3138506,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
fs_8k_rand_rw_70_30: (groupid=7, jobs=8): err= 0: pid=67: Sat Jul 12 14:02:46 2025
read: IOPS=15.6k, BW=122MiB/s (128MB/s)(35.8GiB/300034msec)
slat (usec): min=2, max=113949, avg=26.25, stdev=299.16
clat (usec): min=9, max=344990, avg=5532.25, stdev=6096.07
lat (usec): min=296, max=345060, avg=5558.51, stdev=6109.33
clat percentiles (usec):
| 1.00th=[ 1090], 5.00th=[ 1582], 10.00th=[ 2008], 20.00th=[ 2638],
| 30.00th=[ 3294], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5211],
| 70.00th=[ 5997], 80.00th=[ 6849], 90.00th=[ 8455], 95.00th=[ 11076],
| 99.00th=[ 27395], 99.50th=[ 42206], 99.90th=[ 84411], 99.95th=[100140],
| 99.99th=[141558]
bw ( KiB/s): min=16236, max=175720, per=100.00%, avg=125316.79, stdev=4549.75, samples=4784
iops : min= 2027, max=21963, avg=15662.61, stdev=568.73, samples=4784
write: IOPS=6705, BW=52.4MiB/s (54.9MB/s)(15.3GiB/300034msec); 0 zone resets
slat (usec): min=2, max=91750, avg=27.72, stdev=293.37
clat (usec): min=16, max=507982, avg=6091.30, stdev=7287.95
lat (usec): min=319, max=508001, avg=6119.03, stdev=7298.90
clat percentiles (usec):
| 1.00th=[ 1156], 5.00th=[ 1614], 10.00th=[ 2040], 20.00th=[ 2704],
| 30.00th=[ 3326], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5473],
| 70.00th=[ 6390], 80.00th=[ 7635], 90.00th=[ 10028], 95.00th=[ 13829],
| 99.00th=[ 32113], 99.50th=[ 47973], 99.90th=[ 93848], 99.95th=[117965],
| 99.99th=[185598]
bw ( KiB/s): min= 6766, max=77186, per=100.00%, avg=53724.90, stdev=1968.19, samples=4784
iops : min= 845, max= 9646, avg=6713.81, stdev=246.07, samples=4784
lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.03%, 750=0.14%, 1000=0.42%
lat (msec) : 2=9.27%, 4=30.97%, 10=51.78%, 20=5.39%, 50=1.59%
lat (msec) : 100=0.34%, 250=0.06%, 500=0.01%, 750=0.01%
cpu : usr=1.21%, sys=7.95%, ctx=3971717, majf=0, minf=292
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4692645,2011889,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
hp_128k_rand_rw_70_30: (groupid=8, jobs=32): err= 0: pid=75: Sat Jul 12 14:02:46 2025
read: IOPS=4018, BW=503MiB/s (528MB/s)(148GiB/300066msec)
slat (usec): min=6, max=292516, avg=5504.88, stdev=8929.22
clat (usec): min=1540, max=2165.1k, avg=711916.36, stdev=239278.00
lat (usec): min=1552, max=2165.1k, avg=717419.04, stdev=240328.01
clat percentiles (msec):
| 1.00th=[ 409], 5.00th=[ 468], 10.00th=[ 493], 20.00th=[ 531],
| 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 667],
| 70.00th=[ 760], 80.00th=[ 927], 90.00th=[ 1099], 95.00th=[ 1200],
| 99.00th=[ 1385], 99.50th=[ 1485], 99.90th=[ 1720], 99.95th=[ 1787],
| 99.99th=[ 1938]
bw ( KiB/s): min=60810, max=1184589, per=99.83%, avg=514674.27, stdev=5741.96, samples=19104
iops : min= 462, max= 9243, avg=4015.70, stdev=44.84, samples=19104
write: IOPS=1721, BW=216MiB/s (226MB/s)(63.2GiB/300066msec); 0 zone resets
slat (usec): min=10, max=359814, avg=5709.73, stdev=9138.18
clat (msec): min=7, max=2046, avg=695.67, stdev=235.74
lat (msec): min=7, max=2053, avg=701.37, stdev=236.89
clat percentiles (msec):
| 1.00th=[ 401], 5.00th=[ 456], 10.00th=[ 485], 20.00th=[ 518],
| 30.00th=[ 550], 40.00th=[ 575], 50.00th=[ 609], 60.00th=[ 651],
| 70.00th=[ 735], 80.00th=[ 911], 90.00th=[ 1070], 95.00th=[ 1183],
| 99.00th=[ 1368], 99.50th=[ 1469], 99.90th=[ 1687], 99.95th=[ 1754],
| 99.99th=[ 1905]
bw ( KiB/s): min=17237, max=545509, per=99.83%, avg=220475.23, stdev=2664.79, samples=19101
iops : min= 126, max= 4255, avg=1719.22, stdev=20.82, samples=19101
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.06%, 100=0.07%
lat (msec) : 250=0.16%, 500=11.90%, 750=58.01%, 1000=14.67%, 2000=15.36%
lat (msec) : >=2000=0.01%
cpu : usr=0.23%, sys=1.06%, ctx=1782384, majf=0, minf=1164
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=1205743,516480,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
backup_seq_read: (groupid=9, jobs=2): err= 0: pid=107: Sat Jul 12 14:02:46 2025
read: IOPS=926, BW=927MiB/s (972MB/s)(272GiB/300289msec)
slat (usec): min=17, max=110350, avg=106.94, stdev=480.04
clat (msec): min=166, max=843, avg=276.11, stdev=57.29
lat (msec): min=167, max=843, avg=276.22, stdev=57.30
clat percentiles (msec):
| 1.00th=[ 211], 5.00th=[ 224], 10.00th=[ 230], 20.00th=[ 241],
| 30.00th=[ 247], 40.00th=[ 255], 50.00th=[ 262], 60.00th=[ 271],
| 70.00th=[ 284], 80.00th=[ 296], 90.00th=[ 334], 95.00th=[ 380],
| 99.00th=[ 531], 99.50th=[ 567], 99.90th=[ 651], 99.95th=[ 701],
| 99.99th=[ 827]
bw ( KiB/s): min=244132, max=1377602, per=100.00%, avg=950475.02, stdev=88416.93, samples=1198
iops : min= 237, max= 1345, avg=927.70, stdev=86.48, samples=1198
lat (msec) : 250=34.37%, 500=64.20%, 750=1.49%, 1000=0.04%
cpu : usr=0.17%, sys=4.71%, ctx=37284, majf=0, minf=72
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=278206,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
backup_seq_write: (groupid=10, jobs=2): err= 0: pid=109: Sat Jul 12 14:02:46 2025
write: IOPS=642, BW=644MiB/s (675MB/s)(189GiB/300289msec); 0 zone resets
slat (usec): min=40, max=85335, avg=139.83, stdev=533.79
clat (msec): min=31, max=1776, avg=397.74, stdev=195.36
lat (msec): min=31, max=1798, avg=397.88, stdev=195.37
clat percentiles (msec):
| 1.00th=[ 146], 5.00th=[ 197], 10.00th=[ 224], 20.00th=[ 255],
| 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 326], 60.00th=[ 355],
| 70.00th=[ 409], 80.00th=[ 567], 90.00th=[ 735], 95.00th=[ 802],
| 99.00th=[ 953], 99.50th=[ 1003], 99.90th=[ 1116], 99.95th=[ 1183],
| 99.99th=[ 1502]
bw ( KiB/s): min= 6123, max=1197201, per=100.00%, avg=661196.59, stdev=136296.11, samples=1194
iops : min= 5, max= 1169, avg=645.05, stdev=133.03, samples=1194
lat (msec) : 50=0.03%, 100=0.15%, 250=17.70%, 500=59.10%, 750=14.46%
lat (msec) : 1000=8.19%, 2000=0.52%
cpu : usr=1.81%, sys=2.57%, ctx=30087, majf=0, minf=74
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,193046,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
latency_4k_rand_read: (groupid=11, jobs=1): err= 0: pid=111: Sat Jul 12 14:02:46 2025
read: IOPS=3081, BW=12.0MiB/s (12.6MB/s)(3612MiB/300001msec)
slat (usec): min=4, max=48349, avg=22.29, stdev=148.37
clat (nsec): min=1753, max=49977k, avg=300022.87, stdev=305288.51
lat (usec): min=169, max=50099, avg=322.31, stdev=341.87
clat percentiles (usec):
| 1.00th=[ 198], 5.00th=[ 215], 10.00th=[ 223], 20.00th=[ 235],
| 30.00th=[ 245], 40.00th=[ 255], 50.00th=[ 265], 60.00th=[ 277],
| 70.00th=[ 293], 80.00th=[ 314], 90.00th=[ 363], 95.00th=[ 437],
| 99.00th=[ 791], 99.50th=[ 1303], 99.90th=[ 3916], 99.95th=[ 5145],
| 99.99th=[10945]
bw ( KiB/s): min= 3040, max=15880, per=100.00%, avg=12343.11, stdev=2376.90, samples=599
iops : min= 760, max= 3970, avg=3085.71, stdev=594.27, samples=599
lat (usec) : 2=0.01%, 4=0.04%, 10=0.04%, 20=0.02%, 50=0.01%
lat (usec) : 100=0.01%, 250=34.76%, 500=62.06%, 750=1.99%, 1000=0.38%
lat (msec) : 2=0.39%, 4=0.22%, 10=0.08%, 20=0.01%, 50=0.01%
cpu : usr=1.65%, sys=9.22%, ctx=928837, majf=0, minf=36
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=924585,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=110MiB/s (116MB/s), 110MiB/s-110MiB/s (116MB/s-116MB/s), io=32.3GiB (34.7GB), run=300007-300007msec
Run status group 1 (all jobs):
WRITE: bw=81.0MiB/s (85.0MB/s), 81.0MiB/s-81.0MiB/s (85.0MB/s-85.0MB/s), io=23.7GiB (25.5GB), run=300006-300006msec
Run status group 2 (all jobs):
READ: bw=863MiB/s (904MB/s), 863MiB/s-863MiB/s (904MB/s-904MB/s), io=253GiB (271GB), run=300015-300015msec
Run status group 3 (all jobs):
WRITE: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=170GiB (183GB), run=300015-300015msec
Run status group 4 (all jobs):
READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=30.8GiB (33.0GB), run=300009-300009msec
Run status group 5 (all jobs):
WRITE: bw=77.9MiB/s (81.7MB/s), 77.9MiB/s-77.9MiB/s (81.7MB/s-81.7MB/s), io=22.8GiB (24.5GB), run=300007-300007msec
Run status group 6 (all jobs):
READ: bw=81.8MiB/s (85.8MB/s), 81.8MiB/s-81.8MiB/s (85.8MB/s-85.8MB/s), io=24.0GiB (25.7GB), run=300005-300005msec
WRITE: bw=81.7MiB/s (85.7MB/s), 81.7MiB/s-81.7MiB/s (85.7MB/s-85.7MB/s), io=23.9GiB (25.7GB), run=300005-300005msec
Run status group 7 (all jobs):
READ: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=35.8GiB (38.4GB), run=300034-300034msec
WRITE: bw=52.4MiB/s (54.9MB/s), 52.4MiB/s-52.4MiB/s (54.9MB/s-54.9MB/s), io=15.3GiB (16.5GB), run=300034-300034msec
Run status group 8 (all jobs):
READ: bw=503MiB/s (528MB/s), 503MiB/s-503MiB/s (528MB/s-528MB/s), io=148GiB (158GB), run=300066-300066msec
WRITE: bw=216MiB/s (226MB/s), 216MiB/s-216MiB/s (226MB/s-226MB/s), io=63.2GiB (67.9GB), run=300066-300066msec
Run status group 9 (all jobs):
READ: bw=927MiB/s (972MB/s), 927MiB/s-927MiB/s (972MB/s-972MB/s), io=272GiB (292GB), run=300289-300289msec
Run status group 10 (all jobs):
WRITE: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=189GiB (203GB), run=300289-300289msec
Run status group 11 (all jobs):
READ: bw=12.0MiB/s (12.6MB/s), 12.0MiB/s-12.0MiB/s (12.6MB/s-12.6MB/s), io=3612MiB (3787MB), run=300001-300001msec
Disk stats (read/write):
sdm: ios=27962526/18936904, sectors=1727460128/1104830592, merge=1805400/1198704, ticks=218200447/184577681, in_queue=402816802, util=98.57%
요약
4K 랜덤 읽기 지연시간 P99: 791usec
4K 랜덤 읽기 IOPS: 28.2k
4K 랜덤 쓰기 IOPS: 20.7K
순차 읽기: 863 MiB/s
순차 쓰기: 580 MiB/s
문제 확인
TrueNAS Scale 과 k8s 워커노드가 하나의 물리 장비에 VM으로 떠있는 상태에셔 네트워크 계층은 virtio vNIC를 사용하는 상태
오버헤드가 좀 있어서 그런가 동일 장비여도 성능이 좀 애매하네..
4k IOPS는 ZFS 쓰기 증폭을 감안한다 해도 128K 순차 읽기/쓰기 성능은 좀 심각한듯
NVMe SSD 온도로 인한 스로틀링은 아니고
워커노드 CPU 성능 문제?
CPU iowait 시간이 최대 62%라..
SSD 공식 스펙 상으로는 순차 읽기가 6GB/s, 순차 쓰기가 4GB/s 인데.. 흠…
해당 SSD 구매 직후 개별 SSD 단위로 성능 측정했어야 됐는데 깜빡해버렸더니..
일단 각종 네트워크 오버헤드로 인해 성능이 떨어지는 것 같아서 단계적으로 개선 작업 진행해봐야 할듯
TrueNAS Scale VM에 할당된 vNIC를 virtio 기반 반가상화 인터페이스가 아닌, 물리 NIC에 SR-IOV 활성화 후 PCIe 패스스루로 하드웨어 오프로드 활성화 구성
k8s 워커노드도 마찬가지로 SR-IOV 활성화
Views: 2
