slow spa_sync on reboot

[Richie111]

Hi

I have a freenas box build on a 60 drive storinator.
2 E5260 procs
128 GB ram
2 rocket 750 HBA cards.

the system boots off 2 ssd drives.

for storage there are 60 WD enterprise 6 TB drives

there are 2 zvol's configured each for 3 raidz2's in them

they are presented to a windows hyperv server through 10 gb iscsi

on one of the volumes we were running out of space, i deleted a 45 TB vhdx file but it didn't seem to reclaim all the space. on reboot it is trying to find the zvol and all i get is

I let it run for 48 hours and then rebooted again. it has been about 36 hours since that reboot.

any thoughts would be greatly appreciated.

thanks

--==Richie

 

[jgreco]

Um. Wow. That sounds like a disaster.

I can maybe give you some idea as to what's happened. RAIDZ2 is extremely bad for block storage. When you write a block to a virtual disk, what ends up happening is that RAIDZ2 has to allocate a new, small, block of space and write that data into it. This isn't very efficient in terms of space or performance.

You had three RAIDZ2 vdevs. The IOPS performance of a vdev is approximately going to be that of one of the underlying devices. You essentially have three 48TB vdevs making up each pool. That's like three hard drives worth of IOPS performance.

So now you take and delete a 45TB vhdx file. Since you seem to think that's "to free up some space", rather than "this is big enough that it should be the only thing on this pool", I'm going to guess that you filled the pool way past the 30-50% point, which caused a huge amount of fragmentation to occur.

This means that in order to delete that file, ZFS has to trace down all the fragments and do a head seek for each one. So. Now we get into a guessing game.

If your entire vhdx file is somewhat lightly fragmented, let's say that we can assume on average that there are 1MB contiguous chunks, 45TB/1MB = 45,000,000 fragments that ZFS has to free. If we assign 1 IOPS per fragment to free, that means 45,000,000 IO operations. Your average hard drive can do ~150 IOPS. You have three vdevs, so that might be able to clear ~450 IOPS if you're lucky. That means that the operation would complete in 100,000 seconds.

See how that works? But the problem you're probably facing is that the fragmentation on your pool is a LOT higher than that. So it could easily take a week, maybe even a month, for the operation to complete, depending on how bad it was.

So here's the thing. You made several mistakes.

1) You didn't let ZFS finish an atomic operation (the removal). By resetting the machine, you forced ZFS to resolve this issue upon reboot. The system will not be able to complete the import until the operation is completed.

2) You probably filled a pool you were using for block storage much fuller than the recommended 25-50%. The emptier it is, the less it fragments.

3) You used RAIDZ2 which is exceedingly horrible at block storage. You need mirrors. You can go with three-way mirrors if the single redundancy of two-way mirrors is insufficient.

4) Doubling (quadrupling?) the RAM and adding some L2ARC might help give you some better performance, given the scale of your storage system.

 

[Richie111]

Hi

Thank you for the response. you are correct that the zpool is far fuller than it should be, hence my problem. there is a 500 gb ssd as a l2arc

the performance is horrible, I (now) realize that is because the iops are limited to a single device in the vdev. the l2arc does not help much as these are extremely large files that are not often reread so we are mostly reading from disk not cache. (when I do read or write to cache i get 1 GB per second, otherwise i get more like 10-12 MB)

There is another zpool on here besides the one that I think is causing me trouble. Is there a way to delete the one I messed up so at least the other one comes up?

if I redo the pools as mirrors, would I still have a limitation of 50% utilization?
If my app would be able to use SMB instead of needing block access what would be the max utilization I would want under those circumstances?

thank you again for your assistance.

 

[jgreco]

The L2ARC is of course dependent on usage patterns, but even if it is just caching useful metadata, that's helpful... and you lost it when you rebooted.

You can drop the busted pool, yes, but I'm hesitant to rattle off any incantations off the top of my head, and that's all I have time for right now, sorry. It's totally possible in theory to get the machine and its second pool up and running and then try to import the first pool (which has an unknown amount-of-time to import, so that could be danger-y).

The 50% "limit" is a side effect of how fragmentation affects performance of pools for random writes.

That's what happens to what is effectively the worst case performance of a single disk pool. A single disk has maybe ~150-200 IOPS, right? So if you look carefully, you'll notice that at 10% occupancy, the disk *appears* to have 6000 write IOPS. This is because ZFS can easily lay down most random writes as contiguous blocks. The downside is that both random and sequential writes share the same performance fate; a sequential write on a highly fragmented (50%+ full) pool will tend to have difficulty finding long runs of sequential blocks to write to, so both sequential and random tend to degrade. It is important to note that this is performance over time - on a fresh pool, everything's always fast, so benchmarks on ZFS are deceptive.

Using SMB has the potential to be better for ZFS because ZFS understands the structure of your data in that case, and isn't trying to make guesses.

https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/

 

[Richie111]

Hi

It finished!

on the console it says the following:

Does that mean its still not so happy?
Is there a way to determine if it will do the same thing again on reboot?

thank you again for your help.

 

[jgreco]

Quite frankly, I'm not sure. I've never seen that before. Based on what it is saying, it seems to me like it could very possibly be some sort of corruption, but it could also be that zfsd didn't manage to retrieve the data within an expected timeframe. That's just a few random thoughts though.

If I were you, I would identify which pools those devices were components of, and then see what their health status was. It might also be possible to kill off zfsd and hopefully the system would automatically restart it. I just tried that on one of the filers here and it seems to. My suggestion would be to try that and then see what happens, and report back here.

So it could easily take a week, maybe even a month, for the operation to complete,

Anyways, congratulations, I think you dodged a bit of a bullet there. Please please PLEASE take away some useful lessons from this horrifying experience. It's very depressing to be the guy who has to break the bad news. ZFS can be so awesome but it can also really suck.

 

[cyberjock]

Hi

It finished!

on the console it says the following:

View attachment 11176

Does that mean its still not so happy?
Is there a way to determine if it will do the same thing again on reboot?

thank you again for your help.

Those messages are because you have spare disks in a zpool somewhere. They are normal and can be ignored (unless you don't have spare disks, in which case you have a problem that I can't even think up a scenario where it would be possible).

 

[jgreco]

Cyberjock is correct. Now that I know what causes it, I see that for the spares here too. That's a pretty crappy message to emit for a harmless event.

 

[Richie111]

Thank you. looking at the drive names, they are SSD'd that have not been assigned. It has been working, I am afraid to reboot it again though.

Would you have any rough idea as to what performance I should expect on read and write once I have filled up cache? each zpool seems to max out at under 100 MB per second. I have a (small) pool that is mirrored for testing and performance there is not much faster.

We spend upwards of 40k on each of these boxes (plus the 10 gbe networking) and I would love to get more disk IO from it.

your help is much appreciated.

 

[jgreco]

That configuration is never going to be stellar. You have three RAIDZ2 vdevs per pool. The IOPS performance of a vdev is approximately going to be that of one of the underlying devices. You essentially have three 48TB vdevs making up each pool. That's like three hard drives worth of IOPS performance for a HUGE amount of storage.

Write speeds are impacted by the number of vdevs, the amount of free space on the pool, and the fragmentation on the pool. With ZFS, both what-you-would-think-of-as-random and what-you-would-think-of-as-sequential writes are handled in a similar manner: the system looks for a contiguous run of blocks to stick them in. Random writes can be transformed into something that reduce seeks, while sequential writes may not be allocated contiguously. Keeping large amounts of free space on the pool is a turbocharger for writes, because fragmentation is usually low when there's lots of free space, because it is easy to find an appropriately sized allocation.

Read speeds are dependent on the nature of the read traffic. Absent some sort of mitigation strategy, the disks have to seek around to get data. Since even "contiguous" VM disk file data may not in fact be contiguous, fetching from the pool is bad. The greater the fragmentation of the pool, the slower the reads will be. Free space on the pool helps to mitigate that somewhat, because it offers an increased chance for writes to be grouped reasonably. But the biggest mitigation is ARC, and L2ARC... gobs of it.

So the best "fix" I can suggest is that you tear apart the RAIDZ2's and replace them with mirror vdevs. And make sure you're keeping pool occupancy low, like 30-40%. 60 6TB drives would get you a 180TB pool. which implies you should probably shoot for 256GB of RAM and only using maybe 60-80TB of the pool capacity. Then add lots of L2ARC - at 256GB of RAM you should have no problem doing 1TB of L2ARC, maybe more.

There may be things specific to Hyper-V that would help, I don't know.

 

[Richie111]

Hi

I do have one mirror vdev/pool (6 4 tb drives) that is almost empty. read speed from it is also in the 80-90 MB range.

I have an open (now escalated) ticket with microsoft to see if its anything on their side.

I also mapped an smb share to the same mirror and can 400 MB write (probably due to cache) and 250 on read (not sure if this is still in RAM a day later when I tried to read it)

thanks

 

[jgreco]

What are the speeds like if you're working on it directly from the NAS CLI?

 

[Richie111]

How would I test ?

 

[Richie111]

This is iozone from the cli to the (mirror) CIFS share, i am afraid to write to the iscsi LUN


The top row is records sizes, the left column is file sizes
Writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 179338 399573 651849 1232453 1828508
128 163490 348661 526444 942199 1827299 2027414
256 219184 383917 705107 1062263 1718521 2692393 2533571
512 294748 516743 924123 1302447 1764853 2860238 2811557 2642062
1024 361048 630518 898941 1221964 2064088 2844715 2960401 2859868 2645000
2048 354197 598818 1063399 1513696 1853353 2612664 2712491 3034448 2619037 2687877
4096 372561 693301 1069210 1494316 1830994 2741407 2938334 2769693 3086681 2563468 2370354
8192 421856 644686 1115945 1347345 1741867 2550303 3063510 3010630 2861211 2774122 1717059 2343166
16384 352639 696543 1113338 1454263 1863942 3092991 2893109 3018937 2945692 2808796 2227044 2499816 2404476
32768 0 0 0 0 1860027 2139200 3164404 3186265 3062378 2703000 1921099 2367604 2434495
65536 0 0 0 0 1542387 2870103 2886014 2913885 3028897 3005811 2602078 2303475 2508083
131072 0 0 0 0 1688248 2536174 2343836 2794116 2613916 2190962 2240008 1953473 1967159
262144 0 0 0 0 1500401 2209443 2391080 2419929 1771963 2202371 1894459 1616028 1964863
524288 0 0 0 0 1250925 2310779 2094302 2162020 2203936 1989798 2042193 1635094 1898590
Re-writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 500060 1033216 1679761 2662899 2203800
128 403721 976473 1642783 3199360 1755594 2453642
256 421603 880256 1075025 3017775 2131261 2665657 2639446
512 522399 1026126 1815584 3182373 2307123 3122224 3527371 3304808
1024 423324 819837 1640674 1875690 2774874 3142339 3532609 2694787 2730767
2048 498782 825415 1587551 2656292 2589040 2888532 3185225 3180508 2798201 2805512
4096 429488 828792 1607427 2006805 2406546 3105091 2663212 3155856 2880206 2760347 2469150
8192 476829 836944 1328179 1981095 2048050 2369832 3182693 2990974 2829637 2944580 2335204 2328557
16384 446721 814995 1394135 1402758 2729692 3099408 2689838 3023985 2954176 2295024 2628518 2463345 1911380
32768 0 0 0 0 2477368 2727027 2307269 2600852 2287301 2056090 1898071 1829596 1784949
65536 0 0 0 0 1953856 2605704 2717989 2794104 2608177 2538588 2224204 1922387 2300237
131072 0 0 0 0 1976687 2390525 2215508 2579998 2172829 2074543 1884682 1948971 1996586
262144 0 0 0 0 1709727 1798265 2328557 2748588 2029481 2207806 1692462 1831765 1859164
524288 0 0 0 0 2011054 2799414 2073842 2292500 2317735 2318195 2043379 1787780 1798754
Reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1460430 2662899 4274062 6421025 9318832
128 1363288 1539168 4012317 6406138 5325799 11470204
256 794889 2247235 1842366 5938650 5569035 11207494 12812277
512 1201856 2665014 4195896 5684095 3941742 5019764 6174377 10485468
1024 1216082 2497356 1701108 5593820 4574926 4805258 4415030 5507740 8750850
2048 1421279 2104960 2752473 4707314 3630870 4096676 4071433 5832361 5345969 5095447
4096 1039641 1934058 3202328 3552638 3483484 4188367 3781775 4280194 4592558 4175134 2940345
8192 1247231 1848808 2446089 3159282 3354524 3204958 3206454 4211368 3932757 3358786 3285873 3163062
16384 1260588 1945140 2553222 3008890 3715207 3927095 4222704 4006778 3951937 3078580 3002974 2810864 2829265
32768 0 0 0 0 3457600 3719411 4095588 4215685 2752919 3772499 3059855 2754795 2894194
65536 0 0 0 0 3584328 2792572 3054307 3528563 3232686 3049901 2338653 2540465 2516740
131072 0 0 0 0 3286248 3752425 3946282 3599107 3668810 3734581 2982635 2919980 2701399
262144 0 0 0 0 3305769 3547283 3466317 3699881 3816889 3330381 2364516 2842343 2956266
524288 0 0 0 0 3276940 3533011 3647120 3893132 3470906 3489018 3005288 2684030 2329837
Re-reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1562436 2923952 4564786 8182586 9318832
128 1421023 2660335 4717434 7582312 9795896 10567140
256 1012194 2639446 3043436 7120034 8815202 12228612 12812277
512 1589779 2796910 4732165 5886650 9510316 10911694 12499490 12499490
1024 1539522 2798378 2188177 6776473 9240349 10796646 4966395 11773301 9142007
2048 1132220 2458618 3848872 5521212 5704543 7761014 5832361 8949659 6189580 5611381
4096 1198388 2205712 3663236 4622213 5192166 5777065 4911613 6263611 5986422 4801789 3699523
8192 1233175 2003389 2753668 3339526 3686744 3837016 3070354 4710810 4548665 3790873 3592684 2824984
16384 1258672 1911114 2741125 2907062 3740689 4100254 4255654 4245663 4097077 3136328 3254563 2602635 2842489
32768 0 0 0 0 3609139 3907867 4231389 4347572 3864247 3980529 3344680 2804544 3115130
65536 0 0 0 0 3684255 3736794 3794569 3670431 3850542 3539239 3218379 2590870 2399259
131072 0 0 0 0 3315340 3941415 3696339 4086923 3627989 3489210 3021203 1820288 2786271
262144 0 0 0 0 3599246 3816108 3843588 4132666 3641143 3591698 2483931 2739882 2975659
524288 0 0 0 0 3498098 3870771 3920996 4020888 3743070 3444327 3071202 2683058 2609476
Random Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1392258 2561267 3958892 6421025 9318832
128 969421 2367096 4267461 6406138 6727225 10567140
256 770920 1610276 2812272 6249745 7965107 11091721 12228612
512 1267097 2548017 4305250 6156675 8701110 9859630 11877300 10235583
1024 1369216 2547723 3684119 6479979 8988941 10454984 9142007 10878686 8457895
2048 1325646 2228948 3969827 5655716 6967792 8531869 5032754 8292998 6649556 5507054
4096 1276003 1826906 3105091 4267436 5312585 6847827 5902100 7434579 6169144 4592558 3565171
8192 1179700 1941688 2472847 3656923 4122934 3639106 4528283 4862131 4531269 4065372 3394960 2655351
16384 1135337 1811562 2589883 3058439 3792715 4124866 4295556 4027442 3658053 3069779 3121937 2827751 2584235
32768 0 0 0 0 3525268 3781217 4258925 4284549 3828403 3913097 3385462 2749724 3121356
65536 0 0 0 0 3580220 3672098 3749844 3773162 4001960 3744481 3162795 2683363 2800310
131072 0 0 0 0 3401453 3761668 4126031 3997066 3674572 3636845 3012116 2755574 2772137
262144 0 0 0 0 3182868 3829665 3855232 3888280 3700292 3377357 1781994 2631702 2524306
524288 0 0 0 0 3322270 3746252 3780314 3785859 3801803 3423473 2959687 2694000 2703170
Random Write Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 537079 752329 1879725 3057153 2923952
128 447464 883293 1705404 3124872 2558895 3657016
256 272605 842278 1053922 3159869 3159869 4652146 3667079
512 408281 953248 1625889 2894940 3304808 2975154 4271001 3822466
1024 472347 978841 1376236 2854167 3294177 4048778 3046495 4783849 3659010
2048 495273 900997 1651654 2848305 2698856 4046500 3255235 3717289 3117032 3112514
4096 455677 771096 1369480 2420789 2782703 3524939 2860067 3771811 3330202 3141429 2696655
8192 394419 806305 1223036 2116815 2822895 2777935 3142235 3268991 3124235 2814110 2438797 2244442
16384 447844 825302 1296207 1875234 2566764 3389235 3398453 2932993 2963348 2645518 2424753 2290816 2203337
32768 0 0 0 0 2565604 3043391 3307255 3407368 3025903 2979460 2310177 2398677 2330587
65536 0 0 0 0 1677273 2900602 2991842 3057160 3076665 2678108 2467290 2197426 2376997
131072 0 0 0 0 1887283 2312611 2299388 2732659 2252333 2377766 2053198 1821476 2105401
262144 0 0 0 0 1696691 2694677 2341449 2812461 2000884 2390534 1506314 1728977 1830497
524288 0 0 0 0 1546239 2502413 2361933 2280901 2382608 2078140 1945372 1861322 1761081
Backward Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1232453 2133730 3363612 4564786 2892445
128 1266785 2248149 3657016 5122535 2727923 3445772
256 1195962 2170027 2152625 4553502 3087188 3511189 5347168
512 1365383 1199171 3884698 2381315 4099771 3604335 2975154 3580298
1024 1291406 2220992 3908759 5277632 4114719 3324778 3999762 4064103 3495237
2048 1288849 2131071 3390131 3806235 3871422 3835124 4081105 4319142 4086930 3452815
4096 1054315 1641680 2323234 3152960 3486312 4095510 3402085 4075109 3237937 3404782 2933818
8192 1136843 1709881 2306362 2966955 3216359 3913942 3708229 4098346 3253206 3524869 2791249 2727871
16384 1131337 1495175 2391506 3172233 3536443 3855055 3930914 3913230 3829704 3648149 3079822 2659236 2446768
32768 0 0 0 0 3623125 3833422 4057501 4235301 3663299 3589814 3016472 2469666 2759385
65536 0 0 0 0 3529695 3472837 3901644 3986175 3819987 3484106 2836927 2558603 2588406
131072 0 0 0 0 2830192 3692987 3639060 3694352 3699572 3256445 2907287 2778006 2686941
262144 0 0 0 0 3051877 3717985 3902578 3637108 3771705 3269847 2310554 2881713 2720177
524288 0 0 0 0 3301436 3502706 3610598 3724482 3638262 3415243 2873549 2728534 2768533
Record Rewrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 538156 1033216 1828508 3022727 2772930
128 511887 992724 1885042 2985839 3199360 3759450
256 446684 907792 1514860 2330140 3823789 4929815 4572895
512 498047 873740 1753325 2544997 4610256 5760329 6652558 3459188
1024 485537 958739 1759651 3029305 4920874 6244448 7054742 4876180 3894582
2048 484027 882756 1777789 3103517 4948672 7106133 7262337 6132138 4016229 3052781
4096 451189 899808 1569969 3058654 4779081 7024216 5527974 7408930 4911613 3641495 2590917
8192 470721 936212 1592512 2926772 4855947 6872214 7008178 7663074 5281396 3803041 2452724 1768585
16384 468234 837452 1631562 2693423 4158564 7504743 6414703 7933620 4911408 3521943 2401619 2285407 2235520
32768 0 0 0 0 5184848 7643648 7285082 7876696 5074018 3497191 3184125 2629212 2633242
65536 0 0 0 0 4661491 7356993 7376142 8108060 4206640 3601281 2866601 2484686 2062427
131072 0 0 0 0 4888551 7346258 6590863 7808498 5074974 3639156 3088762 2403400 2528103
262144 0 0 0 0 4598681 7396209 7212475 7126783 4980985 3505539 2232493 2587008 2516449
524288 0 0 0 0 4694897 7452350 7349042 7867230 4756662 4024995 2865446 2598188 2298983
Stride Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1363961 2537060 4018152 6421025 9318832
128 1291158 1967960 4135958 5545860 6045455 3560017
256 897923 1274008 3368013 5117791 6398720 5687020 4117018
512 1368865 2438090 2081286 4973263 7115451 6821616 7115451 3736016
1024 1201790 2188177 3877004 5564829 8391792 8183918 9569770 5821271 4361233
2048 968337 2088074 3402216 4720248 7112016 9228493 9752360 9101380 5212953 3799501
4096 1108178 1820324 2372973 3245890 5826043 6410863 7742845 10985935 5833956 3792628 2908979
8192 1151550 1765858 2397446 727208 4081792 4130865 4196965 11804201 6439465 4705005 2224247 1915386
16384 1139271 1696617 2279645 3007310 3542459 4071588 3858735 4320673 6726121 4342516 2960539 2744847 2696064
32768 0 0 0 0 3746583 3578412 4169516 4098030 4091321 4478909 3648614 2746043 2971087
65536 0 0 0 0 3226463 3759486 3415658 4225912 3562310 3484327 3340567 2957527 2646955
131072 0 0 0 0 3199997 3639590 3816561 3671603 3925711 3546518 2681332 2919778 2964365
262144 0 0 0 0 3147587 3801620 3810619 4037397 3711409 3375791 2695483 2857100 2863335
524288 0 0 0 0 3157715 3590987 3677894 3862042 3693343 3287670 2926191 2709205 2500419
Fwrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 565358 1101022 1947927 3203069 4897948
128 556454 907174 1705404 3124872 3380677 2843510
256 364251 680960 1158540 2557711 2754556 3823789 3770085
512 526111 996139 1574625 1677976 3320137 3481620 3510074 3634839
1024 506433 1008964 1825466 3140041 3379719 2918161 4736370 3794792 4079544
2048 480641 988848 1784066 2384234 3034448 4054139 3542524 3586902 3126106 3070239
4096 432570 851044 1248826 2617364 3717133 3471518 2415343 3908241 3110150 2880688 2809093
8192 480080 812694 1398681 943021 2680625 2944580 3293747 3652258 2919312 2967980 2376552 2484649
16384 438052 809008 1239622 1989803 2169188 2967058 3114861 3045292 3083138 2819051 2528421 2383791 2371042
32768 0 0 0 0 2659072 3167175 3419406 3205363 3084301 1065494 2716839 2482065 2092337
65536 0 0 0 0 2266664 3208911 2929537 3211610 2827996 2686432 2319766 2275539 2390163
131072 0 0 0 0 2176735 2105039 2380659 2427705 2143625 2355001 2036036 1894979 2046594
262144 0 0 0 0 1924669 2405013 2271925 2249466 2334133 2070698 1780300 1863513 1898354
524288 0 0 0 0 1688517 2361476 2179955 2454199 2104880 2084819 1891138 1861607 1856682
Re-fwrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 571375 1033216 1933893 3057153 2203800
128 554157 1065630 1827299 3199360 2035099 2608629
256 377041 900936 1552085 1471270 2392442 2872459 2345409
512 528962 1001715 1854787 3086326 2891043 3140488 3415178 2651850
1024 465437 935151 1406431 1537868 2156318 2884840 3348104 3348104 3066069
2048 482884 964207 1699696 1788524 2619037 2398213 2999482 3083465 2786401 2629459
4096 434814 769197 1438752 2284312 2652931 3103408 2842558 3279348 2710696 2708986 2616965
8192 455012 789900 1309900 2038935 2656378 3167728 3159282 2880642 2936276 2368362 2295882 2504389
16384 424291 829736 1180832 1939649 2566764 2693001 3004024 3107115 3160416 2970778 2491478 2434460 2339879
32768 0 0 0 0 2436264 3207233 3190629 2918656 3041976 2667122 2106737 2310371 2307966
65536 0 0 0 0 2077406 2707201 2589015 2683704 2444928 2420002 1447801 2086014 2014188
131072 0 0 0 0 2129756 2344676 2647271 2348031 1835328 2148055 1959251 1766160 2068985
262144 0 0 0 0 1752285 2288304 2046818 2235348 2196318 1819763 1777922 1817958 1738875
524288 0 0 0 0 1584804 2514678 2221559 2182497 2473873 2055942 1888189 1940714 1874404
Fread Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1013707 1526887 1991276 1879725 1526887
128 1093856 1997245 2608629 3759450 2058509 1332828
256 934659 1855098 1579483 2065659 2371308 3087188 1178892
512 1329871 2124526 3201349 3762197 2995907 2782414 2736325 864942
1024 990352 1590254 1896395 2565989 2509027 3130886 2674649 2746483 1328146
2048 1042239 1906837 1906837 2255282 2849249 2566606 2649737 2813783 2153510 1025565
4096 1093715 1658960 2192482 2848213 2932315 3093907 3000952 3020473 2128907 1736426 760345
8192 1038817 1556231 2205119 2648597 2767419 3004049 2919560 2866463 2297264 1100187 1045550 711487
16384 1086116 1595166 2042680 2451220 2791114 2967058 2686788 2736976 2405149 1866676 1253940 974484 724890
32768 0 0 0 0 2821413 3102192 3124336 2891880 2185288 1593295 1427243 1132388 977831
65536 0 0 0 0 2222172 2625115 2510098 2958577 2161264 1578082 1330246 1244845 1108504
131072 0 0 0 0 2511392 1772989 2839210 2660660 2337856 1736168 1435255 1246013 1171236
262144 0 0 0 0 2534559 2831750 2984811 2979384 2232475 1858266 1410256 1373366 1316380
524288 0 0 0 0 2792404 2761999 2910447 3022966 1982050 1653514 1301945 1350658 1299488
Re-fread Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1119387 2067979 3363612 4988978 5389653
128 1175271 2248149 2985839 4267461 5325799 6727225
256 999940 1897721 2719671 4197489 4652146 6554972 6249745
512 1336492 2207515 3510074 3678421 5495016 6156675 6821616 4195896
1024 1161176 2115949 3241960 3908759 4079544 6059442 6163794 4270175 2918161
2048 886217 2047755 1752044 3574960 3835124 4751580 3029098 4414590 2452301 2156213
4096 1110901 1901520 2767462 3546038 3767675 4055868 4055868 3660895 2628979 2092858 1591346
8192 1064566 1626049 1847615 2643503 3154641 3361415 3312482 3198692 2538433 1393689 1399137 1320827
16384 1052207 1559190 2087100 2500544 2764725 2879532 3116698 2925253 2506289 2078138 1621820 1452480 1443540
32768 0 0 0 0 2831935 2990091 3223935 2968135 2271763 1420061 1594478 1427732 1399983
65536 0 0 0 0 2663834 2830909 3005943 3162940 2495287 1970001 1571828 1433860 1120081
131072 0 0 0 0 2686784 2921997 3032185 3073857 2396674 1922325 1449430 1409786 1335315
262144 0 0 0 0 2670935 2943090 3114130 2910348 2460039 1925998 1587701 1478098 1370422
524288 0 0 0 0 2662114 2932555 3066367 2949247 2304858 1852661 1357903 1364636 1351076

 

[Mlovelace]

This is iozone from the cli to the (mirror) CIFS share, i am afraid to write to the iscsi LUN


The top row is records sizes, the left column is file sizes
Writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 179338 399573 651849 1232453 1828508
128 163490 348661 526444 942199 1827299 2027414
256 219184 383917 705107 1062263 1718521 2692393 2533571
512 294748 516743 924123 1302447 1764853 2860238 2811557 2642062
1024 361048 630518 898941 1221964 2064088 2844715 2960401 2859868 2645000
2048 354197 598818 1063399 1513696 1853353 2612664 2712491 3034448 2619037 2687877
4096 372561 693301 1069210 1494316 1830994 2741407 2938334 2769693 3086681 2563468 2370354
8192 421856 644686 1115945 1347345 1741867 2550303 3063510 3010630 2861211 2774122 1717059 2343166
16384 352639 696543 1113338 1454263 1863942 3092991 2893109 3018937 2945692 2808796 2227044 2499816 2404476
32768 0 0 0 0 1860027 2139200 3164404 3186265 3062378 2703000 1921099 2367604 2434495
65536 0 0 0 0 1542387 2870103 2886014 2913885 3028897 3005811 2602078 2303475 2508083
131072 0 0 0 0 1688248 2536174 2343836 2794116 2613916 2190962 2240008 1953473 1967159
262144 0 0 0 0 1500401 2209443 2391080 2419929 1771963 2202371 1894459 1616028 1964863
524288 0 0 0 0 1250925 2310779 2094302 2162020 2203936 1989798 2042193 1635094 1898590
Re-writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 500060 1033216 1679761 2662899 2203800
128 403721 976473 1642783 3199360 1755594 2453642
256 421603 880256 1075025 3017775 2131261 2665657 2639446
512 522399 1026126 1815584 3182373 2307123 3122224 3527371 3304808
1024 423324 819837 1640674 1875690 2774874 3142339 3532609 2694787 2730767
2048 498782 825415 1587551 2656292 2589040 2888532 3185225 3180508 2798201 2805512
4096 429488 828792 1607427 2006805 2406546 3105091 2663212 3155856 2880206 2760347 2469150
8192 476829 836944 1328179 1981095 2048050 2369832 3182693 2990974 2829637 2944580 2335204 2328557
16384 446721 814995 1394135 1402758 2729692 3099408 2689838 3023985 2954176 2295024 2628518 2463345 1911380
32768 0 0 0 0 2477368 2727027 2307269 2600852 2287301 2056090 1898071 1829596 1784949
65536 0 0 0 0 1953856 2605704 2717989 2794104 2608177 2538588 2224204 1922387 2300237
131072 0 0 0 0 1976687 2390525 2215508 2579998 2172829 2074543 1884682 1948971 1996586
262144 0 0 0 0 1709727 1798265 2328557 2748588 2029481 2207806 1692462 1831765 1859164
524288 0 0 0 0 2011054 2799414 2073842 2292500 2317735 2318195 2043379 1787780 1798754
Reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1460430 2662899 4274062 6421025 9318832
128 1363288 1539168 4012317 6406138 5325799 11470204
256 794889 2247235 1842366 5938650 5569035 11207494 12812277
512 1201856 2665014 4195896 5684095 3941742 5019764 6174377 10485468
1024 1216082 2497356 1701108 5593820 4574926 4805258 4415030 5507740 8750850
2048 1421279 2104960 2752473 4707314 3630870 4096676 4071433 5832361 5345969 5095447
4096 1039641 1934058 3202328 3552638 3483484 4188367 3781775 4280194 4592558 4175134 2940345
8192 1247231 1848808 2446089 3159282 3354524 3204958 3206454 4211368 3932757 3358786 3285873 3163062
16384 1260588 1945140 2553222 3008890 3715207 3927095 4222704 4006778 3951937 3078580 3002974 2810864 2829265
32768 0 0 0 0 3457600 3719411 4095588 4215685 2752919 3772499 3059855 2754795 2894194
65536 0 0 0 0 3584328 2792572 3054307 3528563 3232686 3049901 2338653 2540465 2516740
131072 0 0 0 0 3286248 3752425 3946282 3599107 3668810 3734581 2982635 2919980 2701399
262144 0 0 0 0 3305769 3547283 3466317 3699881 3816889 3330381 2364516 2842343 2956266
524288 0 0 0 0 3276940 3533011 3647120 3893132 3470906 3489018 3005288 2684030 2329837
Re-reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1562436 2923952 4564786 8182586 9318832
128 1421023 2660335 4717434 7582312 9795896 10567140
256 1012194 2639446 3043436 7120034 8815202 12228612 12812277
512 1589779 2796910 4732165 5886650 9510316 10911694 12499490 12499490
1024 1539522 2798378 2188177 6776473 9240349 10796646 4966395 11773301 9142007
2048 1132220 2458618 3848872 5521212 5704543 7761014 5832361 8949659 6189580 5611381
4096 1198388 2205712 3663236 4622213 5192166 5777065 4911613 6263611 5986422 4801789 3699523
8192 1233175 2003389 2753668 3339526 3686744 3837016 3070354 4710810 4548665 3790873 3592684 2824984
16384 1258672 1911114 2741125 2907062 3740689 4100254 4255654 4245663 4097077 3136328 3254563 2602635 2842489
32768 0 0 0 0 3609139 3907867 4231389 4347572 3864247 3980529 3344680 2804544 3115130
65536 0 0 0 0 3684255 3736794 3794569 3670431 3850542 3539239 3218379 2590870 2399259
131072 0 0 0 0 3315340 3941415 3696339 4086923 3627989 3489210 3021203 1820288 2786271
262144 0 0 0 0 3599246 3816108 3843588 4132666 3641143 3591698 2483931 2739882 2975659
524288 0 0 0 0 3498098 3870771 3920996 4020888 3743070 3444327 3071202 2683058 2609476
Random Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1392258 2561267 3958892 6421025 9318832
128 969421 2367096 4267461 6406138 6727225 10567140
256 770920 1610276 2812272 6249745 7965107 11091721 12228612
512 1267097 2548017 4305250 6156675 8701110 9859630 11877300 10235583
1024 1369216 2547723 3684119 6479979 8988941 10454984 9142007 10878686 8457895
2048 1325646 2228948 3969827 5655716 6967792 8531869 5032754 8292998 6649556 5507054
4096 1276003 1826906 3105091 4267436 5312585 6847827 5902100 7434579 6169144 4592558 3565171
8192 1179700 1941688 2472847 3656923 4122934 3639106 4528283 4862131 4531269 4065372 3394960 2655351
16384 1135337 1811562 2589883 3058439 3792715 4124866 4295556 4027442 3658053 3069779 3121937 2827751 2584235
32768 0 0 0 0 3525268 3781217 4258925 4284549 3828403 3913097 3385462 2749724 3121356
65536 0 0 0 0 3580220 3672098 3749844 3773162 4001960 3744481 3162795 2683363 2800310
131072 0 0 0 0 3401453 3761668 4126031 3997066 3674572 3636845 3012116 2755574 2772137
262144 0 0 0 0 3182868 3829665 3855232 3888280 3700292 3377357 1781994 2631702 2524306
524288 0 0 0 0 3322270 3746252 3780314 3785859 3801803 3423473 2959687 2694000 2703170
Random Write Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 537079 752329 1879725 3057153 2923952
128 447464 883293 1705404 3124872 2558895 3657016
256 272605 842278 1053922 3159869 3159869 4652146 3667079
512 408281 953248 1625889 2894940 3304808 2975154 4271001 3822466
1024 472347 978841 1376236 2854167 3294177 4048778 3046495 4783849 3659010
2048 495273 900997 1651654 2848305 2698856 4046500 3255235 3717289 3117032 3112514
4096 455677 771096 1369480 2420789 2782703 3524939 2860067 3771811 3330202 3141429 2696655
8192 394419 806305 1223036 2116815 2822895 2777935 3142235 3268991 3124235 2814110 2438797 2244442
16384 447844 825302 1296207 1875234 2566764 3389235 3398453 2932993 2963348 2645518 2424753 2290816 2203337
32768 0 0 0 0 2565604 3043391 3307255 3407368 3025903 2979460 2310177 2398677 2330587
65536 0 0 0 0 1677273 2900602 2991842 3057160 3076665 2678108 2467290 2197426 2376997
131072 0 0 0 0 1887283 2312611 2299388 2732659 2252333 2377766 2053198 1821476 2105401
262144 0 0 0 0 1696691 2694677 2341449 2812461 2000884 2390534 1506314 1728977 1830497
524288 0 0 0 0 1546239 2502413 2361933 2280901 2382608 2078140 1945372 1861322 1761081
Backward Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1232453 2133730 3363612 4564786 2892445
128 1266785 2248149 3657016 5122535 2727923 3445772
256 1195962 2170027 2152625 4553502 3087188 3511189 5347168
512 1365383 1199171 3884698 2381315 4099771 3604335 2975154 3580298
1024 1291406 2220992 3908759 5277632 4114719 3324778 3999762 4064103 3495237
2048 1288849 2131071 3390131 3806235 3871422 3835124 4081105 4319142 4086930 3452815
4096 1054315 1641680 2323234 3152960 3486312 4095510 3402085 4075109 3237937 3404782 2933818
8192 1136843 1709881 2306362 2966955 3216359 3913942 3708229 4098346 3253206 3524869 2791249 2727871
16384 1131337 1495175 2391506 3172233 3536443 3855055 3930914 3913230 3829704 3648149 3079822 2659236 2446768
32768 0 0 0 0 3623125 3833422 4057501 4235301 3663299 3589814 3016472 2469666 2759385
65536 0 0 0 0 3529695 3472837 3901644 3986175 3819987 3484106 2836927 2558603 2588406
131072 0 0 0 0 2830192 3692987 3639060 3694352 3699572 3256445 2907287 2778006 2686941
262144 0 0 0 0 3051877 3717985 3902578 3637108 3771705 3269847 2310554 2881713 2720177
524288 0 0 0 0 3301436 3502706 3610598 3724482 3638262 3415243 2873549 2728534 2768533
Record Rewrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 538156 1033216 1828508 3022727 2772930
128 511887 992724 1885042 2985839 3199360 3759450
256 446684 907792 1514860 2330140 3823789 4929815 4572895
512 498047 873740 1753325 2544997 4610256 5760329 6652558 3459188
1024 485537 958739 1759651 3029305 4920874 6244448 7054742 4876180 3894582
2048 484027 882756 1777789 3103517 4948672 7106133 7262337 6132138 4016229 3052781
4096 451189 899808 1569969 3058654 4779081 7024216 5527974 7408930 4911613 3641495 2590917
8192 470721 936212 1592512 2926772 4855947 6872214 7008178 7663074 5281396 3803041 2452724 1768585
16384 468234 837452 1631562 2693423 4158564 7504743 6414703 7933620 4911408 3521943 2401619 2285407 2235520
32768 0 0 0 0 5184848 7643648 7285082 7876696 5074018 3497191 3184125 2629212 2633242
65536 0 0 0 0 4661491 7356993 7376142 8108060 4206640 3601281 2866601 2484686 2062427
131072 0 0 0 0 4888551 7346258 6590863 7808498 5074974 3639156 3088762 2403400 2528103
262144 0 0 0 0 4598681 7396209 7212475 7126783 4980985 3505539 2232493 2587008 2516449
524288 0 0 0 0 4694897 7452350 7349042 7867230 4756662 4024995 2865446 2598188 2298983
Stride Read Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1363961 2537060 4018152 6421025 9318832
128 1291158 1967960 4135958 5545860 6045455 3560017
256 897923 1274008 3368013 5117791 6398720 5687020 4117018
512 1368865 2438090 2081286 4973263 7115451 6821616 7115451 3736016
1024 1201790 2188177 3877004 5564829 8391792 8183918 9569770 5821271 4361233
2048 968337 2088074 3402216 4720248 7112016 9228493 9752360 9101380 5212953 3799501
4096 1108178 1820324 2372973 3245890 5826043 6410863 7742845 10985935 5833956 3792628 2908979
8192 1151550 1765858 2397446 727208 4081792 4130865 4196965 11804201 6439465 4705005 2224247 1915386
16384 1139271 1696617 2279645 3007310 3542459 4071588 3858735 4320673 6726121 4342516 2960539 2744847 2696064
32768 0 0 0 0 3746583 3578412 4169516 4098030 4091321 4478909 3648614 2746043 2971087
65536 0 0 0 0 3226463 3759486 3415658 4225912 3562310 3484327 3340567 2957527 2646955
131072 0 0 0 0 3199997 3639590 3816561 3671603 3925711 3546518 2681332 2919778 2964365
262144 0 0 0 0 3147587 3801620 3810619 4037397 3711409 3375791 2695483 2857100 2863335
524288 0 0 0 0 3157715 3590987 3677894 3862042 3693343 3287670 2926191 2709205 2500419
Fwrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 565358 1101022 1947927 3203069 4897948
128 556454 907174 1705404 3124872 3380677 2843510
256 364251 680960 1158540 2557711 2754556 3823789 3770085
512 526111 996139 1574625 1677976 3320137 3481620 3510074 3634839
1024 506433 1008964 1825466 3140041 3379719 2918161 4736370 3794792 4079544
2048 480641 988848 1784066 2384234 3034448 4054139 3542524 3586902 3126106 3070239
4096 432570 851044 1248826 2617364 3717133 3471518 2415343 3908241 3110150 2880688 2809093
8192 480080 812694 1398681 943021 2680625 2944580 3293747 3652258 2919312 2967980 2376552 2484649
16384 438052 809008 1239622 1989803 2169188 2967058 3114861 3045292 3083138 2819051 2528421 2383791 2371042
32768 0 0 0 0 2659072 3167175 3419406 3205363 3084301 1065494 2716839 2482065 2092337
65536 0 0 0 0 2266664 3208911 2929537 3211610 2827996 2686432 2319766 2275539 2390163
131072 0 0 0 0 2176735 2105039 2380659 2427705 2143625 2355001 2036036 1894979 2046594
262144 0 0 0 0 1924669 2405013 2271925 2249466 2334133 2070698 1780300 1863513 1898354
524288 0 0 0 0 1688517 2361476 2179955 2454199 2104880 2084819 1891138 1861607 1856682
Re-fwrite Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 571375 1033216 1933893 3057153 2203800
128 554157 1065630 1827299 3199360 2035099 2608629
256 377041 900936 1552085 1471270 2392442 2872459 2345409
512 528962 1001715 1854787 3086326 2891043 3140488 3415178 2651850
1024 465437 935151 1406431 1537868 2156318 2884840 3348104 3348104 3066069
2048 482884 964207 1699696 1788524 2619037 2398213 2999482 3083465 2786401 2629459
4096 434814 769197 1438752 2284312 2652931 3103408 2842558 3279348 2710696 2708986 2616965
8192 455012 789900 1309900 2038935 2656378 3167728 3159282 2880642 2936276 2368362 2295882 2504389
16384 424291 829736 1180832 1939649 2566764 2693001 3004024 3107115 3160416 2970778 2491478 2434460 2339879
32768 0 0 0 0 2436264 3207233 3190629 2918656 3041976 2667122 2106737 2310371 2307966
65536 0 0 0 0 2077406 2707201 2589015 2683704 2444928 2420002 1447801 2086014 2014188
131072 0 0 0 0 2129756 2344676 2647271 2348031 1835328 2148055 1959251 1766160 2068985
262144 0 0 0 0 1752285 2288304 2046818 2235348 2196318 1819763 1777922 1817958 1738875
524288 0 0 0 0 1584804 2514678 2221559 2182497 2473873 2055942 1888189 1940714 1874404
Fread Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1013707 1526887 1991276 1879725 1526887
128 1093856 1997245 2608629 3759450 2058509 1332828
256 934659 1855098 1579483 2065659 2371308 3087188 1178892
512 1329871 2124526 3201349 3762197 2995907 2782414 2736325 864942
1024 990352 1590254 1896395 2565989 2509027 3130886 2674649 2746483 1328146
2048 1042239 1906837 1906837 2255282 2849249 2566606 2649737 2813783 2153510 1025565
4096 1093715 1658960 2192482 2848213 2932315 3093907 3000952 3020473 2128907 1736426 760345
8192 1038817 1556231 2205119 2648597 2767419 3004049 2919560 2866463 2297264 1100187 1045550 711487
16384 1086116 1595166 2042680 2451220 2791114 2967058 2686788 2736976 2405149 1866676 1253940 974484 724890
32768 0 0 0 0 2821413 3102192 3124336 2891880 2185288 1593295 1427243 1132388 977831
65536 0 0 0 0 2222172 2625115 2510098 2958577 2161264 1578082 1330246 1244845 1108504
131072 0 0 0 0 2511392 1772989 2839210 2660660 2337856 1736168 1435255 1246013 1171236
262144 0 0 0 0 2534559 2831750 2984811 2979384 2232475 1858266 1410256 1373366 1316380
524288 0 0 0 0 2792404 2761999 2910447 3022966 1982050 1653514 1301945 1350658 1299488
Re-fread Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
64 1119387 2067979 3363612 4988978 5389653
128 1175271 2248149 2985839 4267461 5325799 6727225
256 999940 1897721 2719671 4197489 4652146 6554972 6249745
512 1336492 2207515 3510074 3678421 5495016 6156675 6821616 4195896
1024 1161176 2115949 3241960 3908759 4079544 6059442 6163794 4270175 2918161
2048 886217 2047755 1752044 3574960 3835124 4751580 3029098 4414590 2452301 2156213
4096 1110901 1901520 2767462 3546038 3767675 4055868 4055868 3660895 2628979 2092858 1591346
8192 1064566 1626049 1847615 2643503 3154641 3361415 3312482 3198692 2538433 1393689 1399137 1320827
16384 1052207 1559190 2087100 2500544 2764725 2879532 3116698 2925253 2506289 2078138 1621820 1452480 1443540
32768 0 0 0 0 2831935 2990091 3223935 2968135 2271763 1420061 1594478 1427732 1399983
65536 0 0 0 0 2663834 2830909 3005943 3162940 2495287 1970001 1571828 1433860 1120081
131072 0 0 0 0 2686784 2921997 3032185 3073857 2396674 1922325 1449430 1409786 1335315
262144 0 0 0 0 2670935 2943090 3114130 2910348 2460039 1925998 1587701 1478098 1370422
524288 0 0 0 0 2662114 2932555 3066367 2949247 2304858 1852661 1357903 1364636 1351076

I was just crushed to death by your wall of text...

 

[Richie111]

sorry, that was pretty bad.

Maybe a graph of read and write is better

writes