Scrub Quesiton

[raidflex]

Scrub is still going. Only a couple hours left and its at 92%. Still no errors. I still do not understand what is causing the system to take so long.


[root@fileserver] ~# systat -vm zpool

1 users Load 0.17 0.24 0.21 Mar 15 17:30

Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 1178572 12968 2174240 16424 13235k count
All 2313892 15244 1076051k 38668 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 363 cow 4833 total
126 8607 4749 12k 833 1113 4444 2831 zfod 1 ehci0 16
ozfod 821 arcmsr0 18
10.5%Sys 0.4%Intr 14.0%User 0.0%Nice 75.1%Idle %ozfod 2 ehci1 23
| | | | | | | | | | | daefr 2000 cpu0: time
=====>>>>>>> 3033 prcfr 9 em0 irq256
154 dtbuf 32164 totfr ahci0 257
Namei Name-cache Dir-cache 332397 desvn react 2000 cpu1: time
Calls hits % hits % 32436 numvn pdwak
139795 131624 94 12 0 20983 frevn pdpgs
intrn
Disks da0 da1 da2 da3 da4 da5 da6 1624688 wire
KB/t 47.50 47.92 48.34 48.45 48.69 48.78 48.52 716132 act
tps 105 104 103 102 103 103 101 584704 inact
MB/s 4.85 4.84 4.84 4.82 4.88 4.91 4.79 1052 cache
%busy 98 98 98 98 99 99 98 13233084 free
175664 buf




[root@fileserver] ~# zpool iostat 3
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
Data 6.27T 8.23T 174 26 20.8M 1.99M
Data 6.27T 8.23T 234 0 28.4M 0
Data 6.27T 8.23T 224 39 26.7M 171K
Data 6.27T 8.23T 233 0 28.6M 0
Data 6.27T 8.23T 219 0 26.9M 0
Data 6.27T 8.23T 233 0 28.4M 0
Data 6.27T 8.23T 232 0 28.4M 0
Data 6.27T 8.23T 226 0 27.6M 0
Data 6.27T 8.23T 232 0 28.4M 0
Data 6.27T 8.23T 230 0 28.0M 0
Data 6.27T 8.23T 224 0 27.5M 0
Data 6.27T 8.23T 230 0 28.0M 0
Data 6.27T 8.23T 213 39 25.4M 171K
Data 6.27T 8.23T 231 0 28.4M 0
Data 6.27T 8.23T 220 0 26.9M 0
Data 6.27T 8.23T 231 0 28.4M 0
Data 6.27T 8.23T 226 0 27.6M 0
Data 6.27T 8.23T 223 0 27.2M 0
Data 6.27T 8.23T 222 0 27.2M 0
Data 6.27T 8.23T 225 0 27.5M 0
Data 6.27T 8.23T 219 0 26.8M 0
Data 6.27T 8.23T 207 0 25.2M 0
Data 6.27T 8.23T 163 39 19.3M 171K
Data 6.27T 8.23T 162 0 19.8M 0
Data 6.27T 8.23T 196 0 24.0M 0
Data 6.27T 8.23T 231 0 28.2M 0
Data 6.27T 8.23T 234 0 28.6M 0
Data 6.27T 8.23T 234 0 28.5M 0
Data 6.27T 8.23T 233 0 28.6M 0
Data 6.27T 8.23T 231 0 28.2M 0
Data 6.27T 8.23T 233 0 28.6M 0
Data 6.27T 8.23T 235 0 28.4M 0
Data 6.27T 8.23T 225 40 27.0M 179K

 

[raidflex]

This system is waiting for disk-io
( if you compare my n36 that maes > 600 tps and the disks are still < 50% busy )
You might rearrange your disks to for more vdev's each with smaller number of disks. ( most likley raidz )
Not a happy thing to do, but you don't have any choice if you want more speed.

This wouldn't explain why other people with 8+ HDDs in a pool do not have the same issue. I mean we are talking twice as long for my system to do a scrub then most, at least thats what it seems. Previously with Raid 6 hardware using my areca 1220 it was much faster, even while rebuilding the raid array, now I know that is a different setup, but still.

 

[Thomas Blomberg Hansen]

This wouldn't explain why other people with 8+ HDDs in a pool do not have the same issue. I mean we are talking twice as long for my system to do a scrub then most, at least thats what it seems. Previously with Raid 6 hardware using my areca 1220 it was much faster, even while rebuilding the raid array, now I know that is a different setup, but still.

Yours are also quite slower than mine.
I have a new motherboard ready to be tested.

[root@freenas] ~# zpool iostat 3
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
data 8.79T 2.08T 243 26 29.8M 1.90M
data 8.79T 2.08T 313 0 38.3M 0
data 8.79T 2.08T 120 0 14.7M 0
data 8.79T 2.08T 164 0 20.0M 0
data 8.79T 2.08T 146 0 18.0M 0
data 8.79T 2.08T 299 0 36.7M 0
data 8.79T 2.08T 359 0 43.9M 0
data 8.79T 2.08T 391 0 48.2M 0
data 8.79T 2.08T 253 0 31.1M 0
data 8.79T 2.08T 342 45 41.3M 239K
data 8.79T 2.08T 353 0 43.1M 0
data 8.79T 2.08T 269 0 32.8M 0
data 8.79T 2.08T 292 0 35.8M 0
data 8.79T 2.08T 353 0 43.1M 0
data 8.79T 2.08T 397 0 48.7M 0
data 8.79T 2.08T 345 0 42.4M 0
data 8.79T 2.08T 567 0 69.0M 0
data 8.79T 2.08T 427 0 52.1M 0
data 8.79T 2.08T 258 0 31.5M 0
data 8.79T 2.08T 228 47 27.6M 252K
data 8.79T 2.08T 561 0 68.5M 0
data 8.79T 2.08T 245 0 30.1M 0
data 8.79T 2.08T 281 0 34.4M 0
data 8.79T 2.08T 490 0 59.9M 0
data 8.79T 2.08T 441 0 53.9M 0


I am doing the second scrub and its looks like its going to be faster.

 

[raidflex]

Are your HDDs 7200rpm? Also my read/write speeds seems to be much more consistant, although lower then yours.

Also are you using RaidZ1 or RaidZ2?

 

[Thomas Blomberg Hansen]

Are your HDDs 7200rpm? Also my read/write speeds seems to be much more consistant, although lower then yours.

Also are you using RaidZ1 or RaidZ2?

6x2TB (4 WD Green & 2 Seagate) all 7200rpm and 64mb cache.
RaidZ2
8GB ram (never been maxed out)
FreeNAS 8.0.2 all default settings.

 

[raidflex]

6x2TB (4 WD Green & 2 Seagate) all 7200rpm and 64mb cache.
RaidZ2
8GB ram (never been maxed out)
FreeNAS 8.0.2 all default settings.

Well the 7200rpm would matter. I know my Seagate's are 5900rpm/64MB cache.

But I thought the WD green drives were 5400/5900RPM?

 

[Thomas Blomberg Hansen]

Sry you are right, its only 5400rpm for the WD's (WD20EARX) and like you said 5900rpm for the Seagates.

 

[raidflex]

Sry you are right, its only 5400rpm for the WD's (WD20EARX) and like you said 5900rpm for the Seagates.

Well then that still leaves me in the same place I started. Scrub has 1hr left and no errors, its weird everything is showing me its okay, but the numbers are still too low, especially the TP/s.

Only other factor I could think of is the Areca 1220 RAID controller, that maybe there is some setting causing an issue. But as far as I know that setting the controller in JBOD mode it should act as an HBA only, but I could be wrong.

 

[Thomas Blomberg Hansen]

Well then that still leaves me in the same place I started. Scrub has 1hr left and no errors, its weird everything is showing me its okay, but the numbers are still too low, especially the TP/s.

Only other factor I could think of is the Areca 1220 RAID controller, that maybe there is some setting causing an issue. But as far as I know that setting the controller in JBOD mode it should act as an HBA only, but I could be wrong.

Yep, its looks like the only thing left is the controller.
I have a new motherboard with enough sata connections.
I'll post you the new results. It should give and indicator on how much the controller is capable of slowing the system down. Well its the best I can do :)
I'll get dual LAN and 16gb ram too, but with the drives being the bottle neck, should really matter anything.

If the estimated time left is holding up. The second scrub is down from 27h to 17h.

 

[Trianian]

I'm a FreeNAS noob, but when recently watching the ZFS Ninja video, the speaker repeatedly said that RAIDZ-2 is a LOT slower than RAIDZ-1.

Perhaps that's contributing to this issue?

 

[raidflex]

I'm a FreeNAS noob, but when recently watching the ZFS Ninja video, the speaker repeatedly said that RAIDZ-2 is a LOT slower than RAIDZ-1.

Perhaps that's contributing to this issue?

This may be true but Thomas above also is using RaidZ-2 with a similar configuration and his scrub time is half of mine.

I also did check my Areca-1220 manual and it says nothing of JBOD's other then it relays the drives to the OS as a single stand alone drive each, which is what I do want. I still will go in and disable all extended features that might possibly affect performance.

 

[raidflex]

Finally finished with 0 errors.

scrub: scrub completed after 45h16m with 0 errors on Thu Mar 15 20:54:28 2012
config:

NAME STATE READ WRITE CKSUM
Data ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gptid/e902c40d-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e91172b0-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e91f0e06-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e92cba1e-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e93a03e7-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e94867a0-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e956b27d-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0
gptid/e964d6cf-3326-11e1-acd0-e06995ebe0de ONLINE 0 0 0

errors: No known data errors



UPDATE:

I checked my Areca-1220 controller and all extended settings are off including power management, read-ahead-cache and write-back. All drives are set to SATA300-NCQ mode in a JBOD configuration.

 

[peterh]

Why don't you enable read-ahead, it wont hurt you.

BTW i have 4 WDC WD10EARS-00MVWB0 ( seagate green )
[root@fnas] ~# camcontrol identify ada0
pass0: <WDC WD10EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

protocol ATA/ATAPI-8 SATA 2.x
device model WDC WD10EARS-00MVWB0
firmware revision 51.0AB51
serial number WD-WCAZA9224982
WWN 50014ee262d4a59
cylinders 16383
heads 16
sectors/track 63
sector size logical 512, physical 4096, offset 0
LBA supported 268435455 sectors
LBA48 supported 1953525168 sectors
PIO supported PIO4
DMA supported WDMA2 UDMA6

Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes yes
flush cache yes yes
overlap no
Tagged Command Queuing (TCQ) no no
Native Command Queuing (NCQ) yes 32 tags
SMART yes yes
microcode download yes yes
security yes no
power management yes yes
advanced power management no no
automatic acoustic management no no
media status notification no no
power-up in Standby yes no
write-read-verify no no
unload no no
free-fall no no
data set management (TRIM) no

 

[b1ghen]

For a reference my 6x2TB disk RAIDZ2 completes quite a lot faster than what you have been reporting:

pool: tank
state: ONLINE
scrub: scrub completed after 8h35m with 0 errors on Sun Mar 4 19:00:44 2012
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
gpt/disk4 ONLINE 0 0 0
gpt/disk5 ONLINE 0 0 0


This pool is almost filled to the brim, I have only about 500G free from the 6.8T. I don't know if the kind of data stored makes a difference but I think it might, I basically only have large video files (300MB+) stored on my pool.

Specs are as follows:

Motherboard: ASUS M4A88T-M
CPU: AMD Athlon II X3 (4th core unlocked so now it's a quad core)
RAM: 2 x 4GB Kingston ECC DDR3-1333 (8GB total), scrub was just as fast with 4GB before though.
Disks: 6 x Seagate Barracuda Green ST2000DL003 5900rpm

The disks are connected to the onboard SATA ports.

 

[Thomas Blomberg Hansen]

My 2nd scrub finished a bit faster than the first and way faster than raidflex's and way slower than b1ghen's(which my setup is almost identical).
Next time I check in, I have a comparison off raid controller vs. non raid controller etc.

[root@freenas] ~# zpool status -v
pool: data
state: ONLINE
scrub: scrub completed after 25h16m with 0 errors on Fri Mar 16 10:08:52 2012
config:

NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gptid/c2315af9-dafd-11e0-aec7-e06995e518cf ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
gptid/c3217031-dafd-11e0-aec7-e06995e518cf ONLINE 0 0 0
ada5p2 ONLINE 0 0 0 128K repaired
gptid/c474e126-dafd-11e0-aec7-e06995e518cf ONLINE 0 0 0
gptid/c5382338-dafd-11e0-aec7-e06995e518cf ONLINE 0 0 0

errors: No known data errors
[root@freenas] ~#

 

[raidflex]

It's saying both SMART and write cache is not supported, but both are on and supported in the Areca-1220 controller.


root@fileserver] ~# ataidle /dev/da0
Model:
Serial:
Firmware Rev:
ATA revision: unknown/pre ATA-2
LBA 48: no
Geometry: 0 cyls, 0 heads, 0 spt
Capacity: 0MB
SMART Supported: no
Write Cache Supported: no
APM Supported: no
AAM Supported: no

Here are the Read/Write results:

Write:

[root@fileserver] ~# dd if=/dev/zero of=/mnt/Data/test.dat bs=8M count=5000
5000+0 records in
5000+0 records out
41943040000 bytes transferred in 401.683960 secs (104418011 bytes/sec)

Read:

[root@fileserver] ~# dd if=/mnt/Data/test.dat of=/dev/null bs=8M count=5000
5000+0 records in
5000+0 records out
41943040000 bytes transferred in 116.649330 secs (359565202 bytes/sec)

I'm not sure if these results are correct or good for that matter. Any feedback would be appreciated. Thanks.

I was also thinking of trying another scrub, but I do not want it take tie up the server for 2 days and im not sure if canceling the scrub would be damaging.

 

[peterh]

You might be better off seling this controller on ebay and buying a real sata controller
ZFS does all you need and your cpu is way faster then the raid-controllers.
You will most likley get some money out if that switch.

 

[raidflex]

You might be better off seling this controller on ebay and buying a real sata controller
ZFS does all you need and your cpu is way faster then the raid-controllers.
You will most likley get some money out if that switch.

Yeah I may have to. I figured that setting the controller in JBOD mode would have turned it into an HBA only, but I guess not.

The Areca-1220 uses a PCIE x8 interface and SATA 300, which I figured would be fast enough.

Also isn't JBOD offloading the processing to the CPU and not the controller itself?

Also any recommendation on the controller to purchase for at least 8 or more SATA II ports?

 

[Trianian]

The IBM M1015 is recommended quite frequently, $70 to $80 on ebay then update with the IT firmware. There are a couple of threads on it.

 

[raidflex]

The IBM M1015 is recommended quite frequently, $70 to $80 on ebay then update with the IT firmware. There are a couple of threads on it.

Looks like that M1015 is exactly what I need for the system. If I am going to use ZFS I want all the benefits of software raid and for ZFS to have direct access to the HDDs.