Error with a Disk

[DarkCorner]

4 PCs Win10 Pro are connected to TrueNAS SCALE, but they are used regularly only by 2 users and another 2 occasionally.
They have two work folders common to all that they use to start a ERP that resides on another server and to exchange documents between them.
Then in the evening two series of backups start for each PC, from 8PM to 12PM.

I noticed recently that the NAS CPU is always at 100%.
I also found a warning relating to a disk, but neither the datasets nor the individual disks report errors.
How do I identify the disk using the ID reported in the message?

Device /dev/disk/by-partuuid/6d3f2f06-7a91-4ba0-9a2d-fd18b5cab789 is causing slow I/O on pool BCK-RZ1. 2023-09-11 20:51:48 (Europe/Rome)

Last night one of the 3 PCs launched a very cumbersome backup.
This morning we found it stuck with compression at 50%.
No PC could connect to the shared folders and the NAS crashed upon shutdown. We had to force it off using the power button.

TrueNAS version: TrueNAS-SCALE-22.12.0 (I don't trust myself to do the latest update in these conditions)
CPU: AMD A4-5300 APU with Radeon(tm) HD Graphics (Why 1 Core? It has 2 Cores.)
Memory: 9.2GTB (Free: 4GB - ZFS cache 1.8GB - Services 3.3 GB)
3 work HDD WD Red 3TB (1 Pool RAIDZ1 - 412GB Total Allocation)

 

[Redcoat]

3 work HDD WD Red 3TB

Are these SMR disks? See this Resource for ID info: https://www.truenas.com/community/resources/list-of-known-smr-drives.141/ If so, this is a likely cause of your error report.

If this were TrueNAS Core you could use glabel status to get the disk ID, then Storage>Disks to get the Serial Number in order to find the actual drive - I don't know if those work in Scale...

 

[Arwen]

Also, that APU / CPU is pretty old, from 2012 using DDR3 memory. It should have 2 cores as you think, though Wikipedia shows both cores are on the same module. (Back then, CPUs were on single dies... so other tricks were tried to improve density.)

That won't account for the disk slow message, but Redcoat has that covered.

 

[DarkCorner]

If this were TrueNAS Core you could use glabel status to get the disk ID, then Storage>Disks to get the Serial Number in order to find the actual drive - I don't know if those work in Scale...

Is global status a command I type in the terminal or is it an item in the menu? Because I can't find it.

 

[DarkCorner]

Also, that APU / CPU is pretty old, from 2012 using DDR3 memory. It should have 2 cores as you think, though Wikipedia shows both cores are on the same module. (Back then, CPUs were on single dies... so other tricks were tried to improve density.)

This NAS was assembled a while ago, but it has always been kept off in the corner.
The discs, for example, have never been used.

Considering that it is truly a small office with limited needs, a dual core with 10/12 MB still seemed sufficient to me.

So I just had to do a fresh install and put it into production.

 

[DarkCorner]

@Redcoat
In Storage Disk, the model is WDC_WD30EFRX-68AX9N0.

 

[Redcoat]

@Redcoat
In Storage Disk, the model is WDC_WD30EFRX-68AX9N0.

Good! That model is an "older" WR Red and is CMR. Are all 3 HDD's the same model? What you do not want to find is WD30EFAX (which is a SMR disk).

Is global status a command I type in the terminal or is it an item in the menu? Because I can't find it.

You type glabel (not global) status in a terminal window - so glabel status

 

[Redcoat]

Good! That model is an "older" WR Red and is CMR. Are all 3 HDD's the same model? What you do not want to find is WD30EFAX (which is a SMR disk).

You type glabel (not global) status in a terminal window - so glabel status

But I just looked - and I think glabel won't be in Scale. I don't know how to get the disk name in Scale...

 

[Arwen]

Bur I just looked - and I don't think glabel won't be in Scale. I don't know how to get the disk name in Scale...

I think a simply ls would do the trick;
ls -l /dev/disk/by-partuuid/6d3f2f06-7a91-4ba0-9a2d-fd18b5cab789

This NAS was assembled a while ago, but it has always been kept off in the corner.
The discs, for example, have never been used.

Considering that it is truly a small office with limited needs, a dual core with 10/12 MB still seemed sufficient to me.

So I just had to do a fresh install and put it into production.

Understood. It is in someways a good thing to re-use old, but adequate hardware for something new.

The US may have pushed the PC industry forward quite quickly by a few people, (some gamers), doing upgrades every year or 2. But, the average person does not need to do that today.

 

[DarkCorner]

@Redcoat
All 3 discs are the same. I would have bought him a fourth record, but I couldn't find one like it.
If I had to replace the discs I would no longer buy the Red ones, but the Black or even the Blue ones.
It is small office with limited needs.
Maybe I'll program the shutdown in the evening and restart in the morning from the BIOS.

@Arwen
Years ago, a Dual Core was the standard and the Quad Core for medium-high needs.

 

[DarkCorner]

Now that I have identified it with ls -l I have identified sde2, how do I resolve the error message that slows down my entire NAS??
The Short SMART test reports no errors; the long one lasts hours and I will try to launch it after the backups.

 

[DarkCorner]

On the reported disk I ran a SMART test which lasted hours.
Now I have detected the SMART status of all 3 disks.
I'm surprised by the Prefail reports.
The disks were purchased years ago, but used very little, both in terms of hours of work, but also in terms of load.

However for all three disks I see similar results and the final result is "No errors".
So why does the sde2 disk slow down the NAS by keeping the CPU at 100%?

Test on SDC
root@gp-hq-bck-srv[/home/admin]# smartctl /dev/sdc -a smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T3959004 LU WWN Device Id: 5 0014ee 2b3a3ad75 Firmware Version: 80.00A80 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Sep 13 08:47:45 2023 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x02) Offline data collection activity was completed without error. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (41040) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 412) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 177 175 021 Pre-fail Always - 6125 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 232 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8400 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 232 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 85 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 146 194 Temperature_Celsius 0x0022 111 107 000 Old_age Always - 39 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 8377 - # 2 Extended offline Completed without error 00% 336 - # 3 Short offline Completed without error 00% 328 - # 4 Short offline Completed without error 00% 326 - # 5 Short offline Completed without error 00% 326 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.

Test on SDD
root@gp-hq-bck-srv[/home/admin]# smartctl /dev/sdd -a smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T4142913 LU WWN Device Id: 5 0014ee 25e4df2eb Firmware Version: 80.00A80 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Sep 13 08:48:56 2023 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (40080) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 402) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 177 175 021 Pre-fail Always - 6133 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 234 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8401 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 234 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 88 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 145 194 Temperature_Celsius 0x0022 110 107 000 Old_age Always - 40 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 336 - # 2 Short offline Completed without error 00% 328 - # 3 Short offline Completed without error 00% 326 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.

Test on SDE (Disk with error messages)
root@gp-hq-bck-srv[/home/admin]# smartctl /dev/sde -a smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T3958590 LU WWN Device Id: 5 0014ee 2b3a3d36b Firmware Version: 80.00A80 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Sep 13 08:44:55 2023 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (40860) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 410) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 179 177 021 Pre-fail Always - 6033 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 232 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8400 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 232 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 86 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 145 194 Temperature_Celsius 0x0022 109 105 000 Old_age Always - 41 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 8394 - # 2 Extended offline Completed without error 00% 335 - # 3 Short offline Completed without error 00% 328 - # 4 Short offline Completed without error 00% 326 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[DarkCorner]

@Redcoat
@Arwen
Forgive me if I insist, but in your opinion, what state are those three disks in? Do I risk losing data even though they have been used very little?
The last one (sde) is the one with the long SMART Test.

 

[Arwen]

Sorry, I have no opinion on why that disk is reported as slow, nor whether it will impact reliability.

You could try this command, changing the "1" to "2" or "3" seconds if desired.
zpool iostat BCK-RZ1 -v 1

...
If I had to replace the discs I would no longer buy the Red ones, but the Black or even the Blue ones.
It is small office with limited needs.
...

Be advised that some other lines of disks may be SMR. While sometimes SMR disks work okay in some situations, other times they are bad enough people want help fixing the problem. Except the fix is to not use SMR disks.

 

[joeschmuck]

First, open up the timeline for the SWAP file and find out if any swap space was used, especially when the error message occured. Using any SWAP space is a sign of low RAM, but a few kilobytes is not a big deal if it happens once in a blue moon, but routine use of SWAP will slow the system down. I can't say this is what caused your issue but it's worth looking at.

Run a SMART Long test on the drives sdc and sdd as well. Drive sde looks fine from a SMART perspective. Also, you can kick off a SMART test even while a backup operation is being done, the SMART tests always run in background, lowest priority and will pause to let other data operation occur, which means a SMART test would take longer to complete if you are doing some intensive operations.

I would also recommend posting the output of the following two commands for completeness:
zfs list BCK-RZ1 and zpool status

I also recommend that you setup a routine SMART tests for your drives, testing at ~300 hours and then the next time is at ~8000 hours, that is not good practice. Since you rarely spin these drives down, I recommend a SMART Short test every day and a once a week Long test.

As for the TrueNAS software upgrade, I would perform the upgrade to version 22.12.3.3 but if asked to upgrade your pool feature set, just do not do it. It causes no harm and allows you ro roll back to an older version if needed. I never advice upgrading your pool feature set unless you need those specific new features.

Due to your limited amount of RAM, you might not consider upgrading again without knowing if you have enough RAM to run effectively. At some point what were the minimum specs for FreeNAS/TrueNAS years ago become obsolete for the newer versions. And minimum specs may only support minimally.

 

[Redcoat]

@Redcoat
@Arwen
Forgive me if I insist, but in your opinion, what state are those three disks in? Do I risk losing data even though they have been used very little?
The last one (sde) is the one with the long SMART Test.

Late to the party today...
I agree that the test results on sde don't indicate the drive is the problem and support @Arwen's and @joeschmuck's responses/suggestions, especially looking at SWAP usage history - is your memory aequate (it looks to be "on the low end")?
Nobody yet has mentioned power - is your NAS power supply adequately sized for the system?

 

[DarkCorner]

First, open up the timeline for the SWAP file and find out if any swap space was used, especially when the error message occured. Using any SWAP space is a sign of low RAM, but a few kilobytes is not a big deal if it happens once in a blue moon, but routine use of SWAP will slow the system down. I can't say this is what caused your issue but it's worth looking at.
...
I would also recommend posting the output of the following two commands for completeness:
zfs list BCK-RZ1 and zpool status

Swap = 0Kb

sudo zfs list BCK-RZ1 NAME USED AVAIL REFER MOUNTPOINT BCK-RZ1 428G 4.90T 149K /mnt/BCK-RZ1

admin@gp-hq-bck-srv[~]# sudo zpool status pool: Apps state: ONLINE scan: scrub repaired 0B in 00:01:04 with 0 errors on Sun Sep 3 00:01:08 2023 config: NAME STATE READ WRITE CKSUM Apps ONLINE 0 0 0 964c01dc-6a2c-4a07-83be-78cd70ffd68f ONLINE 0 0 0 errors: No known data errors pool: BCK-RZ1 state: ONLINE scan: scrub repaired 0B in 01:16:26 with 0 errors on Sun Sep 3 01:16:29 2023 config: NAME STATE READ WRITE CKSUM BCK-RZ1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 6d3f2f06-7a91-4ba0-9a2d-fd18b5cab789 ONLINE 0 0 0 e8b96b2f-565b-4395-bb46-0429ac7c831a ONLINE 0 0 0 bfda04ff-024a-48dc-bfed-4a8a84e2be87 ONLINE 0 0 0 errors: No known data errors pool: boot-pool state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:00:42 with 0 errors on Wed Sep 13 03:45:45 2023 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 sdb3 ONLINE 0 0 0 errors: No known data errors

 

[DarkCorner]

Nobody yet has mentioned power - is your NAS power supply adequately sized for the system?

The power supply (120W) is present in the case.
Chembro ES34169

 

[DarkCorner]

You could try this command, changing the "1" to "2" or "3" seconds if desired.
zpool iostat BCK-RZ1 -v 1

admin@gp-hq-bck-srv[~]# iostat
Linux 5.15.79+truenas (gp-hq-bck-srv) 09/17/23 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 18.77 0.00 79.91 0.01 0.00 1.30 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd sda 31.26 7.94 284.47 0.00 3391304 121441917 0 sdb 0.80 8.00 4.53 0.00 3414611 1933404 0 sdc 0.28 2.98 21.22 0.00 1271952 9060507 0 sdd 0.29 0.93 23.20 0.00 396126 9906121 0 sde 0.28 1.71 21.23 0.00 731169 9061378 0 sdf 0.00 0.00 0.00 0.00 1 0 0 sdg 0.01 0.00 0.00 0.00 1463 0 0

 

[Redcoat]

"iostat" - not "iostack"