Storage Usage Question - 36 TiB installed, 24TiB available

[Roger Mahler]

Configuration
Server: IBM DX360-M3
Disks: 12+3TiB
RaidZ2
LSI SAS9210-8i
Memory: 24g

Install was almost seamless, struggled a little with configuration but mostly misunderstanding on my side.

Question: the top freenas mount shows 32.5 TiB, a little high from past experiences but ok, however when i get down to the data drives it shows available of 24TiB - quite a loss, is this normal. Wanted to know before loading up with data, i will be moving close to 15 TiB almost immediately.

Thank you.

 

[Bidule0hm]

Yes, top row is in fact the pool so it shows the raw space (including parity, etc...) while the second row is the root dataset and shows the data space only.

 

[Roger Mahler]

Then it is normal for freenas to use that amount of space, 1/3 of the space, for recovery. if this is normal than i will start moving data.

 

[danb35]

Then it is normal for freenas to use that amount of space, 1/3 of the space, for recovery

It may be, or it may not be, depending on how you set up your pool. If (as would not be at all unusual) it's done as two RAIDZ2 vdevs, each with six drives in it, then 1/3 of the space is exactly what would be used for parity.

 

[Roger Mahler]

I looked at that, i have it on one pool - when i did the volume layout it was one pool.

 

[danb35]

That wasn't the question. A pool can consist of one or more vdevs. Your 12-disk pool could consist of one 12-disk vdev, two 6-disk vdevs, or even three 4-disk vdevs (or, for that matter, many other arrangements), all combined into a single pool (which FreeNAS calls a volume). See @cyberjock's ZFS guide (link is in @Bidule0hm's sig up-thread from this post) for quite a bit more detail.

 

[Roger Mahler]

perfect, thank you very much for the info, and especially your quick responses. i will start digging in deeper.

 

[Robert Trevellyan]

In the Storage tab, click on the top level of your pool, then click the Volume Status button at the bottom. Post a screenshot.
or
Run the zpool status command in a shell or via SSH and paste the results between CODE tags.

 

[Roger Mahler]

The following is the Volume Status you requested - thoughts

 

[Roger Mahler]

What i found odd was loosing so much memory, this is what i ended up with. Any help would be greatly appreciated - would like to have a little more storage available from the installed amount. Or have i really configured something incorrectly.

 

[depasseg]

It is a little strange. It looks like that is an 11 drive raid-z2 (purely based on the space). Do you have a spare identified? Or is one drive a different size?

 

[Mlovelace]

Please post the zpool status output either from the shell or the GUI. No one can offer help if you don't provide the requested information.

 

[Roger Mahler]

there are 12X3TiB, 36 TiB in total. Attached is the zpool status, hope i got the right on this time.

and the storage volumes

 

[jde]

Can you also post the output of camcontrol devlist

 

[Roger Mahler]

Attached is the requested camcontrol devlist

 

[jde]

It looks like you have a sas expander between your drives and HBA. Please describe how you have the disk cabling setup. Also, post the output of "camcontrol identify /dev/ses0"

Do you have the LSI card flashed to IT model with updated firmware?

Is the firmware on the sas expander up to date?

Edit:
Checkout these links to see if it might be related to your problem.
https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5093355

https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5087255

 

[Roger Mahler]

I ran the camcontrol identify /dev/ses0 and received the following
root@freenas ~]# camcontrol identify /dev/ses0
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

The LSI card has been upgraded to IT rev 20.

The server this is being installed on is an IBM DX360-M3, the LSI card and the extender can as part of the setup, honestly never thought of upgrading the extender, just the card.

Thank you for the links, i will look into them immediately.

 

[Roger Mahler]

I ran the camcontrol identify /dev/ses0 and received the following
root@freenas ~]# camcontrol identify /dev/ses0
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

The LSI card has been upgraded to IT rev 20.

The server this is being installed on is an IBM DX360-M3, the LSI card and the extender can as part of the setup, honestly never thought of upgrading the extender, just the card.

Thank you for the links, i will look into them immediately.

 

[Roger Mahler]

I will look into upgrading, have two other 360's running that have not had problems (same everything, but staying up to date with drivers is not a bad thing). but not sure that is the problem as all drives report available.

thank you,

roger...

 

[jde]

I agree with your assessment - it would seem unlikely that the gross storage capacity of the disks would report correctly, but the net storage capcity of the pool would be wrong. It would seem more likely if the expander were the problem that both values would either be correct or both would be incorrect. But, as they say, go after the low hanging fruit first.

 

[Roger Mahler]

I greatly appreciate the guidance everyone has provided, are there any other recommendations or thoughts. i have a minimum of 2 more servers to build and am concerned i am missing something.