Older firmware for LSI 9305-16i

[artlessknave]

That's a bit overstated.

probably. i might have been a little melodramatic."you don't need to know the hardware" didn't go down well.
I was trying to nice-ish; complete success not expected.

 

[artlessknave]

650W gold+ PSU

also, it just occurred to me, the PSU brand can matter as well.
a 650W gold+ PSU that is poorly made is not always the same as a well made 650W gold+ PSU, even if they both meet the "gold" spec

 

[Octopuss]

you said, but as you apparently refuse to post any hardware (if I missed it, please let me know), all we have is hearsay.
when I try to ask questions to clear up the mysteries, you try to Jedi your way out of giving any answers....
"the power supply....is fine"
"the hardware only draws 120w"

the 9305 is going to be ~5-20w on its own. each hard drive is ~10-20W idle, a bit more when spinning up. unless you have an ATOM cpu, most cpus are ~50Widle, motherboards are about 20W or so, any IPMI will be ~10W.
if you have 10 drives, that's already over what you claim your hardware uses, just in drives. most motherboards have 4-8 SATA ports, a primary reason to get a 9305 is because the motherboard doesn't have enough ports, which implies 6-10 drives minimum.
we don't know how many drives you have, we don't know the hardware. diagnosing issues blind is a frustrating waste of everyone's time.

I'm not even saying it's the power supply, I'm saying there isn't enough evidence to rule it out, yet, and that simply *following the rules of the forum* would aid that.

See, whenever someone starts to push me, I automatically go into defensive mode and absolutely won't play the game. That's what happened here.
I know my system, you don't, I know how much electricity it draws, you do not. If I didn't know, I wouldn't say it.
I might not know much at all about servers, but I do know something about building a computer in general, and I'm telling you it's not related to the stupid PSU in any way imaginable. If you have two HBA cards that draw about the same amount of watts.... Actually screw it, why do I even waste keyboard on this nonsense?
Here, have at this:

That's what the server is idling at with six SSDs, a HBA, 10Gbit network card and six core Xeon.

 

[jgreco]

I automatically go into defensive mode and absolutely won't play the game. That's what happened here.

Please don't do this. Try to assume that the community is well-meaning, and that we have indeed seen lots of stuff over the years. I do this professionally and have posted a deep dive on figuring out how large a PSU you actually need, and that size has NOTHING to do with the number displayed on a Kill-a-Watt.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

I will note that the Forum Rules, conveniently linked at the top of every page in red, do ask that you post a detailed summary of your hardware. It is definitely a thing for people to come in with undersized PSU's on large systems. People might well think that you have a large system because you have a 9305-16i.

 

[Octopuss]

Is there a way to add system specs into your profile or something at least, like you see on various IT forums? I wouldn't mind filling that in, but being asked for detailed system specs everytime you ask about something and knowing for a fact (that's disputable of course, but one should assume people know what they physically have next to them, especially when it's kinda special kind of equipment) that it's simply not related gets annoying pretty quickly

Please don't do this. Try to assume that the community is well-meaning, and that we have indeed seen lots of stuff over the years.

I am a very, very impatient and short-tempered person, and unless I am absolutely certain of what I do, I ask about it. Otherwise I present stuff as hard facts. Maybe I am an autist in this regard, but I kind of expect everyone else to be the same. Is that not true?

 

[jgreco]

Is there a way to add system specs into your profile or something at least, like you see on various IT forums? I wouldn't mind filling that in, but being asked for detailed system specs everytime you ask about something and knowing for a fact (that's disputable of course, but one should assume people know what they physically have next to them, especially when it's kinda special kind of equipment) that it's simply not related gets annoying pretty quickly

Yes. Profile->Your Account->Signature.

 

[Octopuss]

That sounds like boasting to me, but if there's nothing better available, I guess I have no choice if it helps prevent confusion and thread pollution in future.

 

[jgreco]

What exactly sounds like boasting?

 

[Octopuss]

I always found it bizarre and stupid how people kept writing page long system specs of their badass gaming machines on computer forums. I don't think vast majority of people ever did it for troubleshooting purposes. That's it, basically. Yes I might be weird.

 

[Octopuss]

Ok, there we go. Hopefully it will help me avoid unnecessary back and forth in future.

 

[jgreco]

Ah, I see. For the purposes of gaming systems, that's quite possibly even true. I also don't see the need for clear panels or fans with LED's, which seem to be just for show.

I think it's like modding your car to make it "cool". Cool is in the eye of the beholder. Painting flames on the side of your car is all about impressions. Me, I drive a mostly-stock 2006 Toyota Highlander Hybrid Limited. It was one of the very first batch of HiHy's imported from Japan. Aftermarket add-ons are basically limited to a power inverter and a cell booster. If I decided to hit the pedal on the freeway onramp, I can blow past most traffic other than Teslas. That's just inherent to the vehicle though.

A NAS should be more like that than like a sports car with painted flames. But to that end, if your NAS doesn't have the oomph, we really need to understand what's going on underneath the hood. That will help us identify what might be going wrong. Since a lot of the art of building servers is unusual and different from building gaming PC's, we typically drill down when people are hesitant to share their specs.

 

[artlessknave]

Thank you for putting the hardware. it looks pretty solid, so I think that rules out issues there.

I notice "running ESXi 6.7"

you are running TrueNAS as a VM? if so, that complicates things. can you clarify how that is setup?

 

[Octopuss]

I'm not sure how to clarify. I run TrueNAS virtualized. I don't think there's anything more to say?
Besides the current HBA being somewhat incompatible at random, shutting the VM down every two years or so, I was nothing but happy about this setup.

 

[artlessknave]

there are many ways to virtualize TrueNAS that lead to problems, and only a handful that lead to reliable builds.
at this point, we have no clear idea which you are using; you have demonstrated being knowledgeable though reticent, so I'm hoping you have one of the reliable ways.
when virtualizing, the virtual config is as important as the hardware, and adds a layer of complexity that can cause further issues with troubleshooting.

one way to reduce that is to make an alternate test boot setup that it just TrueNAS, and fire it up to see if it can see all the hardware correctly without the VM layer. then when you are troubleshooting the VM you know for sure the hardware works as expected, and that the issues lie in the hypervisor. rather than guess where the source of the problems are.

 

[jgreco]

I'm not sure how to clarify. I run TrueNAS virtualized. I don't think there's anything more to say?
Besides the current HBA being somewhat incompatible at random, shutting the VM down every two years or so, I was nothing but happy about this setup.

There's lots more to say. It can work out very well or it can totally suck. That's why there are several articles dedicated to the topic. How many cores are dedicated to the VM? How much memory? Is PCIe passthru set up properly? Are you using E1000{,E} or vmxnet3? Did you do anything bad to try to make use of disk resources, such as RDM?

Virtualized, yes, you can end up with a very happy setup if you do it right.

 

[Octopuss]

I repeatedly read that the horrors of virtualizing TrueNAS were things of long time ago, and that seems to be my experience also. Besides that one problem I had never anything weird happen that would indicate something caused directly by virtualization.
And I have neither the time nor the luxury of having another box to altrernate between hardware to test things out. While nice to have, after running this for over two years, I think it's pretty stable.

 

[Octopuss]

There's lots more to say. It can work out very well or it can totally suck. That's why there are several articles dedicated to the topic. How many cores are dedicated to the VM? How much memory? Is PCIe passthru set up properly? Are you using E1000{,E} or vmxnet3? Did you do anything bad to try to make use of disk resources, such as RDM?

Virtualized, yes, you can end up with a very happy setup if you do it right.

Ahh I read you.
Well, those are more like configuration specifics than real problems caused by virtualization, aren't they? But I guess I understand what you mean.
I have two cores dedicated to TrueNAS, I think 12GB of RAM, passthrough is working correctly indeed, and I have no idea what E1000 or RDM is.

 

[artlessknave]

E1000

should be the virtual NIC type. i think esx had intel E1000 and one or 2 others? been awhile, and never used 6.7.

 

[Octopuss]

It's vmxnet 3.
I think I followed a guide how to set this up back then, so it should be correct. I didn't just randomly create a VM.

 

[jgreco]

I repeatedly read that the horrors of virtualizing TrueNAS were things of long time ago, and that seems to be my experience also.

You read wrong then. Here, from just this morning:

An io error bug in truenas11-12 version when using QEMU lun passthrough, raid card dell h730p

env： os:rhel 9 host:dell r730xd raid card:dell h730p (in raid6 mode) virtual machine：rhel 9 QEMU/KVM problem config: error log: freenas (da0:vtscsi0:0:0:2): CAM status: SCSI Status Error freenas (da0:vtscsi0:0:0:2): SCSI status: Check Condition freenas (da0:vtscsi0:0:0:2): SCSI...

www.truenas.com

Yikes.

Besides that one problem I had never anything weird happen that would indicate something caused directly by virtualization.

Very little has changed over the last ten years since I wrote the original virtualization stickies. ESXi is by far the best hypervisor for virtualization, though Proxmox is gaining credibility as well. Using other hypervisors is very hit-or-miss because TrueNAS is incredibly demanding and stressful on the hardware. Using virtual disk abstractions such as RDM is very dangerous because it tends not to be recoverable when something goes wrong. You can often use VM virtual disks (vmdk's for you ESXi folks) but they need to be on a redundant datastore or you can get a stall when something goes wrong. You can get all sorts of problems with stuff like interrupts trying to do PCIe passthru on subpar hypervisors. Networking can be a problem depending on how your hypervisor networking works. People tend to under-resource their virtual NAS because they want to be able to run other stuff on their platform that maxxes out at 32GB of RAM and four cores.

Well, those are more like configuration specifics than real problems caused by virtualization, aren't they?

You are welcome to think of them however you please. It is going to depend on what you mean by "real problems"; there's a lot of PEBCAK that goes on, where people who don't really know what they are doing end up painting themselves into a destructive corner. I guess I don't care to differentiate that between "configuration specifics" and "real problems caused by virtualization". Either way, they can eat your pool and ruin your day.

esx had intel E1000 and one or 2 others?

Depends on the "hardware version" of your VM. Typically the good choices are between E1000 and VMXNET3, though depending on the hw level, you might also be able to select E1000E (enhanced) or VMXNET2 or even VMXNET. Older hardware often supports LANCE (under some other name), and "Flexible" is a weird thing that identifies as the well-supported but low-performance LANCE but magically upgrades at runtime to VMXNET(3?) if VMware Tools is loaded.

I have run into weird issues with VMXNET3 but under 6.7 it seems to work okay unless you vMotion it. Since you shouldn't be vMotioning a NAS, you could run either VMXNET3 or E1000. I still may prefer E1000 because it lets you recover quickly if your hypervisor foo blows up.