SOLVED mps0: mpssas_prepare_remove: Sending reset for target ID 15 CAM status: CCB request aborted by the host

[VioletDragon]

Speaking as someone who does this stuff professionally, I find that most problems can be resolved but it often requires some time and effort, trial and error, to determine what's wrong. It's all very much harder when I have to work through your eyes and ears to proxy things through, and I don't actually get a chance to see your setup directly.

Hi jgreco, sorry about my anger, i have been doing some looking at the power distribution from my power supply to this backplain. I bought this chassis second hand have not touched anything else except install the motherboard, cpu and ram, cards and disks.

The scary thing i have found is that the previous owner has been running 12 disks from a single molex to a 3 molex splitter on a single 15amp rail. i have re-done the cabling to the backplane and the fan wall. i have added 3 molex direct from the Power Supply to the backplane using no splitter, the fans were using 2 dedicated molex from the Power Supply. Lets see if this helps i have noticed that the fan on the Power Supply is not screaming no more and less heat coming from the power supply.

 

[jgreco]

I understand anger, computers can bring out the worst in us humans. ;-) No worries there. It is good, however, that you are sticking with this.

You've just discovered a way in which it's super-important for you to be the eyes and ears, because while I kinda talked about power supply sizing and wire gauges and stuff upstream in post #22, there's no way for me to identify the issue you just found, and quite frankly, since I do this stuff professionally, I automatically attach the first "Molex" from each PSU lead to one port on the backplane and would never use a power splitter there, so it's difficult for me to picture something like what you found when I am running through the list of things that could be affecting you.

Hanging 12 disks off of a single feed could definitely cause voltage sag, and, it may not be what you want to hear, but brownouts do have the potential to do permanent damage to drives, and overdrawing on a PSU rail can stress PSU components as well. Modern designs are pretty elegant and robust, though. Cross your fingers, be mindful about the possibility, but it is only a possibility.

The good news is that you get to a certain point in these issues, and having eliminated the usual suspects, you get to the unusual stuff -- "it's gotta be hardware" -- and most of the time it then becomes expensive to remediate: new backplane, bad drive, bad PSU, bad HBA, etc., and it is really difficult for most home users to do these kinds of swapouts. I think you've hit the "aha" moment and there's a good chance you'll be good to go now, and the best news is that it was a cheap fix. That just makes my day.

I have to wonder if whoever sold you the chassis was having problems with it and got frustrated.

In any case, I hope this is your big fix, if so, please feel free to post a followup down the road and tell us how it went.

 

[VioletDragon]

I understand anger, computers can bring out the worst in us humans. ;-) No worries there. It is good, however, that you are sticking with this.

You've just discovered a way in which it's super-important for you to be the eyes and ears, because while I kinda talked about power supply sizing and wire gauges and stuff upstream in post #22, there's no way for me to identify the issue you just found, and quite frankly, since I do this stuff professionally, I automatically attach the first "Molex" from each PSU lead to one port on the backplane and would never use a power splitter there, so it's difficult for me to picture something like what you found when I am running through the list of things that could be affecting you.

Hanging 12 disks off of a single feed could definitely cause voltage sag, and, it may not be what you want to hear, but brownouts do have the potential to do permanent damage to drives, and overdrawing on a PSU rail can stress PSU components as well. Modern designs are pretty elegant and robust, though. Cross your fingers, be mindful about the possibility, but it is only a possibility.

The good news is that you get to a certain point in these issues, and having eliminated the usual suspects, you get to the unusual stuff -- "it's gotta be hardware" -- and most of the time it then becomes expensive to remediate: new backplane, bad drive, bad PSU, bad HBA, etc., and it is really difficult for most home users to do these kinds of swapouts. I think you've hit the "aha" moment and there's a good chance you'll be good to go now, and the best news is that it was a cheap fix. That just makes my day.

I have to wonder if whoever sold you the chassis was having problems with it and got frustrated.

In any case, I hope this is your big fix, if so, please feel free to post a followup down the road and tell us how it went.

The previous owner had this chassis in Storage he had a Dual Xeon Server inside this chassis so he says but maybe could be the reason why he got rid of it. I have noticed that the fan on the Power Supply is not screaming like it was, its temprature controlled but even on idle the fan was near flat out.

I will keep an eye on it for the next couple of days but i have 3 feeds direct from the Power Supply with no splitters another 2 to the FAN wall and a SATA cable from the Power Supply to the boot drives.

This Power Supply has 3 12v Power Rails +12v1 15amps, +12v2 15amps, +12v3 17amps. but not sure if both are for Molex/SATA or Motherboard Power?

 

[VioletDragon]

This looks more promising.

 

[VioletDragon]

I understand anger, computers can bring out the worst in us humans. ;-) No worries there. It is good, however, that you are sticking with this.

You've just discovered a way in which it's super-important for you to be the eyes and ears, because while I kinda talked about power supply sizing and wire gauges and stuff upstream in post #22, there's no way for me to identify the issue you just found, and quite frankly, since I do this stuff professionally, I automatically attach the first "Molex" from each PSU lead to one port on the backplane and would never use a power splitter there, so it's difficult for me to picture something like what you found when I am running through the list of things that could be affecting you.

Hanging 12 disks off of a single feed could definitely cause voltage sag, and, it may not be what you want to hear, but brownouts do have the potential to do permanent damage to drives, and overdrawing on a PSU rail can stress PSU components as well. Modern designs are pretty elegant and robust, though. Cross your fingers, be mindful about the possibility, but it is only a possibility.

The good news is that you get to a certain point in these issues, and having eliminated the usual suspects, you get to the unusual stuff -- "it's gotta be hardware" -- and most of the time it then becomes expensive to remediate: new backplane, bad drive, bad PSU, bad HBA, etc., and it is really difficult for most home users to do these kinds of swapouts. I think you've hit the "aha" moment and there's a good chance you'll be good to go now, and the best news is that it was a cheap fix. That just makes my day.

I have to wonder if whoever sold you the chassis was having problems with it and got frustrated.

In any case, I hope this is your big fix, if so, please feel free to post a followup down the road and tell us how it went.

Hi jgreco, 2 days of uptime and no CAM Errors seems that adapter was the culprit but it will keep an eye out for a new Power Supply as i want a redundant solution as im running iSCSI to Mail, Web Servers etc which is one of the reasons why i needed this sorted,

Thanks again.

Jack.

 

[VioletDragon]

I understand anger, computers can bring out the worst in us humans. ;-) No worries there. It is good, however, that you are sticking with this.

You've just discovered a way in which it's super-important for you to be the eyes and ears, because while I kinda talked about power supply sizing and wire gauges and stuff upstream in post #22, there's no way for me to identify the issue you just found, and quite frankly, since I do this stuff professionally, I automatically attach the first "Molex" from each PSU lead to one port on the backplane and would never use a power splitter there, so it's difficult for me to picture something like what you found when I am running through the list of things that could be affecting you.

Hanging 12 disks off of a single feed could definitely cause voltage sag, and, it may not be what you want to hear, but brownouts do have the potential to do permanent damage to drives, and overdrawing on a PSU rail can stress PSU components as well. Modern designs are pretty elegant and robust, though. Cross your fingers, be mindful about the possibility, but it is only a possibility.

The good news is that you get to a certain point in these issues, and having eliminated the usual suspects, you get to the unusual stuff -- "it's gotta be hardware" -- and most of the time it then becomes expensive to remediate: new backplane, bad drive, bad PSU, bad HBA, etc., and it is really difficult for most home users to do these kinds of swapouts. I think you've hit the "aha" moment and there's a good chance you'll be good to go now, and the best news is that it was a cheap fix. That just makes my day.

I have to wonder if whoever sold you the chassis was having problems with it and got frustrated.

In any case, I hope this is your big fix, if so, please feel free to post a followup down the road and tell us how it went.

Hey jgreco, not had anymore problems since removing that adapter but i have won a auction 2 Zippy Emacs Redundant Power Supply which is 620watt ive also picked up another 2 modules to have as spares will install later on this evening. Modules are cheap just under £10 each.

Cheers.

Jack.

 

[jgreco]

Awesome, thanks for the followup. Having debugged many problems over the years, I know both the pain and frustration of unsolved annoyances, and also the good feeling of having solved problems like this. Best wishes moving forward! ;-)

 

[VioletDragon]

Awesome, thanks for the followup. Having debugged many problems over the years, I know both the pain and frustration of unsolved annoyances, and also the good feeling of having solved problems like this. Best wishes moving forward! ;-)

No worries. Will let you know how i get one once i installed these Redundant Power Supplies.

 

[VioletDragon]

Awesome, thanks for the followup. Having debugged many problems over the years, I know both the pain and frustration of unsolved annoyances, and also the good feeling of having solved problems like this. Best wishes moving forward! ;-)

Hey jgreco, I have installed the Redundant Power Supplies and no problems like i was having before but i have a drive that died on me relating to bad sectors got the replacement on the way. The drive that has died has the date of 2015 so not done too bad. I have sorted out the cables before putting the cover back on. its a huge Power Supply lol.

 

[HoneyBadger]

That is a long unit for sure. Cables look a little cozy near the front (intake) side - I imagine the fans have enough static pressure to pull through there but it might get a bit loud.

Noted that you're running an iSCSI workload on this. What's the initiator-side OS? If you've got LUNs cut to something like Windows/Linux that will issue its own flush/sync-IO demands that's fine, but I'm a tiny bit concerned if there's a hypervisor in the middle that could potentially be assuming that the SAN-side is handling the data volatility concerns.

 

[VioletDragon]

That is a long unit for sure. Cables look a little cozy near the front (intake) side - I imagine the fans have enough static pressure to pull through there but it might get a bit loud.

Noted that you're running an iSCSI workload on this. What's the initiator-side OS? If you've got LUNs cut to something like Windows/Linux that will issue its own flush/sync-IO demands that's fine, but I'm a tiny bit concerned if there's a hypervisor in the middle that could potentially be assuming that the SAN-side is handling the data volatility concerns.

Chassis has 4 fans in. Where the cabling is where the 5th fan goes not populated.

ISCSI is going over 4G LAGG/LACP to 2 hypervisors running XCP-ng with 12 servers mail and Web servers as well as cloud storage servers. but ISCSI is not the issue here.

Issue was drives randomly removing themselves but that has been resolved now. The drive that has died is 7 years old as date on drive is from 2015.

 

[HoneyBadger]

Chassis has 4 fans in. Where the cabling is where the 5th fan goes not populated.

That's good. The rear fans in the two hot-swap PSUs should be capable of pulling air through there as the fans appear to be fairly high pressure.

ISCSI is going over 4G LAGG/LACP to 2 hypervisors running XCP-ng with 12 servers mail and Web servers as well as cloud storage servers. but ISCSI is not the issue here.

Issue was drives randomly removing themselves but that has been resolved now. The drive that has died is 7 years old as date on drive is from 2015.

The connection over LAGG is not an issue per se, but rather a potential area of improvement. iSCSI derives greater throughput by a method called MPIO (Multi-Path Input Output) which does not work correctly over LAGG/LACP. Changing your configuration now might prove challenging however.

The absence of an SLOG and use of asynchronous writes however may be an issue if you ever experience an "unexpected downtime" on the FreeNAS machine (eg: complete power loss, HBA failure, kernel panic) - XCP-ng and VMware vSphere both expect remote iSCSI LUNs to have their own non-volatile write cache, and the default sync=standard doesn't provide this. See the excellent resource from @jgreco here that explains this:

Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

Enabling sync=always on spinning disks will have serious negative consequences on your performance (read: "crippling") so you may need to consider an SLOG device if the risk of async is a concern.

 

[VioletDragon]

That's good. The rear fans in the two hot-swap PSUs should be capable of pulling air through there as the fans appear to be fairly high pressure.



The connection over LAGG is not an issue per se, but rather a potential area of improvement. iSCSI derives greater throughput by a method called MPIO (Multi-Path Input Output) which does not work correctly over LAGG/LACP. Changing your configuration now might prove challenging however.

The absence of an SLOG and use of asynchronous writes however may be an issue if you ever experience an "unexpected downtime" on the FreeNAS machine (eg: complete power loss, HBA failure, kernel panic) - XCP-ng and VMware vSphere both expect remote iSCSI LUNs to have their own non-volatile write cache, and the default sync=standard doesn't provide this. See the excellent resource from @jgreco here that explains this:

Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

Enabling sync=always on spinning disks will have serious negative consequences on your performance (read: "crippling") so you may need to consider an SLOG device if the risk of async is a concern.

I have had problems with 10G high CPU usage and un-responsive Network on the iSCSI Network and anyway i wanted a Ha configuration so im using 2 switches for Ha. Ive tried different 10G cards but all had the same issue so went back to 4G LAGG over 1G Network which is more reliable.

I will look into the SLOG but i am not breaking my environment as i have Mail Servers for Business as well as my parents business and Web Servers. I think this is as good as i am going to get with this hardware.

I will most likely buy a DAS for iSCSI but what i have will do for now. I cant afford all the latest stuff which is the problem.

I have turned off sync=always.

 

[VioletDragon]

Awesome, thanks for the followup. Having debugged many problems over the years, I know both the pain and frustration of unsolved annoyances, and also the good feeling of having solved problems like this. Best wishes moving forward! ;-)

Just another update all problems are cured. Server is happily running again and working great. Replaced a drive that is 2 years over the warranty but other than that its working great again. No problems like before with drives randomly detaching.

These new Power Supplies are great, i can monitor them in the boards IPMI :D

Can be marked as solved.

Thanks.