Idle reboot bug: aacraid0: COMMAND 0xfffffe00007bdfa0 TIMEOUT AFTER 3857 SECONDS, shutting down controller...done

[bsdimp]

Has anybody tried my patch? I'd like to get this issue resolved and I think that my patch is more correct than the earlier ones proposed, but lack hardware to test my belief.

 

[mdedetrich]

Has anybody tried my patch? I'd like to get this issue resolved and I think that my patch is more correct than the earlier ones proposed, but lack hardware to test my belief.

So unfortunately I wasn't about to test the patch because I didn't have a good time to reset our machines that have the AACRAID card. I am going to do an upgrade of the systems roughly during the next week and I am also willing to test the patch at the same time (the machines run 24/7 with around a petabyte of data).

The only issue is a support way of patching the TrueNAS FreeBSD kernel, there seems to be zero documentation on this and someone already asked in https://www.truenas.com/community/threads/how-to-run-custom-version-of-freebsd-in-freenas.74524/ with no response. Ideally I would like to add the patched kernel as an addition entry as a boot option so if there is some issue I can quickly swap back to the official kernel.

I created an upstream TrueNAS ticket on how to document this at https://jira.ixsystems.com/browse/NAS-112209

 

[jgreco]

TrueNAS doesn't support this because it's an appliance, not a general purpose FreeBSD host, and it doesn't have a full set of tools onboard to compile. In order to do this very bad idea, you have to stage a similar FreeBSD host, do your build there, and propagate the file back, and then every time something comes along that messes with this, like an upgrade, do that process again.

 

[mdedetrich]

TrueNAS doesn't support this because it's an appliance, not a general purpose FreeBSD host, and it doesn't have a full set of tools onboard to compile. In order to do this very bad idea, you have to stage a similar FreeBSD host, do your build there, and propagate the file back, and then every time something comes along that messes with this, like an upgrade, do that process again.

Thats fine, but I was also talking about creating a manual TrueNAS build on another FreeBSD host to create a patched FreeBSD kernel, I couldn't find any documentation on this.

There is sparse documentation on how to build TrueNAS from scratch but thats pretty much it.

 

[jgreco]

Thats fine, but I was also talking about creating a manual TrueNAS build on another FreeBSD host to create a patched FreeBSD kernel, I couldn't find any documentation on this.

There is sparse documentation on how to build TrueNAS from scratch but thats pretty much it.

Really? It's right there with the source code.

https://github.com/truenas/core-build

But in general this isn't encouraged or supported.

I get the feeling that you're expecting a beginner-level HOWTO that walks you through all the exact steps of taking a patch, applying it, and building, I don't know that such a thing exists. I mean, it's easy for someone who has worked with FreeBSD for years and who regularly builds kernels and stuff like that to do, but it is kinda like driving to the store... a lot of things that are just assumed knowledge, like how to open the car door, how the seatbelt works, which way to turn the key, how to put the car into drive, which turns to make, etc. Building a new FreeNAS system is going to be very dependent on having some existing experience with building a FreeBSD system from sources, combined with being able to follow the general directions included in the Github repo. Any instructions written a year ago won't be completely applicable today, and any instructions written today would likely be somewhat outdated a year from now. And there's just no incentive for anyone to maintain such a thing, because you're not supposed to be doing it to begin with.

 

[mdedetrich]

Really? It's right there with the source code.

https://github.com/truenas/core-build

But in general this isn't encouraged or supported.

I don't know what you mean by "right there", if you mean the actual FreeBSD source then it isn't, it instead gets pulled by the build system which appears to be custom and written in python (hence the documentation part)

I get the feeling that you're expecting a beginner-level HOWTO that walks you through all the exact steps of taking a patch, applying it, and building, I don't know that such a thing exists. I mean, it's easy for someone who has worked with FreeBSD for years and who regularly builds kernels and stuff like that to do, but it is kinda like driving to the store... a lot of things that are just assumed knowledge, like how to open the car door, how the seatbelt works, which way to turn the key, how to put the car into drive, which turns to make, etc. Building a new FreeNAS system is going to be very dependent on having some existing experience with building a FreeBSD system from sources, combined with being able to follow the general directions included in the Github repo. Any instructions written a year ago won't be completely applicable today, and any instructions written today would likely be somewhat outdated a year from now.

Of course it exists, I do it all the time on Linux (I use Archlinux/Manjaro) and they have extensive documentation and even build systems/package managers that allow you to apply patches/diffs (to both packages and the kernel) such as https://github.com/Jguer/yay. Its okay if the reason for it not existing is because TrueNAS doesn't have enough people to work on the product but claiming this is the norm for open source projects of this scale is disingenuous.

I mean its fine, I will figure this out myself it will just take a bit more time since I am not that familiar with FreeBSD.

And there's just no incentive for anyone to maintain such a thing, because you're not supposed to be doing it to begin with.

And how do you expect this to get tested then? I mean you are describing a catch 22 situation. An upstream committer of FreeBSD took his own time creating a patch to fix this specific problem and I doubt its going to get merged until it gets tested (which I am willing to do acknowledging the risks).

This is the whole spirit and ethos of open source, if you don't want people testing the system outside of what I suppose is ixSystems then why even bother making the TrueNAS build open?

 

[Ericloewe]

You make an important and valid point, but if the answers were easy we wouldn't be discussing it. Since you're willing to dive into this, my best suggestion is to share your feedback on the experience with the community and, more importantly, with the devs.

In any case, if you want to do this by building TrueNAS, I'd try applying the patch after the make update and before the make release steps, judging by the instructions in the Readme. This is just from a quick glance by someone who hasn't actually tried this before, so let us know how it goes.

 

[jgreco]

Of course it exists, I do it all the time on Linux (I use Archlinux/Manjaro) and they have extensive documentation and even build systems/package managers that allow you to apply patches/diffs (to both packages and the kernel) such as https://github.com/Jguer/yay. Its okay if the reason for it not existing is because TrueNAS doesn't have enough people to work on the product but claiming this is the norm for open source projects of this scale is disingenuous.

I made no such claim about "open source projects of this scale." But, this also isn't an environment like Linux or FreeBSD, where code contributions are routinely expected and accepted from the general public.

You have missed the point, which is that FreeNAS/TrueNAS is an appliance firmware. It is not really something that you are expected to be able to patch or modify to your own taste. Because it is based on FreeBSD, and because iXsystems has graciously made the source code available, it is actually possible for people with requisite skill and abilities to made modifications or apply patches, but this does not somehow cause it to become equivalent to the FreeBSD/Linux situation. It still requires some experience with the underlying systems, and because beginners are not expected to be making changes to the system firmware, there are no HOWTO style documents, so I call B.S. on:

Of course it exists, I do it all the time on Linux (I use Archlinux/Manjaro) and they have extensive documentation and even build systems/package managers

this. If this is true, please build me a Ubiquiti Unifi AP firmware. I mean, that's built on Linux, right? So this should be easy for you to do?

And how do you expect this to get tested then? I mean you are describing a catch 22 situation. An upstream committer of FreeBSD took his own time creating a patch to fix this specific problem and I doubt its going to get merged until it gets tested (which I am willing to do acknowledging the risks).

I am not describing a catch 22 situation. I could apply Warner's patch easily. The problem is, I'm not going to go out and put a flaky RAID controller into one of my systems in an effort to "fix" something that is only broken for people who haven't followed the hardware recommendations.

I don't expect this to get tested until someone who has both motivation and experience wants to test it.

This is the whole spirit and ethos of open source, if you don't want people testing the system outside of what I suppose is ixSystems then why even bother making the TrueNAS build open?

Well, seeing as how none of us here are iXsystems employees, I can only speculate that iXsystems is very generous and is more committed to the concept of open source development than many other corporate consumers such as Ubiquiti. On the other hand, they are a small team, and they cannot really afford to tilt at windmills like Adaptec.

Part of what we've tried to do here in the forums is to guide people towards the hardware that is known to work well, because it lets iX focus their limited resources on making an awesome thing even better, rather than worrying about bugs found in random jankydevices. The open source angle lets them benefit from the experience and contributions of seasoned developers who can target portions of the system that they've not yet gotten around to.

The impression I get from you is that you seem to feel someone owes you a HOWTO about how to do this thing that you want, and because it's not a thing most people do, it seems implied that you feel some level of step-by-step beyond what iXsystems has already provided is due you.

You are absolutely welcome to

test[...] the system outside of what I suppose is ixSystems

which is why it is open source. That should be obvious. This doesn't mean anyone owes you a detailed step-by-step if you do not have the basic understanding of how to do these things.

An upstream committer of FreeBSD took his own time creating a patch to fix this specific problem

Yup, and I kinda imagine that this was instigated, and Warner popped up in this thread, because you somehow began that process, and Warner's awesome for trying to help (as has been the case for decades). But perhaps you should reflect on this a bit. You seem to be implying that "we've" wasted Warner's time producing a patch which isn't being tested. That is technically correct, but I submit that this could be more accurately be summed as YOU have wasted Warner's time, if you didn't have the ability to apply a patch that you were given.

I don't see a practical solution to this problem that doesn't involve someone ELSE getting involved and donating time to do a build, and then testing on a RAID card, or maybe iX will have a developer step in and provide you with your more detailed HOWTO, which is of course taking them away from more important things. What a mess.

At the end of the day, the pragmatic thing to have done here would have been to just use the known-highly-compatible HBA's inexpensively available.

 

[mdedetrich]

Unfortunately as you can tell from the reply at https://github.com/freebsd/freebsd-src/pull/523#issuecomment-1418071819 the patch is now dead and unless someone else has the time to take up testing it or some documentation is provided on how to run TrueNAS with a patched kernel, so you know, people can actually test bug fixes in an open source project.

If https://ixsystems.atlassian.net/browse/NAS-112209 gets resolved and some documentation gets created, then someone can ping me and I might be able to find some time to look into it.

 

[mdedetrich]

Yup, and I kinda imagine that this was instigated, and Warner popped up in this thread, because you somehow began that process, and Warner's awesome for trying to help (as has been the case for decades). But perhaps you should reflect on this a bit. You seem to be implying that "we've" wasted Warner's time producing a patch which isn't being tested. That is technically correct, but I submit that this could be more accurately be summed as YOU have wasted Warner's time, if you didn't have the ability to apply a patch that you were given.

I am sorry but this mentality is just hogwash. I work full time as an open source developer as well as maintaining various projects and if a project is unable to even document how the community should help contribute then thats a fault of the project, not people like myself. The mentality of your post just demonstrates that.

I manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups using these aacraid cards and I was willing to test this even for months/years to see if its working, so I guess you can pat yourself on the back for a wasted opportunity because it seems its more important to "discourage people to not test official setups" (just typing this sentence feels so wrong) rather than provide basic documentation/instructions which is all I was asking for. I unfortunately don't have the time anymore to figure out how to do this myself anymore but if someone provides instructions on how to do it, I can look into it.

 

[jgreco]

I am sorry but this mentality is just hogwash. I work full time as an open source developer as well as maintaining various projects

That's nice. I maintain Diablo and also run infrastructure for NTP.ORG, along with having been a FreeBSD contributor since the 386BSD days, and a major documentation force here on the forums.

if a project is unable to even document how the community should help contribute then thats a fault of the project, not people like myself. The mentality of your post just demonstrates that.

Lots of projects are hesitant to try to micromanage issues that are actually a result of upstream sources; iXsystems has occasionally attempted to do so in the past, and it has generally been a bit problematic.

You seem a bit confused about what this project is, though. This is not a project to build a general purpose NAS package that works with random hardware. This is a project to build the firmware for TrueNAS Enterprise, a very specific hardware product sold by iX. Because they're super-cool, because they like to exploit the community to locate bugs before their paying customers, and because it was easy to do, iX has allowed the community to use a slightly feature-reduced version for free. This costs them essentially nothing; FreeBSD has been polished as a server OS for many years, and especially the vendor-provided drivers for stuff like Chelsio, Intel, and LSI are super high quality. iXsystems has made it clear that they do not care to spend developer time trying to make stuff like the Broadcom or Emulex ethernet stuff work, and there's a large variety of other hardware that is dodgy or simply not suitable, and iX has made it clear that they're not really interested in supporting a custom fork of FreeBSD to fix driver issues for janky hardware. This means that the path to success for TrueNAS is to make your hardware platform more closely mirror what iX is selling, not kvetching about how the community or project is at fault for not doing what you'd like to see done.

I manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups

Congratulations. About 20 years ago, my shop was putting out some of the first 24-bay server machines running FreeBSD for large scale storage applications. I remember when a petabyte was two dozen racks filled with such storage servers.

and I was willing to test this even for months/years to see if its working, so I guess you can pat yourself on the back for a wasted opportunity because it seems its more important to "discourage people to not test official setups"

This feels like you're trying to shift the blame here. You were invited to test the patch Warner provided, and I even provided loose guidance and a pointer to the build instructions.

you have to stage a similar FreeBSD host, do your build there, and propagate the file back, and then every time something comes along that messes with this, like an upgrade, do that process again.

It's right there with the source code.

https://github.com/truenas/core-build

This isn't that hard to do.

rather than provide basic documentation/instructions which is all I was asking for.

I wouldn't expect a

full time [...] open source developer

to need handholding through the process of building a FreeBSD kernel. I already provided the link to the TrueNAS-specific bits; the general FreeBSD instructions are over in the FreeBSD Handbook. You seem to think that the project owes you some kind of bespoke documentation/instructions on exactly how to do this, but the problem with that is that is that it would require someone to actually figure this out for you, and none of us have a vested interest in solving this problem. We're just community members too, and most of us are not that interested in solving this problem for your off-brand obsolete RAID controller from a decade ago.

I'm sure we have all the parts in inventory here, certainly a 12-bay is easy, X9 board too, and I know we've got some Adaptec ASR-somethingsomething's (71605?) as well, but quite frankly I don't understand what my motivation would be to waste several hours on a build and then figure out how to build your custom firmware image and document that for you. Most of the open source developers I work with understand this problem and at a minimum know to ask nicely rather than expecting, demanding, or whining for an outcome. There's this weird thing in open source that's happened in recent years where people act like they're paying customers and they're owed answers. I personally find this distasteful and my hair bristles at it. Doesn't make me want to jump out of my chair and run back to the shop to find answers.

 

[mdedetrich]

You seem a bit confused about what this project is, though. This is not a project to build a general purpose NAS package that works with random hardware. This is a project to build the firmware for TrueNAS Enterprise, a very specific hardware product sold by iX. Because they're super-cool, because they like to exploit the community to locate bugs before their paying customers, and because it was easy to do, iX has allowed the community to use a slightly feature-reduced version for free. This costs them essentially nothing; FreeBSD has been polished as a server OS for many years, and especially the vendor-provided drivers for stuff like Chelsio, Intel, and LSI are super high quality. iXsystems has made it clear that they do not care to spend developer time trying to make stuff like the Broadcom or Emulex ethernet stuff work, and there's a large variety of other hardware that is dodgy or simply not suitable, and iX has made it clear that they're not really interested in supporting a custom fork of FreeBSD to fix driver issues for janky hardware. This means that the path to success for TrueNAS is to make your hardware platform more closely mirror what iX is selling, not kvetching about how the community or project is at fault for not doing what you'd like to see done.

Then they should either not use FreeBSD or they should strip the FreeBSD source of drivers that they do not support. You are creating a false dichotomy here, if iXSystems is using FreeBSD then by definition their system is supporting more then they claim. There is an overlap here whether people realize it or not.

Congratulations. About 20 years ago, my shop was putting out some of the first 24-bay server machines running FreeBSD for large scale storage applications. I remember when a petabyte was two dozen racks filled with such storage servers.

Nice work on missing the entire point I was making (hint it wasn't to show off, it was to demonstrate I was willing to test it on a non trivial setup for an extended period of time).

This feels like you're trying to shift the blame here. You were invited to test the patch Warner provided, and I even provided loose guidance and a pointer to the build instructions.

Yes and as I claimed before, I am not that familiar with TrueNAS build system and the documentation on how to run it with a custom kernel is completely missing. I created an official iXSystems ticket on this as well and it still hasn't received any responses.

You are also twisting what Warden's frustration was, he wasn't complaining that I specifically wasn't testing it but rather that the discussion happening both on this forum and the the ticket wasn't getting anywhere constructive and he is completely correct.

You seem to think that the project owes you some kind of bespoke documentation/instructions on exactly how to do this, but the problem with that is that is that it would require someone to actually figure this out for you, and none of us have a vested interest in solving this problem. We're just community members too, and most of us are not that interested in solving this problem for your off-brand obsolete RAID controller from a decade ago.

For starters no one owes anyone anything, this is open source software not a contract (not sure where you got that from). I am merely stating that not having documentation is generally not good for an open source project and the fact that you stated that no one has a "vested interest" already states where the problem is.

I'm sure we have all the parts in inventory here, certainly a 12-bay is easy, X9 board too, and I know we've got some Adaptec ASR-somethingsomething's (71605?) as well, but quite frankly I don't understand what my motivation would be to waste several hours on a build and then figure out how to build your custom firmware image and document that for you. Most of the open source developers I work with understand this problem and at a minimum know to ask nicely rather than expecting, demanding, or whining for an outcome. There's this weird thing in open source that's happened in recent years where people act like they're paying customers and they're owed answers. I personally find this distasteful and my hair bristles at it. Doesn't make me want to jump out of my chair and run back to the shop to find answers.

TBH, I think you are just overly defensive (from what it seems like having being stuck in an ixSystems bubble?) and are not willing to listen to constructive criticism.

For starters you seem to imply that I am asking everyone to do all of the effort, I am not. If I was to test TrueNAS with this patched kernel, I still have to set it up on all of the hardware, monitor it and then deal with the risk that the patch might make things worse. This is not a case of some random internet person asking someone to do everything.

What you seem to be conveniently ignoring is this is just a case of pragmatism, as is quite clear I am not that familiar with TrueNAS/FreeBSD so its actually far more efficient for someone that is completely familiar with TrueNAS to document what I was suggesting and then I would handle the building/testing rather than figuring out everything for myself. I am not forcing anyone to do anything here and there isn't some grand expectation of this, but if it doesn't occur then it makes things more difficult for everyone (which is exactly what happened). This isn't some weird or foreign concept, this is how well maintained open source projects are meant to work in building a community otherwise you end up in the case where almost all of the work needs to be done by a small number of people. Yes it does require effort to generate such documentation, but as a return you create a lot of "free" labor from the community (and in this specific case, that "free" labor is my labor).

And on the point of open source projects, the ones that don't provide documentation (or more generally don't make it easier for contributors) are projects that on the long term tend to die and if they do survive its because they are completely beholden to a single company which creates its own set of problems that often just kicks the can down the road (fun fact, the reason why I don't have that much time now is I am maintaining a newly created hard fork of an open source software that changed its license from an open one to a business one, and a large part why the project changed its license is because of effects that was created as a result of the project mainly being driven by a single company).

 

[jgreco]

Then they should either not use FreeBSD or they should strip the FreeBSD source of drivers that they do not support. You are creating a false dichotomy here, if iXSystems is using FreeBSD then by definition their system is supporting more then they claim. There is an overlap here whether people realize it or not.

This is ridiculous. An open source project like FreeBSD contains drivers for lots of things that will never be used; the audio, wifi, Bluetooth, and SCSI tape changer devices come immediately to mind as great examples. There are also lots of system executables that will never be used, such as finger(1) or talk(1). You are essentially talking about creating a custom fork of FreeBSD. That's been done, it was called TrueOS, and it was a difficult-to-maintain disaster that mostly existed because one of the developers was also using it as a platform for PC-BSD. iXsystems has experience maintaining a fork, and it's a lot of work. TrueOS didn't even "strip the FreeBSD source of drivers that they do not support"; this is just insanity. It was basically just to integrate stuff like the latest video drivers and stuff like that to make the desktop environment work.

by definition their system is supporting more then they claim.

By definition this is a steaming pile; they never made any claims about what they support. What we push here in the forums is a pragmatic set of answers that is known to work; what kind of a shitheel would I be if I advocated for new users to use things known not to work well? Sure, go ahead and use USB-attached hard drives with your little $50 ITX mainboard on 8GB of RAM and a Realtek ... it's all fine, it'll work... but it won't work well (or maybe just barely) and that's no fair to anyone. You don't seem to understand that there's going to be some stuff out there that isn't really acceptable for use with the NASware. This isn't really anyone's fault. The PC world is full of subpar race-to-the-bottom hardware that barely works on Windows. iXsystems isn't going to go to any significant effort to make that work, and here on the forums, we're not going to recommend it, even if it might be "supported" by FreeBSD.

Nice work on missing the entire point I was making (hint it wasn't to show off, it was to demonstrate I was willing to test it on a non trivial setup for an extended period of time).

I didn't miss the point. You missed mine. My point is that I've been doing this a hella-long time and have been testing a variety of stuff in production at scale for an extended period of time. I seeded this forum with a lot of wisdom earned from the School of Hard Knocks, and I've also carefully tried to gather as much additional wisdom about what works and what doesn't in the decade-plus I've been a participant here.

Yes and as I claimed before, I am not that familiar with TrueNAS build system and the documentation on how to run it with a custom kernel is completely missing. I created an official iXSystems ticket on this as well and it still hasn't received any responses.

Running it with a custom kernel is simply not an option that anyone on the development team conceived of. The way you run a custom kernel is to build a custom firmware and install it. If you don't like it, you roll back to the previous boot environment. That should be obvious to anyone who has updated TrueNAS once or twice. There's no "documentation" for this other than "do the obvious."

I already linked you to the build resources for TrueNAS as well.

For starters no one owes anyone anything, this is open source software not a contract (not sure where you got that from).

Probably from what comes across in this conversation as a perennially entitled attitude where people owe you various things. This is reinforced because you seem to be mostly talk, not appearing to have taken any action on Warner's request to try a patch.

I am merely stating that not having documentation is generally not good for an open source project and the fact that you stated that no one has a "vested interest" already states where the problem is.

There's plenty of documentation. The FreeBSD Handbook. Design and Implementation by McKusick and Watson. UTSL. You aren't even following guidance and patches provided to you on a forum. You keep suggesting that there's NO documentation when what you really mean is that no one has written a Linux style HOWTO to do the very weird and very specific thing that you want to do that isn't really that hard to do. It's open source. You see a documentation shortcoming. Address it.

TBH, I think you are just overly defensive (from what it seems like having being stuck in an ixSystems bubble?) and are not willing to listen to constructive criticism.

I don't have anything to do with iXsystems except that now and then they send me a coffee mug or stickers or one time they sent an awesome Beastie plushie; you can see him being strangled in my avatar. I do occasionally reprocess iXsystems gear in the shop here for my customers. I like their stuff, but that's about it. I don't control the direction of the project, I'm not a developer for the project, I do write documentation here on the forum, but you seem to be laboring under an illusion that you can provide your "constructive criticism" to me and something is magically going to happen as a result. It probably won't. If I had that sort of sway, there's a lot of stuff that'd be just a little different about this project.

For starters you seem to imply that I am asking everyone to do all of the effort, I am not. If I was to test TrueNAS with this patched kernel, I still have to set it up on all of the hardware, monitor it and then deal with the risk that the patch might make things worse. This is not a case of some random internet person asking someone to do everything.

Functionally null paragraph. You already claimed to "manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups using these aacraid cards" so presumably you have the lab resources to test this. Yet you don't seem to have actually bothered to build and test Warner's patch. It feels to me like you will next blame this on a "lack of documentation". So if you are not asking everyone else to do the work here, and you are not doing the work here, this seems like another vapid thread where someone pointlessly argues that their RAID controller should be just fine but damn it their mfi0 keeps throwing timeouts, just ignoring the obvious fix in favor of boring argument.

What you seem to be conveniently ignoring is this is just a case of pragmatism, as is quite clear I am not that familiar with TrueNAS/FreeBSD so its actually far more efficient for someone that is completely familiar with TrueNAS to document what I was suggesting and then I would handle the testing.

So, someone write up a Linux HOWTO of how to do this, because the available instructions are not specific enough. So let me help you here. If I go over to https://github.com/truenas/core-build we see:

Requirements:​

• Hardware
  • CPU: amd64-compatible 64-bit Intel or AMD CPU.
  • 16GB memory, or the equivalent in memory plus swap space
  • at least 80GB of free disk space
• Operating System
  • The build environment must be FreeBSD 13.x (or 13-STABLE)

So you prepare your FreeBSD 13-amd64 system. I'll clue you in that this could be a chroot environment on your NAS.

Then you do this crap:

Procedure​

• Install git
  pkg install -y git
  rehash
  
  
• Clone the build repository (/usr/build is used for this example):
  git clone https://github.com/truenas/build /usr/build
  
  
• Install Dependencies
  cd /usr/build
  make bootstrap-pkgs
  python3 -m ensurepip
  pip3 install six
  
  
• First-time checkout of source:
  make checkout
  

A FreeNAS release is built by first updating the source, then building:

make update
make release
To build the SDK version:

make update
make release BUILD_SDK=yes
Clean builds take a while, not just due to operating system builds, but because poudriere has to build all of the ports. Later builds are faster, only rebuilding files that need it.

Use make clean to remove all built files.

Then you get your results.

Results​

Built files are in the freenas/_BE subdirectory, /usr/build/freenas/_BE in this example.

ISO files: freenas/_BE/release/TrueNAS-13-MASTER-{date}/x64/.

Update files: freenas/_BE/release/.

Log files: freenas/_BE/objs/logs/.

Then you install your ISO from the ISO directory. Once you can do all this, then you go back and patch your tree with Warner's patch, repeat the "make release", then install your customized ISO.

This is just SOOOOO complicated, I can see why you need bespoke instructions.

I am not forcing anyone to do anything and there isn't some grand expectation of this, but if it doesn't occur then it makes things more difficult for everyone (which is exactly what happened). This isn't some weird or foreign concept, this is how well maintained open source projects are meant to work in building a community otherwise you end up in the case where almost all of the work needs to be done by a small number of people. Yes it does require effort to generate such documentation, but as a return you create a lot of "free" labor from the community.

I feel little need to respond to this steaming pile of BS. TrueNAS is a commercial product with paid developers. Your impressions of how you feel an open source project should work has almost zero relevance. They're not writing it for you and you are not the intended customer. Most likely no one over at iX cares if your obsolete deprecated non-recommended RAID card doesn't work and they're happy to let upstream address the problem, as has been their pattern in recent years.

And on the point of open source projects, the ones that don't provide documentation (or more generally don't make it easier for contributors) are projects that on the long term tend to die and if they do survive its because they are completely beholden to a single company which creates its own set of problems that often just kick the can down the road

That's nice. Apparently iXsystems is dying twelve years into the project to the tune of $100M annual revenue.

Anyways this is meaningless because I have spent a lot of futile time trying to impart clue.

 

[mdedetrich]

This is ridiculous. An open source project like FreeBSD contains drivers for lots of things that will never be used; the audio, wifi, Bluetooth, and SCSI tape changer devices come immediately to mind as great examples. There are also lots of system executables that will never be used, such as finger(1) or talk(1). You are essentially talking about creating a custom fork of FreeBSD. That's been done, it was called TrueOS, and it was a difficult-to-maintain disaster that mostly existed because one of the developers was also using it as a platform for PC-BSD. iXsystems has experience maintaining a fork, and it's a lot of work. TrueOS didn't even "strip the FreeBSD source of drivers that they do not support"; this is just insanity. It was basically just to integrate stuff like the latest video drivers and stuff like that to make the desktop environment work.

Of course its ridiculous, thats the point I am making because its a sequitur of your original statement. TrueNAS uses FreeBSD, which is an open source kernel + OS. That means they are benefiting from FreeBSD, otherwise they would have to build their own kernel and operating system. This means its in their interest to also help the development of FreeBSD because it also helps back, even if that initial help isn't directly relevant

By definition this is a steaming pile; they never made any claims about what they support. What we push here in the forums is a pragmatic set of answers that is known to work; what kind of a shitheel would I be if I advocated for new users to use things known not to work well? Sure, go ahead and use USB-attached hard drives with your little $50 ITX mainboard on 8GB of RAM and a Realtek ... it's all fine, it'll work... but it won't work well (or maybe just barely) and that's no fair to anyone. You don't seem to understand that there's going to be some stuff out there that isn't really acceptable for use with the NASware. This isn't really anyone's fault. The PC world is full of subpar race-to-the-bottom hardware that barely works on Windows. iXsystems isn't going to go to any significant effort to make that work, and here on the forums, we're not going to recommend it, even if it might be "supported" by FreeBSD.

This is entirely a strawman so its not worth addressing

I didn't miss the point. You missed mine. My point is that I've been doing this a hella-long time and have been testing a variety of stuff in production at scale for an extended period of time. I seeded this forum with a lot of wisdom earned from the School of Hard Knocks, and I've also carefully tried to gather as much additional wisdom about what works and what doesn't in the decade-plus I've been a participant here.



Running it with a custom kernel is simply not an option that anyone on the development team conceived of. The way you run a custom kernel is to build a custom firmware and install it. If you don't like it, you roll back to the previous boot environment. That should be obvious to anyone who has updated TrueNAS once or twice. There's no "documentation" for this other than "do the obvious."

No one is suggesting that under normal production use people should use TrueNAS with custom kernels, what is being suggested is to allow people in the community to easily test fixes/bugs (which are often patches to the kernel) to see if the fixes/bugs work.

Your entire train of thought seems to imply there is no distinction between testing and production use

I already linked you to the build resources for TrueNAS as well.



Probably from what comes across in this conversation as a perennially entitled attitude where people owe you various things. This is reinforced because you seem to be mostly talk, not appearing to have taken any action on Warner's request to try a patch.

I did actually attempt this, but it was over a year ago and I came across some issues which I have already forgotten. I would have to dig up my custom machine I used to build FreeBSD

There's plenty of documentation. The FreeBSD Handbook. Design and Implementation by McKusick and Watson. UTSL. You aren't even following guidance and patches provided to you on a forum. You keep suggesting that there's NO documentation when what you really mean is that no one has written a Linux style HOWTO to do the very weird and very specific thing that you want to do that isn't really that hard to do. It's open source. You see a documentation shortcoming. Address it.

That isn't what I was suggesting at all, I would read my statement again

I don't have anything to do with iXsystems except that now and then they send me a coffee mug or stickers or one time they sent an awesome Beastie plushie; you can see him being strangled in my avatar. I do occasionally reprocess iXsystems gear in the shop here for my customers. I like their stuff, but that's about it. I don't control the direction of the project, I'm not a developer for the project, I do write documentation here on the forum, but you seem to be laboring under an illusion that you can provide your "constructive criticism" to me and something is magically going to happen as a result. It probably won't. If I had that sort of sway, there's a lot of stuff that'd be just a little different about this project.

Your statements give a strong implication that you speak on behalf of iXSystems, as if you are speaking on an altar for them.

Functionally null paragraph. You already claimed to "manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups using these aacraid cards" so presumably you have the lab resources to test this. Yet you don't seem to have actually bothered to build and test Warner's patch. It feels to me like you will next blame this on a "lack of documentation". So if you are not asking everyone else to do the work here, and you are not doing the work here, this seems like another vapid thread where someone pointlessly argues that their RAID controller should be just fine but damn it their mfi0 keeps throwing timeouts, just ignoring the obvious fix in favor of boring argument.

Well your feelings and assumptions are completely wrong so you should probably look into why that is the case. Hint, my name is quite public and if you did some very basic research you would easily find that I spend a lot of time contributing to other projects.

So, someone write up a Linux HOWTO of how to do this, because the available instructions are not specific enough. So let me help you here. If I go over to https://github.com/truenas/core-build we see:



So you prepare your FreeBSD 13-amd64 system. I'll clue you in that this could be a chroot environment on your NAS.

Then you do this crap:



Then you get your results.



Then you install your ISO from the ISO directory. Once you can do all this, then you go back and patch your tree with Warner's patch, repeat the "make release", then install your customized ISO.

This is just SOOOOO complicated, I can see why you need bespoke instructions.

As I said there was issues in building this and the documentation has been updated since I tried this over a year go.

I feel little need to respond to this steaming pile of BS. TrueNAS is a commercial product with paid developers.

As is a lot of other open software, nothing new here

Your impressions of how you feel an open source project should work has almost zero relevance. They're not writing it for you and you are not the intended customer.

As is also the case for a lot of open source software, again nothing new here.

Most likely no one over at iX cares if your obsolete deprecated non-recommended RAID card doesn't work and they're happy to let upstream address the problem, as has been their pattern in recent years.

Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)

That's nice. Apparently iXsystems is dying twelve years into the project to the tune of $100M annual revenue.

Anyways this is meaningless because I have spent a lot of futile time trying to impart clue.

Yes and you are also missing the point entirely because its not about the financials of the company.

 

[Ericloewe]

Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)

And they do contribute back, they just don't care about aacraid. In part because the vendor doesn't care, and who wants to deal with a vendor who doesn't care?

 

[jgreco]

Of course its ridiculous, thats the point I am making because its a sequitur of your original statement. TrueNAS uses FreeBSD, which is an open source kernel + OS. That means they are benefiting from FreeBSD, otherwise they would have to build their own kernel and operating system. This means its in their interest to also help the development of FreeBSD because it also helps back, even if that initial help isn't directly relevant

They've contributed a huge amount to BSD's development over the years, including ZFS, kernel iSCSI, and a variety of other things. Your argument is feeble.

This is entirely a strawman so its not worth addressing

No, it is exactly the point. You want your random bit of hardware to work. It's not known to work and has been problematic in the past. No one is beating down the door to fix this decade-old deprecated device. But when you cannot come up with a reasonable response, it's "a strawman" and "not worth addressing".

No one is suggesting that under normal production use people should use TrueNAS with custom kernels, what is being suggested is to allow people in the community to easily test fixes/bugs (which are often patches to the kernel) to see if the fixes/bugs work.

Yes, so go ahead and do so. It's buildable. I've pointed to the instructions. What's your hangup?

Well your feelings and assumptions are completely wrong so you should probably look into why that is the case. Hint, my name is quite public and if you did some very basic research you would easily find that I spend a lot of time contributing to other projects.

Was I supposed to be impressed? So not.

Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)

The FreeBSD Foundation and iXsystems have a long history of collaboration. It isn't clear what you're going on about.

Your statements give a strong implication that you speak on behalf of iXSystems, as if you are speaking on an altar for them.

Oh. Just figured it out. Perhaps something's wrong with reading or comprehension. Because you were clearly told

none of us here are iXsystems employees

and obviously I don't carry an iXsystems badge. Or perhaps you simply can't deal with it when someone knows the score and talks about it? You know, I really don't care anymore. You've been given the directions, you've been given a proposed patch, you've done jack diddly squat with it as far as I can tell, and you come across (at least to me) with an aura of undue entitlement, which makes you unpleasant to talk to.

Anyways it looks like someone closed the thread while I was writing this, so I'll take the opportunity to wish you good luck with your RAID card and the patch.