Has anybody tried my patch? I'd like to get this issue resolved and I think that my patch is more correct than the earlier ones proposed, but lack hardware to test my belief.
TrueNAS doesn't support this because it's an appliance, not a general purpose FreeBSD host, and it doesn't have a full set of tools onboard to compile. In order to do this very bad idea, you have to stage a similar FreeBSD host, do your build there, and propagate the file back, and then every time something comes along that messes with this, like an upgrade, do that process again.
Thats fine, but I was also talking about creating a manual TrueNAS build on another FreeBSD host to create a patched FreeBSD kernel, I couldn't find any documentation on this.
There is sparse documentation on how to build TrueNAS from scratch but thats pretty much it.
Really? It's right there with the source code.
https://github.com/truenas/core-build
But in general this isn't encouraged or supported.
I get the feeling that you're expecting a beginner-level HOWTO that walks you through all the exact steps of taking a patch, applying it, and building, I don't know that such a thing exists. I mean, it's easy for someone who has worked with FreeBSD for years and who regularly builds kernels and stuff like that to do, but it is kinda like driving to the store... a lot of things that are just assumed knowledge, like how to open the car door, how the seatbelt works, which way to turn the key, how to put the car into drive, which turns to make, etc. Building a new FreeNAS system is going to be very dependent on having some existing experience with building a FreeBSD system from sources, combined with being able to follow the general directions included in the Github repo. Any instructions written a year ago won't be completely applicable today, and any instructions written today would likely be somewhat outdated a year from now.
And there's just no incentive for anyone to maintain such a thing, because you're not supposed to be doing it to begin with.
make update
and before the make release
steps, judging by the instructions in the Readme. This is just from a quick glance by someone who hasn't actually tried this before, so let us know how it goes.Of course it exists, I do it all the time on Linux (I use Archlinux/Manjaro) and they have extensive documentation and even build systems/package managers that allow you to apply patches/diffs (to both packages and the kernel) such as https://github.com/Jguer/yay. Its okay if the reason for it not existing is because TrueNAS doesn't have enough people to work on the product but claiming this is the norm for open source projects of this scale is disingenuous.
Of course it exists, I do it all the time on Linux (I use Archlinux/Manjaro) and they have extensive documentation and even build systems/package managers
And how do you expect this to get tested then? I mean you are describing a catch 22 situation. An upstream committer of FreeBSD took his own time creating a patch to fix this specific problem and I doubt its going to get merged until it gets tested (which I am willing to do acknowledging the risks).
This is the whole spirit and ethos of open source, if you don't want people testing the system outside of what I suppose is ixSystems then why even bother making the TrueNAS build open?
test[...] the system outside of what I suppose is ixSystems
An upstream committer of FreeBSD took his own time creating a patch to fix this specific problem
I am sorry but this mentality is just hogwash. I work full time as an open source developer as well as maintaining various projects and if a project is unable to even document how the community should help contribute then thats a fault of the project, not people like myself. The mentality of your post just demonstrates that.Yup, and I kinda imagine that this was instigated, and Warner popped up in this thread, because you somehow began that process, and Warner's awesome for trying to help (as has been the case for decades). But perhaps you should reflect on this a bit. You seem to be implying that "we've" wasted Warner's time producing a patch which isn't being tested. That is technically correct, but I submit that this could be more accurately be summed as YOU have wasted Warner's time, if you didn't have the ability to apply a patch that you were given.
I am sorry but this mentality is just hogwash. I work full time as an open source developer as well as maintaining various projects
if a project is unable to even document how the community should help contribute then thats a fault of the project, not people like myself. The mentality of your post just demonstrates that.
I manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups
and I was willing to test this even for months/years to see if its working, so I guess you can pat yourself on the back for a wasted opportunity because it seems its more important to "discourage people to not test official setups"
you have to stage a similar FreeBSD host, do your build there, and propagate the file back, and then every time something comes along that messes with this, like an upgrade, do that process again.
rather than provide basic documentation/instructions which is all I was asking for.
full time [...] open source developer
Then they should either not use FreeBSD or they should strip the FreeBSD source of drivers that they do not support. You are creating a false dichotomy here, if iXSystems is using FreeBSD then by definition their system is supporting more then they claim. There is an overlap here whether people realize it or not.You seem a bit confused about what this project is, though. This is not a project to build a general purpose NAS package that works with random hardware. This is a project to build the firmware for TrueNAS Enterprise, a very specific hardware product sold by iX. Because they're super-cool, because they like to exploit the community to locate bugs before their paying customers, and because it was easy to do, iX has allowed the community to use a slightly feature-reduced version for free. This costs them essentially nothing; FreeBSD has been polished as a server OS for many years, and especially the vendor-provided drivers for stuff like Chelsio, Intel, and LSI are super high quality. iXsystems has made it clear that they do not care to spend developer time trying to make stuff like the Broadcom or Emulex ethernet stuff work, and there's a large variety of other hardware that is dodgy or simply not suitable, and iX has made it clear that they're not really interested in supporting a custom fork of FreeBSD to fix driver issues for janky hardware. This means that the path to success for TrueNAS is to make your hardware platform more closely mirror what iX is selling, not kvetching about how the community or project is at fault for not doing what you'd like to see done.
Nice work on missing the entire point I was making (hint it wasn't to show off, it was to demonstrate I was willing to test it on a non trivial setup for an extended period of time).Congratulations. About 20 years ago, my shop was putting out some of the first 24-bay server machines running FreeBSD for large scale storage applications. I remember when a petabyte was two dozen racks filled with such storage servers.
Yes and as I claimed before, I am not that familiar with TrueNAS build system and the documentation on how to run it with a custom kernel is completely missing. I created an official iXSystems ticket on this as well and it still hasn't received any responses.This feels like you're trying to shift the blame here. You were invited to test the patch Warner provided, and I even provided loose guidance and a pointer to the build instructions.
For starters no one owes anyone anything, this is open source software not a contract (not sure where you got that from). I am merely stating that not having documentation is generally not good for an open source project and the fact that you stated that no one has a "vested interest" already states where the problem is.You seem to think that the project owes you some kind of bespoke documentation/instructions on exactly how to do this, but the problem with that is that is that it would require someone to actually figure this out for you, and none of us have a vested interest in solving this problem. We're just community members too, and most of us are not that interested in solving this problem for your off-brand obsolete RAID controller from a decade ago.
I'm sure we have all the parts in inventory here, certainly a 12-bay is easy, X9 board too, and I know we've got some Adaptec ASR-somethingsomething's (71605?) as well, but quite frankly I don't understand what my motivation would be to waste several hours on a build and then figure out how to build your custom firmware image and document that for you. Most of the open source developers I work with understand this problem and at a minimum know to ask nicely rather than expecting, demanding, or whining for an outcome. There's this weird thing in open source that's happened in recent years where people act like they're paying customers and they're owed answers. I personally find this distasteful and my hair bristles at it. Doesn't make me want to jump out of my chair and run back to the shop to find answers.
Then they should either not use FreeBSD or they should strip the FreeBSD source of drivers that they do not support. You are creating a false dichotomy here, if iXSystems is using FreeBSD then by definition their system is supporting more then they claim. There is an overlap here whether people realize it or not.
by definition their system is supporting more then they claim.
Nice work on missing the entire point I was making (hint it wasn't to show off, it was to demonstrate I was willing to test it on a non trivial setup for an extended period of time).
Yes and as I claimed before, I am not that familiar with TrueNAS build system and the documentation on how to run it with a custom kernel is completely missing. I created an official iXSystems ticket on this as well and it still hasn't received any responses.
For starters no one owes anyone anything, this is open source software not a contract (not sure where you got that from).
I am merely stating that not having documentation is generally not good for an open source project and the fact that you stated that no one has a "vested interest" already states where the problem is.
TBH, I think you are just overly defensive (from what it seems like having being stuck in an ixSystems bubble?) and are not willing to listen to constructive criticism.
For starters you seem to imply that I am asking everyone to do all of the effort, I am not. If I was to test TrueNAS with this patched kernel, I still have to set it up on all of the hardware, monitor it and then deal with the risk that the patch might make things worse. This is not a case of some random internet person asking someone to do everything.
What you seem to be conveniently ignoring is this is just a case of pragmatism, as is quite clear I am not that familiar with TrueNAS/FreeBSD so its actually far more efficient for someone that is completely familiar with TrueNAS to document what I was suggesting and then I would handle the testing.
https://github.com/truenas/core-build
we see:Requirements:
- Hardware
- CPU: amd64-compatible 64-bit Intel or AMD CPU.
- 16GB memory, or the equivalent in memory plus swap space
- at least 80GB of free disk space
- Operating System
- The build environment must be FreeBSD 13.x (or 13-STABLE)
Procedure
A FreeNAS release is built by first updating the source, then building:
- Install git
pkg install -y git
rehash
- Clone the build repository (/usr/build is used for this example):
git clone https://github.com/truenas/build /usr/build
- Install Dependencies
cd /usr/build
make bootstrap-pkgs
python3 -m ensurepip
pip3 install six
- First-time checkout of source:
make checkout
make update
make release
To build the SDK version:
make update
make release BUILD_SDK=yes
Clean builds take a while, not just due to operating system builds, but because poudriere has to build all of the ports. Later builds are faster, only rebuilding files that need it.
Use make clean to remove all built files.
Results
Built files are in the freenas/_BE subdirectory, /usr/build/freenas/_BE in this example.
ISO files: freenas/_BE/release/TrueNAS-13-MASTER-{date}/x64/.
Update files: freenas/_BE/release/.
Log files: freenas/_BE/objs/logs/.
I am not forcing anyone to do anything and there isn't some grand expectation of this, but if it doesn't occur then it makes things more difficult for everyone (which is exactly what happened). This isn't some weird or foreign concept, this is how well maintained open source projects are meant to work in building a community otherwise you end up in the case where almost all of the work needs to be done by a small number of people. Yes it does require effort to generate such documentation, but as a return you create a lot of "free" labor from the community.
And on the point of open source projects, the ones that don't provide documentation (or more generally don't make it easier for contributors) are projects that on the long term tend to die and if they do survive its because they are completely beholden to a single company which creates its own set of problems that often just kick the can down the road
Of course its ridiculous, thats the point I am making because its a sequitur of your original statement. TrueNAS uses FreeBSD, which is an open source kernel + OS. That means they are benefiting from FreeBSD, otherwise they would have to build their own kernel and operating system. This means its in their interest to also help the development of FreeBSD because it also helps back, even if that initial help isn't directly relevantThis is ridiculous. An open source project like FreeBSD contains drivers for lots of things that will never be used; the audio, wifi, Bluetooth, and SCSI tape changer devices come immediately to mind as great examples. There are also lots of system executables that will never be used, such as finger(1) or talk(1). You are essentially talking about creating a custom fork of FreeBSD. That's been done, it was called TrueOS, and it was a difficult-to-maintain disaster that mostly existed because one of the developers was also using it as a platform for PC-BSD. iXsystems has experience maintaining a fork, and it's a lot of work. TrueOS didn't even "strip the FreeBSD source of drivers that they do not support"; this is just insanity. It was basically just to integrate stuff like the latest video drivers and stuff like that to make the desktop environment work.
This is entirely a strawman so its not worth addressingBy definition this is a steaming pile; they never made any claims about what they support. What we push here in the forums is a pragmatic set of answers that is known to work; what kind of a shitheel would I be if I advocated for new users to use things known not to work well? Sure, go ahead and use USB-attached hard drives with your little $50 ITX mainboard on 8GB of RAM and a Realtek ... it's all fine, it'll work... but it won't work well (or maybe just barely) and that's no fair to anyone. You don't seem to understand that there's going to be some stuff out there that isn't really acceptable for use with the NASware. This isn't really anyone's fault. The PC world is full of subpar race-to-the-bottom hardware that barely works on Windows. iXsystems isn't going to go to any significant effort to make that work, and here on the forums, we're not going to recommend it, even if it might be "supported" by FreeBSD.
No one is suggesting that under normal production use people should use TrueNAS with custom kernels, what is being suggested is to allow people in the community to easily test fixes/bugs (which are often patches to the kernel) to see if the fixes/bugs work.I didn't miss the point. You missed mine. My point is that I've been doing this a hella-long time and have been testing a variety of stuff in production at scale for an extended period of time. I seeded this forum with a lot of wisdom earned from the School of Hard Knocks, and I've also carefully tried to gather as much additional wisdom about what works and what doesn't in the decade-plus I've been a participant here.
Running it with a custom kernel is simply not an option that anyone on the development team conceived of. The way you run a custom kernel is to build a custom firmware and install it. If you don't like it, you roll back to the previous boot environment. That should be obvious to anyone who has updated TrueNAS once or twice. There's no "documentation" for this other than "do the obvious."
I did actually attempt this, but it was over a year ago and I came across some issues which I have already forgotten. I would have to dig up my custom machine I used to build FreeBSDI already linked you to the build resources for TrueNAS as well.
Probably from what comes across in this conversation as a perennially entitled attitude where people owe you various things. This is reinforced because you seem to be mostly talk, not appearing to have taken any action on Warner's request to try a patch.
That isn't what I was suggesting at all, I would read my statement againThere's plenty of documentation. The FreeBSD Handbook. Design and Implementation by McKusick and Watson. UTSL. You aren't even following guidance and patches provided to you on a forum. You keep suggesting that there's NO documentation when what you really mean is that no one has written a Linux style HOWTO to do the very weird and very specific thing that you want to do that isn't really that hard to do. It's open source. You see a documentation shortcoming. Address it.
Your statements give a strong implication that you speak on behalf of iXSystems, as if you are speaking on an altar for them.I don't have anything to do with iXsystems except that now and then they send me a coffee mug or stickers or one time they sent an awesome Beastie plushie; you can see him being strangled in my avatar. I do occasionally reprocess iXsystems gear in the shop here for my customers. I like their stuff, but that's about it. I don't control the direction of the project, I'm not a developer for the project, I do write documentation here on the forum, but you seem to be laboring under an illusion that you can provide your "constructive criticism" to me and something is magically going to happen as a result. It probably won't. If I had that sort of sway, there's a lot of stuff that'd be just a little different about this project.
Well your feelings and assumptions are completely wrong so you should probably look into why that is the case. Hint, my name is quite public and if you did some very basic research you would easily find that I spend a lot of time contributing to other projects.Functionally null paragraph. You already claimed to "manage over 1 petabytes of data on many 24 bay server machines on TrueNAS with different setups using these aacraid cards" so presumably you have the lab resources to test this. Yet you don't seem to have actually bothered to build and test Warner's patch. It feels to me like you will next blame this on a "lack of documentation". So if you are not asking everyone else to do the work here, and you are not doing the work here, this seems like another vapid thread where someone pointlessly argues that their RAID controller should be just fine but damn it their mfi0 keeps throwing timeouts, just ignoring the obvious fix in favor of boring argument.
As I said there was issues in building this and the documentation has been updated since I tried this over a year go.So, someone write up a Linux HOWTO of how to do this, because the available instructions are not specific enough. So let me help you here. If I go over tohttps://github.com/truenas/core-build
we see:
So you prepare your FreeBSD 13-amd64 system. I'll clue you in that this could be a chroot environment on your NAS.
Then you do this crap:
Then you get your results.
Then you install your ISO from the ISO directory. Once you can do all this, then you go back and patch your tree with Warner's patch, repeat the "make release", then install your customized ISO.
This is just SOOOOO complicated, I can see why you need bespoke instructions.
As is a lot of other open software, nothing new hereI feel little need to respond to this steaming pile of BS. TrueNAS is a commercial product with paid developers.
As is also the case for a lot of open source software, again nothing new here.Your impressions of how you feel an open source project should work has almost zero relevance. They're not writing it for you and you are not the intended customer.
Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)Most likely no one over at iX cares if your obsolete deprecated non-recommended RAID card doesn't work and they're happy to let upstream address the problem, as has been their pattern in recent years.
Yes and you are also missing the point entirely because its not about the financials of the company.That's nice. Apparently iXsystems is dying twelve years into the project to the tune of $100M annual revenue.
Anyways this is meaningless because I have spent a lot of futile time trying to impart clue.
And they do contribute back, they just don't care about aacraid. In part because the vendor doesn't care, and who wants to deal with a vendor who doesn't care?Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)
Of course its ridiculous, thats the point I am making because its a sequitur of your original statement. TrueNAS uses FreeBSD, which is an open source kernel + OS. That means they are benefiting from FreeBSD, otherwise they would have to build their own kernel and operating system. This means its in their interest to also help the development of FreeBSD because it also helps back, even if that initial help isn't directly relevant
This is entirely a strawman so its not worth addressing
No one is suggesting that under normal production use people should use TrueNAS with custom kernels, what is being suggested is to allow people in the community to easily test fixes/bugs (which are often patches to the kernel) to see if the fixes/bugs work.
Well your feelings and assumptions are completely wrong so you should probably look into why that is the case. Hint, my name is quite public and if you did some very basic research you would easily find that I spend a lot of time contributing to other projects.
Yes, and the point is they are relying on upstream (FreeBSD) for their product to exist. Point is they should foster that, because otherwise if it wasn't for FreeBSD their own product wouldn't even exist (or it would be worse)
Your statements give a strong implication that you speak on behalf of iXSystems, as if you are speaking on an altar for them.
none of us here are iXsystems employees