TrueNas jira is incredibly slow

[overeat]

The https://jira.ixsystems.com/ is super slow.

Jira has a file called batch.js which is 16MB and takes 2 minutes to download on my connection (a 1 GBPS connection, but in Asia)

Makes it really frustrating when browsing for bugs or attempting to submit them. I've already given up a couple times, and I assume others have too.

This is a known bug (feature) with Jira. You guys should check your server to make sure that file is not generated on the server on each page load (that would be CPU intensive). If possible you'd want to make sure it's at the very least generated rarely and cache'd statically on the server in a file (or redis or some memory cache) for quick serving.

I'd assume you guys are aware of this issue, but a quick search in the forums didn't turn up any complaints, so I figure I'd add one.

Some references:

Loading...

jira.atlassian.com

How to troubleshoot long page load times in Jira server | Jira | Atlassian Documentation

confluence.atlassian.com

Loading...

jira.atlassian.com

 

[overeat]

The file doesn't appear to get cached in the browser, which is the main issue, so for instance, I just submitted a bug and upon submission and page reload, it's downloading batch.js again.

I double checked and I dont have the "Disable Cache while DevTools is open" option on in Chrome Inspector.

 

[jgreco]

There've been lots of complaints about Jira being slow here in the forums.

I don't really know anything about what iXsystems has done with Jira. I do run the infrastructure that powers a different Jira Datacenter instance, and we haven't noticed these sorts of issues with it, in what I believe is a relatively out-of-the-box setup. Having been a Jira user for many years, I can say that Jira tends to get a bit laggy once you get "too much stuff" in it, and I wouldn't be shocked if the iX Jira had a ton of tickets.

It sometimes takes a lot of time and effort to get these things working well, and in my opinion iX isn't doing a very good job. The forums were recently moved to a CDN model where clearly next-to-no testing had been done, which resulted in notifications and other updates being just totally broken. Having undone that trainwreck, the user activity index is now busted... go to any user profile summary (click on a username), click on the "Messages" count, like yours is "8", and the results haven't updated since July (yours says no posts found).

I don't hold out high hopes. :-(

 

[ChrisRJ]

What @jgreco writes is troubling to me, since it demonstrates a certain attitude towards quality. For me the latter is, unfortunately, in line with how I have perceived the handling of the TrueNAS 12.0 release. Coming from an enterprise software background, my view is that a reputation for quality is probably the biggest asset you can have in the enterprise market.

I would like to appeal to iXsystem to think about this. From where I stand you have been damaging your brand pretty consistently over the last 12 months. Not to the point that the ship cannot be turned around. But let this continue for another year and my next NAS will not be TrueNAS, which I would hate.

Happy to discuss details!

 

[jgreco]

What @jgreco writes is troubling to me, since it demonstrates a certain attitude towards quality. For me the latter is, unfortunately, in line with how I have perceived the handling of the TrueNAS 12.0 release. Coming from an enterprise software background, my view is that a reputation for quality is probably the biggest asset you can have in the enterprise market.

That might be a bit overly dramatic; I perceive the web issues here to be an issue with an IT department without sufficient resources. IT is a sufficiently expansive thing that you're not likely to find even a small group of people with all the requisite skills, and that expertise in arcane areas is expensive.

A former iX staffer once described an attempt to set up their own FreeNAS download site at Hurricane Electric as a "dumpster fire" during a discussion of how to set up a download system for a different project. Apparently the point was to "use a CDN to avoid the headache." This totally missed on technical merit; trying to set up a download site for ISO-sized things downloaded at scale by every end user is indeed at least vaguely difficult. But we were talking about a package that was only 10MBytes, rarely downloaded directly by end users, and on top of it, I've done infrastructure engineering for USENET for the past 30 years; even an ISO download site is easy in comparison.

It was the embodiment of the old "beware of programmers carrying screwdrivers" thing. Know your limits, and you can't be afraid to hire in specialist help now and then. The whole forum migration thing felt very much like a repeat of the download site fiasco, where people didn't really grasp the totality of the issues, and then didn't have the correct skill set, and (worst of all) didn't realize it.

Knowing what you know and what you don't is a real hazard for tech people. Anyone who has been observant here through any Windows discussion will note that I scrupulously avoid discussions of Samba, SMB, AD, Windows permissions, and a variety of other topics, my knowledge of them possibly being less than the average reader's. I also prefer to avoid web stuff, because, while I can generally solve things if I dig hard enough, there's a lot of people out there that do it far more easily than I do.

I would like to appeal to iXsystem to think about this. From where I stand you have been damaging your brand pretty consistently over the last 12 months. Not to the point that the ship cannot be turned around. But let this continue for another year and my next NAS will not be TrueNAS, which I would hate.

You have a better alternative? Let me remind you that a lot of people ended up here as Nexenta refugees or from other less-great NASware platforms, and that was a decade ago when FreeNAS was much less fully-featured than it is now.

Missteps are a part of the software development world, and part of the price we pay as free users of this commercial product is that we help to shake the bugs and issues out. Expecting commercial-grade perfection for free is unrealistic. If you really can't afford the occasional misstep and rollback, perhaps you should consider doing what some of us do, which is to wait for awhile before updating key systems, and let other people be the cannon fodder. FreeNAS and the freeware user base are an integral part of the iX development cycle, because it is inherently super-difficult for a small developer group to cover every possible base and use case.

The overall trajectory of the product has been fantastic over the past decade, and I don't think you'll find a lot of better options out there.

 

[Basil Hendroff]

Makes it really frustrating when browsing for bugs or attempting to submit them. I've already given up a couple times, and I assume others have too.

Community members spend the majority of their time on the forum and only occasionally on JIRA. It may be frustrating for us, but spare a thought for iXSystem employees who do the reverse i.e. spend the majority of their time on JIRA managing their workflow and not a lot of time on the forum.

 

[NugentS]

Hopefully (for them) they are local to the server. But probably not

 

[ChrisRJ]

@jgreco, thanks for taking the time and elaborate the way you did!

Missteps are a part of the software development world, and part of the price we pay as free users of this commercial product is that we help to shake the bugs and issues out. Expecting commercial-grade perfection for free is unrealistic. If you really can't afford the occasional misstep and rollback, perhaps you should consider doing what some of us do, which is to wait for awhile before updating key systems, and let other people be the cannon fodder.

I was probably not clear enough on what I wanted to convey, sorry. My core point was not so much about us free users, but those who pay for support. If I am in the market for commercial support, I also have some expectations. As to deferring the update, that is indeed my approach and the reason why I am still on 11.3u5. But, perhaps wrongly, I am under the impression that is not an option for support-paying customers anymore.

 

[overeat]

The purpose of my post was to point out the actual Jira issue which makes it slow and provide some steps the admin of that system could do to resolve the issue.

Jira being "slow" as a whole is not the issue. This batch.js not being cached properly is the actual issue which makes Jira SUPER slow and could be fixed, thus making Jira less slow and tolerable.

I believe it's a fixable problem with caching and making sure that file is served with the proper expiry headers so it only needs to be downloaded once....and not forced to be redownloaded per page load. Alternatively perhaps the issue is changing of the URL of batch.js between page requests due to CDN config, which is causing it to be force reloaded.

It's a known issue for Jira which other Jira users encounter and it should be fixable (i assume).

Here's some things to look into:

How to troubleshoot long page load times in Jira server | Jira | Atlassian Documentation

confluence.atlassian.com

issue with missing cache folder, which can cause this as well:

Loading...

jira.atlassian.com

 

[jgreco]

I was probably not clear enough on what I wanted to convey, sorry. My core point was not so much about us free users, but those who pay for support. If I am in the market for commercial support, I also have some expectations. As to deferring the update, that is indeed my approach and the reason why I am still on 11.3u5. But, perhaps wrongly, I am under the impression that is not an option for support-paying customers anymore.

I would imagine that commercial TrueNAS support goes through their field engineering team, and that they don't expect paying customers to be using their Jira. The whole "it's an appliance" line comes into play, I would think/hope. Remember, they actually sell TrueNAS boxes and have presales and postsales engineering support...

Jira offers a licensing tier for open source software projects that is "free", with part of the requirement being that it be available for public use (don't recall the exact terms of the deal). Since Jira is kinda pricey, I imagine that iX might have pursued this option. It could well be that they have Jira set for open access as a "contractual compliance" checkoff; in such a case, I could imagine not wanting to spend a lot of time or money to tune it up as long as it worked for the people who needed it on a daily basis. I don't actually know which choices are being made there, or why, of course, but having been on the bad side of "having to fix this sort of thing" now and then, I understand that these things can happen suboptimally sometimes.

 

[jgreco]

The purpose of my post was to point out the actual Jira issue which makes it slow and provide some steps the admin of that system could do to resolve the issue.

Jira being "slow" as a whole is not the issue. This batch.js not being cached properly is the actual issue which makes Jira SUPER slow and could be fixed, thus making Jira less slow and tolerable.

I believe it's a fixable problem with caching and making sure that file is served with the proper expiry headers so it only needs to be downloaded once....and not forced to be redownloaded per page load. Alternatively perhaps the issue is changing of the URL of batch.js between page requests due to CDN config, which is causing it to be force reloaded.

It's a known issue for Jira which other Jira users encounter and it should be fixable (i assume).

Here's some things to look into:

How to troubleshoot long page load times in Jira server | Jira | Atlassian Documentation

confluence.atlassian.com

issue with missing cache folder, which can cause this as well:

Loading...

jira.atlassian.com

I can pretty much guarantee that the "right people" at iXsystems are not going to be seeing this here on the forums. We get minimal visits from iX staff. You could try posting a ticket to Jira and putting a short explanation, those links, and maybe a link to this thread in there and see if that causes this to get escalated in the right direction... that's my best suggestion that could be productive.

 

[Arwen]

And we should all "vote" for that ticket!

Supposedly with 10 votes, Jira tickets get attention. Perhaps their is some background process that checks for 10 votes, and creates an alert or E-Mail.

 

[overeat]

I can pretty much guarantee that the "right people" at iXsystems are not going to be seeing this here on the forums. We get minimal visits from iX staff. You could try posting a ticket to Jira and putting a short explanation, those links, and maybe a link to this thread in there and see if that causes this to get escalated in the right direction... that's my best suggestion that could be productive.

And we should all "vote" for that ticket!

Supposedly with 10 votes, Jira tickets get attention. Perhaps their is some background process that checks for 10 votes, and creates an alert or E-Mail.

I found a ticket which is still open and has other people complaining here:

https://jira.ixsystems.com/browse/NAS-109699

Please upvote, and lets get this in front of people.

 

[no_connection]

Jira said nope to wanting attention. It's sentient I tell you! That is why you have to download a 16MB! script that should NEVER happen in web design.

 

[Jaron]

@no_connection I was working on implementing some of the potential fixes at that exact time. Should be back now. Let me know if you see a difference, however it is doubtful.
@j0rd Thank you for the links to potential solutions. I have explored most of these, and much more, and agree it is definitely related to a caching issue. We hve suspected for some time now, that it is also related to a misbehaving plugin combination as well. We do attempt to work on it here and there, but have kind of given up hope for many reasons. The biggest of which was our plans to move to Jira Cloud. Unfortunately, doing this has presented itself with more challenging issues, which as stated above somewhere, we havent dedicated our limited resources into solving yet.

I appreciate your patience, and welcome more potential solutions and ideas if you find any. Please add them to the ticket above as it does ping me each time it is touched or commented on and I rarely check the forum threads.
I doubt we put any elaborate effort into fixing the current Server Edition since it is end of life practically. We will most likely put our efforts toward migration to the Cloud.
Please bare with us as we continue to plan for this soon

 

[Jaron]

@jgreco The issues experienced during the CDN migration should have been resolved by now. We did do testing before pulling the trigger however those issues didnt creep up at that time. Once we put our finger on the issues we implemnted a fix soon thereafter. It took coordination with multiple departments to finally clear it up. Are you still experiencing these issues?
Also, I appreciate your insight above. We do try to limit ourselves to what we can and cannot do, and finding someone that can be that needed expert can be and even bigger challenge. Lots of fires in this dumpster, and often not enough fighters to put them out. We will continue working hard to get them put out as fast as we can.

 

[Etorix]

@Jaron Thanks for your explanations. I've nevertheless upvoted the Jira ticket. iXSystems really needs to get the message that their interaction with users is lacking

 

[jgreco]

@jgreco The issues experienced during the CDN migration should have been resolved by now.

Search indexing is still massively broken as outlined in the example above.

 

[Jaron]

Search indexing is still massively broken as outlined in the example above.

Thank you! I was not aware.. We will look into it.

 

[overeat]

So I went to login to Jira today and I'm following in chome inspector.

Downloaded batch.js on first page load.

Then I login and I'm presented with this batch.js download again:

https://jira.ixsystems.com/s/2879e6eadc9a851d0e1f5256b42da298-CDN/-hs1z89/816001/lsbyeq/6f6b36277ab48041a95e98eeeb4989eb/_/download/contextbatch/js/atl.dashboard,atl.general,jira.general,jira.global,jira.dashboard,-_super/batch.js?agile_global_admin_condition=true&jag=true&locale=en-US&slack-enabled=true&user-logged-in=true

After logging in, I click over to this "jira.ixsystems unbearably slow" ticket and the URL changes, thus forcing redownload:

https://jira.ixsystems.com/s/0a0d4119f54d6caea7e75e4717ee803b-CDN/-hs1z89/816001/lsbyeq/f2d9ddc684174fa3fbf521bd35802bcf/_/download/contextbatch/js/jira.browse.project,jira.view.issue,atl.general,project.issue.navigator,jira.general,jira.global,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en-US&richediton=true&slack-enabled=true&user-logged-in=true

So perhaps the issue is the changing of the URL between page requests.

Check -XX:ReservedCodeCacheSize=512m variable to make sure it's large enough to store all the code cache between requests.

Health Check: JVM Code Cache | Confluence | Atlassian Documentation

confluence.atlassian.com

If that doesn't work and isn't the issue


Disable Batch Mode

Web Resource module

developer.atlassian.com

So Jira is doing something stupid here. I think Jira is trying to be "Smart" here and instead of loading all JS/CSS for all pages, it's trying to be specific and only inject the specific JS which is required for that specific page. Problem is when you load another page, it needs another JS file, so Jira injects that now and you're forced to redownload another 16MB batch.js

I'd suggest seeing if you can disable batch.js completely. I'm not sure if this is the setting, but it's the best I can come up with (I don't have Jira installed)

If that doesn't disable batch.js

Web Resource Context to Force concatenation

Web resource

developer.atlassian.com

There is a way to force Jira to load all JS files in a "context", but I feel this might end up like whack a mole, unless you know every file that might be needed to every page. If you did know this, you could probably add them all to this context and then it should be fairly static between page requests.

CDN needs to disable cookies for specific paths, thus will allow caching and not serve files per user

Configure your CDN for Jira Data Center | Administering Jira applications Data Center 9.15 | Atlassian Documentation

confluence.atlassian.com