FreeNAS Jira instance painfully, horribly slow

[taupehat]

Sorry for title gore but it is Halloween. Also the title is accurate. We have our own Jira here at work so I have some basis for comparison, and the one maintained by IXsystems is, to be direct, unacceptably slow. Since I'm not a sysadmin on that box I can't guess what the bottleneck might be but somewhere it's choked out bad. It really shouldn't take over a minute to load an issue.

 

[Jaron]

We have had other complaints, but usually find it is overseas users that have extrememly long load times due to latency. we do have a plugin that tends to cause this issue especially on higher than normal latency. Here locally onsite and off it seems to load fine.
could you send me a traceroute or mtr between yourself and jira.ixsystems.com? Send direct message.

 

[taupehat]

Direct message sent.

 

[Oriann]

I am having same problem. Last time reporting a bug almost take my whole day.

Tracert here...that exponential ping rise is like a THX art in cinema, just pop up popcorn and get a 3D glasses
Tracing route to jira.ixsystems.com [38.109.202.235]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms OPNrouter.localdomain [192.168.2.1]
2 10 ms 9 ms 12 ms pppoe-1.ttc.praha.*****.cz [185.*******.0]
3 8 ms 8 ms 10 ms 100.121.0.1
4 34 ms 11 ms 14 ms te0-7-0-8-s202.bb1.pop1.brn.sloane.cz [77.48.21.129]
5 24 ms 27 ms 25 ms cz-pra-pop115-rb1-vla2121.net.upc.cz [84.116.221.93]
6 18 ms 21 ms 19 ms cz-prg01a-ra4-vla2119.net.upc.cz [84.116.221.78]
7 29 ms 24 ms 30 ms de-fra11b-rc1-ae-33-0.aorta.net [84.116.135.5]
8 30 ms 29 ms 30 ms cz-prg01a-ra4-xe-4-3-2-0.aorta.net [84.116.130.62]
9 * * * Request timed out.
10 30 ms 30 ms 32 ms be2813.ccr41.ams03.atlas.cogentco.com [130.117.0.121]
11 181 ms 173 ms 164 ms be2182.ccr21.lpl01.atlas.cogentco.com [154.54.77.246]
12 163 ms 165 ms 164 ms be3042.ccr21.ymq01.atlas.cogentco.com [154.54.44.162]
13 167 ms 164 ms 164 ms be3259.ccr31.yyz02.atlas.cogentco.com [154.54.41.205]
14 164 ms 164 ms 164 ms be2993.ccr21.cle04.atlas.cogentco.com [154.54.31.225]
15 163 ms 164 ms 164 ms be2717.ccr41.ord01.atlas.cogentco.com [154.54.6.221]
16 175 ms 176 ms 177 ms be2831.ccr21.mci01.atlas.cogentco.com [154.54.42.165]
17 188 ms 194 ms 187 ms be3035.ccr21.den01.atlas.cogentco.com [154.54.5.89]
18 189 ms 187 ms 187 ms be2201.agr12.den01.atlas.cogentco.com [154.54.31.102]
19 188 ms 187 ms 187 ms te0-0-2-0.nr12.b006545-1.den01.atlas.cogentco.com [154.24.13.54]
20 187 ms 189 ms * te0-0-2-0.nr11.b004491-1.den01.atlas.cogentco.com [154.24.53.106]
21 195 ms 194 ms 194 ms 38.142.161.74
22 188 ms 192 ms 187 ms 38.109.202.235

 

[Jaron]

We see this frequently with connections from overseas. One or more of the plugins loaded adds a terrible amount of latency for some reason and we havent narrowed it down. We will continue to work on it and do what we can to mitigate this further. I apologize for the trouble it is causing you.

 

[Oriann]

We see this frequently with connections from overseas. One or more of the plugins loaded adds a terrible amount of latency for some reason and we havent narrowed it down. We will continue to work on it and do what we can to mitigate this further. I apologize for the trouble it is causing you.

Ok thank you, I am looking forward to some improvements

 

[jgreco]

10 30 ms 30 ms 32 ms be2813.ccr41.ams03.atlas.cogentco.com [130.117.0.121]
11 181 ms 173 ms 164 ms be2182.ccr21.lpl01.atlas.cogentco.com [154.54.77.246]

It would be interesting to know why round trip jumps from 30 to 181ms between Amsterdam and London.

traceroute to 77.48.21.129 (77.48.21.129), 30 hops max, 60 byte packets
1 gi0-0-0-19.3.agr15.den01.atlas.cogentco.com (66.250.250.49) 0.814 ms 0.813 ms
2 te0-1-0-15.ccr22.den01.atlas.cogentco.com (154.54.41.105) 0.798 ms te0-1-0-15.ccr21.den01.atlas.cogentco.com (154.54.40.81) 1.127 ms
3 be3036.ccr22.mci01.atlas.cogentco.com (154.54.31.90) 12.097 ms be3035.ccr21.mci01.atlas.cogentco.com (154.54.5.90) 12.133 ms
4 be2831.ccr41.ord01.atlas.cogentco.com (154.54.42.166) 23.760 ms be2832.ccr42.ord01.atlas.cogentco.com (154.54.44.170) 23.807 ms
5 be2765.ccr41.ord03.atlas.cogentco.com (154.54.45.18) 24.102 ms 24.049 ms
6 us-chi01a-ri3-ae-6-0.aorta.net (213.46.190.157) 23.696 ms 23.688 ms
7 us-was03a-rd1-ae-9-0.aorta.net (84.116.146.146) 168.340 ms 168.632 ms
8 us-was02a-ri2-ae-3-0.aorta.net (84.116.130.202) 168.346 ms 168.348 ms
9 cz-prg02a-ra2-xe-9-0-2-0.aorta.net (84.116.130.106) 169.006 ms 168.984 ms
10 cz-prg01a-ra4-ae-33-0.aorta.net (84.116.135.6) 174.980 ms 180.865 ms
11 cz-pra-pop115-rb1-vla2119.net.upc.cz (84.116.221.77) 168.991 ms 168.859 ms
12 te0-7-0-8-s202.bb1.pop1.brn.sloane.cz (77.48.21.129) 177.121 ms 177.504 ms

The return path from Denver seems to indicate that there is a huge latency jump between Chicago and Washington on aorta.net, which is Virgin Cable's backbone.

The conclusion here is that there is some ISP component to this problem because they are not delivering packets efficiently.

 

[adrianwi]

It's always painfully slow for me in the UK on Virgin Media

 

[Patrick M. Hausen]

Really interesting path inside Cogent from here (German Telekom). But more important: no AAAA record. Bad iXsystems!

Code:

$ traceroute jira.ixsystems.com
traceroute to jira.ixsystems.com (38.109.202.235), 64 hops max, 52 byte packets
 1  opnsense.ettlingen.hausen.com (217.29.46.41)  1.043 ms  0.328 ms  0.328 ms
 2  fritzbox.ettlingen.hausen.com (192.168.1.1)  1.226 ms  1.618 ms  0.938 ms
 3  62.156.244.43 (62.156.244.43)  148.056 ms  150.129 ms  164.575 ms
 4  62.156.247.82 (62.156.247.82)  152.508 ms  154.901 ms  150.265 ms
 5  pd900c976.dip0.t-ipconnect.de (217.0.201.118)  162.344 ms  166.425 ms  166.354 ms
 6  80.157.201.198 (80.157.201.198)  161.342 ms  164.301 ms  200.099 ms
 7  be3186.ccr41.fra03.atlas.cogentco.com (130.117.0.1)  136.632 ms  164.992 ms
    be3187.ccr42.fra03.atlas.cogentco.com (130.117.1.118)  172.452 ms
 8  be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141)  175.151 ms
    be2813.ccr41.ams03.atlas.cogentco.com (130.117.0.121)  157.211 ms
    be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141)  173.851 ms
 9  be12488.ccr42.lon13.atlas.cogentco.com (130.117.51.41)  273.288 ms
    be12194.ccr41.lon13.atlas.cogentco.com (154.54.56.93)  171.458 ms  186.253 ms
10  be3042.ccr21.ymq01.atlas.cogentco.com (154.54.44.162)  192.263 ms
    be3043.ccr22.ymq01.atlas.cogentco.com (154.54.44.166)  200.413 ms
    be2317.ccr41.jfk02.atlas.cogentco.com (154.54.30.185)  197.451 ms
11  be3599.ccr21.alb02.atlas.cogentco.com (66.28.4.237)  197.755 ms
    be2890.ccr22.cle04.atlas.cogentco.com (154.54.82.245)  164.346 ms
    be3599.ccr21.alb02.atlas.cogentco.com (66.28.4.237)  140.432 ms
12  be2717.ccr41.ord01.atlas.cogentco.com (154.54.6.221)  139.369 ms
    be2878.ccr21.cle04.atlas.cogentco.com (154.54.26.129)  137.670 ms
    be2717.ccr41.ord01.atlas.cogentco.com (154.54.6.221)  138.398 ms
13  be2718.ccr42.ord01.atlas.cogentco.com (154.54.7.129)  147.787 ms
    be2832.ccr22.mci01.atlas.cogentco.com (154.54.44.169)  158.225 ms
    be2831.ccr21.mci01.atlas.cogentco.com (154.54.42.165)  172.994 ms
14  be2832.ccr22.mci01.atlas.cogentco.com (154.54.44.169)  179.442 ms  171.894 ms  173.512 ms
15  be3036.ccr22.den01.atlas.cogentco.com (154.54.31.89)  219.735 ms
    be2201.agr12.den01.atlas.cogentco.com (154.54.31.102)  211.795 ms
    te0-0-2-0.agr13.den01.atlas.cogentco.com (154.54.30.10)  212.230 ms
16  be2068.agr11.den01.atlas.cogentco.com (154.54.28.30)  222.461 ms
    te0-0-2-0.agr13.den01.atlas.cogentco.com (154.54.30.10)  221.000 ms
    te0-0-2-3.nr12.b006545-1.den01.atlas.cogentco.com (154.24.11.202)  224.932 ms
17  te0-0-2-3.nr11.b004491-1.den01.atlas.cogentco.com (154.24.53.102)  239.812 ms
    te0-0-2-0.nr11.b004491-1.den01.atlas.cogentco.com (154.24.53.106)  232.869 ms
    te0-0-2-3.nr11.b004491-1.den01.atlas.cogentco.com (154.24.53.102)  245.034 ms
18  te0-0-2-0.nr11.b004491-1.den01.atlas.cogentco.com (154.24.53.106)  240.258 ms
    38.142.161.74 (38.142.161.74)  185.578 ms  191.424 ms
19  38.109.202.235 (38.109.202.235)  211.703 ms  213.935 ms
    te0-0-2-1.nr11.b006545-1.den01.atlas.cogentco.com (154.24.17.118)  217.161 ms

 

[jgreco]

Cogent is often a mess internally. Their lack of deterministic pathing tends to result in a higher percentage of out-of-order packet delivery.

You can use the Cogent looking glass to see what your return path looks like.

https://www.cogentco.com/en/14-network/looking-glass

 

[Oriann]

For 2 days I want to reply to issue asignee on Jira - I am not able to reach your site, because it wont load all source files...

Please can you do something about it ? Temporarily disable that plugin causing troubles or anything...anything that will allow me to continue work on bug I have reported ?
Loading bugtracker is really a desperation not just for me.

 

[Jamberry]

Same Problem. Init7 is the ISP

7 17 ms 18 ms 17 ms r1dus1.core.init7.net [77.109.129.10]
8 17 ms 17 ms 19 ms ddf-b2-link.ip.twelve99.net [62.115.54.65]
9 23 ms * 22 ms hbg-bb4-link.ip.twelve99.net [62.115.115.50]
10 22 ms 21 ms 22 ms hbg-b1-link.ip.twelve99.net [213.155.135.85]
11 23 ms 21 ms 22 ms be3044.rcr11.b015763-1.ham01.atlas.cogentco.com [130.117.14.37]
12 23 ms 23 ms 23 ms be3221.ccr42.ham01.atlas.cogentco.com [154.54.57.109]
13 28 ms 27 ms 29 ms be2816.ccr42.ams03.atlas.cogentco.com [154.54.38.209]
14 160 ms 200 ms 202 ms be2183.ccr22.lpl01.atlas.cogentco.com [154.54.58.69]
15 122 ms 121 ms 123 ms be3043.ccr22.ymq01.atlas.cogentco.com [154.54.44.166]
16 179 ms 121 ms 121 ms be3260.ccr32.yyz02.atlas.cogentco.com [154.54.42.89]
17 143 ms 202 ms 123 ms be2994.ccr22.cle04.atlas.cogentco.com [154.54.31.233]
18 144 ms 121 ms 174 ms be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
19 161 ms 197 ms 202 ms be2832.ccr22.mci01.atlas.cogentco.com [154.54.44.169]
20 205 ms 207 ms 201 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
21 204 ms 207 ms 198 ms te0-0-2-3.agr13.den01.atlas.cogentco.com [154.54.30.230]
22 181 ms 201 ms 202 ms te0-0-2-1.nr11.b006545-1.den01.atlas.cogentco.com [154.24.17.118]
23 173 ms 204 ms 200 ms te0-0-2-3.nr11.b004491-1.den01.atlas.cogentco.com [154.24.53.102]
24 147 ms 255 ms 201 ms 38.142.161.74
25 212 ms 201 ms 145 ms 38.109.202.235