Strange and Odd Behaviour (Exceptionally Slow Connections, Laggy, High Latency etc)

Status
Not open for further replies.
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
Howdy Everyone,

Will try to give as much information as I can, I am a FreeNAS/BSD/Unix newbie, I know my way around a little, but that is simply because a lot of people have done a lot of work before me to figure things out, But These errors, after looking around, dont seem to be discussed ..

The Hardware

I've got Two (2) HP ML10v2's with 5 x 3TB Drives in a RaidZ1-0 setup. Both Machines have 8 GB of Memory, There are 4 Drives in the Drive Cage that come with the Unit, and one external off the Onboard BUS. They both have two inbuilt NIC's that I've teamed together following instructions (but they dont seem to LAG the way I thought they would (My Windows Server (DL380 G7)) has 4 1GB Nic's and when teamed together, increases bandwidth, but for some reason with the LAG setup on these two machines, it seems to take it in turns even though its setup, but this isnt the worry.

The Software

I've got two (2) 16GB Pen Drives (verbatim USB's) plugged into the two front USB ports on the ML10's, and they are in a Mirror-0 arrangement, so if one karks it, the other has the latest setup, and I do regular config backups as well whenever I change anything major, rather than trying to remember what I did last.

Running :

System Information
Hostname abc_primary_nas Edit
Build FreeNAS-9.10.2-U5 (561f0d7a1)
Platform Intel(R) Pentium(R) CPU G3240 @ 3.10GHz
Memory 8128MB
System Time Fri Jul 14 13:14:06 AEST 2017
Uptime 1:14PM up 4 days, 20:27, 1 user
Load Average 0.48, 0.48, 0.43

The Issue(s):

The box's have been running for about 2 months, without any major issues. I've lost one drive in the primary box during that time, it got degraded automatically one night, I got the message, I got a new drive, replaced it after I Id'd it, and its been running okay ever since. In the last 3-4 weeks, I've moved about 400GB of email from the win-2003 mail server (all the old stuff from the past 5 years) onto the NAS, for backup, as its not needed, but just in case, ever since then, it has been exceptionally slow, and doing strange things.

I have both machines setup in a RSYC backup, where the main server pushes the data to the backup server, and the backup server simply dumps a copy the mirror image hardware/hdd's/memory, and setup, the only things that are different are the IP's and domain names, I've tried to keep them exactly the same all the way through.

There are two RSYNC modules setup, one for the storage portion, and one for the mail backup portion,

Whenever I select Anything, it will take a good 10-30 seconds to load or bring up a new page, where previously it would be a few seconds and that is about it. I dont know if it has something to do with the data its transferring between the two machines, or the load being places on the server,

I enabled SSH the other week, simply so I could get putty to work, it doesnt have any certificates etc loaded, and its purely on the local network.

I guess the question is where can I start to try and diagnose this, There are a lot of Errors coming up in the tail screen that runs down the bottom, where there used to be none.. I have a feeling, that the RSYNC is the cause, its moving lots of files, from the primary to the secondary, and I dont know why, it should have transferred all the files over a week ago when I first moved them? But there are RSYNC errors coming up?

Any thoughts would be great...


Some Config Screens



Some History

Code:


	raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Jul 14 13:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 13:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 13:29:04 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:29:11 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:29:20 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:29:22 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 13:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 13:30:11 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:30:13 abc_primary_nas manage.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 13:32:27 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 13:32:27 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Stop refresh


Code:

Jul 14 00:00:00 abc_primary_nas newsyslog[7087]: logfile turned over due to size>100K
Jul 14 00:00:00 abc_primary_nas syslog-ng[1875]: Configuration reload request received, reloading configuration;
Jul 14 00:00:01 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 00:00:01 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 00:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 00:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 00:24:00 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 00:24:00 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 00:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 00:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 00:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 00:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 01:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 01:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 01:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 01:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 01:24:48 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 01:24:48 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 01:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 01:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 01:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 01:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 02:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 02:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 02:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 02:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 02:25:17 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 02:25:17 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 02:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 02:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 02:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 02:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 03:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 03:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 03:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 03:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 03:26:05 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 03:26:05 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 03:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 03:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 03:30:01 abc_primary_nas cachetool.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 03:30:09 abc_primary_nas cachetool.py: [common.pipesubr:66] Popen()ing: klist
Jul 14 03:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 03:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 04:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 04:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 04:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 04:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 04:26:16 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 04:26:16 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 04:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 04:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 04:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 04:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 05:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 05:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 05:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 05:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 05:27:13 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 05:27:13 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 05:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 05:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 05:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 05:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 06:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 06:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 06:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 06:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 06:27:50 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 06:27:50 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 06:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 06:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 06:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 06:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 07:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 07:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 07:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 07:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 07:17:40 abc_primary_nas rsyncd[4754]: file has vanished: "/Backups/VSS Bi Monthly Backup of Source/192.168.1.32/ABC-SERVICE/Backup Set 2017-07-06 030006/Backup Files 2017-07-09 030006/SBSE____.N8F" (in storage)
Jul 14 07:28:05 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 07:28:05 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 07:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 07:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 07:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 07:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 08:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 08:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 08:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 08:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 08:28:39 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 08:28:39 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 08:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 08:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 08:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 08:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 09:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 09:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 09:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 09:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 09:29:20 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 09:29:20 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 09:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 09:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 09:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 09:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 10:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 10:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 10:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 10:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 10:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 10:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 10:30:00 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 10:30:00 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 10:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 10:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 11:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 11:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 11:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 11:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 11:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 11:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 11:30:50 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 11:30:50 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 11:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 11:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 12:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 12:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 12:15:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 12:15:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 12:30:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 12:30:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 12:31:32 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 14 12:31:32 abc_primary_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Jul 14 12:45:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 12:45:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 13:00:00 abc_primary_nas rsync: @ERROR: chroot failed
Jul 14 13:00:00 abc_primary_nas rsync: rsync error: error starting client-server protocol (code 5) at main.c(1662) [sender=3.1.2]
Jul 14 13:00:48 abc_primary_nas manage.py: [rollbar:1097] Exception while posting item ConnectionError(ProtocolError('Connection aborted.', BadStatusLine("''",)),)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1095, in _send_payload
	_post_api('item/', payload, access_token=access_token)
  File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1136, in _post_api
	verify=SETTINGS.get('verify_https', True))
  File "/usr/local/lib/python2.7/site-packages/rollbar/lib/transport.py", line 7, in post
	return _session.post(*args, **kw)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 522, in post
	return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 596, 
Jul 14 13:00:48 abc_primary_nas manage.py: in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
	raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Stop refresh
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
One other thing... I did run a zpool scrub on the main pool.. and after 4 days, and working at about 5ish mb per minute for most of it, it still have 96 hours to go... i had to cancel it because the CPU was running at 80-90% and it was degrading performance quite dramatically...
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
How is this 5th drive attached? Seems like you have a core filesystem performance problem. Can you create a data with compression turned off so we can run a test? Run this dd if=/dev/zero of=/mnt/tank/dataset/10gig.dat bs=1M count=10000

Make the of= path appropriate to your system.

Sent from my Nexus 5X using Tapatalk
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
markbris said:
One other thing... I did run a zpool scrub on the main pool.. and after 4 days, and working at about 5ish mb per minute for most of it, it still have 96 hours to go... i had to cancel it because the CPU was running at 80-90% and it was degrading performance quite dramatically...
Click to expand...
I don't like the sound of that at all. Disable all the rsync stuff and lets see if performance looks nominal after that.
 
Jailer

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
Ericloewe said:
I don't like the sound of that at all. Disable all the rsync stuff and lets see if performance looks nominal after that.
Click to expand...

Howdy

I've done that, on both 1.3 and 1.4 to see what happens, will monitor over the next 24 hours.
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
Jailer said:
Please explain how this drive is connected.
Click to expand...
Howdy

There are Five (5) 3TB WD Purple Drives in a Raidz0-1 configuration, Four of these drives are in the 4 port drive cage, that is connected onto the main mother board, they can be configured with the onboard controller (Some variant on a HP Raid Controller), but they are in ACPI Mode (from memory), the fifth drive is plugged into the onboard sata port (Main Board has two SATA Ports).

Hope this helps.

According to HP

HP ProLiant ML10 v2 Server G3240

Pentium G3240 3.1GHz
4x 3.5" Bay Controller
2x 5.25" Bays
2x Gigabit Ethernet port

HP Dynamic Smart Array B120i Controller
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
The issue is most definitive coming from RSYNC (not the module, but what its doing)... I disabled it an hour ago... Unit responds great again, quick and snappy...

BUT

am still getting

Code:

Jul 16 12:44:35 abc_backup_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 16 12:44:35 abc_backup_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Stop refresh


Every 15-30 minutes.. any ideas what this is?
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
SweetAndLow said:
How is this 5th drive attached? Seems like you have a core filesystem performance problem. Can you create a data with compression turned off so we can run a test? Run this dd if=/dev/zero of=/mnt/tank/dataset/10gig.dat bs=1M count=10000

Make the of= path appropriate to your system.

Sent from my Nexus 5X using Tapatalk
Click to expand...

Howdy SnL

Have done this on both the servers now,

Both report similar settings
Code:

[root@abc_primary_nas /mnt/ABC_Primary/storage]# dd if=/dev/zero of=/mnt/ABC_Primary/storage/10gig.dat bs=1M count=10000			
10000+0 records in																												 
10000+0 records out																												 
10485760000 bytes transferred in 4.584274 secs (2287332711 bytes/sec)															   
[root@abc_primary_nas /mnt/ABC_Primary/storage]#
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
markbris said:
Howdy SnL

Have done this on both the servers now,

Both report similar settings
Code:

[root@abc_primary_nas /mnt/ABC_Primary/storage]# dd if=/dev/zero of=/mnt/ABC_Primary/storage/10gig.dat bs=1M count=10000			
10000+0 records in																												 
10000+0 records out																												 
10485760000 bytes transferred in 4.584274 secs (2287332711 bytes/sec)															   
[root@abc_primary_nas /mnt/ABC_Primary/storage]#
Click to expand...
You didn't turn compression off in that dataset. Try again....

Sent from my Nexus 5X using Tapatalk
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
markbris said:
The issue is most definitive coming from RSYNC (not the module, but what its doing)... I disabled it an hour ago... Unit responds great again, quick and snappy...

BUT

am still getting

Code:

Jul 16 12:44:35 abc_backup_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name
Jul 16 12:44:35 abc_backup_nas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
Stop refresh


Every 15-30 minutes.. any ideas what this is?
Click to expand...
You can ignore this error, it's not a real error.

Sent from my Nexus 5X using Tapatalk
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
SweetAndLow said:
You didn't turn compression off in that dataset. Try again....

Sent from my Nexus 5X using Tapatalk
Click to expand...
SnL

Is there a switch to use for turning it off? Or do we turn off compression in the gui for the array etc (can you even do that once it has been enabled!!)

Are we going to try a throughput test?
 
Last edited:
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
markbris said:
SnL

Is there a switch to use for turning it off? Or do we turn off compression in the gui for the array etc (can you even do that once it has been enabled!!)

Are we going to try a throughput test?
Click to expand...
With the check box in the GUI. Either make a new testing dataset or just turn it off in a existing dataset. Yes it can be turned on and off.
markbris said:
Do you know what it relates to, or a way to stop it coming up?
Click to expand...
Related to a bug fix for the avaton boards think. Happens to everyone like once an hour I think. Sounds be 2 messages. Can't turn it off. You could upgrade to 11 which might not have that problem.

Sent from my Nexus 5X using Tapatalk
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
SweetAndLow said:
With the check box in the GUI. Either make a new testing dataset or just turn it off in a existing dataset. Yes it can be turned on and off.Related to a bug fix for the avaton boards think. Happens to everyone like once an hour I think. Sounds be 2 messages. Can't turn it off. You could upgrade to 11 which might not have that problem.

Sent from my Nexus 5X using Tapatalk
Click to expand...
Upgraded to 11 on the backup machine, now have all kinds of kernal panics, and database errors... ha... hence why I do things on the backup.. will reformat the USB sticks, and put the old version back on, and probably kill the ZFS and rebuild it (its the backup after all!)
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
Ericloewe said:
I don't like the sound of that at all. Disable all the rsync stuff and lets see if performance looks nominal after that.
Click to expand...
Have had it turned off for a week, and performance is good.. so it seems to be around the RSYNC issues.
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
SweetAndLow said:
How is this 5th drive attached? Seems like you have a core filesystem performance problem. Can you create a data with compression turned off so we can run a test? Run this dd if=/dev/zero of=/mnt/tank/dataset/10gig.dat bs=1M count=10000

Make the of= path appropriate to your system.

Sent from my Nexus 5X using Tapatalk
Click to expand...

Okay After 4 Days doing a scrub, its back to normal, and I've managed to get 10 to run the 10gb creation...

Here is the output

Code:

[root@abc_primary_nas /mnt/ABC_Primary/storage]# dd if=/dev/zero of=/mnt/ABC_Primary/storage/10gig.dat bs=1M count=10000			
10000+0 records in																												 
10000+0 records out																												 
10485760000 bytes transferred in 27.623091 secs (379601255 bytes/sec)


Doesnt Seem *to* bad... any thoughts?
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Sounds like you were overloading the CPU with rsync, which dragged everything down.
 
M

markbris

Dabbler
Joined
Apr 29, 2017
Messages
16
This is what I have done, I've scapped the backup machine, and reloaded it from source, updated the os to the latest release in the 9x stream, and put it back into service, your all right, I was overloading both machines with two RSYC tasks, so I've had one running for 4 days now backing up about 6TB... its nearly done.. Thought it would have been a LOT quicker than this, Just need to figure out why its still using 50% of the CPU on one RSYC task...
 
Stux

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Check your memory usage/swap etc in the Reporting panel. You're looking for lots of swap usage and/or a large percentage of Inactive Memory.
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Strange and Odd Behaviour (Exceptionally Slow Connections, Laggy, High Latency etc)"

Similar threads

X
  • Locked
Error Message Spamming Console - /sbin/dmidecode
Replies
1
Views
1K
m0nkey_
m0nkey_
F
  • Locked
New alert found today, anyone know what it is?
Replies
1
Views
1K
dlavigne
D
O
  • Locked
SOLVED Rsync error question
Replies
3
Views
8K
onekanoobie
O
bbox
  • Locked
Reading the console: "mDNS_Register_internal: ERROR!!"
Replies
2
Views
1K
bbox
bbox
VioletDragon
  • Locked
Rsync not working.
Replies
2
Views
2K
VioletDragon
VioletDragon
Top