Read slow after write.

[skyyxy]

Hi everyone here
I have my own Truenas core server in the office.
Truenas core 13.0 u2
xeon 1620v3
64GB reg ecc ddr4
24*8TB HGST HDD
2TB nvme L2ARC cache and without SLOG(because I dont have best device for SLOG)
6 x Intel x520 10Gb network link to PCs without switch.
set sync=disable (for best write speed)

I upload 100GB file to the server, after that read speed drop down to even 2MB/S-10MB/S(normally should 700MB-1GB/S), then I restart the server and back to normal.

Why its happen and what should I do to fix it. Thanks a lot.

I remember the webgui have some settings about limit how much ram for write cache, Maybe I should do somthing there?

 

[MrGuvernment]

Why did you feel you needed L2ARC drive or how did you think it would help your configuration?

• How are your drives configured? how manyt vDevs?
• What are you using to test the speeds?
• Have you done any testing on the server it's self using dd to get raw numbers of your drive performance?
• Have you done any iperf testing to and from your server to rule out any network issues?

 

[joeschmuck]

I upload 100GB file to the server, after that read speed drop down to even 2MB/S-10MB/S(normally should 700MB-1GB/S), then I restart the server and back to normal.

Can you be specific on the steps you do? Example: 1) transfer a single 100GB video file from a Windows PC #1 via SMB to TrueNAS. 2) Read that same file back from the NAS to Windows PC #2 via SMB and it's slow. Have you tried this from only one PC or all of the other PC's?

Also, provide screen captures of the network speed graphs for each operation.

How old is this TrueNAS setup? Have your had any hardware changes in the past month or two? Or why do you think this started happening? You probably have more insight than we do since you operate that system and we do not.

@MrGuvernment has questions which all should be addressed as everyone will want to know those pieces of information as well.

 

[skyyxy]

Why did you feel you needed L2ARC drive or how did you think it would help your configuration?

• How are your drives configured? how manyt vDevs?
• What are you using to test the speeds?
• Have you done any testing on the server it's self using dd to get raw numbers of your drive performance?
• Have you done any iperf testing to and from your server to rule out any network issues?

thanks for reply.
. 1 VDEV and raidz2
. aja speed test
. I will test by dd and fio in monday.
. not yet but will can do it with vdev speed test together.

something forgotten. the hba is Lsi 2308 with 6ports connnet to the back pannel and each port has 4hdds. the server has working good for 1.5year and going wird like I said in these 3months.

I still feel its about write cache in memory. because after restart the server that the performance will back to normal and will happen in days or weeks again and I need restart the server again. If it about the hardware that will not going back after restart and keep days and weeks. I will try to limit write cache size of memory first and looks forward.

 

[skyyxy]

Can you be specific on the steps you do? Example: 1) transfer a single 100GB video file from a Windows PC #1 via SMB to TrueNAS. 2) Read that same file back from the NAS to Windows PC #2 via SMB and it's slow. Have you tried this from only one PC or all of the other PC's?

Also, provide screen captures of the network speed graphs for each operation.

How old is this TrueNAS setup? Have your had any hardware changes in the past month or two? Or why do you think this started happening? You probably have more insight than we do since you operate that system and we do not.

@MrGuvernment has questions which all should be addressed as everyone will want to know those pieces of information as well.

thanks for reply.
my 100GB file not only 1.is around 2x files.

I transfer the file from mac os to server via smb around 700-1000MB per s.

mins later try to read them in Davinci Resolve very slow and test by aja speed test just 2-10MB per s.

the TN core server was 1.5years and works good, just going wird in these 3months for few times and every time I need reboot the server then can keep for days or weeks.(upgrade os from FREENAS 11 TO 13.2U2 by clean install and inport the data pool,but not upgrade the zfs version yet.)

my understood is after reboot the memory has totally wiped the speed comes up again. but I dont know how to limit the write cache in memory or if its about sync=dusable or else?

 

[joeschmuck]

but not upgrade the zfs version yet.)

Outstanding news! This means you could roll back to your previous version if needed, and you might want to do this to prove that the problem is not something else.

Since you upgraded from FreeNAS 11, might I ask what your HBA is flashed to right now? I'm not the expert here but read this, see if you are running the correct HBA firmware.

Detailed newcomers' guide to crossflashing LSI 9211/9300/9305/9311/9400/94xx HBA and variants

There's hundreds of threads on crossflashing variants of LSI 9211/9300/9305/9311/94xx cards here, on ServeTheHome, and on blogs. But they don't always discuss the subtleties of doing it on an EFI board for users who may never have used EFI...

www.truenas.com

 

[skyyxy]

Outstanding news! This means you could roll back to your previous version if needed, and you might want to do this to prove that the problem is not something else.

Since you upgraded from FreeNAS 11, might I ask what your HBA is flashed to right now? I'm not the expert here but read this, see if you are running the correct HBA firmware.

Detailed newcomers' guide to crossflashing LSI 9211/9300/9305/9311/9400/94xx HBA and variants

There's hundreds of threads on crossflashing variants of LSI 9211/9300/9305/9311/94xx cards here, on ServeTheHome, and on blogs. But they don't always discuss the subtleties of doing it on an EFI board for users who may never have used EFI...

www.truenas.com

yep, That's the point why I not upgrade zfs verion to system remind me, Because maybe I need rollback to the old system version.
My HBA card upgrade from LSI 9200-8I to LSI 9300-8I newest FW from offical website (2016 year version I remember), and works well.
Tomorrow I will try to set the sync = disable to Standard and watch days or weeks, Hope its about it.