Home

Forums

Stories

Knowledge Base

Business Community > Switches > TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

< Switches

TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.

TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.

Gerco

2023-12-03 21:39:00

Posts: 7

Helpful: 1

Solutions: 1

Stories: 0

Registered: 2020-01-01

TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

2023-12-03 21:39:00

Model: TL-SG3428X

Hardware Version: V1

Firmware Version: 1.0.12 Build 20230602 Rel.76586

I recently bought this 3428x switch two transfer data between two Hp Proliant ML 310e servers (one is master, other is slave, a spare copy).

Both these servers have a HP Ethernet 10Gb 2-Port 560SFP+ Full Profile network card installed in the 8x PCI-e slots. Both servers are connected to the TP Link TL-SG3428X switch and both have TRUENAS Scale installed on bare metal.

One HP server is connected through a HP Aruba 10G SFP+ To SFP+ DAC cable to the switch. The other is connected through transceivers (TP-Link 10G SFP+ in the switch and Microtik 10G SFP+ in the HP; MM / lc-lc) with an OM3 cable. The switch reports both servers are connected at 10G speed, full duplex.

I can access the TrueNas scale gui of both hp servers and they seem to be working great. I can copy files from f.e. my synology and pc (both 1G connection) to either of the machines just fine. Speed 113 mb/s.

Now, here is my issue. When I want to replicate data from one server to the other, the speed is very slow (300 kb/s). When I use SCP to transfer a file, the tranfers stops after a few seconds, message is --stalled--. When I use rsync, it will copy at very slow speeds too. I have set the MTU to 9000 in both truenasses and switch.

When I connect the two HP's directly via that DAC cable, I can copy files at high(er) speed. I prefer for data transfers to travel through the switch because I will upgrade my PC and Synology to 10G networking as well.

Does anyone have a clue what is wrong here? Are there specific settings in the switch I should need to alter before I can transfer at 10G speed (or close to it)?

Should replace the DAC for two transceivers (the DAC might not be compatible to run at high speed?)

Anything else?

Thank you!

2 Reply

Clive_A

2023-12-04 02:50:34

Posts: 5981

Helpful: 2947

Solutions: 1375

Stories: 0

Registered: 2023-06-09

Re:TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

2023-12-04 02:50:34

Hi @Gerco

Thanks for posting in our business forum.

Main server---SW---Copy server

Can you first rule out the possibility of the DAC? So, it is simple. Between the main and the copy server, if you perform a file transfer, what's the best speed? Is this speed normal?

If yes, we can rule out the possibility of the cable.

When you use a DAC to connect both servers directly, bypassing the switch, what's the "high(er)" speed you can get?

When you sync the main and copy, is there anything else going on both servers? Have you ruled out the possibility of the CPU throttle? SSD/HDD bottleneck？

300KB/s = 3Mbps. It is very strange.

And, you mentioned you've enabled the jumbo frame. So, if you enable the jumbo frame, it is very recommended to set all devices in this network, which means both servers and SW need to be set to the jumbo frame of 9,000. Can you Wireshark and verify that the jumbo frame is working? Monitor its packet size.

Have you checked what's on truenas forum about this? Any support from truenas to locate the issue?

https://www.truenas.com/community/threads/rsync-stalled.26506//

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update! ☛Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.

Gerco

LV1

2023-12-04 20:32:04 - last edited 2023-12-04 20:36:00

Posts: 7

Helpful: 1

Solutions: 1

Stories: 0

Registered: 2020-01-01

Re:TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

2023-12-04 20:32:04 - last edited 2023-12-04 20:36:00

@Clive_A

Hi Clive,

Thank you for your extensive reply. You've given me quite a bit to look into, which I will.

However, since I am 'playing around' with my two servers with my crucial data on it, I am first making a second back-up to my Synology, just to make sure I am not losing any data at the process. This will take 2 days, I think. Plus, I am going to Brussels, Belgium, tomorrow morning and won't return until Wednesday evening.

So allow me a couple of days before I can try any of your suggestions and reply to your questions.

To answer some of your questions already:

When you use a DAC to connect both servers directly, bypassing the switch, what's the "high(er)" speed you can get?

What I do seem to recall (but I'll reverify) is that using SCP to transfer a large file between the two servers, directly connected through the DAC, the speed went up tp 340MB/s where I would expect ~900MB/s. I do think I did not change the mtu from 1500 to 9000, which I think could/would explain this, right?

When you sync the main and copy, is there anything else going on both servers? Have you ruled out the possibility of the CPU throttle? SSD/HDD bottleneck？

No, nothing else was running on both servers.
I didn't rule out the CPU throttle possibility. Both servers have been running ProxMox for over a year without any issues when it comes to CPU throttling. I will check though.
That also applies for a possible SSD/HDD bottleneck.

I will check that TrueNas forum link, thank you. I will also try a direct link between the two servers, using two transceivers and OM3 fiber cable.

Kind regards,

Gerco van der Boon

Netherlands

TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

TL-SG3428x 1.0 / two device connected at 10Gb but data transfer sufficates

Information

Voters 0

Tags