T2600G-28TS - Problem with LACP and Jumbo Frame

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
12

T2600G-28TS - Problem with LACP and Jumbo Frame

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
T2600G-28TS - Problem with LACP and Jumbo Frame
T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-16 02:51:25
Model :

Hardware Version :

Firmware Version :

ISP :

Hi guys,
I'm writing here because I need your precious help.
I think to probably have a little problem of configuration on my network and I'llI try to explain in simple way the problem and my network's configuration. Thanks to all for support and patience !
I have an ASUS AC87U Router, a T2600G-28TS and two identical client with AMD FX 8350 processor, 32 Gb of ram and a SSD as primary disk.
In each of these two clients I have installed a
HP 331T Quad Port 1GbE Network Adapter (which is moved by a Bradcom chipset and which is managed using Bradcom Advanced Control Suite 4).
Using Broadcom Control Suite I create a Team using all 4 lan port and on my T2600G-28TS I did a link aggregation (4 lan port of the switch for the first pc and other 4 lan port of the switch for the second PC).
The cables are CAT6A FTP,
The problem is that when I try to transfer data from PC1 to PC2 (form a source folder that on PC1 is on SSD to a target folder that on PC2 is always on a SSD) I don't see throughput of the Team adpater which I can see regularly working under lan device of my Windows 10.
Team device show 4Gbit connection while data transfer is of 130 Mbyte (so more near to a 1Gbit connection).
In my idea I think to reach 300-400 Mbyte of speed but it isn't. There is something wrong in my idea?

I did some test using iperf and i decide to work on the same pc as server and as client infact from a DOS command line I launched this command :
iperf3.exe -s -i 5

and from a second DOS command line :
iperf3.exe -c ipserver -t 60 -i 5

and this is the result:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.00 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.00 sec 26.2 GBytes 3.74 Gbits/sec receiver

It seem to be correctly aggregated but I can see this speed when I move data from PC1 to PC2. Can be a reason? Could I check something on the switch?

The second problem is jumbo frame.
I set MTU 9000 to all 4 lan adapter of the HP 331T Quad Port 1GbE Network Adapter, so Team adapter has 9000 MTU.
I try this command “ping IP address –f –l 9000" but the answer is that packet need to be fragmented and I don't undestand the reason.

On the switch (for the first PC) in LAG session I have







1 Active LACP 1/0/17,1/0/19,1/0/21,1/0/23




Hash Algorithm SRC IP + DST IP

Detail info :









































































Group Number:
LAG1

LAG Type:
Active LACP

Port Status:
Enable

Speed:
Auto

Flow Control:
Disable

Ingress Bandwidth (bps):
--

Egress Bandwidth (bps):
--

Broadcast Control (bps):
--

Multicast Control (bps):
--

UL Control (bps):
--

QoS Priority:
CoS 0

Join VLAN:
1




I don't know reason why bps field are blank.

I hope to be sufficiently clear in the exposition. If you need more info please tell me .
Thanks you very much for time and support that you will dedicated to me.





  0      
  0      
#1
Options
13 Reply
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-17 06:37:00

AINEMORF wrote

It seem to be correctly aggregated but I can see this speed when I move data from PC1 to PC2. Can be a reason? Could I check something on the switch?


You selected SRC IP + DST IP as the LAG hash algorithm. Should use SRC MAC + DST MAC to make use of more than one LAG channel.

As for the jumbo frames: sometimes the 9kB specification includes TCP headers, sometimes not, so try to set a slightly lower MTU.
༺ 0100 1101 0010 10ཏ1 0010 0110 1010 1110 ༻
  0  
  0  
#2
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-17 17:01:10
Thanks a lot for your answer.
I made the change you indicated to me but I can't see any improvement but I have a doubt.
I know that 802.3ad standard (which is the standard for Link aggregation) does not increase the bandwith for a single conversation.
This means that also if I have a quad lan port on each of the two computers if a transfer a single file of 4 Gb, from a pc to another, the speed is of 1 Gbit/s.
I could see the improvement only if I transfer 4 file of 4 Gb. In this situation I could see 4 transfer process which can go ad 1 Gbit/s.

Now the doubt : BASF (Broadcom Advanced Control Suite driver) has 3 option when I go to create team :

1-Load balancing and failover (it is not my case because I'm trying to improve performances of speed)
2-802.3ad Link Aggregation using Link Aggregation Control Protocol (LACP)
3-FEC/GEC Generic Trunking

I choose option 2 and the suite create my virtual lan adpater (called Team 1).
Doing this, if I go on the web pages of the switch under SWITCHING-LAG i see the Member correctly populated:


Global Config
Hash Algorithm: SRC MAC + DST MAC
LAG Table
Select Group Number Description Member Operation
1 Passive LACP 1/0/18,1/0/20,1/0/22,1/0/24 Edit | Detail
2 Passive LACP 1/0/17,1/0/19,1/0/21,1/0/23 Edit | Detail


In other way do I have a change to see for a SINGLE transfer a speed > 1 Gbit/s? (working on configuration of my switch or on configuration of my pci lan quad adapter)

On youtube there is a video (linustechtips link aggregation) which speaks about RSS and SMB3.0 and he tells that he was able to transfer a single file at 3.60Gbit (you can also in the video) because his configuration splits traffic on the 4 cable. Infact if we see the video and his W10 perfromances monitor we can see 4 lan apdator and not 1 (he has no teaming). I try also in this way enablin RSS and SMB3-0 but nothing (I have tremendous performances).

Thanks a lot fot your support.
  0  
  0  
#3
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-17 19:47:29
You are correct for the question about jumbo frame
Infact if I try this command from PC1 : ping PC2 -f -l 8970 it gives us a correct result (also if from PC2 I try : ping PC1 -f -l 8970).
We are very very near to the limit of 9000 for maximun MTU.
  0  
  0  
#4
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-19 08:36:22

AINEMORF wrote


In other way do I have a change to see for a SINGLE transfer a speed > 1 Gbit/s? (working on configuration of my switch or on configuration of my pci lan quad adapter)


IMHO this depends on implementation of the interfaces used for the LAG. If the 4 ports are independent interfaces they should have different MACs even in a LAG, at least on the system I did use channel bonding on (server running IRIX). But Windows and even Linux might use another scheme. I'm not familiar with those. Currently I just use LAGs between TP-Link switches, not servers or even client devices.

Maybe it's easier to consider buying a 10Gbit/s interface card to increase bandwidth of a single system/single session.
༺ 0100 1101 0010 10ཏ1 0010 0110 1010 1110 ༻
  0  
  0  
#5
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-21 03:17:55
Hi,
thanks a lot for your answer.
I took some steps forward and I wanted to share with the results.
Let's start by saying that, as the standard says, LACP does not increase speed but increases troughtput and these are extremely different concepts.
If, as in my case, we have quad lan adapters we have a transfer rate of 1 Gbit / s. The difference is that we have more lanes (many as the ports made available by the network card that we will aggregate and configure on the switch) .
This operation allows you to transfer 4 files simultaneously at a rate of 1Gbit / s.
If you want to increase the speed you do not have to teaming and use the LACP protocol, but you have to work on the individual adapters focusing on the parameters: jumbo frames, receive buffers, transmission buffer, RSS queus.
You need to have enabled the SMB 3.0 protocol and have a processor that has a good number of physical cores (for example 8).
In this scenario with my FX8350 (4 core) and this setup

Ethernet 15 802.3az EEE (Copper Only) Disable
Ethernet 15 Flow Control Auto Negotiation
Ethernet 15 Interrupt Moderation Enabled
Ethernet 15 Jumbo Mtu 8970
Ethernet 15 Large Send Offload V2 (IPv4) Disabled
Ethernet 15 Large Send Offload V2 (IPv6) Disabled
Ethernet 15 Maximum Number of RSS Queues RSS 4 Queues
Ethernet 15 ARP Offload Enabled
Ethernet 15 NS Offload Enabled
Ethernet 15 Priority & VLAN Priority & VLAN Enabled
Ethernet 15 Receive Buffers Maximum
Ethernet 15 Receive Side Scaling Enabled
Ethernet 15 Speed & Duplex (Copper Only) Auto Negotiation
Ethernet 15 TCP/UDP Checksum Offload (I... Rx & Tx Enabled
Ethernet 15 TCP/UDP Checksum Offload (I... Rx & Tx Enabled
Ethernet 15 Transmit Buffers 600
Ethernet 15 Virtual Machine Queues Disabled
Ethernet 15 VMQ VLAN Filtering Disable
Ethernet 15 Wake on Magic Packet Enabled
Ethernet 15 Wake on Pattern Match Enabled
Ethernet 15 EEE Control Policies (Coppe... Optimal Power and Performance
Ethernet 15 Network Address D89D672DA0D9
Ethernet 15 VLAN ID 1
Ethernet 15 Ethernet@WireSpeed (Copper ... Enable
Ethernet 15 WOL Speed (Copper Only) Lowest Speed Advertised

I have 220 Mbyte/s transfer rate. The video on youtube showed 450MByte/s but that guys have 8 processor so I think 220 MByte/s can be a good results.

BUT this was only an experiment because i want to team the 4 lan port into a virtual team card and i want to use LACP protocol

NOW THE new PROBLEM :

I want to create on TPLINK T2600G-28TS switch three different LAG group. Every of this group is made by three ports.

LAG Table









Select Group Number Description Member Operation
























[/TD]
[TD][FONT=Arial]1
Passive LACP 1/0/17,1/0/19,1/0/21,1/0/23 [COLOR=#3366FF]Edit | Detail
[/TD]
[TD]2
Passive LACP 1/0/18,1/0/20,1/0/22,1/0/24 Edit | Detail
[/TD]
[TD]3
Passive LACP N/A Edit | Detail





[/FONT]

Group number 1 is the port of PC1, group number 2 the port of pc 2 and group 3 the port of the NAS that at the moment I don't have but I'll buy in the future.

Port Config





UNIT: 1LAGS


























Select Port Type Description Status Speed Duplex Flow Control Jumbo LAG
[/TD]
[TD]
[/TD]
[TD]
DisableEnable 10M100M1000MAuto HalfFullAuto DisableEnable DisableEnable [/TD]








Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper









Copper










Copper










Copper










Copper










Copper










Copper










Copper










Copper










Copper










Copper










Copper










Copper








[TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD] [TD]
[FONT=Arial]1/0/1[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/2[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/3[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/4[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/5[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/6[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/7[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/8[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/9[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/10[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/11[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/12[/TD]
[TD]Enable
Auto Auto Disable Disable ---
[/TD]
1/0/13LAG_3_Nas Enable Auto Auto Disable Disable ---
[/TD]
1/0/14LAG_3_Nas Enable Auto Auto Disable Disable ---
[/TD]
1/0/15LAG_3_Nas Enable Auto Auto Disable Disable ---
[/TD]
1/0/16LAG_3_Nas Enable Auto Auto Disable Disable ---
[/TD]
1/0/17LAG_1_Matteo Enable 1000M Full Enable Enable LAG 1
[/TD]
1/0/18LAG_2_Serena Enable 1000M Full Enable Enable LAG 2
[/TD]
1/0/19LAG_1_Matteo Enable 1000M Full Enable Enable LAG 1
[/TD]
1/0/20LAG_2_Serena Enable 1000M Full Enable Enable LAG 2
[/TD]
1/0/21LAG_1_Matteo Enable 1000M Full Enable Enable LAG 1
[/TD]
1/0/22LAG_2_Serena Enable 1000M Full Enable Enable LAG 2
[/TD]
1/0/23LAG_1_Matteo Enable 1000M Full Enable Enable LAG 1
[/TD]
1/0/24LAG_2_Serena Enable 1000M Full Enable Enable LAG 2




If I try in this configuration a single transfer this is slow only 40 Mbyte/s and if I try multiple transfers, for example 4, speed decrease and it is divided a 10 MByte/s for single transfer .
[/FONT]
Obviously here there is something wrong because it works opposite to the standard of 802.3ad...
But were is the error? Do you see something wrong?
Please help me...
Thanks
  0  
  0  
#6
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-21 09:36:20
I don't know whether this is the error, but AFAIK all your systems need to process Jumbo frames if one device does to get full benefits of using Jumbo frames. IMHO, LAG makes only sense if you have at least a number of different sessions (with different IPs or MACs) equal the number of ports in the LAG. That's not the case if you transfer 4 files from the same server system to the same client system.
༺ 0100 1101 0010 10ཏ1 0010 0110 1010 1110 ༻
  0  
  0  
#7
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-21 23:05:13

R1D2 wrote

I don't know whether this is the error, but AFAIK all your systems need to process Jumbo frames if one device does to get full benefits of using Jumbo frames. IMHO, LAG makes only sense if you have at least a number of different sessions (with different IPs or MACs) equal the number of ports in the LAG. That's not the case if you transfer 4 files from the same server system to the same client system.


This is my logic error. Infact also if I start 4 transfer process ip is the same so LACP doesn't work...OK now I have undertstood.
Only a last question...(this is my last problem, I hope, about this discussion). For what reason I have a transfer speed of only 65 Mbyte/s, when a transfer a file from PC1 to PC2 using teaming lan virtual adapter of the PC1 and PC2 and the LACP configuration on the TP-Link switch described in the previos, post against 119 MByte/s when I use standard integrated Lan adapter with the same transfer test (same folder, same file). This difference has no sense...

Thanks
  0  
  0  
#8
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-21 23:44:00

AINEMORF wrote

For what reason I have a transfer speed of only 65 Mbyte/s, when a transfer a file from PC1 to PC2 using teaming lan virtual adapter of the PC1 and PC2 and the LACP configuration on the TP-Link switch described in the previos, post against 119 MByte/s when I use standard integrated Lan adapter with the same transfer test (same folder, same file).


Maybe due to customized settings? Just a guess, I'm not a PC expert. But from what I see above, hardware off-loading for "large send" (whatever large is in this context) is disabled, so check if with standard settings it is enabled. On my routers, disabling HW off-loading reduces NAT throughput by 66% (from 900 Mbit/s down to 300 Mbit/s). Usually standard settings are tuned to the maximum already, so I would not fiddle too much with it.
༺ 0100 1101 0010 10ཏ1 0010 0110 1010 1110 ༻
  0  
  0  
#9
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-22 04:18:44
YOU ARE RIGHT!!!
The problem is the tuning on the driver's parameters.
Now I try to explain all my test hoping to be useful for someone.
Like you told me I set the standard parameters and transfer speed has reached 117MByte/s.
Later I try to modify jumbo frame with catastrofic results and this is interesting because if I work without team, all the lan adapter of the quad lan adapter can support MTU of 8970 but this value is not support by the virtual lan adapter (Team 1).
As second thing I try to change Transmission buffer and Receive buffers setting the first to 600 and the second to Maximum and I could see to to changing this couple of parameters change the behaviour of the adpater infact also if in all the tests I made the transfer speed is always about 114-117MByte/s it is different the way how you can reach this speed.
The milestone is to keep 1500 MTU, later if we keep default value for buffer we can see that when we start a transfer process speed follows these values 30-40-50-80-90-114MByte/s while if we increase this value transfer process arrives to 114MByte/s faster.

Thanks

Now I was thinking to another thing... but in this argument I'm just ignorant...
If we set different VLAN? A different VLAN for every lan port of the quad lan adapter... We could have a situation where we have different IPs and different MACAddress...at this point if we do multiple transfer processes we could see a situation of 4 transfer process where every of this transfers go ad 114 MByte/s... Please try only to follow my idea because honestly I never set up a VLAN and I don't have good basic about it
If you can give me some suggestions and your opinions on this argument I will appreciate it very much.

In other words..I'd like to create a scenario where I can see working the quad lan adapter and LACP configuration on the switch.

Thanks a lot
  0  
  0  
#10
Options
Re:T2600G-28TS - Problem with LACP and Jumbo Frame
2018-04-23 20:32:03
I suppose that last reply is not pertinent to the topic ?
  0  
  0  
#12
Options

Information

Helpful: 0

Views: 2556

Replies: 13