DPI Not Accurate/Upload Not Included

DPI Not Accurate/Upload Not Included

DPI Not Accurate/Upload Not Included
DPI Not Accurate/Upload Not Included
2024-10-18 04:36:04 - last edited Thursday
Model: ER7206 (TL-ER7206)   EAP245  
Hardware Version: V2
Firmware Version: 2.1.2

Hi Team,

Perhaps you can help, have a ER7206 with a Docker Controllere on the latest release of Firmware + Controller updates for some reason DPI is not accurate at all, no upload usage gets logged at all, my EAP245 is plugged into the gateway along with my main PC and Server images below

 

Anyone else have the same issue? as can see lots of traffic missed mostly P2P and normal file transfers via HTTPS

  0      
  0      
#1
Options
1 Accepted Solution
Re:DPI Not Accurate/Upload Not Included-Solution
2024-10-22 01:25:25 - last edited Thursday

Hi @ZainexZA 

Thanks for posting in our business forum.

ZainexZA wrote

  @Clive_A 

 

Hi Clive this is indeed correct, traffic is 99% encrypted as per SSL certs on most downloads these days hence why i can still understand why Youtube stats are far off, but how does Unifi manage to solve this? do they use a Man in the middle cert that users can install to investigate the packet flow? qBittorent should be listed let me check. 

 

 

 

So for these the app flow should be the same unless a standard port must be used if you can somehow guide me in the right direction or somehow i can capture packets white wiresharck to send to to RnD so we can get this on par with Unifi and other DPI solutions this would be great, reason for the large traffic amounts is due to the lab i am running i need to get this info sorted for various reasons or understand what the shortfalls are that i can explain these to our clients to push the aweness and present other focus points on why Omada is right for them and what it can and cannot do.

Not asking for a miricale here but a intermidate solution even if this is users being uable to update the DPI Database themselves as i know when DPI is enabled it pulls the latest info from a cloud source somewhwere if i am not mistaken? i have the Q2025 Roadmap for Omada and know this is a feature our users will love and hence if i can help fix it or explain why xyz is not tracked this would be most helpful sadly i have no direct contact with HQ nor RnD from my current role. As you can see Steam is being tracked 100% accurate and i am pushing the limits on this almost over 2TB so far on steam alone QUIC sure cannot track due to how the protocol works but not sure how it picks up Mobile Youtube and tracks the usuage. Also why Upload is not counted on DPI as it should count downstream and upstream traffic? If you can replicate my lab this would be good!


Currently running Omada Cloud Trail (30 Days Standard) ER7206 v2 and a SG2210MP v4.20 switch with a EAP245 all firmware up to date till the latest release.

I am currently Employed by TP-Link ZA as Technical Sales and would share my info to you via DM so we can perhaps solve this inernally and give our users more function and help improve the Omada Ecosystem, do let me know!

Since I am not working with the other vendors, I cannot say for sure that's a middleman cert to crack and log. That's a way to see the traffic flow but I think it is not safe and possible to do that as a solution sold on the market. Potential danger if the cert is modified.

As for some patterns that are correctly recognized, they should be stable and clear DST and ports recognized. Some patterns are stable, unlike the download-ware like bit download which may contain sensitive files that violate the DMCA. It could masquerade itself so the traffic might not be correctly told.

For ISPs, they simply cut off the high-bandwidth connection instead of recognizing it via its pattern time in certain regions. Or they use the database or tag to identify certain IP addresses to tell if they are bit-download servers. That requires a daily update to a database which is something that we don't have. The database is updated usually along with the Controller update.

 

Correct understanding of the DPI pattern recognition. DPI now is based on the pattern provided by vendors. It may not catch up with the latest changes in the traffic pattern if an app has changed. The traffic pattern may be changed and the database requires a follow-up update. So, it got the margin to take errors.

 

Further details regarding how DPI and what source the DPI database comes from, could be confidential. I don't have any details about it either since this goes way deep and some information is not made for the general public understanding. If you are an employee, you can contact the HQ and escalate this matter and the dev can get to you. The local company to HQ support channel is different from others.

You can send an email to support.smb@tp-link.com. Or check if there is a source with your local team leader who can relay you to the HQ via a different source. Sorry that I have not been assigned to a local team before so I don't know how your local team works.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
Recommended Solution
  0  
  0  
#7
Options
6 Reply
Re:DPI Not Accurate/Upload Not Included
2024-10-19 12:50:21

To add more info to this have tried different controller versions, have tried a full reset, have tried rebooting devices so no upload traffic gets detected, its not near accurate, youtube on Desktop is not tracked but somehow android works.
 

This is quite an issue as would like to offer this as an alternative to Unifi Systems for my clients that use a full stack.

  0  
  0  
#2
Options
Re:DPI Not Accurate/Upload Not Included
2024-10-21 02:06:16

Hi @ZainexZA 

Thanks for posting in our business forum.

We are not providing support to the Docker. If you could try this in the LINUX or Windows version and provide the version number you are testing with, that would be great.

And for the tests, please make sure everything is up-to-date.

 

The traffic summary counts all traffic instead of the real-time Internet consumption. It means regardless of whether the traffic is to the WAN or internal LAN transfer, they'll be calculated.

 

P2P or file transfer, that's up to the app you have. It could be classified as P2P instead of an accuracy issue.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
  1  
  1  
#3
Options
Re:DPI Not Accurate/Upload Not Included
2024-10-21 03:24:52

  @Clive_A Hi Clive,


Please see below have tried Linux/Windows + Currently on Omada Cloud with a trail license, still the same P2P Qbittoerent is the P2P Application that is in use funny enough downloads it does track, upload traffic however not so much:

 

More info below:

 

System Time

Oct 21, 202405:21:21 am

Uptime

1day(s) 16h 43m 21s

Data Storage

Data of Omada-MougrinetCloud is hosted in Europe

Controller Version

5.14.26.60

 

 

Thus far cloud has been the most accurate see some Youtube traffic is even detected now where as with the Controller-Self Hosted it was not no matter the OS

  0  
  0  
#4
Options
Re:DPI Not Accurate/Upload Not Included
2024-10-21 04:06:52 - last edited 2024-10-21 05:51:40

Hi @ZainexZA 

Thanks for posting in our business forum.

ZainexZA wrote

  @Clive_A Hi Clive,


Please see below have tried Linux/Windows + Currently on Omada Cloud with a trail license, still the same P2P Qbittoerent is the P2P Application that is in use funny enough downloads it does track, upload traffic however not so much:

 

More info below:

 

System Time

Oct 21, 202405:21:21 am

Uptime

1day(s) 16h 43m 21s

Data Storage

Data of Omada-MougrinetCloud is hosted in Europe

Controller Version

5.14.26.60

 

 

 

 

Thus far cloud has been the most accurate see some Youtube traffic is even detected now where as with the Controller-Self Hosted it was not no matter the OS

I am under the impression that you are a professional user and understand how DPI works. So no need for me to explain how it works.

 

So, now you are saying it is not accurate, except for the pictures from the Omada Controller and statement regarding the non-accuracy summary and DPI, logically, it takes two to tango. What do you have to show me that it is not accurate? The summary of your software which consumes that much traffic but the interface did not show properly?

That should be provided as well so I can judge that there's a problem with the traffic count. Something that reflects the wrong counts between two things. One's correct, one's wrong.

 

And is the name of the app you mentioned listed in the DPI list?

Does it encrypt or fake its traffic?

The picture you've shown means the overall Appflow is not correct. No correct calculation at all? Or which one of the counts is inaccurate?

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
  1  
  1  
#5
Options
Re:DPI Not Accurate/Upload Not Included
2024-10-21 17:17:32

  @Clive_A 

 

Hi Clive this is indeed correct, traffic is 99% encrypted as per SSL certs on most downloads these days hence why i can still understand why Youtube stats are far off, but how does Unifi manage to solve this? do they use a Man in the middle cert that users can install to investigate the packet flow? qBittorent should be listed let me check. 

 

 

So for these the app flow should be the same unless a standard port must be used if you can somehow guide me in the right direction or somehow i can capture packets white wiresharck to send to to RnD so we can get this on par with Unifi and other DPI solutions this would be great, reason for the large traffic amounts is due to the lab i am running i need to get this info sorted for various reasons or understand what the shortfalls are that i can explain these to our clients to push the aweness and present other focus points on why Omada is right for them and what it can and cannot do.

Not asking for a miricale here but a intermidate solution even if this is users being uable to update the DPI Database themselves as i know when DPI is enabled it pulls the latest info from a cloud source somewhwere if i am not mistaken? i have the Q2025 Roadmap for Omada and know this is a feature our users will love and hence if i can help fix it or explain why xyz is not tracked this would be most helpful sadly i have no direct contact with HQ nor RnD from my current role. As you can see Steam is being tracked 100% accurate and i am pushing the limits on this almost over 2TB so far on steam alone QUIC sure cannot track due to how the protocol works but not sure how it picks up Mobile Youtube and tracks the usuage. Also why Upload is not counted on DPI as it should count downstream and upstream traffic? If you can replicate my lab this would be good!


Currently running Omada Cloud Trail (30 Days Standard) ER7206 v2 and a SG2210MP v4.20 switch with a EAP245 all firmware up to date till the latest release.

I am currently Employed by TP-Link ZA as Technical Sales and would share my info to you via DM so we can perhaps solve this inernally and give our users more function and help improve the Omada Ecosystem, do let me know!

  0  
  0  
#6
Options
Re:DPI Not Accurate/Upload Not Included-Solution
2024-10-22 01:25:25 - last edited Thursday

Hi @ZainexZA 

Thanks for posting in our business forum.

ZainexZA wrote

  @Clive_A 

 

Hi Clive this is indeed correct, traffic is 99% encrypted as per SSL certs on most downloads these days hence why i can still understand why Youtube stats are far off, but how does Unifi manage to solve this? do they use a Man in the middle cert that users can install to investigate the packet flow? qBittorent should be listed let me check. 

 

 

 

So for these the app flow should be the same unless a standard port must be used if you can somehow guide me in the right direction or somehow i can capture packets white wiresharck to send to to RnD so we can get this on par with Unifi and other DPI solutions this would be great, reason for the large traffic amounts is due to the lab i am running i need to get this info sorted for various reasons or understand what the shortfalls are that i can explain these to our clients to push the aweness and present other focus points on why Omada is right for them and what it can and cannot do.

Not asking for a miricale here but a intermidate solution even if this is users being uable to update the DPI Database themselves as i know when DPI is enabled it pulls the latest info from a cloud source somewhwere if i am not mistaken? i have the Q2025 Roadmap for Omada and know this is a feature our users will love and hence if i can help fix it or explain why xyz is not tracked this would be most helpful sadly i have no direct contact with HQ nor RnD from my current role. As you can see Steam is being tracked 100% accurate and i am pushing the limits on this almost over 2TB so far on steam alone QUIC sure cannot track due to how the protocol works but not sure how it picks up Mobile Youtube and tracks the usuage. Also why Upload is not counted on DPI as it should count downstream and upstream traffic? If you can replicate my lab this would be good!


Currently running Omada Cloud Trail (30 Days Standard) ER7206 v2 and a SG2210MP v4.20 switch with a EAP245 all firmware up to date till the latest release.

I am currently Employed by TP-Link ZA as Technical Sales and would share my info to you via DM so we can perhaps solve this inernally and give our users more function and help improve the Omada Ecosystem, do let me know!

Since I am not working with the other vendors, I cannot say for sure that's a middleman cert to crack and log. That's a way to see the traffic flow but I think it is not safe and possible to do that as a solution sold on the market. Potential danger if the cert is modified.

As for some patterns that are correctly recognized, they should be stable and clear DST and ports recognized. Some patterns are stable, unlike the download-ware like bit download which may contain sensitive files that violate the DMCA. It could masquerade itself so the traffic might not be correctly told.

For ISPs, they simply cut off the high-bandwidth connection instead of recognizing it via its pattern time in certain regions. Or they use the database or tag to identify certain IP addresses to tell if they are bit-download servers. That requires a daily update to a database which is something that we don't have. The database is updated usually along with the Controller update.

 

Correct understanding of the DPI pattern recognition. DPI now is based on the pattern provided by vendors. It may not catch up with the latest changes in the traffic pattern if an app has changed. The traffic pattern may be changed and the database requires a follow-up update. So, it got the margin to take errors.

 

Further details regarding how DPI and what source the DPI database comes from, could be confidential. I don't have any details about it either since this goes way deep and some information is not made for the general public understanding. If you are an employee, you can contact the HQ and escalate this matter and the dev can get to you. The local company to HQ support channel is different from others.

You can send an email to support.smb@tp-link.com. Or check if there is a source with your local team leader who can relay you to the HQ via a different source. Sorry that I have not been assigned to a local team before so I don't know how your local team works.

Best Regards! If you are new to the forum, please read: Howto - A Guide to Use Forum Effectively. Read Before You Post. Look for a model? Search your model NOW Official and Beta firmware. NEW features! Subscribe for the latest update!Download Beta Here☚ ☛ ★ Configuration Guide ★ ☚ ☛ ★ Knowledge Base ★ ☚ ☛ ★ Troubleshooting ★ ☚ ● Be kind and nice. ● Stay on the topic. ● Post details. ● Search first. ● Please don't take it for granted. ● No email confidentiality should be violated. ● S/N, MAC, and your true public IP should be mosaiced.
Recommended Solution
  0  
  0  
#7
Options

Information

Helpful: 0

Views: 202

Replies: 6

Related Articles