When connecting to a virtual server via modem over the internet, it needs to check which IP address I'm connecting from. If some IPs on the blacklist are connecting to NAT, it should DROP connection request or if connection request comes from a whitelist it can ALLOW that connection. 

Since the modem lacks these functions, there is no way to prevent unauthorized IP addresses from attempting to access the virtual server.