Have you secured 5GHz network on VR900v? It seems it's unsecured and you may try to secure it firstly. If you disable 2.4GHz network on VR900v, does RE580 connect to 5GHz?

How far between your brother's home and yours? The ANT2409A is a directional antenna, and you may not get good signal on your home wireless devices. It's better get another outdoor devices in your...