@Rizuan Hello,
If RE300 failed to detect router's 5Ghz : It is recommended to change 5Ghz channel to 36/40/44/48 and follow the thread below to troubleshoot:
[Troubleshooting] TP-Link Range Extender is not detecting the 5GHz Signal of the Main Router
https://community.tp-link.com/en/home/forum/topic/265818
If RE300 detected router's 5Ghz wifi but failed to connect to 5Ghz wif: please verify 5Ghz Wi-Fi password of main router and set up RE again:
Go to Settings-> WiFi on your phone, Tap (or Long press the router's WiFi name ->Forget / Delete the network
Please reconnect to your router Wi-Fi, and it will ask for a password, will it accept your 5Ghz router password?