From what i gathered and and understand what you said in the other house is where your isp router is.

If your now tp link router (I'm guessing in a out house or adjacent building) is connected to your main isp router amd has the dhcp turned on as well as your tp link router you could be getting double nat issues,

If what im saying is correct i would put your tp link router into AP mode amd put the tp link ssid amd password the same as the main router ssid amd password so atleast when you go between buildings you can join either router but atleast turn the tp link router dhcp off as thay could be causing a conflict on the network which is why your xbox is disconecting. Im not too clued up on double nat but im sure someone else more clued up will correct me or atleast have another idea why this is happening and offer more advice 👍🏻👍🏻