It seems the only way to make sure you are on the same server is if you accept invitation to play from them.