Yes, the thing with friends is if you want them to be in your server, they have to start the game for the first time from your invitation. If they already are playing they already are on another server (usually.)