Your in-game friends are already on the same server - thats why you could see them and add them in the first place.

Friends added by facebook can be on any server but you will only see the matches of fb friends on the same server.

IF your friend had added you as fb friend before you started the game and sent you a gift/invite you would be on the same server and wouldn't have an issue.

Hindsight is a wonderful thing and highlights we don't know what we need to know until its too late