You don't have to remove them from Facebook, you can just remove them from your friends in game