You can't remove a friend from game if you have him on facebook. The only way is to remove him from facebook friends