Is not such a good idea to unfriend him, after that you won't be able to add him again.