There is an option. Next to his name there is a small minus friend sign ,,click on that.
Also he cannot just add you ,you had to have accepted his friend request .