It depends on how you are friend's with them. If its through fb, then its not that simple (unless as Vlad said, you unfriend them on fb).

However, if you've friended them just through the game, try clicking on their picture and the screen opens up with their team's info. In the box with their picture, you should see a very small red box. If you click this, it should unfriend them from your list.

I have just over 100 "friends" in here, but less than 12 of them are fb friends.