It's annoying for sure, but when you add a 'friend', make sure you can make contact on Fb or whatever.
You can stop a friendly match by pressing the red 'X' on your todays matches, but sometimes you only got minutes....that's most of the time not enough reaction time.

And like herdo and Gafaton just said: removing those unwanted 'friends'....