You have cool friends lol
The only thing you can do is to delete your "friends" when you want to get rid of friendly games.
But you are right, it should get fixed in a way that you need to accept the match.