I have 20 friends and no one challenge me to friendly games without communicating first.
Find friends that respect you or get a better team that wins all friendly - than you won't have to worry about people challenging your team.