Probably your opponents are also his friends, because his friends are also likely grouped to your league.

I usually avoid attending matches between friends except one of the friend's team is abandoned, or one of the team is actually my alternate team.

I used to unfriend some friends in similar situation. But sometimes you may better be patient and take it easy.