i don't think it happens every match for you. i just think that since it's a negative you just pull it out of your memory and forget all the other single yellow games.
and if it happens so much, get players that don't foul... start playing on easy if it's such a problem.