It's all in the rules. A trophy only shows once it's been won, so if you play someone for the first time and win then you gain the trophy, their count will not go down.
If they play you back though and they win then your count will go down 1 and theirs will increase 1