No, you do not lose them, you can build up the number of reds and greens for as long as you like. You only lose blues if you lose a game that you have not attended.