Your doing nothing wrong, players seem to lose morale quite randomly.