Sometimes it depends if you are thrashing opponents 10-0, then your defense has nothing to do and can get a poor rating, just like a gk with nothing to do. But if you are letting goals in, then put him on the bench for a while and see if he improves after that or just sell.