I've long be training physical and defense on my GK, and I think if your defense and physical difference don't exceed 10 points in quality, he will be fine.

I usually get the physical and defense same level for D and GK and it worked fair enough for me.

More to the subjest, D and GK can train with minimum points in attack without much issues, but upfield players cannot train with minimum defense else they will receive cards too easily.