Don't worry about it because anyway your account is secured thru Facebook or Google account and if you chose the "Play Now" option and your account is not assigned to anything then in fact is even more secure as long it can be used only from that device.

With you or without you as friend nothing changes, he have the same visibility regarding your details (Your name and your avatar picture).

Basically with you on friends or without is just the same thing. Cheers!