Great question! It was never an issue for me. The address was necessary because, of course, food had to be delivered.

With that said, I suppose there's probably an issue related to the types of emails being sent and the purpose. Mine were always personal emails (i.e. not mass marketing emails), and the emails always had a touch of personalization as well in order to build trust. I'd often also connect with the person on LinkedIn so they could see more info about me.

As you note, trust building is the most difficult part, and this is where high levels of personalization really help.