I think that the person already would have a problem themselves with it. He may already have candida himself, which may flare up with contact with you.

But that is not your fault, both partners usually have to address their diet when that occurs.