To let go of judgment does not mean that you don't see what they do. It means that you recognize their behavior as a form of conditioning, and you see it and accept it as that. You don't construct an identity out of it for that person.
That liberates you as well as the other person from identification with conditioning, with form, with mind. That liberates you as well as the other person from identification with conditioning, with form, with mind. The ego then no longer runs your relationships.