You were quite within your rights to refuse to cooperate with him.

你完全有权拒绝与他合作。