I could tell that my refusal to allow him to ruffle me infuriated him.

我可以断定，我拒绝让他激怒我使他很恼火。