His supporters fell away as his popularity declined.

随着他的名望下降，他的支持者渐渐离他而去。