If it's not the close proximity that irritates you, it's the sheer pushiness of your fellow diners.

惹恼你的不是因为距离接近，而是同来的食客的催促。