how was chillingworth injuring dimmesdale without inflicting any physical injury?

Was it mental torture? Did Dimmesdale suffer (mental) agony?

NO