The zigzag shape of rock layers in mountains is primarily caused by folding of rock. Folding occurs when tectonic forces compress and deform rock layers, leading to the formation of folds that can create zigzag patterns. Therefore, the correct response is:
folding of rock