Japan feared it would become a colony primarily because it had seen this occur in South and Southeast Asia. During the late 19th and early 20th centuries, many countries in these regions were subject to colonization by Western powers, leading Japan to be acutely aware of the potential threat posed by imperialism. The experience of neighboring countries that lost their sovereignty to colonial powers served as a cautionary example for Japan, motivating the nation to modernize and strengthen itself to avoid a similar fate.
The other options you mentioned, like the Iwakura Mission or events during the Meiji Restoration, played roles in Japan's modernization efforts, but the overarching fear of colonization was heavily influenced by the historical context of Western imperialism in nearby regions.