The author means that popular sovereignty "undermined the Missouri Compromise" by indicating that it reopened debate over the border between free and slave states. While the Missouri Compromise had established a clear division between free and slave territories along the thirty-sixth parallel, the principle of popular sovereignty allowed new territories to decide for themselves whether to permit slavery, thereby challenging the previously agreed-upon boundaries and leading to further conflict.
Therefore, the correct response is: It reopened debate over the border between free and slave states.