When you rub the balloon on the sweater, you gather more blue negative charges (electrons) from the sweater onto the balloon. When you bring the charged balloon close to the wall, you'll notice that the wall gets attracted to the balloon. This happens because the negative charges in the balloon push away some of the negative charges in the wall, causing the positive charges in the wall to be left behind.
So, the positive charges in the wall move closer to the balloon because they are attracted to the negative charges on the balloon. This shows that while the electrons (negative charges) are moving from the sweater to the balloon, the positive charges in the wall are moving toward the balloon, but not physically moving away like the negative charges in the wall.
In summary, negative charges move from the sweater to the balloon, while the positive charges in the wall are attracted and "move" closer to the balloon without actually leaving their spot in the wall. This is why we see an attraction between the balloon and the wall.