The set S has the property that if a is in S, then a squared + a is also in S. What set could be S? The answer is {1,0}. How do you get this answer?

Question

What numbers squared are itself?

one and zero

But what exactly is a set and what is a?

I'm not sure, because the way you've stated the problem S should have more elements than just 1 and 0. You said
"...if a is in S, then a squared + a is also in S."
If I understand this correctly, then a is in S => a² + a = a(a+1) is in S.
However, there are no restrictions placed on a, so it could be integers, rationals or real numbers.
You wrote, "The answer is {1,0}," but I don't see any rational for why those are the only possible numbers that can be in S. What I conclude is that if 0 is in S then 1 is in S, but if 1 is in S then 1*2=2 is in S, which means 2*3 is in S... Which means S is the triangular numbers, but this is only true if 0 is in S to begin with. Is there something you're omitting from the original question?