To yield a square that is similar to the original square but not congruent, you would need to apply a transformation that changes its size while maintaining its shape. The only transformation from the provided options that achieves this is dilation.
So, the correct response would be:
B: reflection and dilation
The reflection would change the orientation of the square, and the dilation would change its size, producing a similar but not congruent square.