The group that stood against Andrew Jackson on the removal of Native Americans was:
the Supreme Court
The Supreme Court ruled in cases such as Worcester v. Georgia (1832) that Native American nations were distinct communities and that the state of Georgia had no authority to impose its laws on them. However, Jackson famously ignored the Court's decision, leading to the forced removal of Native Americans known as the Trail of Tears.