The main reason Congress initially refused to annex Texas in the 1830s and 1840s was largely tied to the issue of slavery and the balance of power between slave and free states. Texas was a slaveholding territory, and its annexation would add a significant pro-slavery state to the Union, potentially upsetting the delicate balance between slave and free states in Congress.
Additionally, there were concerns about the implications of annexing Texas on U.S.-Mexico relations, as Mexico still considered Texas to be part of its territory following the Texas Revolution. There were also fears of the potential for increased conflict with Mexico, which could lead to war.
Political opposition also played a role, with some factions, particularly the Whig Party, wary of the expansionist policies of President James K. Polk and his Democratic Party, which favored annexation. Ultimately, it wasn't until the 1845 election and explicitly pro-annexation sentiment, combined with changing political dynamics, that Texas was successfully annexed.