The XYZ Affair created significant problems for President John Adams primarily because of the backlash and divisions it caused within his own party and among the public. While some points addressed your question are partially accurate, the most relevant issue is:
Members of his own party were upset with Adams for not asking Congress for war.
During the XYZ Affair, there was substantial pressure from the Federalist Party and some factions within the political landscape for Adams to take a stronger military stance against France after the diplomatic incident where French officials demanded bribes from American diplomats. Although there was a strong inclination towards war among many Federalists, Adams opted for diplomacy and avoided declaring war. This decision angered some Federalists, who saw it as a failure to stand up against French aggression.
Additionally, the situation weakened Adams' position within the Federalist Party as he faced opposition from those who wanted a more aggressive military response. His approach to the crisis created rifts within the party and affected his popularity among voters, undermining his administration's stability during a period of heightened tensions between the United States and France. Thus, Adams’ choice to seek a peaceful resolution ultimately led to challenges in maintaining party unity and support.