how do i figure this out? if all incomes were equalized by government taxes and transfer payments, how much would the average household in each income quintile gaine (via transfers) or lose (via taxes)?

Hummm, for real or as a hypothetical exercise. The math would be the same for both. However, for real you would need to find and income distribution of the US. And trust me, that could be a real challenge.

That said hypothetically -- first compute the overall average income (Y). Next, compute the average income in each quintile class. (y1, y2, ... y5). Thus, average taxes in each quintile are positive amounts of yi-Y. Transfers are positive amounts of Y-yi.