The hardest part of building a prototype can vary depending on the specific project and the individuals involved. However, many often find that "time commitment" is a significant challenge. Prototyping can be a time-intensive process that requires careful planning, iterations, and adjustments, which can be difficult to manage alongside other project responsibilities.
If you need me to choose one from the options you've provided, "time commitment" is a commonly perceived difficulty.