The sentence that has an infinitive or infinitive phrase functioning as a direct object is:
Their first task was to build barracks and other permanent structures.
In this sentence, "to build barracks and other permanent structures" is the infinitive phrase that serves as the direct object of the verb "was."