The most fitting word to complete the sentence is A. propriety.
The sentence would read: "The mysterious guest's costume seemed to have no propriety."
This implies that the costume lacked appropriateness or suitability for the occasion, adding to its mysterious nature.