The detail from the excerpt that infers that the narrator and his valet are trespassing at the chateau is:
The chateau appears to be abandoned.
This suggests that they are entering a place that is not currently inhabited or intended for visitors, which supports the idea of trespassing.