Question

Let Θˆ be an estimator of a random variable Θ, and let Θ˜=Θˆ−Θ be the estimation error.

a) In this part of the problem, let Θˆ be specifically the LMS estimator of Θ. We have seen that for the case of the LMS estimator, E[Θ˜∣X=x]=0 for every x. Is it also true that E[Θ˜∣Θ=θ]=0 for all θ? Equivalently, is it true that E[Θˆ∣Θ=θ]=θ for all θ?

b) In this part of the problem, Θˆ is no longer necessarily the LMS estimator of Θ. Is the property Var(Θ)=Var(Θˆ)+Var(Θ˜) true for every estimator Θˆ?

Answer 1

a) NO
B) NO

Answer 2

a) No, it is not necessarily true that E[Θ˜∣Θ=θ]=0 for all θ. While the LMS estimator has the property that E[Θ˜∣X=x]=0 for every x, this does not imply that E[Θ˜∣Θ=θ]=0 for all θ. It is possible for the estimation error to have a non-zero mean even when conditioned on the true parameter value.

b) Yes, the property Var(Θ)=Var(Θˆ)+Var(Θ˜) is true for every estimator Θˆ. This can be shown by using the definition of variance and properties of conditional expectations:

Var(Θ) = E[(Θ - E[Θ])^2]
= E[(Θ - E[Θˆ + Θ˜])^2]
= E[(Θ - E[Θˆ] - E[Θ˜])^2]
= E[(Θ - Θ)^2 + (Θ - E[Θˆ])^2 + (Θ - E[Θ˜])^2 + 2(Θ - E[Θˆ])(Θ - E[Θ˜])]
= Var(Θˆ) + Var(Θ˜) + 2E[(Θ - E[Θˆ])(Θ - E[Θ˜])]
= Var(Θˆ) + Var(Θ˜)

So, the variance of the estimator Θ is equal to the sum of the variances of its estimation error and the true parameter.