3.3 Parameter estimation
Very often we know the waveform of the signal that we are searching for in the data in terms of a finite
number of unknown parameters. We would like to find optimal procedures of estimating these parameters.
An estimator of a parameter
is a function
that assigns to each data set the “best” guess of the
true value of
. Note that because
depends on the random data it is a random variable. Ideally
we would like our estimator to be (i) unbiased, i.e., its expectation value to be equal to the
true value of the parameter, and (ii) of minimum variance. Such estimators are rare and in
general difficult to find. As in the signal detection there are several approaches to the parameter
estimation problem. The subject is exposed in detail in [81, 82]. See also [149] for a concise
account.
3.3.1 Bayesian estimation
We assign a cost function
of estimating the true value of
as
. We then associate with an
estimator
a conditional risk or cost averaged over all realizations of data
for each value of the
parameter
:
where
is the set of observations and
is the joint probability distribution of data
and
parameter
. We further assume that there is a certain a priori probability distribution
of the
parameter
. We then define the Bayes estimator as the estimator that minimizes the average risk defined
as
where E is the expectation value with respect to an a priori distribution
, and
is the set of
observations of the parameter
. It is not difficult to show that for a commonly used cost function
the Bayesian estimator is the conditional mean of the parameter
given data
, i.e.,
where
is the conditional probability density of parameter
given the data
.
3.3.2 Maximum a posteriori probability estimation
Suppose that in a given estimation problem we are not able to assign a particular cost function
.
Then a natural choice is a uniform cost function equal to 0 over a certain interval
of the parameter
.
From Bayes theorem [28] we have
where
is the probability distribution of data
. Then, from Eq. (50) one can deduce that for each
data point
the Bayes estimate is any value of
that maximizes the conditional probability
.
The density
is also called the a posteriori probability density of parameter
and the
estimator that maximizes
is called the maximum a posteriori (MAP) estimator. It is
denoted by
. We find that the MAP estimators are solutions of the following equation
which is called the MAP equation.
3.3.3 Maximum likelihood estimation
Often we do not know the a priori probability density of a given parameter and we simply assign to it a
uniform probability. In such a case maximization of the a posteriori probability is equivalent to
maximization of the probability density
treated as a function of
. We call the function
the likelihood function and the value of the parameter
that maximizes
the maximum likelihood (ML) estimator. Instead of the function
we can use the function
(assuming that
).
is then equivalent to the likelihood ratio [see
Eq. (37)] when the parameters of the signal are known. Then the ML estimators are obtained by solving
the equation
which is called the ML equation.