M-estimates for location and scale

Huber (1964) introduced the class of M-estimates. Suppose that $x_1, \ldots, x_n$ are independent and identically distributed with density $f \left( x, \theta \right)$. The M-estimate of $\theta$ is

$$T_n = \arg \min_\theta \sum_{i=1}^n\rho \left( x_i, \theta \right),$$

where $\rho$ is a user-chosen function. When $\rho \left( x, \theta \right) = - \log f \left( x, \theta \right)$, T_n is the maximum likelihood estimate of $\theta$. We can also define T_n implicitly by the equation

$$\sum_{i=1}^n\psi \left( x_i, T_n \right) = 0$$

where $\psi = \partial \rho / \partial \theta$. Assume the following location model. Let $x_1, \ldots, x_n$ be a random sample satisfying

$$x_i = \mu + \epsilon_i \hspace{.2in} i=1, \ldots, n$$ (2)

where $\epsilon_i$ are independent errors with common distribution F₀ and finite variance $\sigma^2$. The parameter of interest is $\mu$. The cumulative distribution function F of the x_i's satisfies $F \left( x \right) = F_0 \left( \left( x - \mu \right) / \sigma \right)$. In this case it is natural to take $\psi \left( \cdot, \cdot \right)$ to be a function of a single argument, namely the difference $x_i - \theta$. The defining equation becomes

$$\sum_{i=1}^n\psi \left( x_i - T_n \right) = 0.$$ (3)

One important drawback of this definition is that in general, if b denotes a real number,

$$T_n \left( b \ x_1, \ldots, b \ x_n \right) \ne b \ T_n \left( x_1, \ldots, x_n \right).$$

That is, the resulting estimate might not be scale equivariant. To overcome this problem we couple our equation with a scale estimate $S_n = S_n \left( x_1, \ldots, x_n \right)$ that satisfies

$$S_n \left( a x_1, \ldots, a x_n \right) = a S_n \left( x_1, \ldots, x_n \right)$$

and instead of using equation (3) we solve

$$\frac1n \sum_{i=1}^n\psi \left( \left( x_i - T_n \right) / S_n \right) = 0.$$ (4)

Possible choices for S_n are the M-estimates of scale

$$\frac1n \sum_{i=1}^n\chi \left( \left( x_i - T_n \right) / S_n \right) = b$$ (5)

where b is a constant and $\chi$ is again a user-chosen function. The procedure of simultaneously solving equations (4) and (5) is known in the literature as Huber's Proposal 2 (see Huber, 1964). However, on top of the numerical inconvenience of solving this system of two non-linear equations, the robustness properties of the resulting estimates are not satisfactory. Another choice for the scale estimator is to solve the following equation independently of (4)

$$\frac1n \sum_{i=1}^n\chi \left( \frac{x_i - \tilde{T_n}}{S_n} \right) = b$$ (6)

where $\tilde{T_n}$ is an auxiliary location estimate that is generally different from T_n in (4). For example, we can use $\tilde{T_n} = \mbox{median} \left( x_i \right)$. Under very mild conditions the estimates T_n and S_n are strongly consistent, i.e., $T_n \rightarrow \mu$ and $S_n \rightarrow \sigma$ almost surely. It is well known (see for example Huber, 1981) that if $\psi$ is odd, $\chi$ is even and the underlying distribution is symmetric, S_n and T_n are asymptotically independent and T_n satisfies

$$\sqrt{n} \left( T_n - \mu \right) \longrightarrow N \left( 0, \ \mbox{AV} \left( F, T \right) \right),$$

with asymptotic variance given by

$$\mbox{AV} \left( F, T \right) = \sigma^2 \ \frac{ E_F \ps... ... \left( \left( X - \mu \right) / \sigma \right) \right)^2 }.$$ (7)