What Is the Generalized Inverse of a Matrix? Je Gill, jgill@ucdavis.

edu
[Edited out of Gill and King: What to do When Your Hessian is Not Invertible: Alternatives to Model Respecication in Nonlinear Estimation.]

The Generalized Inverse

This research note briey reviews the mathematical and statistical considerations behind the generalized inverse. The literature on the theory and application of the generalized inverse is vast and spans several elds. Here we summarize some of the fundamental principles. See Harville (1997) for further details.

1.1

The Many Generalized Inverse Matrices

Any matrix, A, can be decomposed as A = L D U where, D= Drr 0 , 0 0 (1.1)

(pq )

(pp)(pq )(q q )

and both L (lower triangular) and U (upper triangular) are non-singular (even given a singular A). The diagonal matrix Drr has dimension and rank r corresponding to the rank of A. When A is non-negative denite and symmetric, then the diagonals of Drr are the eigenvalues of A. If A is non-singular, positive denite, and symmetric, as in the case of a proper invertible Hessian, then Drr = D (i.e. r = q ) and A = LDL . The matrices L, D, and U are all non-unique unless A is nonsingular. By rearranging 1.1 we can diagonalize any matrix as D = L1 AU1 = Drr 0 . 0 0 (1.2)

Now dene a new matrix, D created by taking the inverses of the non-zero (diagonal) elements of D: D r r 0 . D = (1.3) 0 0 If DD = Iqq then we could say that D is the inverse of D. However, this is not true: DD = Instead, we notice that: DD D = 1 0 0 0 Drr 0 Drr 0 = D. = 0 0 0 0 Drr 0 0 0 D r r 0 = 1 0 0 0 0 0

So D is a generalized inverse of D because of the extra structure required. Note that this is a generalized inverse not the generalized inverse since the matrices on the right side of (1.1) are non-unique. By rearranging (1.1) and using (1.3) we can dene a new q p matrix: G = U1 D L1 . The importance of the generalized inverse matrix G is revealed in the following theorem.1
1 The generalized inverse is also sometimes referred to as the conditional inverse, pseudo inverse, and g-inverse.

Theorem 1.1 (Moore 1920) G is a generalized inverse of A since AGA = A The new matrix G necessarily has rank r since the product rule states that the result has rank less than or equal to the minimum of the rank of the factors, and AGA = A requires that A must have rank less than or equal to the lowest rank of itself or G. Although G has innitely many denitions that satisfy Theorem 1.1, any one of them will do for our purposes. For example, in linear regression, the tted values dened as X GX y , with G as the generalized inverse of X X , X as a matrix of explanatory variables and y as the outcome variable are invariant to the denition of G. In addition, we only use our pseudo-variance as a rst approximation to the surface of the true posterior, and we will improve it in our importance resampling stage.2

1.2

The Unique Moore-Penrose Generalized Inverse Matrix

Moore (1920) and (unaware of Moores work) Penrose (1955) reduced the innity of generalized inverses to one unique solution by imposing four reasonable algebraic constraints, all met by the standard inverse. If 1. general condition: AGA = A, 2. reexive condition: GAG = G, 3. normalized condition: (AG) = GA, and 4. reverse normalized condition: (GA) = AG then this G matrix is unique. The proof is lengthy, and we refer the interested reader to Penrose (1955). There is a vast literature on generalized inverses that meet some subset of the Moore-Penrose condition. A matrix that satises the rst two conditions is called a reexive or weak generalized inverse and is order dependent. A matrix that satises the rst three conditions is called a normalized generalized inverse. A matrix that satises the rst and fourth conditions is called a minimum norm generalized inverse. Because the properties of the Moore-Penrose generalized inverse are intuitively desirable, and because of the invariance of important statistical results to the choice of generalized inverse, we follow standard statistical practice by using this form from now on. The implementations of the generalized inverse in Gauss and Splus are both the Moore-Penrose version. The Moore-Penrose generalized inverse is also easy to calculate using QR factorization. QR factorization takes the input matrix, A, and factors it into the product of an orthogonal matrix, Q, and a matrix, R, which has a triangular leading square matrix (r) followed by rows of zeros corresponding to the dierence in rank and dimension in A: A= r . 0

This factorization is implemented in virtually every professional level statistical package. The Moore-Penrose generalized inverse is produced by: G = r1 0 Q where 0 is the transpose of the zeros portion of the R matrix required for conformability.
2 Note in addition that AG is always idempotent (GAGA = G(AGA) = GA), and rank(AG) = rank(A). These results hold whether A is singular or not.

1.3

Numerical Examples

As a means of motivating a simple numerical example of how the generalized inverse works, we develop a brief application to the linear model where the X X matrix is noninvertible because X is singular. In this context, the generalized inverse provides a solution to the normal equations (Campbell and Meyer 1979, p.94), and both the tted values of Y and the residual error variance are invariant to the choice of G (Searle 1971, p.169-71). We use the Moore Penrose generalized inverse. Let 5 2 5 9 2 11 1 2 X= Y= 3 5 2 3 2.95 1 3 2 (Our omission of the constant term makes the numerical calculations cleaner but is not material to our points.) Applying the least squares model to these data (X is of full rank) yields the coecient vector = (X X)1 X Y = (222.22, 11.89, 215.22) , b tted values, and variance matrix 57283.95 1580.25 56395.06 = 1580.25 187.65 1491.36 . 56395.06 1491.36 55550.62 What we call the standardized correlation matrix, a correlation with standard deviations on the diagonal, is then 239.34 0.48 0.99 0.46 . Cs = 0.48 13.69 0.99 0.46 235.69 Now suppose we have a matrix of explanatory aects that is identical to X except that we have changed the bottom left number from 2.95 to 2.99 5 2 5 2 1 2 X2 = 3 2 3 2.99 1 3 Using the same Y outcome vector and applying the same least squares calculation now gives 2 = (1111.11, 11.89, 1104.11) , b and = (11.22, 2.11, 2.78, 2.00) . Y = Xb = (11.22, 2.11, 2.78, 2.00) . Y

However, the variance-covariance matrix reacts sharply to the movement towards singularity as seen in the standardized correlation matrix: 1758.79 0.48 0.99 0.48 Cs = 0.48 20.12 0.99 0.48 1753.35 3

Indeed, if X3 = 2.999, then X X is singular (with regard to precision in Gauss and Splus) and we must use the generalized inverse. This produces 3 = GX Y = (1.774866, 5.762093, 1.778596) , b and = XGX Y = (6.2431253, 1.3448314, 0.8637996, 4.8965190) . Y

The resulting pseudo-variance matrix (calculated now from G 2 ) produces larger standard deviations for the rst and third explanatory variable, reecting greater uncertainty, again displayed as a standardized correlation matrix: 16328.7257311 0.4822391 0.9999999 18.6815417 0.4818444 Cs = 0.4822391 0.9999999 0.4818444 16323.6599450

References
Campbell, S. L. and C. D. Meyer, Jr. 1979. Generalized Inverses of Linear Transformations. New York: Dover Publications. Gill, Phillip E., G. H. Golub, Walter Murray, and M. A. Sanders. 1974. Methods for Modifying Matrix Factorizations. Mathematics of Computation 28, 505-35. Gill, Phillip E. and Walter Murray. 1974. Newton-Type Methods for Unconstrained and Linearly Constrained Optimization. Mathematical Programming 7, 311-50. Gill, Phillip E., Walter Murray, and M. H. Wright. 1981. Practical Optimization. London: Academic Press. Harville, David A. Matrix Algebra From a Statisticians Perspective. New York: Springer. Matthews, A. and D. Davies. 1971. A Comparison of Modied Newton Methods for Unconstrained Optimization. Computer Journal 14, 213-94. Moore, E. H. 1920. On the Reciprocal of the General Algebraic Matrix. (Abstract) Bulletin of the American Mathematical Society 26, 394-5. Penrose, R. A. 1955. A Generalized Inverse for Matrices. Proceedings of the Cambridge Philosophical Society 51, 406-13. Searle, S. R. 1971. Linear Models. New York: Wiley & Sons.