ebooksclub.org the_em_algorithm_and_extensions__wiley_series_in_probability_and_statistics_(1)

Download Ebooksclub.org  the_em_algorithm_and_extensions__wiley_series_in_probability_and_statistics_(1)

Post on 21-May-2015




2 download

Embed Size (px)


will help a lot in your research work


  • 1. The EM Algorithm and Extensions Second EditionGeoffrey J. McLachlan The University o Queensland f f Department o Mathematics and Institutefor Molecular Bioscience St. Lucia, AustraliaThriyambakam Krishnan Cranes Sofiware International Limited Bangalore. IndiaA JOHN WILEY & SONS, INC., PUBLICATION

2. This Page Intentionally Left Blank 3. The EM Algorithm and Extensions 4. WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Noel A . C. Cressie, Garrett M. Fitzmaurice, Iain M. Johnstone, Geert Molenberghs, David W. Scott, Adrian F. M. Smith, Ruey S. Tsay, Sanford Weisberg Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall, Jozef L. Teugels A complete list of the titles in this series appears at the end of this volume. 5. The EM Algorithm and Extensions Second EditionGeoffrey J. McLachlan The University o Queensland f f Department o Mathematics and Institutefor Molecular Bioscience St. Lucia, AustraliaThriyambakam Krishnan Cranes Sofiware International Limited Bangalore. IndiaA JOHN WILEY & SONS, INC., PUBLICATION 6. Copyright 02008 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 1 River Street, Hoboken, NJ 07030, (201) 748-601 1, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at www.wiley.com.Library of Congress Cataloging-in-Publieation Data: McLachlan, Geoffrey J., 1946The EM algorithm and extensions / Geoffrey J. McLachlan, Thriyambakam Krishnan. - 2nd ed. p. cm. ISBN 978-0-471-20170-0 (cloth) I . Expectation-maximization algorithms. 2. Estimation theory. 3. Missing observations (Statistics) I. Krishnan, T. (Thriyambakam), 193- 1 . Title. 1 QA276.8.M394 2007 519.5'4Uc22 2007017908 Printed in the United States of America I 0 9 8 7 6 5 4 3 2 1 7. To Beryl, Jonathan, and Robbie 8. This Page Intentionally Left Blank 9. CONTENTSPREFACE TO THE SECOND EDITIONxixPREFACE TO THE FIRST EDITIONxxiLIST OF EXAMPLESxxv1 GENERAL INTRODUCTION11.1Introduction11.2Maximum Likelihood Estimation31.3Newton-Type Methods51.3.1Introduction51.3.2Newton-Raphson Method51.3.3Quasi-Newton Methods61.3.4Modified Newton Methods61.4Introductory Examples81.4.1Introduction81.4.2Example I . 1: A Multinomial Example8 vii 10. viiiCONTENTS1.4.3Example 1.2: Estimation of Mixing Proportions13 18EM Algorithm181S . 2Example 1.3: Censored Exponentially Distributed Survival Times201.5.3E- and M-Steps for the Regular Exponential Family22IS.4Example 1.4: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)231.5.5Generalized EM Algorithm241.5.6GEM Algorithm Based on One Newton-Raphson Step241S.7EM Gradient Algorithm251.5.8 1.6Formulation of the EM Algorithm Mapping26EM Algorithm for MAP and MPL Estimation261.6.1Maximum a Posteriori Estimation261.6.2Example 1.5: A Multinomial Example (Example 1.1 Continued)27Maximum Penalized Estimation271.6.3 1.7Brief Summary of the Properties of the EM Algorithm281.8History of the EM Algorithm291.8.1Early EM History291.8.2Work Before Dempster, Laird, and Rubin (1977)291.8.3EM Examples and Applications Since Dempster, Laird, and Rubin (1977)311.8.4Two Interpretations of EM321.8.5Developments in EM Theory, Methodology, and Applications 331.9Overview of the Book1.10 Notations 2 EXAMPLES OF THE EM ALGORITHM36 37 412.1Introduction412.2Multivariate Data with Missing Values422.2.1422.2.2Numerical Illustration452.2.3 2.3Example 2.1 : Bivariate Normal Data with Missing Values Multivariate Data: Bucks Method45Least Squares with Missing Data472.3.147Healy-Westmacott Procedure 11. CONTENTS2.3.2ixExample 2.2: Linear Regression with Missing Dependent Values472.3.3Example 2.3: Missing Values in a Latin Square Design492.3.4Healy-Westmacott Procedure as an EM Algorithm492.4Example 2.4: Multinomial with Complex Cell Structure512.5Example 2.5: Analysis of PET and SPECT Data542.6Example 2.6: Multivariate &Distribution (Known D.F.)582.6.1ML Estimation of Multivariate t-Distribution582.6.2Numerical Example: Stack Loss Data61Finite Normal Mixtures612.7.1Example 2.7: Univariate Component Densities612.7.2Example 2.8: Multivariate Component Densities642.7.32.7Numerical Example: Red Blood Cell Volume Data65 66Introduction662.8.2Specification of Complete Data662.8.3E-Step692.8.4M-Step702.8.5Confirmation of Incomplete-Data Score Statistic702.8.6M-Step for Grouped Normal Data712.8.7 2.9Example 2.9: Grouped and Truncated Data2.8.12.8Numerical Example: Grouped Log Normal Data72Example 2.10: A Hidden Markov AR( 1) model3 BASIC THEORY OF THE EM ALGORITHM73 773.1Introduction773.2Monotonicity of the EM Algorithm783.3Monotonicity of a Generalized EM Algorithm793.4Convergence of an EM Sequence to a Stationary Value793.4.1Introduction793.4.2Regularity Conditions of Wu (1983)803.4.3Main Convergence Theorem for a Generalized EM Sequence813.4.4A Convergence Theorem for an EM Sequence823.5Convergence of an EM Sequence of Iterates833.5.1Introduction833.5.2Two Convergence Theorems of Wu (1983)833.5.3Convergence of an EM Sequence to a Unique Maximum Likelihood Estimate84 12. XCONTENTS3.5.4 3.6Constrained Parameter Spaces84Examples of Nontypical Behavior of an EM (GEM) Sequence853.6.1Example 3.1 : Convergence to a Saddle Point853.6.2Example 3.2: Convergence to a Local Minimum883.6.3Example 3.3: Nonconvergence of a Generalized EM Sequence90Example 3.4: Some E-Step Pathologies933.6.4 3.7Score Statistic953.8Missing Information953.8.1Missing Information Principle953.8.2Example 3.5: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)96Rate of Convergence of the EM Algorithm993.9.1Rate Matrix for Linear Convergence993.9.2Measuring the Linear Rate of Convergence1003.9.3Rate Matrix in Terms of Information Matrices1013.9.4Rate Matrix for Maximum a Posteriori Estimation1023.9.5Derivation of Rate Matrix in Terms of Information Matrices1023.9.63.9Example 3.6: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)1034 STANDARD ERRORS AND SPEEDING UP CONVERGENCE1054.1Introduction1054.2Observed Information Matrix1064.2. IDirect Evaluation1064.2.2Extraction of Observed Information Matrix in Terms of the Complete-Data Log Likelihood1064.2.3Regular Case1084.2.4Evaluation of the Conditional Expected Complete-Data Information Matrix108Examples1094.2.5 4.3Approximations to Observed Information Matrix: i.i.d. Case1144.4Observed Information Matrix for Grouped Data1164.4.11164.4.2 4.5Approximation Based on Empirical Information Example 4.3: Grouped Data from an Exponential Distribution117Supplemented EM Algorithm120 13. CONTENTSxi4.5.1Definition1204.5.2Calculation of J ( k) via Numerical Differentiation1224.5.3Stability1234.5.4Monitoring Convergence1244.5.5Difficulties of the SEM Algorithm1244.5.6Example 4.4: Univariate Contaminated Normal Data1254.5.7Example 4.5: Bivariate Normal Data with Missing Values1284.6Bootstrap Approach to Standard Error Approximation1304.7Bakers, Louis, and Oakes Methods for Standard Error Computation1314.7.1Bakers Method for Standard Error Computation1314.7.2Louis Method of Standard Error Computation1324.7.3Oakes Formula for Standard Error Computation1334.7.4Example 4.6: Oakes Standard Error for Example 1.11344.7.5Example 4.7: Louis Method for Example 2.41344.7.6Bakers Method for Standard Error for Categorical Data1354.7.7Example 4.8: Bakers Method for Example 2.4136Acceleration of the EM Algorithm via Aitkens Method1374.8.1Aitkens Acceleration Method1374.8.2Louis Method1374.8.3Example 4.9: Multinomial Data1384.8.4Example 4.10: Geometric Mixture1394.8.5Example 4.1 1: Grouped and Truncated Data. (Example 2.8 Continued)1424.84.9An Aitken Acceleration-Based Stopping Criterion4.10 Conjugate Gradient Acceleration of EM Algorithm142 1444.10.1Conjugate Gradient Method1444.10.2A Generalized Conjugate Gradient Algorithm1444.10.3Accelerating the EM Algorithm1454.1 1 Hybrid Methods for Finding the MLE 4.1 1.1Introduction4.1 1.2 Combined EM and Modified Newton-Raphson Al