Why are some statistics problems essentially

optimization problems?15th Jan 2017

Chien-Yi Tsai

IntroductionOperations Research is a difficult, yet incredibly useful discipline. Today’s topic will explore the similarity between statistics and optimization problems.

What is Operations Research? (OR)• OR is the discipline of applying advanced analytical methods to help make better

decisions. In practice, OR aims to derive optimal or near-optimal solutions to complex problems.

About Me• I am a first year Master’s student of Industrial and Operations Engineering at the

University of Michigan

What is Optimization?Optimization problems search for the best solution out of all feasible solutions when given a set of constraints.

What is the “best” solution? • If the goal is to maximize profit or minimize the cost of a system, the best solution

will do this while respecting constraints such as meeting demand and not exceeding resource capacity.• This is also known as the optimal solution.

How is this related to Statistics?• The Maximum Likelihood Estimator (MLE) is an important method in Statistics

for defining parameters, given existing data, that maximize the probability of those observations occurring.

• For example, you may want to know the average weight for a class of 40 students, but only have records for half the class. Taking the average of these 20 observations might seem like the “most likely” average for the class.

• MLE can actually estimate values for the mean and variance or other parameters that in turn maximize the likelihood that our 20 original observations occur again randomly.

“Most likely”?• Statistically speaking, the term “most likely” is used to indicate that

something has the highest chance of occurring or being observed.

• Likewise, it means that something has the “highest density” in the probability density function.

The Comparison• An optimization problem, used in Operations Research, might seek a

solution for obtaining the MOST profit.

• A statistics problem using the MAXIMUM likelihood estimator wants to find the MOST likely and HIGHEST density parameters.

• Both of these fields are searching for the “greediest” answer!

In other words, through the lens of Optimization• A statistics problem that utilizes the MLE method is essentially the

same as the following optimization model:

• Simply put, MLE is a problem that tries to find an estimator x that maximizes the (joint) probability density function l(x) with respect to some constraints C.

Does that not look beautiful?Two seemly unrelated disciplines, Optimization and Statistics, are actually related!