KDnuggets Home » News » 2015 » Mar » Opinions, Interviews, Reports » All Machine Learning Models Have Flaws ( 15:n07 )

All Machine Learning Models Have Flaws

This classic post examines what is right and wrong with different models of machine learning, including Bayesian learning, Graphical Models, Convex Loss Optimization, Statistical Learning, and more.

By John Langford (Microsoft, Hunch.net)

(Gregory Piatetsky: I recently came across another classic and valuable post by John Langford, and he kindly agreed to repost it in KDnuggets).

Attempts to abstract and study machine learning are within some given framework or mathematical model. It turns out that all of these models are significantly flawed for the purpose of studying machine learning. I've created a table (below) outlining the major flaws in some common models of machine learning.

The point here is not simply "woe unto us". There are several implications which seem important.

  1. Machine LearningThe multitude of models is a point of continuing confusion. It is common for people to learn about machine learning within one framework which often becomes there "home framework" through which they attempt to filter all machine learning. (Have you met people who can only think in terms of kernels? Only via Bayes Law? Only via PAC Learning?) Explicitly understanding the existence of these other frameworks can help resolve the confusion. This is particularly important when reviewing and particularly important for students.
  2. Algorithms which conform to multiple approaches can have substantial value. "I don't really understand it yet, because I only understand it one way". Reinterpretation alone is not the goal - we want algorithmic guidance.
  3. We need to remain constantly open to new mathematical models of machine learning. It's common to forget the flaws of the model that you are most familiar with in evaluating other models while the flaws of new models get exaggerated. The best way to avoid this is simply education.

Here is a summary what is wrong with various frameworks for learning. To avoid being entirely negative, I added a column about what's right as well.

Bayesian Learning

What is right: Handles the small data limit. Very flexible. Interpolates to engineering.

What is wrong:
  2. Computationally difficult problems are commonly encountered.
  3. Human intensive. Partly due to the difficulties above and partly because "first specify a prior" is built into framework this approach is not very automatable.

Graphical/generative Models

What is right: Relative to pure Bayesian systems, this approach is sometimes computationally tractable. More importantly, the graph language is natural, which aids prior elicitation.

What is wrong:

Convex Loss Optimization

What is right: Mathematically clean solutions where computational tractability is partly taken into account. Relatively automatable.

What is wrong:
  2. Limited models. Although switching to a convex loss means that some optimizations become convex, optimization on representations which aren't single layer linear combinations is often difficult.

Gradient Descent
Methodology: Specify an architecture with free parameters and use gradient descent with respect to data to tune the parameters.

What is right: Relatively computationally tractable due to (a) modularity of gradient descent (b) directly optimizing the quantity you want to predict.

What is wrong:
  2. Overfitting is a significant issue.

Kernel-based learning
Methodology: You chose a kernel K(x,x') between datapoints that satisfies certain conditions, and then use it as a measure of similarity when learning.

What is right: People often find the specification of a similarity function between objects a natural way to incorporate prior information for machine learning problems. Algorithms (like SVMs) for training are reasonably practical - O(n2) for instance.



What is right: A focus on getting something that works quickly is natural. This approach is relatively automated and (hence) easy to apply for beginners.

What is wrong: The boosting framework tells you nothing about how to build that initial algorithm. The weak learning assumption becomes violated at some point in the iterative process.

Online Learning with Experts

What is right: This is an effective automated method to extract performance from a pool of predictors.


Learning Reductions
Methodology: You solve complex machine learning problems by reducing them to well-studied base problems in a robust manner.

What is right: The reductions approach can yield highly automated learning algorithms.


PAC Learning

What is right: The focus on computation is pretty right-headed, because we are ultimately limited by what we can compute.

What is wrong: There are not many substantial positive results, particularly when D is noisy. Data isn’t IID in practice anyways.

Statistics Statistical Learning Theory

What is right: There are substantially more positive results than for PAC Learning, and there are a few examples of practical algorithms directly motivated by this analysis.


Decision tree learning
Methodology: Learning is a process of cutting up the input space and assigning predictions to pieces of the space.

What is right: Decision tree algorithms are well automated and can be quite fast.


Algorithmic complexity
Methodology: Learning is about finding a program which correctly predicts the outputs given the inputs.

What is right: Any reasonable problem is learnable with a number of samples related to the description length of the program.

What is wrong: The theory literally suggests solving halting problems to solve machine learning.

RL, MDP learning
Methodology: Learning is about finding and acting according to a near optimal policy in an unknown Markov Decision Process.



RL, POMDP learning
Methodology: Learning is about finding and acting according to a near optimaly policy in a Partially Observed Markov Decision Process

What is right: In a sense, we’ve made no assumptions, so algorithms have wide applicability.

What is wrong: All known algorithms scale badly with the number of hidden states.


Original: hunch.net/?p=224


Sign Up

By subscribing you accept KDnuggets Privacy Policy