Hey!
In machine learning it’s common to think “simpler is better”
- Simpler models are easier to explain and understand. You can easily put a trend-line from Linear Regression in a Power Point slide, but it’s hard to show people what the thousands of parameters in a Neural Network mean.
- Simpler models are less likely to overfit your data. A complex model is capable of fitting data with complex shapes, so it can stretch itself to fit outlier data points that you shouldn’t include.
- Simpler models are faster and cost less money. Who wants slow code that costs a lot of money??
Ideally you want your ML solution to be as simple as possible, while still performing the task well. Therefore, when faced with an option between a simpler solution and a complex solution, it’s common to start by choosing the simpler option. You can make it complex later if you need to.
In psychology, a mental model that is used to quickly make decisions is called a razor. They allow you to cut through a problem and choose a solution.
Occam’s Razor is a mental model that conveys this idea of “simpler is better”. It states that when you are choosing between competing hypothesis or explanations, the simplest one is likely to be the best. Unnecessary complexity should be avoided.
Caveat: “Best” does not mean the most correct or accurate. “Best” means the solution is probably efficient, practical, and gets the job done good enough. If you need a very accurate solution, then you might need to add more complexity to your approach.
In machine learning and in life, Occam’s Razor is a tool we can use to quickly make decisions that will likely lead us to the best answers.
I hope you found this month’s newsletter thought provoking!
Michael Hammer
Read all my Newsletters here: https://michaelphammer.com/newsletter/