In the previous posts Models splinter when you look at them closely – Part 1 and Part 2, I introduced Emmanuel Derman’s book Models. Behaving. Badly. and discussed a few different types and features of models.
In this post I’ll conclude my reflections on Derman’s book with some benefits and risks of models.
Models help us in at least two ways, Derman says, that to me seem somewhat contradictory.
The first is that the process of developing a model builds intuition in the team. They examine real-world data, capture the relationships, and test the model under various scenarios to understand, not how the system definitely will behave, but the ways that it may behave under different conditions. In many situations the possible behaviour is obvious in hindsight: we aren’t able to remember how blind we were to the possibilities.
The second is to transform this intuition into something “formulaic so that anyone can benefit from the insight”. Often games are created where players attempt to achieve a certain outcome by making choices over a period of time. The contradictory part is that by making it formulaic or packaged, the chance to develop intuition is reduced. Of course I’m not proposing that everyone build an entire model from scratch, but whether games translate into deeper understanding or insight depends on the setup, accompanying methods, debrief, and follow up application. Otherwise it was just a game, and the insight hasn’t been transferred.
Derman states that digesting a model to the level of unconscious competence increases your power. Unconscious competence refers to the state where we can do something reliably well without a high degree of attention. Driving a car in routine traffic is an example. If we can make business decisions reliably well under a range of conditions then that is powerful.
The challenge is that just as our ability to drive well in routine conditions may give us a false sense of our ability in challenging conditions such as black ice, we may not realize that we have moved beyond our mental models and business competence, and continue making unconscious decisions where we don’t have competence. The actions of managers under changes to oil price, regulation, government, technology demonstrate what can happen.
Models splinter when you look at them closely. What does this mean?
Models contain many simplifications that reflect the judgment of subject matter experts and modelers. Simplifications are necessary to contain the scope of the model, and focus attention on elements of interest.
On the other hand, simplifications in models can be dangerous. For example, small changes in excluded variables could make large cumulative changes in results over time (a missed “butterfly effect”).
Another weakness may be that the relationships captured in the model do not reflect any understanding about why things happen and instead attempt to codify statistical correlations. A good model, then, must be developed to a level of detail that reflects low-level causation between its components.
Derman compares models to theories. Theories are an explanation based on observation and reasoning developed through a search for why things happen. Fully developed mathematically defined theories, he argues, are virtually indistinguishable from reality, while models remain closer to metaphor or analogy. He states that “financial modeling is not the physics of markets” and “when models in the social sciences fail, they fail bluntly, with no hint as to what went wrong and no clue as to what to do next.”
While I don’t agree with his implication that all social sciences models must perform this way, he does highlight that we have well-developed mathematical relationships and theories that underpin models in the natural sciences and do not yet have and equally advanced understanding of social sciences.
One point that Derman misses is that unlike physicists and philosophers who pursue the ideal, we need to compare performance to a baseline and seek improvement. In most cases the baseline is our existing individual unconscious mental models, or simple checklists and spreadsheets. Group-designed formal quantitative models then can provide a significant step forward in power, as long as continual attention is paid to their performance and the validity of assumptions, boundaries, and limits.
We’ve covered a lot of ground in this series of posts inspired by Derman’s book Models. Behaving. Badly. I hope that this series of posts has:
- introduced you to, or reminded you of, some aspects of modeling
- made you curious
- caused you to reflect on your own use of models.
Interested in models or have comments? Let me know at firstname.lastname@example.org.
And sign up for the newsletter!
Disclosure: I have no personal or business relationship with Derman or his book.
Reference: Derman, Emmanuel. Models. Behaving. Badly. Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. New York. Simon and Schuster Free Press. 2012