Every step of the improvement cycle has its own pitfalls, but perhaps the most blatant and preventable errors occur in the collection and analysis of results to determine the effects of actions taken.
It’s in this step that certain specialized skills are necessary, and unfortunately, far from universal. Even among the highly trained, logical and statistical mistakes are occasionally made.
So what can you do? First, know what you don’t know. I’ll mention here and in future posts to some of the areas that you need to be aware of.
Second, develop those skills. To improve in any area means that you need to develop your improvement skills in addition to the core functional skills. An athlete who wants to improve performance learns how to train. A writer learns how to edit. An actor learns how to rehearse. These are all process skills – how to get better – not just hitting the ball, writing a paragraph, or acting a scene.
Third, get a second opinion (or third, or… .) In high stakes situations, organizations use “red” teams, a group of reviewers who is given permission to be critical about the thinking and actions of the team being reviewed.
What are a few of the pitfalls?
- inconsistent collection of data
- excluding or dismissing some data points that don’t seem to fit
- interpreting data to confirm what is already expected or believed
- reaching conclusions before sufficient data has been collected
- presenting results that aren’t statistically significant – there is a certain degree of variation that is expected (“noise” in the system) that can’t be said to mean anything
- delays in effects – any action that has a delay in its effect can be difficult to follow
- confusing correlation with causation – events may align in time, but may not have any cause and effect relationship – to propose a causation, a plausible mechanism for that cause to produce that effect needs to be put forward
- ignoring multiple interconnected variables – to rigorously demonstrate an effect, we want to keep all other things constant as we change one factor. In practice this is extremely difficult, even in a laboratory setting, and it makes us susceptible to seeing what we want to see.
In the sleep example, how do the multiple variables come into play? I chose to focus on temperature as an action, out of a long list of possible factors. How could I keep all of the other factors constant?
Short answer: I couldn’t. If I collect data for a long enough period of time, there might be a case to say that natural variation in the other factors that are not consciously being changed are less likely to be contributing to any trend I observe, than the specifically tracked temperature.
Let’s consider the short list of factors from the last post. I do have control over my screen time, and can attempt to keep some consistency over it. I could track sleep medication, how much and when. I can perhaps get away with assuming that sleep apnea is relatively constant. I can track my alcohol and exercise. Even the temperature itself is subject to the ambient outside temperature with the window open, whether I used the oven or fireplace in the evening, etc.
But some of these weren’t in my measurement system! Am I now going to start tracking a lot more factors? In any case it gets interesting because my tracking of other factors may result in changes to my behaviours in those areas.
I might see that my alcohol use fluctuates, and because I want to make the effect of temperature change more clear, I reduce my variation in alcohol use, whether consciously or more incidentally. This in itself could have a larger impact on sleep than the temperature change I chose as my conscious action.
Almost every improvement initiative has these sorts of effects – focusing on an area for improvement; the selection of actions, the measurement of data; the process of analysis and interpretation; the communication and integration of results; learning about the domain and about the improvement process. All of these can affect the system, even for this personal example.
When I see positive results, that’s great! But I might think it was because I changed the temperature, when actually it was changing my pattern of alcohol use. This type of conflation of actions and results is common in every area of life and business.
Exercise
1. What don’t you know about your improvement area?
2. What formal learning have you had about improvement?
3. Who could you consult as a red team?
4. How might your data collection and interpretation be affected by the pitfalls mentioned above?
5. How might your system be changing unintentionally as you go along?
About Learning
We need to understand what is at the edge of knowledge and what is well-integrated. We don’t need to blindly accept the mainstream, but if we choose to question or reject it, we need to invest more into learning and understanding deeply before following another path. (And here lies the failure of education systems to teach the basic techniques for evaluating knowledge and for identifying the interaction between fact and belief.) It’s for some reason common for some people to reject equally what’s well-established and what is more provisional.
There are many frontiers in our understanding of how things work. And sometimes things change. But by itself this fact doesn’t provide a basis for rejecting current knowledge. Changing one piece requires that change to make sense in the context of everything else we know. It’s our responsibility to dig deeper into how things are interconnected as part of the questioning process.
Where we have detailed, reliable, and proven knowledge that can be used to design new processes, materials, drugs, treatments, products, etc. then it’s an extremely bold claim to reject one small piece of this interconnected knowledge system. Such extraordinary claims require a careful approach of study, experiment, demonstrations, and repetition to be considered valid.
An example is the rejection of vaccines. To say one doesn’t believe in the effectiveness of vaccines to prevent life-threatening illnesses is similar to saying that antibiotics don’t work. I can only assume that people would not refuse antibiotics when they show up very sick at the hospital. Yet both have well-understood, and hidden, mechanisms. Both have short- and long-term trade-offs. Neither area is at the edge of understanding medicine.
Psychoactive drugs are a different story. The mechanisms, consistency of effects, side-effects, and other aspects of many of these drugs is not well-known. This is a leading edge of medical treatment and a healthy skepticism with regard to their use seems well-justified.