I’ve mentioned unintended consequences. This is not as complicated as it may sound, so I’ll illustrate with a few examples.
Category: About Learning
I’ve described a rigorous method for improvement, fairly well-known as plan-do-check-act, with my own embellishments. As described it takes a scientific approach in the face of many unknown factors, to narrow down cause and effect relationships.
Without getting into the details of data analysis and statistics, I have introduced just a few things that you might watch out for as you interpret the data collected after performing the selected actions for a significant period of time – at a minimum, 6 to 10 data points.
- What has changed since you set up the improvement process?
- Is there anything that is obviously not working or is causing problems?
- What is obviously working or beneficial?
- Does the data analysis support what seems to be obvious?
- Are there any clear trends emerging compared to the baseline?
- Is there anything about the measurement system that needs to be adjusted?
- Do you need to collect information for a longer period of time to establish a new baseline before making further adjustments?
- What do you need to try to make sure that the effects you are observing are a result of the cause you believe is at work?
- What are some alternate explanations for what you are observing?
- Have you performed several improvement cycles and reached the end of your exploration such that it’s time to end this initiative and shift your focus to something different?
Every step of the improvement cycle has its own pitfalls, but perhaps the most blatant and preventable errors occur in the collection and analysis of results to determine the effects of actions taken.
- inconsistent collection of data
- excluding or dismissing some data points that don’t seem to fit
- interpreting data to confirm what is already expected or believed
- reaching conclusions before sufficient data has been collected
- presenting results that aren’t statistically significant – there is a certain degree of variation that is expected (“noise” in the system) that can’t be said to mean anything
- delays in effects – any action that has a delay in its effect can be difficult to follow
- confusing correlation with causation – events may align in time, but may not have any cause and effect relationship – to propose a causation, a plausible mechanism for that cause to produce that effect needs to be put forward
- ignoring multiple interconnected variables – to rigorously demonstrate an effect, we want to keep all other things constant as we change one factor. In practice this is extremely difficult, even in a laboratory setting, and it makes us susceptible to seeing what we want to see.
The third stage in the improvement cycle is taking action. The nature of the actions will of course depend on the improvement area. The following is true of any type of improvement, but more would have to be added for larger or more complex initiatives.
- having the highest proven probability of achieving the desired result
- proportionate to the nature and degree of the problem
- well-understood benefits, costs, trade-offs, possible side effects within the detailed understanding of the situation developed in the previous step
- clearly defined — SMART (specific, measurable, agreed-upon, realistic, time-bound)
- that everyone understands, accepts, can commit to, and follow through on
- may have been tried before
- independent, or dependence well-known — can be done sequentially or in meaningful groups — if we make multiple changes we’ll never know what made the difference
- temperature
- screen time (blue light)
- sleep medication
- sleep apnea
- alcohol
- exercise
- having the highest proven probability of achieving the desired result — this is very hard to judge without better definition of the specific situation. Depending on the severity of the underlying issues and the nature of the sleep problem, any of these could be sufficient to make the difference I have in mind.
For this example I’m going to consider them as equally probable at the outset.
- proportionate to the nature and degree of the problem — I defined the problem as being not too severe, so I’ll be willing to try personal interventions before pursuing medical relief.
I’ll rule out sleep apnea and sleep medication as early actions.
- well-understood benefits, costs, trade-offs, possible side effects within the detailed understanding of the situation developed in the previous step — adjusting things that affect the temperature, changing my screen time and using a blue light filter, reducing or changing the time of alcohol consumption, and getting more exercise, are all easily understood with just a bit of thinking through. They could have effects on other people, and could take time.
For this example I’ll suppose I have a consistent morning exercise program so I feel that I’m getting the known benefit of exercise for sleep, and not doing it in the evening when it could be a stimulant. I’m now left with three factors.
- that everyone understands, accepts, can commit to, and follow through on — for a non-severe personal example, this may be the critical decision factor for early actions: what am I willing to do, what do I feel like doing, what do I believe will make a difference, etc. Nothing wrong with that. When the risks are higher, this criterion can lead us astray, choosing what’s easy instead of what’s necessary, and that’s the reason it’s placed after the previous criteria.
In this case I’m going to choose what I can follow through on, something that’s easy to try: temperature. I’m not dismissing the other factors, just choosing one to try first. My research has shown that it’s a valid and reasonable thing to try.
- clearly defined — SMART (specific, measurable, agreed-upon, realistic, time-bound) — this criterion and the previous one are a back-and-forth process: what I can commit to depends on what is defined
How will I define the action? I think it’s okay in this case to actually do a few different things that affect the temperature because I’m really just interested in whether adjusting temperature will work for me, so that if I’m having trouble sleeping I’ll know that I should look at this factor. If I wanted to be more precise, I could change one temperature factor at a time, but that seems like overkill for this example.
I’ve already set the time frame at 2 months. My package of actions will be, without getting into the exact details: turning down the heat one degree; keeping a window open at night; switching to a lighter blanket; lighter night clothing; preventing my bedroom from getting hot during the day by using a window shade
- may have been tried before — people will often say “I’ve tried that” but very rarely would they have made a change in a structured way, and so they may not actually know whether that action was implemented consistently, long enough, or what other factors may have been at play
Maybe I have the observation that I sleep well in winter when it’s a bit cooler. But it’s also darker, and I have heavier blankets, and my activity levels are different, and…
So it’s hard to know which one of those things actually made a difference.
- independent, or dependence well-known — can be done sequentially or in meaningful groups — if we make multiple changes we’ll never know what made the difference.
I think that I can make the changes to temperature independently of other factors. They shouldn’t change anything else in my long factors list. If changes to temperature were going to affect a sleep partner negatively, for example, the impact of any conflict would mean that the temperature factor wouldn’t be independent.
I do want to know at the end of my trial whether the change in temperature actually made a difference. If the case were true that there were several changes that I could make as a package, and I was willing to make and continue all of them, then if they worked, I would be “stuck” with continuing all of them, not knowing whether some of them could be eliminated with no effect. I could try experimenting then, of course, or I can just do it now and build up a set of factors that I am confident works consistently and by itself, for me. I’m not going to achieve scientific rigour, but I’ll at least not be flying by the seat of the pants.
- thermostat heat setting
- window state — closed, ajar, open wide
- blankets used
- clothing worn
- day shade state — closed, partially or part of the time, open
- the temperature at bedtime as reported in the same weather app at the same time each night
- an open-ended notes measure to capture any other impressions, factors or changes that might affect the data. We’ll talk about where we have to be careful in interpretation later.
This may all feel like overkill for the example I’ve given, but a) if I had more serious sleep problems that were affecting my work, relationships, mental health, etc. or b) I was considering a more complex situation, the same method could be used and its structure and usefulness would be more obvious and necessary.
- Reading longer articles and books
- Lectures or documentaries
- Structured creative or non-fiction writing
- Playing music or creating art
- Formal study
- Playing a sport, yoga, martial arts
- Strategy games
- Meditation, prayer, contemplation, reflection
- Nature-based activity: bird-watching, star-gazing
What could you do to build attention?
One of the components of the improvement cycle is understanding the current situation. There’s a balance between being sufficiently informed and getting stuck in research and interpretation. For the current example I’ll suggest it’s important to explore a few areas:
- importance of area of improvement (sleep better) — this would have been considered earlier in the process of setting purpose but it’s worth revisiting at this stage
- personal and interaction history
- understanding the system
- importance of area of improvement — why is this area of improvement? what are the desired benefits and goals? what is the cost or benefit of doing nothing? what is your level of commitment to taking action and achieving results?
- personal and interaction history — how has this situation come to be? what actions have been taken informally or informally and what was the result? how have other people, groups, physical locations, and other external circumstances affected this situation?
- understanding the system — what are the reactions to actions? is there anything that encourages or discourages changes or the desired results? if action is taken, would it make sense that there would be delays before the final effects are seen? how simple or complex is the situation? are there many different parts? are the parts and their interactions well-known by anyone? by you?
- deciding on and planning an improvement process
- determining the current situation (understanding the system and setting a baseline)
- making changes (taking action)
- seeing what the effects are (collecting, analyzing, and interpreting the results), and
- adjusting the process and the actions
- update the averages
- review the table and graphs
- consider any obstacles that I’m encountering in making the changes
- consider whether I need to make a major change to the improvement system as a whole — but I don’t want to tamper too early or often otherwise I lose the value of the measurement system every time I change it
Let’s move to the next category of factors that might be used to evaluate information sources when doing research: social.