I’ve described a rigorous method for improvement, fairly well-known as plan-do-check-act, with my own embellishments. As described it takes a scientific approach in the face of many unknown factors, to narrow down cause and effect relationships.
Sometimes a different approach can be taken to improvement. In the sleep example, I could say, there are a lot of factors that are well-established by solid research to be beneficial to better sleep, for most people. Unless I have specific doubts, I don’t need to prove that a specific method is effective.
I could measure my baseline, select several aspects to change simultaneously, keep track of them as I go, and measure the results. If I get positive results, then I’m happy. Nothing wrong with this method in terms of getting results. I don’t care which factors in particular worked, and as long as I keep all of the changes in place, then I should be able to maintain the new improved state.
The limitations are that I don’t know which of the factors helped the most, which could be discontinued, and I can’t make statements about just one of the factors. I really need to say “I changed a bunch of things, and I don’t know what worked in particular, but it’s improved so I’m happy.“
This method doesn’t apply if I throw into the mix unproven methods. If I do that, I can end up with a false positive: a belief that the unproven method had a positive effect when in fact is was some or all of the proven methods. This can also happen if I don’t track my baseline or my behaviour across the proven factors, and just add one unproven method. I see positive results and I think it was the unproven method. But it was actually something that changed in one or more other factors, but I wasn’t keeping track.
This is what’s referred to as anecdotal evidence: it’s a valid personal observation that hasn’t been conducted with any rigour or controls. In one sense there isn’t harm for some situations, as long as the benefits are achieved and there aren’t any unintended consequences. But a person can’t be justified in stating that the unproven method is now supported, even for themselves, because they haven’t really approached it in any kind of structured way.
Similar effects occur in organizations when several initiatives are undertaken in parallel. Everyone takes credit for any benefits achieved, but in fact some initiatives may have been very helpful and others quite unhelpful, or positive results may be from external factors unrelated to the initiatives. In politics, taking credit for results achieved in the presence of global economic conditions over which no single body has much control, or policies which happened to be in place rather than intentionally designed for a situation, is everyday behaviour.
About Learning
What is the shape/path of progress in learning? In school you might have experienced a fairly steady progression through each grade, a relatively straight line from knowing little to knowing enough to do the assignment, pass the test, get a passing grade. There might have been some bumps: a missed assignment, a failed test, a poor or failing grade, a change of courses.
Learning a sport, at least at the beginning, is a similar process. You observe, try, practice, and improve your skills in spurts and plateaus.
Advanced study and research at a mastery level, and exploration of complex problems and situations, often proceeds differently. Major setbacks, long plateaus, backsliding, abandoning a line of inquiry or action completely, starting from the beginning, making major changes in methods, dealing with injury, insult, or rejection, and straight-up failures and crashes, are part of the possible territory.
Recognizing that the path will often not be easy or straightforward, that you can feel lost or like a failure for long periods, will help you be persistent in the face of difficulties: you’ll expect them as being normal, rather than something to be feared.