Know Your Limitations of a Study
Consider the Population
Setting/Environment. You don't want to pick an MCD value that's appropriate for somebody who is a subacute rehabilitation who just had a stroke maybe three-four weeks ago versus somebody who's had a stroke for three or four years and is in a community dwelling. They will have a different progression. You're going to expect a different level of improvements. Another example of setting and environment is a test that was studied, in the United States where we have one of the highest rates of obesity and comorbidities, versus Okinawa, which is one of the healthiest areas in the world and has the most number of centenarians per capita.
Pathology/Medical Co-morbidities. Something else to consider is pathology and medical comorbidities. So a test that you would use for somebody that has MS is not necessarily a test that is going to be appropriate for somebody who has has a stroke.
Age. Studies ranging from 65 to 85 may not be appropriate for centenarians. Age is very important when we're establishing norms and a lot of tests that you'll see that have gone through large subsets of populations will stratify out their norms based off of age We expect a little bit of a decline as the person gets older whether that's an age-related change or pathological, you have to look at the research for that. We do expect some different values to be had between the different age ranges. So you wouldn't use a test that was appropriate for a 60-year-old in a 100-year-old.
Gender. Gender is also an important consideration.
Consider the Design
When considering the design of the study, you have to look and see how they performed it
Retrospective vs prospective. What were their methods? A retrospective study is where you go back in time. For example, if you're doing a fall study, you're looking at any falls that that person had in the past six months. For a prospective study, you're looking at falls that a person has starting from a specific time point and moving forward. Typically, a prospective study has a lot more weight to it, because you're collecting data in real-time, whereas in retrospective, you are expecting people to be able to recall information from months ago when they may not even remember what they've had for breakfast last week. I know I certainly can't even remember what I had for dinner last night, let along being able to recall maybe a nontraumatic fall that happened three months ago.
Reliability and validity. You want to make sure that the test is measuring what it's supposed to and that it measures it consistently.
Power analysis. A power analysis is also very important. It is seeing how many subjects or patients you need in a specific subset of data to get a statistically important value and to be able to make sure that your data is supported.
Training prior to conducting the study. For a lot of the outcome measures that are used in validation studies, you want to make sure that everybody is trained in a similar manner. I'm sure, as many of you guys in the clinic have seen the Berg being performed differently than you perform the Berg. Now, there is one specific way you're supposed to do a Berg Balance Test, but either through variation of training or something that's more clinically quicker and more convenient for you or others have fallen into a habit of performing it differently. So we're not always necessarily testing the test the way it should be.
Novice vs expert clinician. You should consider if it was a novice or an expert clinician that was conducting the study and doing the testing. That is because somebody who has done it either for many years might have it down pat, but it might be older science, or somebody who's newer might have newer science, or they might not be as consistent in the way that they're delivering the test because they don't have that experience. So it goes both ways.
