Evaluating an Evaluation

Quayhagen, M. P., Quayhage, R. R.Hendrix, R. C., Jackson, J. E., Snyder, L., & Bower, D. (2000). Coping with dementia: Evaluations of four nonpharmacological interventions. International Psychogeriatrics, 12(2), 249-265.

Quayhagen et al. (2000) performed an outcome-based (summative) evaluation of four nonpharmacological interventions directed at individuals diagnosed with dementia and their caregiving spouses. Using an experimental design, Quayhagen et al. randomly assigned patients with specific forms of dementia to one of four intervention groups or a wait-list control group. Although valuable data was collected through the pre-test/post-test experimental design of the evaluation, numerous limitations were revealed.

The pre-test/post-test design of the evaluation enabled Quayhagen et al. (2000) to statistically examine the outcomes of each of the interventions by comparing results between both individual baselines (pre-test) as well as to a no-intervention control group. Employing a pre-test helps to eliminate idiosyncratic differences in the results while comparisons between the control group and the interventions provides greater statistical power to detect differences in outcomes due to the interventions as a whole. The fact that participants who completed the interventions had higher education than the general population, and were primarily white (93%) adults with specific forms of dementia (e.g., 70% were diagnosed with Alzheimer’s dementia) the generalizability of the results is restricted; however, some effort to account for these differences was made. For example, participants were first randomly assigned to either the control or intervention group; next those assigned to the intervention group were randomly assigned further to one of the four interventions being evaluated. Random assignment of participants facilitates the balance of pre-existing characteristics across all groups; in other words random assignment in experimental designs grants a greater likelihood that groups of participants will be ‘equal’ than if a quasi-experimental design is used. Indeed, preliminary analyses indicated that the treatment groups did not differ on age or education and ethnicities were spread out among the groups randomly. Further strengths of the evaluation include an adequately trained team using a variety of training methods (e.g., role playing, videotaping) and a pre- and post-test assessment team who was blind to the assigned treatments. Thorough training ensures that participants from the same groups are receiving identical treatments even though different individuals may be administering them. Using a blind assessment team helps to eliminate potential biases that individuals may have regarding specific treatments (e.g., one may believe that cognitive, rather than support-system interventions are more beneficial which may translate into the results of the assessment).

Various weaknesses of the evaluation are also apparent indicating a need for future research in this area. For example, only 15 patient/caregiver dyads agreed to be wait-listed for treatment potentially limiting the statistical power of the analyses with this group. Additionally, participants who agreed to be part of the wait-list control may have sought out other community or medical-based interventions/treatments the eight week wait between their pre- and post-tests; no efforts were made to determine whether or not this occurred. Thus, the control group may not be a no-treatment control group but rather a group who engaged in other treatments outside of the four being evaluated. There was also a high attrition rate for one of the interventions that limits the conclusions that can be drawn. For example, those who withdrew from the treatment may not have found it beneficial, while the select few who remained in the treatment may have. Thus, the efficacy of the treatment may be more positively biased than had those who withdrew completed the treatment and been included in the analyses. The 8 week treatment period may also not have been long enough to fully appreciate the benefits of the treatments; perhaps great gains may have occurred had the interventions and evaluations been more longitudinal in nature. Lastly, although one of the goals of the evaluation was to determine the impact of the interventions on both the patient and caregiver, a number of caregivers did not return the final, qualitative questionnaire. This questionnaire was the primary indicator of the caregivers’ evaluation of the treatments however, no results were reported regarding whether or not caregivers from specific treatments were more or less apt to return it. Thus, the reader is left wondering if the caregivers who failed to return the questionnaire felt the assigned treatment was in fact beneficial.