Pilot Testing Feedback

Pilot Testing Feedback

After pilot testing the first draft of the survey on five people, my pilot participants made a number of recommendations for improvement. If I intended to use this survey in real-life circumstances, far more pilot testing would be done. Not only would revisions be made based on the first set of participants’ feedback, but I would also pilot test my revisions to make sure the appropriate changes were made. Additionally, the stakeholders (e.g., public transportation system administrators) who are interested in the final results of the survey would also be consulted to ensure that the appropriate questions are being asked in order to address the questions/concerns that warranted the survey.

The recommendations based on the first set of pilot responses have been incorporated into the second draft of my survey below. The following is a summary of the recommendations made by my pilot participants.

Include demographic questions to improve generalizability of the findings and to evaluate what populations are represented when the survey is complete. Potential characteristics of participants that may influence their decision to use/not use public transit may also be revealed.

In the first draft of the survey, I did not include any demographic questions. Collecting demographic data will help describe the characteristics of participants who complete the survey. Themes may emerge based on demographic information that will shed light on characteristics of people who use or do not use public transit. For example, it may be that younger participants are more likely to use transit compared to older participants or that income level impacts the choice to use public transit. Reasons why people use transit (or do not use transit) may also be impacted by age. For example, older participants may feel that the amount of time it takes to get to their destinations on the bus are too lengthy, while younger participants may feel the amount of time spent to arrive to their destination is sufficient. Demographic data will help organize and generalize the findings to other groups and lead to stronger conclusions regarding the use of transit in different age groups and income levels. Including a demographic question about gender will help ensure that the participants who complete my survey are not predominantly from one gender or another, and again will lead to a clear understanding regarding the generalizability of the findings.

Organization of the survey could be improved.

By grouping questions that are related to each other together, participants’ progress through the survey will be facilitated and seem more logical. Grouping related questions will also help participants keep the relevant/required information in mind to make their ratings/responses rather than moving from topic to topic and back again. This may also reduce participants’ potential frustration with completing the survey and will improve the chances that they complete the survey in full.

Include statements regarding the purpose of the survey and/or questions.

The original version of my survey did not have any background information provided. A few of my pilot participants asked what the purpose of the questionnaire was (besides for an assignment J). Thus, the second version of my survey included a few brief statements to tell participants the overall purpose of the questionnaire and to potentially increase their motivation to complete the survey.

A clear definition of what is meant by the statements “I use public transit” and “I do not use public transit” is required.

There was some confusion regarding what column, or section of questions that my pilot participants should complete. For example, one participant told me that during the summer months, she would have completed the “I do not use public transit” section and during the winter months she would complete the “I use public transit section.” Additional questions from pilot participants regarding this issue were: What is meant by ‘using public transit on a regular basis’ (e.g., biking or walking to work but busing for groceries and errands)? Does using public transit on a regular basis mean the use of transit every day or for the current week, month or year? Including a clear definition of exactly what is meant by “using public transit” will aid the development of clear conclusions when data collection is complete. It will also increase the internal validity of the survey as I will be more sure that I am in fact measuring what I intended to measure. Including this definition may also increase the amount of participants who complete the survey – if participants are not confused and have a clear understanding of what is expected of them, they are more likely to complete the survey in its entirety.

Some questions needed clarification and/or additional options.

Some of the original questions on the survey were not entirely clear to my participants. For example, I failed to include an option for transit users who use the U-PASS transit pass or bus tickets and had only included a cash fair or monthly bus pass in the original survey. Any questions that my pilot participants were left wondering about, I reviewed and revised for clarification. For example, “Satisfaction with public transit” can be broken down into a number of components. Participants may be satisfied with the transport system itself but may be dissatisfied with the bus drivers or location(s) of the bus terminals; double-barreled questions should be avoided as much as possible. Breaking broad questions down into more specific questions will help make more specific conclusions when analyzing the data. In addition, some additional questions for people who do not use public transit were also suggested and incorporated in the revised survey to help make comparisons between those who use and those who do not use public transit once data collection is complete.

Include an open-ended question at the end for any additional comments or feedback regarding transportation or the survey itself.

The use of closed-ended questions and forced-choice questions limits the type and quality of participants’ responses. There may be areas of the public transportation system that are not covered in the survey that participants want to discuss or comment on. Thus, including a final open-ended question for participants to add any comments that they see fit may help me get a better understanding of perceptions of the public transit system or related areas of transportation that warrant further investigation. These final comments may also clarify participants’ responses or add detail to responses.

Notes on formatting the final survey.

In doing a bit of a literature search for proper survey design, I came across the following research article that outlines the best practices for formatting paper-based surveys. Once the questions on my survey have been finalized, I would implement as many of the recommended practices as possible in order to improve my chances of having the most number of people complete the survey, in full, as possible. If a survey is easy to read and follow, it is likely that the response rate for the survey will increase. In my past experiences with data collection, entry and analysis, nothing is more frustrating than investing a great amount of time, energy and money into collecting data, only to find during data entry that some questions were inadvertently missed due to formatting that is hard to follow. If questions were skipped because they were of a sensitive nature, this is treated differently during data analysis than simple missing data (e.g., forgetting to circle gender because it appeared to be a sub-question due to formatting problems). A brief summary of the practices that are recommended in the article have been outlined below.

Fanning, E. (2005). Formatting a paper-based survey questionnaire: Best practices. Practical Assessment, Research & Evaluation, 10 (12), 1-14. Available online: http://pareonline.net/getvn.asp?v=10&n=12

My survey is rather short, so it may fit on a one or two page design. In these instances, 8 ½ x 11” paper should be used, the pages should not be double-sided and should be stapled in the top left corner. Regardless of page layout, response rates increase when the survey looks official (e.g., use letterheads and return addresses). When possible, surveys should also include a colored front cover with instructions that will help generate motivation in the respondents. If a back cover is used, questions should not be included on this page, rather a thank you and space for additional comments should be included here.

Directions should be included at the beginning of the survey to inform participants about how to complete the survey and what to expect. If the survey is mailed out to respondents, the deadline date for completion should also be included as well as a return, addressed envelope.

Demographic questions should not start a survey as these are rather uninteresting; personal questions should also be saved for later in the survey. At the beginning of the questionnaire you are still trying to build participants’ motivation to complete the survey and are still gaining their trust. Thus, both demographic questions and sensitive/personal questions should be saved for later on in the survey or at the end of the survey. The most interesting questions and questions that apply to all participants should be at the forefront of the survey.

The questions themselves should be grouped by content and type. Questions should be presented in a logical order to help participants move through the survey without having to continually change topics. The use of color or space between question groupings will also help participants realize that a shift in thinking or content area is required. Questions should be written in a heavy or dark font while the responses should be in a lighter shade or font. The spaces where participants are to provide responses should be consistently used. For example, one column on the page could be reserved for the questions that are asked, while the second column is the consistently used space for participants to provide their answers.

Although blue and yellow has been shown to improve response rates when compared to simple black and white questions, contrast has been proven to be more effective than color. Thus, slightly shading the background of different sections of questions will help improve response rate and will indicate to participants that a specific section of questions is beginning/ending. The use of shading can aid participants’ navigation through the questionnaire thereby improving response rates.

Surveys should be kept as short as possible. This is not to say that important questions should be omitted for the sake of brevity, or than a tiny font should be used to visually shrink the size of the survey. Rather, when designing questionnaires be cognizant of your potential participants’ time, commitment and interest.

A Proposal to Evaluate EFry's Court Worker Program

The Elizabeth Fry Society of Saskatchewan (EFry): A Proposal to Evaluate the Court Worker Program

Brief Background

Through a number of programs, the EFry society of Saskatoon offers a variety of services and support for women who come into conflict with the law. The current evaluation proposal will focus on one of the programs that EFry offers, the Court Worker (CW) program.

The primary goal of the CW program is to make their clients’ progress through the Criminal Justice System (CJS) “more humane” by providing information about the court and justice system (e.g., what to expect in court, clarification of sentence and charges) as well as support to the person in conflict with the law (e.g., through referrals, counseling, child care arrangements). Incarcerated women in Canada often have a lower education than incarcerated men as well as the general Canadian population (Statistics Canada, 2008; http://www.statcan.gc.ca/pub/85-002-x/85-002-x2008001-eng.pdf) indicating a need to clarify the CJS and its court procedures to those who are charged with an offense. Additionally, each female entering the CJS is an individual in that they have different backgrounds, responsibilities, living situations, income levels, etc.. Thus, each woman requires individualized services and support – something that EFry attempts to provide to women through their CW program.

Who should be involved? How will they be engaged?

Those who deliver the services and run the program. Staff and volunteers from EFry receive training regarding court procedures, the criminal justice system, substance abuse, aboriginal and status of women issues, as well as community resources available. Following this training, the staff and volunteers are in the courts daily, supporting and educating women about the CJS process. Their input regarding the delivery of the program will be integral in determining whether or not the program is being implemented as intended. Confidentiality is a high priority at EFry, thus the staff and volunteers will facilitate recruitment of clients who are willing to be participants in the current evaluation. Board directors will also help ensure that volunteers and staff are aware that the evaluation is being conducted and to encourage the staff and volunteers to facilitate the evaluation whenever possible (e.g., recruitment).

Those who receive the services. Clients who receive support services and information regarding the CJS process will also be engaged in the evaluation process. Willing participants will be asked to complete a CJS Understanding questionnaire, which will assess the degree to which clients understand the CJS and court process prior to their engagement with the court as well as following the delivery of EFry’s services. This questionnaire will be developed alongside the directors and staff to ensure that it accesses information that is provided to clients through the CW program (e.g., sentencing practices and options). Semi-structured interviews will also be conducted with participants to evaluate their needs (e.g., is childcare required, is transportation to court proceedings required, etc.) before EFry’s delivery of support services. The same semi-structured interview will again be performed following the delivery of support services however additional questions will be included to evaluate whether or not the client feels that her support needs were met and what could be done to improve these services.

The focus the evaluation. What questions will the evaluation seek to answer? What type of evaluation will you use?

Please refer to the attached logic model to help clarify the focus of the proposed goals- and outcome-based evaluation. The primary purpose of the evaluation will be to determine whether or not clients’ understanding of the CJS is improved through the services that EFry offers as well as if EFry is meeting their clients’ support needs during the CJS process.

In brief, the evaluation will seek to answer the following questions:

1. Are clients gaining a better understanding of the CJS process?
a. What are the common areas of clients’ misunderstanding of the CJS process?
b. Is sufficient information provided to clients or are there areas/aspects of the CJS process that require further elaboration and clarification by staff and volunteers?

2. Are clients’ support needs being met to their satisfaction?
a. What are common needs among clients?
b. Are there needs and/or support services that most clients access? (Should funding and/or resources to this/these services be increased?)
c. Are there needs and/or support services that most clients do not access? (Should funding and/or resources to this/these services be reduced?)

To answer the above-mentioned questions, clients’ understanding of the CJS process is required prior to the delivery of EFry services as well as following the delivery of services. Additionally, clients’ support needs must be assessed both prior to, and following the implementation of support. To determine whether or not clients’ understanding of the CJS process is increased and that their support needs are met to their satisfaction, a comparison between their pre- and post-program responses on the questionnaire and interviews will be required. For example, if their understanding of the CJS process is increased, clients should be able to provide more information regarding sentencing practices and options as well as increased knowledge about the types and quantity of community services and support. If clients’ support needs are met, their post-program responses during the semi-structured interview should indicate that most, if not all of the needs they discussed during the pre-program interview were met. Clients’ satisfaction with the program should also be high (rated on a Likert-type scale).

Who will use the evaluation? How will they use it?

Performing the current evaluation will help determine if EFry is in fact making the CJS process “more humane” by educating their clients and providing support services based on each individual’s needs. The results of the evaluation will also inform the society about areas of misunderstanding that women have regarding the CJS process and may identify areas where more information and training is needed for staff, volunteers and future clients. Common, wide-spread needs of clients may also be identified through the evaluation which would help EFry determine what community resources and support services are required the most to help make decisions regarding the allotment of resources and money in the future. One of the principles of the Canadian Association of EFry societies is to develop policies, positions and acts based on the common needs and interests affecting women in Canada. Thus, information to help develop these policies and acts will also be obtained through the evaluation.

Evaluating the ECS Programming for Children with Severe Disabilities

ASSIGNMENT 2: WHAT MODEL IS APPROPRIATE TO EVALUATE THIS PROGRAM AND WHY?

Based on the limited information provided, the ECS Programming for Children with Severe Disabilities appears to have the goal of helping to improve the functioning of children with severe disabilities in a learning environment either at home or at a center (e.g., daycare, kindergarten). Considering that Individual Program Plans, with clear objectives and goals, are designed for each participant based on his/her needs, the desired outcome of each program may be different. However, one overall goal that should be apparent in each individual program would be to make some sort of gains in the child’s development and/or functioning; in other words, there should be benefits to participating in the program otherwise the program is no longer feasible.

Before finalizing the decision of which approach or model to use to evaluate the ECS Program, a consultation with the stakeholders would have to occur. Stakeholders, or those who are interested in having the evaluation done, should have some input in the direction that evaluation will take. There is likely a reason why the stakeholders have requested that an evaluation be done (such as conflict or complaints from participants, budget requirements, the desire to replicate the program elsewhere, etc.) and part of my job as an evaluator would be to help answer the questions/needs of those stakeholders. Designing the evaluation should be a collaborative effort between the evaluator and stakeholders.

POSSIBLE MODELS/APPROACHES TO THE EVALUATION

The model that I would recommend to evaluate the ECS Program would depend on what question(s) stakeholders want answered through my evaluation. However, the following describes potential models for evaluating the ECS Program and the situations under which each model may be more appropriate to apply.

Summative Goals-Based Evaluation: The Goal is to Determine Whether or Not the Program is Working as Intended

A potential goal of the evaluation may be to determine whether or not the intended objectives and goals of the Individual Program Plans are actually being achieved. There could be numerous motivations for performing a summative goals-based evaluation on the ECS Program. A few examples include identifying areas of the program that should be retained, cut or improved upon in the future, determining whether or not, and to what extent the goals of the program are being reached and the costs associated with obtaining those goals. If the purpose of the evaluation is to examine the extent to which the goals of the program are being achieved, the primary evaluation model that I would recommend would be a goals-based evaluation.

Outcomes-Based Evaluation: The Goal is to Determine if Participants are Benefiting from the Program

Perhaps for publicity reasons the stakeholders would like to examine the efficacy of the programs being applied to each participant. Or, the program may be working so well (or so poorly) that an assessment of participant outcomes is required to make decisions about the future implementation of the program. In these instances, an outcomes-based evaluation may be best suited; the evaluation would aim to assess the benefits and drawbacks of completing the program. In other words, an outcomes-based evaluation would help determine whether or not participants demonstrate improvement from participating in the program and whether those improvements outweigh the costs associated with it. Such an evaluation would help stakeholders decide whether or not to keep or eliminate the program (or certain aspects of it).

Process Evaluation: The Goal is to Replicate the Program Elsewhere or to Inform Stakeholders about the Costs and Benefits of the Program

Perhaps the program has been previously evaluated and has been shown to be quite successful in achieving its goals and there is now a desire to replicate the program elsewhere. Maybe there have been complaints regarding the program, or changes to the program have been made over the years and the expenditure of resources needs to be assessed. In these instances, I would recommend a process evaluation which provides a detailed description of the program’s process, or how the program actually works or runs. The results obtained from a process evaluation could facilitate the implementation of the program at different locations, reveal how resources (including staff, money, etc.) are being utilized and identify potential inefficiencies in the program. If the process of the program is to be evaluated, a process evaluation would be best suited.

Formative Evaluation: The Program is Relatively New and the Goal is to Obtain Feedback regarding Program Improvement

It may be the case that the stakeholders want feedback regarding areas of the program to improve or to test-run the implementation of the program. This would be more likely to occur if the program is somewhat in its infancy and is still being developed. The goal of a formative evaluation of the ECS Program would be to identify areas of the program that could be developed or improved, and to help guide the program towards that improvement.

An Eclectic Approach: There are Numerous Questions to be Answered

Depending upon the purpose of the evaluation and the questions that stakeholders want answered, one model may prove to be better suited than the others. However, it may also be that a more eclectic approach, or a mixture of the models, would be preferred. For example, the stakeholders may want an evaluation of participants’ outcomes (outcomes based) as well as to test run new materials (formative) while attempting to replicate the program at another location (process). Answering all of these questions would not be permitted by strictly following one model while disregarding others; rather a mixture of the models and methods may be the best vehicle to arrive at the desired answers.

Depending on the questions that stakeholders want answered and the resources available to complete the evaluation, the direction and scope of the evaluation may change. My job as an evaluator of the ECS Program would be to first identify the purpose of the evaluation through consultation with stakeholders and to then decide on the most appropriate model or models to follow in order to provide the stakeholders with the information they need to make their decisions.

Evaluating an Evaluation

Quayhagen, M. P., Quayhage, R. R.Hendrix, R. C., Jackson, J. E., Snyder, L., & Bower, D. (2000). Coping with dementia: Evaluations of four nonpharmacological interventions. International Psychogeriatrics, 12(2), 249-265.

Quayhagen et al. (2000) performed an outcome-based (summative) evaluation of four nonpharmacological interventions directed at individuals diagnosed with dementia and their caregiving spouses. Using an experimental design, Quayhagen et al. randomly assigned patients with specific forms of dementia to one of four intervention groups or a wait-list control group. Although valuable data was collected through the pre-test/post-test experimental design of the evaluation, numerous limitations were revealed.

The pre-test/post-test design of the evaluation enabled Quayhagen et al. (2000) to statistically examine the outcomes of each of the interventions by comparing results between both individual baselines (pre-test) as well as to a no-intervention control group. Employing a pre-test helps to eliminate idiosyncratic differences in the results while comparisons between the control group and the interventions provides greater statistical power to detect differences in outcomes due to the interventions as a whole. The fact that participants who completed the interventions had higher education than the general population, and were primarily white (93%) adults with specific forms of dementia (e.g., 70% were diagnosed with Alzheimer’s dementia) the generalizability of the results is restricted; however, some effort to account for these differences was made. For example, participants were first randomly assigned to either the control or intervention group; next those assigned to the intervention group were randomly assigned further to one of the four interventions being evaluated. Random assignment of participants facilitates the balance of pre-existing characteristics across all groups; in other words random assignment in experimental designs grants a greater likelihood that groups of participants will be ‘equal’ than if a quasi-experimental design is used. Indeed, preliminary analyses indicated that the treatment groups did not differ on age or education and ethnicities were spread out among the groups randomly. Further strengths of the evaluation include an adequately trained team using a variety of training methods (e.g., role playing, videotaping) and a pre- and post-test assessment team who was blind to the assigned treatments. Thorough training ensures that participants from the same groups are receiving identical treatments even though different individuals may be administering them. Using a blind assessment team helps to eliminate potential biases that individuals may have regarding specific treatments (e.g., one may believe that cognitive, rather than support-system interventions are more beneficial which may translate into the results of the assessment).

Various weaknesses of the evaluation are also apparent indicating a need for future research in this area. For example, only 15 patient/caregiver dyads agreed to be wait-listed for treatment potentially limiting the statistical power of the analyses with this group. Additionally, participants who agreed to be part of the wait-list control may have sought out other community or medical-based interventions/treatments the eight week wait between their pre- and post-tests; no efforts were made to determine whether or not this occurred. Thus, the control group may not be a no-treatment control group but rather a group who engaged in other treatments outside of the four being evaluated. There was also a high attrition rate for one of the interventions that limits the conclusions that can be drawn. For example, those who withdrew from the treatment may not have found it beneficial, while the select few who remained in the treatment may have. Thus, the efficacy of the treatment may be more positively biased than had those who withdrew completed the treatment and been included in the analyses. The 8 week treatment period may also not have been long enough to fully appreciate the benefits of the treatments; perhaps great gains may have occurred had the interventions and evaluations been more longitudinal in nature. Lastly, although one of the goals of the evaluation was to determine the impact of the interventions on both the patient and caregiver, a number of caregivers did not return the final, qualitative questionnaire. This questionnaire was the primary indicator of the caregivers’ evaluation of the treatments however, no results were reported regarding whether or not caregivers from specific treatments were more or less apt to return it. Thus, the reader is left wondering if the caregivers who failed to return the questionnaire felt the assigned treatment was in fact beneficial.