Pilot Testing Feedback

Pilot Testing Feedback

After pilot testing the first draft of the survey on five people, my pilot participants made a number of recommendations for improvement. If I intended to use this survey in real-life circumstances, far more pilot testing would be done. Not only would revisions be made based on the first set of participants’ feedback, but I would also pilot test my revisions to make sure the appropriate changes were made. Additionally, the stakeholders (e.g., public transportation system administrators) who are interested in the final results of the survey would also be consulted to ensure that the appropriate questions are being asked in order to address the questions/concerns that warranted the survey.

The recommendations based on the first set of pilot responses have been incorporated into the second draft of my survey below. The following is a summary of the recommendations made by my pilot participants.

Include demographic questions to improve generalizability of the findings and to evaluate what populations are represented when the survey is complete. Potential characteristics of participants that may influence their decision to use/not use public transit may also be revealed.

In the first draft of the survey, I did not include any demographic questions. Collecting demographic data will help describe the characteristics of participants who complete the survey. Themes may emerge based on demographic information that will shed light on characteristics of people who use or do not use public transit. For example, it may be that younger participants are more likely to use transit compared to older participants or that income level impacts the choice to use public transit. Reasons why people use transit (or do not use transit) may also be impacted by age. For example, older participants may feel that the amount of time it takes to get to their destinations on the bus are too lengthy, while younger participants may feel the amount of time spent to arrive to their destination is sufficient. Demographic data will help organize and generalize the findings to other groups and lead to stronger conclusions regarding the use of transit in different age groups and income levels. Including a demographic question about gender will help ensure that the participants who complete my survey are not predominantly from one gender or another, and again will lead to a clear understanding regarding the generalizability of the findings.

Organization of the survey could be improved.

By grouping questions that are related to each other together, participants’ progress through the survey will be facilitated and seem more logical. Grouping related questions will also help participants keep the relevant/required information in mind to make their ratings/responses rather than moving from topic to topic and back again. This may also reduce participants’ potential frustration with completing the survey and will improve the chances that they complete the survey in full.

Include statements regarding the purpose of the survey and/or questions.

The original version of my survey did not have any background information provided. A few of my pilot participants asked what the purpose of the questionnaire was (besides for an assignment J). Thus, the second version of my survey included a few brief statements to tell participants the overall purpose of the questionnaire and to potentially increase their motivation to complete the survey.

A clear definition of what is meant by the statements “I use public transit” and “I do not use public transit” is required.

There was some confusion regarding what column, or section of questions that my pilot participants should complete. For example, one participant told me that during the summer months, she would have completed the “I do not use public transit” section and during the winter months she would complete the “I use public transit section.” Additional questions from pilot participants regarding this issue were: What is meant by ‘using public transit on a regular basis’ (e.g., biking or walking to work but busing for groceries and errands)? Does using public transit on a regular basis mean the use of transit every day or for the current week, month or year? Including a clear definition of exactly what is meant by “using public transit” will aid the development of clear conclusions when data collection is complete. It will also increase the internal validity of the survey as I will be more sure that I am in fact measuring what I intended to measure. Including this definition may also increase the amount of participants who complete the survey – if participants are not confused and have a clear understanding of what is expected of them, they are more likely to complete the survey in its entirety.

Some questions needed clarification and/or additional options.

Some of the original questions on the survey were not entirely clear to my participants. For example, I failed to include an option for transit users who use the U-PASS transit pass or bus tickets and had only included a cash fair or monthly bus pass in the original survey. Any questions that my pilot participants were left wondering about, I reviewed and revised for clarification. For example, “Satisfaction with public transit” can be broken down into a number of components. Participants may be satisfied with the transport system itself but may be dissatisfied with the bus drivers or location(s) of the bus terminals; double-barreled questions should be avoided as much as possible. Breaking broad questions down into more specific questions will help make more specific conclusions when analyzing the data. In addition, some additional questions for people who do not use public transit were also suggested and incorporated in the revised survey to help make comparisons between those who use and those who do not use public transit once data collection is complete.

Include an open-ended question at the end for any additional comments or feedback regarding transportation or the survey itself.

The use of closed-ended questions and forced-choice questions limits the type and quality of participants’ responses. There may be areas of the public transportation system that are not covered in the survey that participants want to discuss or comment on. Thus, including a final open-ended question for participants to add any comments that they see fit may help me get a better understanding of perceptions of the public transit system or related areas of transportation that warrant further investigation. These final comments may also clarify participants’ responses or add detail to responses.

Notes on formatting the final survey.

In doing a bit of a literature search for proper survey design, I came across the following research article that outlines the best practices for formatting paper-based surveys. Once the questions on my survey have been finalized, I would implement as many of the recommended practices as possible in order to improve my chances of having the most number of people complete the survey, in full, as possible. If a survey is easy to read and follow, it is likely that the response rate for the survey will increase. In my past experiences with data collection, entry and analysis, nothing is more frustrating than investing a great amount of time, energy and money into collecting data, only to find during data entry that some questions were inadvertently missed due to formatting that is hard to follow. If questions were skipped because they were of a sensitive nature, this is treated differently during data analysis than simple missing data (e.g., forgetting to circle gender because it appeared to be a sub-question due to formatting problems). A brief summary of the practices that are recommended in the article have been outlined below.

Fanning, E. (2005). Formatting a paper-based survey questionnaire: Best practices. Practical Assessment, Research & Evaluation, 10 (12), 1-14. Available online: http://pareonline.net/getvn.asp?v=10&n=12

My survey is rather short, so it may fit on a one or two page design. In these instances, 8 ½ x 11” paper should be used, the pages should not be double-sided and should be stapled in the top left corner. Regardless of page layout, response rates increase when the survey looks official (e.g., use letterheads and return addresses). When possible, surveys should also include a colored front cover with instructions that will help generate motivation in the respondents. If a back cover is used, questions should not be included on this page, rather a thank you and space for additional comments should be included here.

Directions should be included at the beginning of the survey to inform participants about how to complete the survey and what to expect. If the survey is mailed out to respondents, the deadline date for completion should also be included as well as a return, addressed envelope.

Demographic questions should not start a survey as these are rather uninteresting; personal questions should also be saved for later in the survey. At the beginning of the questionnaire you are still trying to build participants’ motivation to complete the survey and are still gaining their trust. Thus, both demographic questions and sensitive/personal questions should be saved for later on in the survey or at the end of the survey. The most interesting questions and questions that apply to all participants should be at the forefront of the survey.

The questions themselves should be grouped by content and type. Questions should be presented in a logical order to help participants move through the survey without having to continually change topics. The use of color or space between question groupings will also help participants realize that a shift in thinking or content area is required. Questions should be written in a heavy or dark font while the responses should be in a lighter shade or font. The spaces where participants are to provide responses should be consistently used. For example, one column on the page could be reserved for the questions that are asked, while the second column is the consistently used space for participants to provide their answers.

Although blue and yellow has been shown to improve response rates when compared to simple black and white questions, contrast has been proven to be more effective than color. Thus, slightly shading the background of different sections of questions will help improve response rate and will indicate to participants that a specific section of questions is beginning/ending. The use of shading can aid participants’ navigation through the questionnaire thereby improving response rates.

Surveys should be kept as short as possible. This is not to say that important questions should be omitted for the sake of brevity, or than a tiny font should be used to visually shrink the size of the survey. Rather, when designing questionnaires be cognizant of your potential participants’ time, commitment and interest.