Halfway through collecting survey responses for your master’s thesis, you fix a setting in the form. In Excel, you see blanks for the online booking outcome. An archived copy of the form shows that the earlier version skipped this question for everyone who tried to book online. You can keep those students’ valid answers to the earlier questions and report booking outcomes for those who provided one. Start by putting the two versions side by side. Then use the records you kept to match each row to the version the respondent saw.
Check each question affected by the change. Record who should have seen it, who was shown it and who answered. A mistake in how the form moves people between sections can limit the result for one question while leaving their earlier answers usable.
Check what respondents saw
- Keep the evidence. Save the original export, both versions of the form and any records of when the change was made. A screenshot of the current settings shows how the survey works today. Use an archived copy or a screenshot from before the fix to confirm the earlier rule.
- Follow the path for each answer. A filter question determines which section a respondent sees next. In both versions, trace each option up to the next question shown to everyone. Record the exact wording, response options and order. The Statistical Office in Poznań’s guide explains how to plan and check these links between questions.
- Assign a version where the evidence allows. A version code in the export or a separate form with its own response file can help. If you edited the same form, the submission time alone can mislead because someone may have opened the earlier form before the change. Mark rows you cannot confirm as “version unknown”. A blank cell by itself cannot tell you which version someone saw.
If a row contains an answer to a question that should have been hidden, check the version assignment and the mapping between questions and spreadsheet columns. If the inconsistency remains, flag that response as unresolved for the question you are analysing and keep the original entry. The U.S. Census Bureau’s standard for checking survey data includes checking that respondents follow the expected paths and looking for conflicting answers.
An Excel example with one routing error and two results
In this fictional study, students describe their most recent attempt to book time in a university recording studio in the previous 30 days. For the first question, they choose “online”, “in person” or “I did not try to book”. The second asks, “Did that attempt result in a booking for the time you wanted?” Students can choose “yes”, choose “no” or leave it blank. Those who did not try to book should skip the second question.
A routing error in Form A also sent people who chose “online” past the second question. After spotting the problem, the researcher launched a separate Form B with the corrected rule and identical questions and response options. Every route in either form led to the same required question after the booking section. Only submitted forms were counted, both exports were kept, each student responded once, and recruitment and collection worked the same way in both versions.
Count the responses in each version
- Form A, 24 people. Eight tried to book online. None saw the second question. Ten tried in person and gave eight “yes” answers and two “no” answers. The other six did not try to book.
- Form B, 26 people. Twelve tried to book online. Nine answered “yes”, two answered “no” and one left the question blank. Eight tried in person. Five answered “yes”, two answered “no” and one left it blank. The other six did not try to book.
In a working copy of the spreadsheet, retain the answers to both questions and add two columns, “Version” and “Question 2 status”. Mark the eight online blanks in Form A as “not shown due to error”. Mark the two blanks among people who tried to book in Form B as “shown, no answer”. For the 12 people who did not try to book, use “skipped as intended”. Mark each recorded “yes” or “no” as “answered”.
Calculate booking method and booking outcome separately
- Method of the last attempt in the full sample. Both versions asked the same first question before the routing error could affect anyone. Twenty of the 50 participants tried to book online (40%). To describe only the people who attempted a booking, use 38 as the base. In that group, 20 out of 38 tried online (52.6%).
- Outcome of online attempts. In Form B, filter for “online” and count nine “yes” and two “no” answers. Nine of the 11 people who answered secured the time they wanted (81.8%). Of the 20 people who tried online, eight never saw the question and one left it blank. Dividing nine by 20 gives the share with a recorded “yes” in a group that also includes people whose outcome is unknown. That would be misleading as a booking success rate.
- Outcome of in-person attempts. Both forms showed the question in the same way to this group. You can report 13 “yes” answers among the 17 people who answered across both forms (76.5%). One of the 18 people who tried in person left the question blank.
If you compare booking methods, use Form B for both groups. The figures are nine out of 11 (81.8%) for online attempts and five out of seven (71.4%) for in-person attempts. Both groups saw the same question during the same period. Students chose their own booking method and the groups are small, so these figures describe the observed answers without establishing which method works better.
Check whether the meaning changed as well
In the example, the fix changed only who saw the question. If Form B had asked about securing “any available time” in place of “the time you wanted”, a “yes” would mean something different. Some people may have booked an alternative time. Report that question separately for each version, even if the columns have the same name. A CBOS report on holiday trips describes a similar comparability issue after changes to the scope of questions and the period they covered.
After fixing a typo, check that the question’s meaning, response options, order and intended respondents stayed the same. If the range of answers or the period in the question changed, record the difference before calculating any results. When neither version can give you the result you need for the full sample, narrow what you report and discuss with your supervisor whether further data collection is needed.
Record the change in your methods and the counts in your results
Methods. Form B was launched after a routing error was identified between the question on booking method and the question on booking outcome. In Form A, students who tried to book online were not shown the outcome question. Booking method was reported for both versions together. The percentage of successful online attempts was calculated among those who answered the question in Form B.
Results. Twenty of the 50 participants reported attempting to book online. In Form B, nine of the 11 students who reported the outcome of their online attempt said they secured the time they wanted (81.8%). Of the 20 who tried online, eight did not see the question because of the routing error and one left it unanswered. The result covers the 11 students who answered in Form B.
Check that the counts in your methods, results, chart captions and conclusions match. If a caption says “n = 50” but the chart describes 11 people from Form B, correct the caption. In an appendix, show both versions or identify the exact rule that changed and which questions it affected. See also How to describe a research instrument in your thesis: an original questionnaire and an existing scale.
Questions about errors and study limitations
Can I describe a calculation error as a study limitation?
Correct a fixable error first and check its impact on other results. A limitations paragraph does not replace correcting a denominator, coding rule or formula. If information cannot be recovered, report the gap and its consequences without inventing values.
How can I connect a limitation to its effect on the conclusions?
Identify a specific feature of the procedure, the finding or claim it affects, and the consequence. Then explain what you actually did to address it and what remains uncertain. A generic list mentioning sample size and time pressure is not enough.
Sources
- Urząd Statystyczny w Poznaniu (2024). Jak przygotować i przeprowadzić badanie ankietowe? Poznań, pp. 1-2.
- Badora, B. (2013). Wyjazdy wypoczynkowe Polaków. Fundacja Centrum Badania Opinii Społecznej, research report BS/15/2013, p. 1.
- U.S. Census Bureau (2021). Statistical Quality Standard C2: Editing and Imputing Data, requirements C2-3.1 and C2-4.
Discuss your results after a change to the survey form
Gather the survey versions, an anonymised export and your research questions. Tell us what changed and where the issue shows up in your chapter. We can agree on which parts of the data, calculations and write-up need checking.
