GESIS Pretest Database: Documentation Questions Details - Project „PubBias“ – Questions on assessing the prevalence and extent of publication bias (English Version)

Question Text:

Why did you not include all experimental groups in the paper(s)?

Instruction:

Check all that apply.

Answer Categories:

Some of the analyses were tied to a subsample of the experimental groups, so that not all groups were needed
The experiment was not designed or conducted properly.
The experimental analysis did not produce the expected results.
The experimental analysis did not work.
The experimental analysis has not yet been completed.
I am/ We are still planning a publication with the missing experimental group(s).
Other:

Recommendations:

The results of the pretest indicate that there are no issues with these two questions, so they can be left as they are. Since we recommend visually distinguishing the “Not applicable” response option in Question 4 from the other options, the same should be done for Question 6.

Cognitive Techniques:

Process Probing, Specific Probing, General Probing

Findings for Question:

Information on the aim and purpose of the test:
Question 6 determines whether an experimental design was used in the study and, if so, whether all experimental groups in the study were reported in the research papers. Respondents who answer “no” to Question 6 are then presented with Question 7, which asks why not all experimental groups were included.
The pretest was designed to determine whether the list of response options in Question 7 is complete or whether additional reasons should be included.

Findings:
During the cognitive interview, three participants stated that they had previously analyzed experimental data as part of a research project but had not taken all experimental groups into account. The reasons for this are listed in Table 6 (frequency distribution for Question 7).
All of these participants stated that the list of response options in Question 7 was complete, which was due, among other things, to the inclusion of an open-ended “Other” category (“I think it’s complete, especially since it’s a semi-open-ended question. And it makes perfect sense to ask it as a ‘check all that apply’ question,” TP03).

Question Topic:

Job and career/ Job situation & professional activity

Construct:

Completeness of reporting in experimental designs