GESIS Pretest Database: Documentation Questions Details - Project „PubBias“ – Questions on assessing the prevalence and extent of publication bias (English Version)

Question Text:

What was the nature of results of the initially planned analyses?

Instruction:

Check all that apply.

Answer Categories:

Lack of statistical significance
Null effect
Opposite direction as expected
Other
I do not recall

Recommendations:

The results of the pretest suggest that the response options for both questions are complete and that no additional options need to be added. However, we recommend rephrasing the response option “Feedback at conferences…” so that it also includes feedback received not at conferences but in other contexts. A possible wording would be:
Feedback from colleagues, peers, or supervisors led to the change in objectives.

Cognitive Techniques:

Process Probing, Specific Probing, General Probing

Findings for Question:

Information on the aim and purpose of the test:
Questions 9 and 10 are asked only of those respondents who indicated in Question 8 that they had deviated in any way from their original research objectives or research questions. Question 9 captures the reasons for the deviation, while Question 10 captures the specific results of the original analyses, provided that, according to Question 9, the respondent decided to take a different approach to analyzing the data based on those results.
The pretest was designed to determine whether the lists of response options in both questions are complete or whether additional options should be included.

Findings:
Seven participants indicated in one of the two versions of Question 8 that they had deviated in some way from their original research objectives or questions. The reasons for this are listed in Table 8 (Frequency Distribution for Question 9).

All of these participants stated that the list of response options in Question 9 was complete; however, this was due in part to the inclusion of an open-ended “Other” category, which three of the seven participants utilized.

Three participants pointed out that the response option “Feedback at conferences…” was too narrowly defined and could be expanded to include feedback from colleagues, co-authors, or mentors:

“The concept of ‘feedback at conferences’ could be generalized. That is, ‘feedback’ or ‘conversations’ in general, not just at conferences.” (TP08)
"I would also include co-authors and supervisors when they provide feedback. The latter would likely be a common occurrence for doctoral students.” (TP04)

Two test persons indicated in Question 9 that the originally planned analyses had not yielded the expected results and that a different evaluation approach had therefore been chosen. These two persons were presented with Question 10 and asked what the results of the original analyses were like (see Table 9). Here, too, both test persons indicated that the list of answer options was complete.

Question Topic:

Job and career/ Job situation & professional activity

Construct:

Reasons for the change in research objectives