Project „PubBias“ – Questions on assessing the prevalence and extent of publication bias (English Version)

Question Text:

Answer Categories:

Alternative Answer Format Tested:

Recommendations:

Cognitive Techniques:

Findings for Question:

Information on the aim and purpose of the test: 
Question 8 assesses the extent to which respondents pursued the originally planned research objectives or questions. The two versions of the question that were tested differ in terms of the response scale used. 
The purpose of the test was to determine which of the two versions works better or is preferred by the test persons.

Findings: 
In both versions, the participants placed themselves at the upper end of the response scale and explained that they had either not deviated at all from their original goals or research questions, or had deviated only slightly. 

• “We’ve kept the research questions virtually unchanged. And the analyses we’ve now carried out actually correspond exactly to what I had envisioned back then.” (TP04, Answer Question 8, Version 1: Definitely yes)  
• “The proposal we had written for this was vague about what we wanted to achieve with this study. So we did pursue the goal outlined in the proposal, and of course we fleshed it out further, expanded on some parts, and adjusted the direction a bit.” (TP02, Answer Question 8, Version 1: Mostly yes)  
• "Well, for the most part, the objectives and the research questions are already in place. But we haven’t actually performed the exact analyses we had planned.” (TP08, Answer Question 8, Version 1: Mostly yes) 

Half of the test persons stated that they preferred the first version of the question, while the other half preferred the second version. The first version was preferred primarily because of the additional use of “yes” and “no,” which made the scale clearer for some respondents. The second version, on the other hand, was preferred because the middle category was more clearly defined and the scale was shorter. 

• “Well, I thought the first version was better because it still included ‘yes’ and ‘no.’ That way, it was somehow clearer whether you agreed or disagreed. I found that more intuitive.” (TP05)
• “A clear preference for the second version, since the intermediate categories in the first version lack distinctiveness. The categories ‘Somewhat yes’ and ‘Somewhat no’ could mean anything or the same thing, respectively.” (TP01)
• “I think the second version is a little clearer because we have five categories here instead of six. You don’t have to think as hard about it; it’s a little easier to answer.” (TP06) 

With the exception of one test person (TP04), all respondents chose the same answer option—that is, either the first or the second—for both versions of the question. Test person 04 chose the answer “Definitely yes” for Version 1, but “Mostly” for Version 2, explaining her choice as follows: 

“I think ‘definitely yes’ means: Yes, I definitely agree with the statement that I pursued the original goals. Although I would also include the idea that I’ve pursued them about 98 percent of the way, not necessarily 100 percent. By ‘completely,’ I mean that I did exactly what I had envisioned back then. So on the second scale here, I would say ‘mostly.’” (TP04)

Question Topic:

Construct: