Project „PubBias“ – Questions on assessing the prevalence and extent of publication bias (English Version)

Question Text:

Answer Categories:

Recommendations:

Cognitive Techniques:

Findings for Question:

Information on the aim and purpose of the test: 
Question 14 is asked of all respondents who indicated in the previous question that their expectations were (at least partially) not met. The question seeks to determine the nature of the discrepancy between the respondents’ expectations and the actual results. 
The purpose of the test was to determine whether the list of response options is complete or whether additional options should be included. 

Findings:  
Nine of the ten participants were presented with Question 14 based on the filter guidance or their response to Question 13. Two of these participants selected the “Other” category and explained that their research question was (partly) exploratory in nature and that they therefore had no expectations regarding the results (TP02), or that although an expected effect was found, it was not statistically significant (TP06). Test person 06 stated that she would not describe this result as a “null effect.” 

• “In some cases, we didn’t have any specific hypotheses. That can also be the case, for example, when developing a scale. In that sense, you don’t have any expectations either. You just want to see if it’s perhaps two- or three-dimensional, or if the construct is this way or that.” (TP02)
• "To some extent, I’m looking for ‘non-significance’ here. What we expected, of course, was an effect in a certain direction, but we always assume, of course, that this effect would be significant. And that would actually be my answer here. So I’d say that while it’s in the same direction as expected, the results weren’t significant—both for the moderation effect and for one or two main effects we expected. ‘Opposite direction’ and ‘null effect’ are clear terms, though they’re really a bit loaded when referring to an actual null result and not just an effect that’s small or simply not significant. I’m missing “not significant as expected” here.“ (TP06) 

With the exception of Test Person 06, all test persons stated that the list of response options was complete, particularly since an open-ended “Other” category was also provided. However, four test persons (TS03, 05, 08, 10) pointed out that they would have preferred a multiple-choice option for this question.

• “I can only check one box, right? I could have checked several things here. Well, I think I just mentioned the main effect—it went in a completely different direction than expected. But we also had one or two predictors where we thought there would be a certain effect, and they just didn’t have any effect. So I could have checked ‘zero effect’ as well.” (TP05)
• “In my case, I’d say ‘no effect’ right now. Because that fits best. But there could be several things that apply here. Multiple answers should be allowed here.” (TP08)

Question Topic:

Construct: