Project „PubBias“ – Questions on assessing the prevalence and extent of publication bias (English Version)

Question Text:

Answer Categories:

Alternative Answer Format Tested:

Recommendations:

The second version of Question 12 was preferred by the majority of test persons and was described as more unambiguous than the first version. However, several test persons pointed out that when using Version 2 of the question, little variation in respondents’ answers is to be expected: Nearly all would likely select “Yes, partially,” as this is a common result in social science studies. We therefore recommend using a combination of the two question versions or response scales and removing the term “main” from the question. In addition, we recommend adding a residual category that allows respondents to indicate that they have not performed any significance tests. This could be relevant, for example, for respondents who use Bayesian statistical methods instead of null hypothesis significance testing: 

Did you find statistically significant results when analyzing the data (by whatever standard you use to interpret statistical significance)? 
Yes, all results were statistically significant 
Yes, most results were statistically significant 
Yes, but only some results were statistically significant 
No, none of the results were statistically significant  
I/We did not conduct tests of significance

Cognitive Techniques:

Findings for Question:

Information on the aim and purpose of the test: 
Question 12 assesses whether the results of the central analyses were statistically significant. The two versions of the question that were tested differ in terms of the response scale used.  
The goal of the testing was to determine which of the two versions works better or is preferred by the test persons. 

Findings: 
In both versions, the participants placed themselves at the upper end of the response scale and explained that they had found (for the most part) statistically significant results. 

• “In this study, we’re looking for mechanisms of action, and we’re finding them in the data. Not as strongly as expected, but we’re definitely finding what we’re looking for.” (TP01, Answer Question 12, Version 1: Definitely yes) 
• “I’d say ‘Mostly yes’ right now. We have several papers. We have several AVs. So it’s a diverse range of analyses that we’ve conducted. And I’d say that, for the most part, we found what we expected. We didn’t find everything we expected.” (TP08, Answer Question 12, Version 1: Mostly yes)
• “We only had a few significant results.” (TP02, Answer Question 12, Version 1: Somewhat yes) 

With the exception of one participant (TP08), all participants stated that they preferred the second version of the question because the response scale was clearer and made it easier to answer the question. 

• “That fits much better. We found some stark differences, but in some cases none at all. I find the ‘Yes, partially’ option here much easier to work with than the first version.” (TP03 Answer Question 12, Version 1: Definitely yes; Answer Question 12, Version 2: Yes, partially)  
• “I definitely think the second one is more appropriate here, because it’s relatively clear how to categorize oneself. And there isn’t as much room for interpretation between the response categories.” (TP04, Answer Question 12, Version 1: Mostly yes; Answer Question 12, Version 2: Yes, partially)  
• “‘Partially’ is exactly the term I needed—that is, ‘in part.’ This version is much simpler.” (TP09, Answer Question 12, Version 1: Somewhat yes; Answer Question 12, Version 2: Yes, partially) 

Test person 08 pointed out that the response scale in question version 2 did not provide enough differentiation; three other test persons also noted this.

• “‘Yes, partially’ could mean all sorts of things. There isn’t enough variation on the scale.” (TP08)  
• “Almost everyone here will probably select ‘Yes, partially.’ Simply because that’s the typical finding in the social sciences.” (TP01)  
• “The second one is much easier to answer, but of course it yields much less precise results. As a respondent, I’d prefer that; as a researcher, it naturally depends on what you’re trying to find out.” (TP02) 
• “Sure, you’ll probably get a lot of ‘yes, partially’ answers with this version. But in terms of cognitive effort, it’s already better—much easier.” (TP06)

Question Topic:

Construct: