We recommend splitting Question 18 into two questions: one asking what (emotional) consequences the rejection(s) have for the respondents personally, and the other asking whether they submitted the paper(s) elsewhere. Question 19 asks whether a resubmission is planned, so this aspect does not need to—and should not—be included in Question 18.
For the first question, we recommend removing the response option “I anticipated the rejection of the paper,” as this does not constitute a consequence. Furthermore, the (recommended) “None of the above” category should be defined as an exclusive response category. Possible wording for the two questions would be as follows:
Question 1:
Does any of the following apply to you as a consequence of the rejection(s) of the paper(s)?
I am frustrated with the journal review process.
I have lost confidence in my work.
I feel embarrassed to reveal that I have performed a study that was not deemed good enough by peers.
I have lost interest in an “unsuccessful” study.
Other:
None of the above
Question 2:
Have you already resubmitted the paper(s)?
Yes, I have resubmitted the paper(s) without major changes.
Yes, I have resubmitted the paper(s) after making revisions according to the suggestions.
No, I have not (yet) resubmitted the paper(s).
Information on the aim and purpose of the test:
Question 18 examines the consequences that the rejection of one or more of the research papers has (had) for the respondents. As with Question 17, it is asked only of those respondents who indicated in Question 16 that all or some of the papers related to the study were rejected.
The purpose of the test was to determine whether the list of response options is complete or whether additional options should be included.
Findings:
All ten participants stated during the cognitive interview that one or more of their research papers had been rejected at some point during the peer-review process. Consequently, all participants were asked Question 18.
The cognitive interviews revealed three issues with the question. First, it is unclear whether the two categories at the end of the answer list (“No, …”) are intended to be mutually exclusive or not—that is, whether it should be possible to indicate both a consequence of the rejection and that the paper(s) were or will be resubmitted. Of the seven test persons who selected at least one of the two “No” categories, six also clicked on one or more substantive categories. Several test persons pointed out that they perceived the top response options and the two bottom ones as belonging to two separate questions.
A second problem is that the two “No” categories are phrased in the present tense and therefore do not apply to situations in which the research papers have already been resubmitted. Two test persons (TP07, 10) used the “Other” category to describe this situation (see Table 16 above).
One test person also pointed out that the response option “I anticipated the rejection of the paper” did not really fit the question, since this was not a consequence (“That’s not a consequence!”; TP01).
