Category-Selection Probing, Difficulty Probing, Specific Probing, General Probing
Information on the aim and purpose of the test:
Question 2 addresses the status of the (primary) survey instrument used in the study at the time of data collection. A distinction is made here between newly developed, previously published, and tested/validated instruments. The question was asked of all participants during the pretest.
The goal of the pretest was to determine whether the response categories were understood as intended, whether respondents found it difficult to answer the question, and whether a response option such as “I do not recall” should be added.
Findings:
All test persons were able to answer the question. Their reasons for choosing a particular answer option were consistent with the option they selected.
- “I took these items from a study that uses repeated items, which have been developed, validated, and published.” (TP10, Answer: published and validated)
- “I’d say ‘published and unvalidated.’ Those were items from the literature, but they weren’t what I consider to be validated. By ‘validated,’ I mean that you weed out items, then perform exploratory and confirmatory factor analysis, and then test the construct validity again, and so on. And that wasn’t done in this case.” (TP02, Answer: published and unvalidated)
- “I didn’t use a scale—for example, from the ESS—that I adopted exactly as is; instead, we based our approach at the time on previous, published studies. But for some items, there were no scales available, which means we had to adapt some of them and, in some cases, design new questions. For that reason, the other options didn’t work for me, especially since we didn’t pretest them either. It’s not as if we conducted any cognitive interviews.” (TP03, response: newly developed but not pretested)
Three participants reported that they found it difficult to answer the question or that they found the answer options unclear. One participant (TP09) explained that in her study, she had used questions from various sources for which multiple answer options were applicable. She therefore suggested allowing multiple answers for this question.
“Some of the questions were taken directly from ALLBUS, ISSP, etc., and some were from the ESS, which we then adapted slightly. And those weren’t tested. I’d need to select multiple options if I were answering for individual questions. For the questionnaire as a whole, I’d stick with the answer ‘newly developed but not pretested,’ because it was developed from scratch and never tested in its entirety. The majority of the questions would be ‘published and validated,’ but some were adapted and not tested.” (TP09, Answer: newly developed but not pretested)
The other two test persons pointed out that the terms “published,” “validated,” and “pretested” were ambiguous and could be interpreted in different ways by respondents.
- “I’m having a little trouble with this ‘validated’ part. Do I have to validate these? Or others? Is that something you know about questions that have been taken from studies, for example?” (TP07)
- “What do ‘validated,’ ‘unvalidated,’ and ‘pretested’ mean here? Many researchers might not immediately know how to interpret these terms. What exactly does it mean for a measurement instrument to be validated? What needs to be done for it to be considered published? I mean, if you develop a new item yourself, it’s easier—you know whether you’ve developed it from scratch and pretested it or not. But when you’re working with existing items, perhaps only the person who published them knows. But what does ‘validated’ mean in that case? For me, that’s actually the key term that isn’t entirely clear.” (TP10)
