PaCo - Mechanisms of panel conditioning in longitudinal surveys: Questions on social desirability, gender roles, and environment (English Version)

Befund zur Multi-Item-Skala Empfehlungen zur Multi-Item-Skala

Do the test persons have problems understanding and/or answering the question?

All test persons answered question 14 using almost the entire width of the scale. None of the test persons asked spontaneous questions about the question text or showed any other signs of ambiguity.

For both questions, three test persons selected a value to the left of the middle of the scale, indicating that a person would "rather not" or "not at all" be embarrassed (scale values 1 to 3) to admit the respective action publicly. Two test persons chose the mean value (scale value 4), and five test persons chose a value indicating that this was "rather" or "very embarrassing" (scale values 5 to 7).

Eight of the ten test persons used the same side of the response scale when answering the items, i.e. they either chose a value to the left or right of the middle both times or the middle value both times. The two test persons who changed the side of the scale gave the following reasons for doing so:

• • TP 02 responded that admitting to pornography use was "rather embarrassing" (scale score 5) because the question spoke of "very frequent" use; she, on the other hand, rated admitting to stealing as "rather not embarrassing" (scale score 2) because she herself admitted to having stolen a piece of gum once as a child, and estimated that most people have such experiences and could admit to it openly as an adult.
  • TP 09 rated publicly admitting to pornography use as "not at all embarrassing" (scale score 1), because this is allowed to be said much more openly nowadays than when she was young; admitting to theft, on the other hand, she rated as "very embarrassing" (scale score 7), because this is an offense to the general public.

Do respondents understand that this is about assessing how desirable characteristics are in society, not for them personally?

The construct of the question was social desirability, i.e., the test persons were asked to indicate how desirable these characteristics are in society and thus to provide an assessment of social acceptance. Follow-up questions were used to test whether test persons erroneously stated exclusively their personal judgment regarding the actions.

Six of the ten test persons clearly and consciously distinguished between these two facets when answering the question (TP 01, 02, 03, 05, 08, 10). These test persons all chose a scale value between 2 and 6, i.e. one of the possible gradations next to the scale endpoints.

• • Test person 10 stated that it was "rather embarrassing" (scale value 5) to admit to viewing pornographic content in public. She justified her answer by saying that this was "still a bit of a taboo subject. When asked whether she would also personally find it embarrassing to admit this in public, she answered in the negative. (“No. So, if I did that, I would have no problem admitting that in public. I could stand by it.”).
  • The two test persons who had chosen the mean explained this by saying that there were both some people who would be very embarrassed and others who would not be embarrassed: “I think it varies from case to case, and that a few people tend to be more closed off about this issue, and a few people tend to be more open. That's why I couldn't say that as a blanket statement.” (TP 01)

One test person initially answered the question only in relation to herself but noticed and corrected her mistake directly when answering the probing question (TP 06). When completing the questionnaire, she stated that it was "very embarrassing" (scale value 7) to admit to frequently viewing pornographic content in public. On second read-through, she remarked, “Due to this, I have to say now, I don't think I answered that correctly. I started from myself.” However, this would have changed her answer only slightly: “Due to that, I should actually have taken 6. I'm very embarrassed, but whether everyone feels that way, I don't know. Above all, I don't know how it is with young people, whether it has changed. But I do think that when people are asked about it due to that, they tend not to admit it.”

Two test persons clearly answered the question based exclusively on their personal feelings and did not include the public's view in their answers. Both selected extreme values on the scale:

• • “Now I [started] from myself. I think [it's] normal to talk about it, and I don't have any problems talking about it. It may be that many people have problems with it, but I don't find it embarrassing at all.” (TP 04; Answer: Scale value 1)
  • “I'm starting from myself now. If one of my acquaintances or friends said, 'I often look at pornographic content,' I would find that embarrassing. Yes, definitely.” (TP 07; Answer: Scale value 7)

Finally, one test person distinguished whether most people would be embarrassed to admit viewing pornographic content from whether society accepts it (TP 09). The test person herself would not be embarrassed to admit this. In her estimation, however, most people would still be embarrassed because they would be afraid of the judgment of others. However, the attitude of society has changed in recent decades to become more tolerant. For this reason, the test person chose the value 1 "not embarrassing at all", because "most people would [indeed] be embarrassed, but it is not actually embarrassing".

Question: Although some test persons answered the question not from the point of view of society but for themselves personally, we recommend leaving the question as it is. In our opinion, there is no problem of understanding here that could be solved by an alternative formulation of the question. Rather, some of the respondents seem to have difficulty reflecting on the attitude of society as a whole, due to the fact that this is a relatively complex cognitive task. On the basis of the cognitive interviews, no statement can be made as to how large the proportion of respondents is who cannot or do not want to perform this cognitive task. 

Response options: No changes recommended.