Findings Web Probing:
Respondents were randomly assigned to one of the three question versions. Responses did not differ significantly between the question versions (χ2(8,792) = 2.939,
p = .938).
Responses differed significantly between countries, with German respondents being generally more likely to work in joint private-public organisations or companies and less likely to work in the public sector than UK and Polish respondents (χ2(8,792) = 39.212,
p < .001).
Moreover, the chosen sector differed based on the respondents’ working status. Self-employed respondents were significantly more likely to choose “private sector” than respondents who were employed or in atypical working situations (χ2(8,792) = 59.962,
p < .001). Of the nine respondents who chose the answer category “Other, namely:”, eight were self-employed or in atypical working situations.
The open-ended answers of the respondents who named another sector showed that certain groups of respondents had difficulty naming their sector. For one, low-qualified workers sometimes had difficulty recognizing that they work in the private sector. This applied to a cleaning aid working in a private household (which the respondent did not conclude to be the private sector) and a respondent who stocked shelves at a (presumably private, for-profit) food retailer. Secondly, some self-employed or freelancers who seemed to receive contracts from different sectors did not seem to realize that their own business was part of the private sector. One respondent owned a business as a gardener, growing and selling plants. Another respondent worked as a detective. A third worked freelance as an alternative, non-medical practitioner. Finally, some respondents working in sectors that can be run privately, publicly, or jointly seemed uncertain in which sector their business was active. These respondents worked in health care, a pharmaceutical company, oil and gas exploration, and a cultural institution.
Did respondents answer correctly?
Directly following the closed survey question (which sector of the economy respondents work in), a probe asked respondents to describe in their own words which sector they work in (P1_Q14). These responses were used to check whether respondents had answered the survey question correctly. One in ten respondents gave a non-substantive answer to the probing question (10%,
n = 78), that is, they left the text field empty or inserted random characters or other non-codable content. In 76% (
n = 600) of all cases, respondents’ answers to the open-ended probe coincided with their survey responses, that is they either clearly confirmed that respondents had chosen the correct sector or at least gave no reason to assume otherwise. In 7% of responses (
n = 55), the responses were ambiguous, making it impossible to determine whether a respondent had classified the sector correctly. For instance, one respondent who chose “private sector” worked in waste collection but did not specify whether he/she worked for the municipality or a private sub-contractor. The same applied to other respondents who worked in other sectors that are partially public, partially private, such as health or education, and who did not provide details on their organisation’s structure. In 5% (
n = 38) of responses, respondents had clearly chosen the incorrect sector. A common misunderstanding was that respondents whose work included dealing with the general population, such as workers in supermarkets, chose “public sector”. Examples of this misunderstanding are
“I work in a supermarket”, “I work with the customers who want help with the house and/or garden” or
“client service, car diagnostics”. Finally, 3% of respondents (
n = 21) answered that they didn’t know which sector they worked in (
“I have my own gardening business and I am the only person working for this business so maybe this could be privately owned”).
Based on the analysis of the probing question, there were no significant differences in the share of correct responses to Q14 depending on the question version, that is, whether clarifications of the sector were shown (1) on demand via ToolTips, (2) to all respondents alongside the response options, or even if (3) no clarifications were presented (χ2(8,792) = 8.747,
p = .364). Likewise, there were no significant differences in the correctness of the survey responses by country (χ2(8,792) = 4.739,
p = .785).
However, the correctness of the response significantly differed depending on the respondents’ answer to the survey question (χ2(12,783) = 239.394,
p < .001). In other words, respondents who indicated that they worked in the private sector were most likely to have chosen the correct sector (85%,
n = 486). Respondents who had chosen the public, joint private-public or not-for-profit sector were significantly more likely to have given an incorrect response (for instance by misunderstanding public sector to mean that their customers are the general public), or to at least be so ambiguous in their description of their sector that their response remained unclear.
Did respondents’ confidence that they chose the correct answer differ depending on the question version?
Confirming the results of the analysis of the open-ended probe, there were no significant differences in respondents’ confidence ratings depending on question version (χ2(6,792) = 3.109,
p = .795) or by country (χ2(6,792) = 9.739,
p = .136).
Respondents who had indicated that they worked in the private sector were significantly more likely to be “very sure” of their response as compared to respondents who chose one of the other sectors (χ2(3,792) = 11.238,
p = .011).
Findings Cognitive Interviews:
In both countries, about half of the respondents (DE:
n = 7; PL:
n = 9) categorized themselves as working in the private sector, while most of the others answered that they worked in the public sector (DE:
n = 6; PL:
n = 4). In Germany, one respondent changed his answer from “in the private sector” to “in a joint private-public organization or company” while answering, and two respondents did not know where they should categorize themselves. In Poland, two respondents classified themselves as working in the not-for-profit sector. One respondent did not understand the term “sector” and finally answered “other” and inserted that he worked in the “construction sector” (PL12).
Did respondents answer correctly and were they confident regarding their answers?
The majority of respondents classified themselves correctly (DE:
n = 12; PL:
n = 14). In Germany, two self-employed respondents referred to their main clients (e.g., publicly run schools or theatres) rather than their own business, causing them to incorrectly answer that they worked in the public sector or joint private-public organisation. Moreover, another self-employed respondent wanted to select both the private and public sector in which his clients work, and only chose the correct answer “private sector” because the question layout forced this. Two other respondents could not classify themselves at all. One of them worked as a nurse, in an institution that could be in the public or private sector, while the other respondent clearly worked in the private sector at a manufacturer.
In Poland, two respondents reacted overwhelmed by the question and the definitions. One of them incorrectly answered with “public sector”, despite clearly working in the private sector as a security guard. The other respondent finally chose the “other” category, explaining that he worked in “construction”.
Most respondents (DE:
n = 12; PL:
n = 14) claimed to be “very confident” that their answers were correct. Among the six respondents who were not “very confident” of their response were the two German respondents who could not answer the question. Four respondents were not confident regarding the company’s ownership, because the company had a complex structure (DE05, DE06, PL16) or they had just started working for it (DE01).
Finally, two self-employed respondents (DE09, DE15) answered that they were very confident regarding their answers, but addressed problems defining whether they should indicate the sector of the clients they work for or their own company:
- “I can't answer that. Why? Because I work for all kinds of sectors, several companies. So, for me it would be the private sector, but also state-owned enterprises and public-private enterprises.” (DE09, public sector)
- “Difficult, that's a question I find hard to answer. If I classify my own company, then I work in the private sector […] When I look at my clients, I work in many countries and in many sectors, but I don't think that's what is meant here.” (DE15, private sector)
Having multiple jobs or businesses did not cause confusion regarding this question. All respondents who had multiple jobs or businesses correctly referred to their main job or business only.
Were the ToolTips used?
The majority of respondents read at least one of the ToolTips. However, respondents in Germany were more likely to read at least one definition (DE:
n = 15; PL:
n = 10). The respondents who did not use the ToolTips explained that they did not need them because they knew the definitions of the sectors or could clearly assign themselves to one sector:
- “I knew the definitions of the sectors, so I didn’t need this help.” (DE09)
- “It was clear for me. I did not have to use the instruction.” (PL01)
Six respondents (DE:
n = 4; PL:
n = 2) read the ToolTips for selected sectors only. In all cases, these respondents were fairly certain of their response, but wanted to check the correctness of their answer. Some respondents did this by reading through the definition of their chosen sector, while others read through sectors that they were not as familiar with:
- “I immediately knew that my sector is the ‘public’ sector. I also knew immediately what ‘not-for-profit’ sector means. Regarding the other [sectors], I wanted to be sure whether they mean what I believe they mean.” (DE10, public sector)
- “I only read through the sector that applies to me. I didn’t bother to read the others.” (DE11, public sector)
Three respondents reported technical difficulties using a tablet or because the mouseover boxes with the dark background and white font strained their eyes.
Were the definitions in the ToolTips perceived as informative?
Four respondents (two from each country) criticized that the definitions of the sectors were too abstract and suggested adding examples to the clarifications:
- “I don't get these explanations at all. It would be good if examples were given so that one could better imagine what is included [in which category] […] I don't know whether it is the public sector or the private-public sector.” (DE02, don’t know)
- “The explanations are sufficient, but perhaps an example could be added to each of them.” (PL11, public sector)
In Poland, one respondent specifically criticized that the definition of the “not-for-profit” sector was not comprehensible (PL01), and another noted that the definitions should also include how to classify “foundations”, as they can contain both private and public funding (PL06).
Did respondents prefer receiving clarifications via ToolTips or alongside the response options?
Respondents showed a preference for the clarifications to be shown alongside the response options (DE:
n = 11; PL:
n = 9). The reasons for this were that the definitions were well visible, and the definitions could be compared to each other:
- “Since I don't understand at all what all these terms mean, I think it would be better if the explanations were displayed directly.” (DE02)
- “The definitions are available right away and easier to use, I can compare them right away.” (PL06)
- “This [version] seems more straightforward to me. All explanations appear at once. [In the other version] you have to do something to look for these explanations.” (PL13)
Four respondents from Germany and six from Poland preferred the version with ToolTips. These respondents explained that the ToolTips led to less text being displayed on the screen at once and that the format did not force them to read unnecessary information:
- “The great thing about the ToolTips is that as soon as I go into another field, they disappear, and it doesn't feel so overloaded.” (DE06)
- “I've already gotten used to the ToolTips; they are easier on the eyes, less exhausting.” (DE12)
- “In simple working situations, you do not need to look at them because you know immediately. I understand that if someone works in several sectors and cannot make up their mind, these hints are necessary.” (PL14)
In Poland, one respondent was indifferent towards the format. In Germany, one respondent suggested a mix of the two formats, with the clarifications being shown on demand in the response options (DE09).
Summary:
- Based on both web probing and cognitive interviews, self-employed respondents had more problems categorizing themselves than employed respondents.
- Moreover, respondents who reported that they worked in the private sector generally answered correctly, while respondents who claimed to work in the public sector were more often subject to a false understanding of the term public sector.
- Some respondents had difficulties understanding the clarifications of the sectors and would have preferred examples alongside the explanations to illustrate the economic sectors.
- The cognitive interviews revealed that most respondents preferred the clarifications of the sectors to be shown alongside the response options. Besides the personal preference, there were no differences in the correctness of the answers depending on the question format (based on web probing).
