Regarding the question format, we recommend retaining the two open-ended questions to capture the occupational status.
Q5 (job title)
Q6 (job description)
Findings Web Probing:
As shown in Table 64, 70% (n = 551) of the respondents provided sufficient information in their answers to questions 5 and 6 to allow their occupation to be classified at the 4-digit ISCO-08 level. There were no significant differences in response quality between countries (χ2(4,792) = 1.981, p = .739). Respondents aged 50 or older were significantly more likely to provide codable responses (75%, n = 234) than younger respondents (66%, n = 318) (χ2(4,792) = 13.816, p = .008). Self-employed respondents (79%, n = 180) were more likely to explain their occupation in more detail and thus to provide codable answers than respondents who were employed (67%, n = 186) or in atypical working situations (65%, n = 186) (see Table 65; χ2(4,792) = 21.164, p < .001).
Little surprisingly, the mean number of words was higher in response to Q6, which required respondents to describe the main tasks of their (main) job, than to Q5, which asked them to name their job title. Moreover, responses that could be coded at the 4-digit ISCO-08 level were significantly longer than responses that could not be coded at this level (see Table 66; Q5: T(763) = -3.774, p < .001; Q6: T(763) = -7.497, p < .001). However, even codable responses to Q5 merely averaged slightly over two words, meaning that many responses of only one to two words length could be successfully assigned an ISCO code. Minimum requirements on response length cannot be recommended.
Findings Cognitive Interviews:
Did the answers to the open-ended questions allow for classifying respondents’ occupations at the 4-digit ISCO-08 level?
As shown in Table 67, all 16 German respondents and 11 Polish respondents provided sufficient information in their answers to questions 5 and 6 to allow their occupation to be classified at the 4-digit ISCO-08 level.
Did respondents read the instructions in both questions (Q5 and Q6) and how understandable did they find them?
With the exception of two Polish respondents (PL01, PL06), who argued that they found no need to read the instructions because the questions were clear, all participants indicated that they had read the instructions when answering the two questions. Most found both instructions “very” or “rather understandable”. Two problems arose regarding the instruction for question 5 (job title).
Firstly, participant DExx rated the instruction “rather not understandable”, because he was not sure whether he had answered in the desired level of detail. In his opinion, the instruction was not clear, especially because he felt that the two terms in the first example (forwarding merchant / commercial employee) were very similar:
Secondly, participant DExx pointed out that the example for civil servants was not well chosen, as the title “Studienrat/rätin” (study council) was rather unspecific (and thus the opposite of “exact” as requested by the instruction):
Did respondents find it difficult to answer both questions in an open-ended format?
Three Polish respondents, who were older and less educated, reported problems with writing down their professional activities (Q6) on their own. The other respondents (in both countries) had no difficulties answering the two questions in an open-ended format.
However, many of the Polish respondents found it difficult to answer both questions due to three reasons. Firstly, eleven participants said it was unclear whether the questions referred to their profession by education or their current occupation, which in four cases were not related at all:
Secondly, five respondents reported difficulties because of the variety, flexibility, or complexity of their tasks:
Thirdly, three self-employed respondents reported difficulties in defining their occupation and describing their professional activities because they were unsure whether to focus on activities related to company management (reporting, controlling, advertising) or their area of market activities (e.g., making furniture,; fitting appliances).
Three participants (two German, one Polish) indicated that they had difficulty deciding how much text they were supposed to enter in the text box(es):
One German participant stated that she did not know the exact title of her (unskilled) job and therefore simply entered “jobber”:
Did respondents prefer other question formats (e.g., with closed answer options) to indicate their occupation?
Participants were presented with the “search tree” implemented in the Survey Codings Occupation Database and asked to select their occupation from the options displayed. Most respondents had difficulty finding their occupation among the categories presented and it took them a long time to select an option. With the exception of one participant, who immediately identified his occupation (“Transport, logistics, port, airpoirt” → “Driver” → “Courier”), none of the German respondents could find their actual occupation in the search tree. In the Polish sample, another 12 participants indicated that they could not find their correct occupation in the search tree.
Most respondents (n = 19) selected the occupation closest to their actual one, one German respondent only selected a job at the top level of the search tree (“Care, children, welfare, social work”), and four respondents only selected a job at the second level of the search tree (e.g., “Construction, fittings, housing” → “Fitting”). Four German respondents did not select any category because they felt that none of the categories fitted their jobs. In the following, we present examples of two of the problems mentioned above:
Three German respondents not only had problems finding the right occupation in the search tree, but also found the functionality of the search tree unintuitive and did not realise that they could click on the categories to go to more specific subcategories.
When asked which question format (open-ended questions vs. search tree) respondents would prefer to indicate their occupation, the majority of participants (n = 26) stated that they would prefer to answer two-open ended questions. They justified this preference firstly because they could not find their occupation in the search tree, and secondly because they could answer more quickly and accurately if they typed in their job title and job description rather than having to read an extensive list of options. One respondent had no preference as to the question format and the five respondents who preferred the search tree (DE: n = 2; PL: n = 3) argued that – even if they had difficulty finding their occupation – this format could in principle avoid typing errors and guaranteed that respondents answered in the way intended by the researchers and with the desired level of detail.
Summary:
