Findings Web Probing:
As shown in Table 64, 70% (
n = 551) of the respondents provided sufficient information in their answers to questions 5 and 6 to allow their occupation to be classified at the 4-digit ISCO-08 level. There were no significant differences in response quality between countries (χ2(4,792) = 1.981,
p = .739). Respondents aged 50 or older were significantly more likely to provide codable responses (75%,
n = 234) than younger respondents (66%,
n = 318) (χ2(4,792) = 13.816,
p = .008). Self-employed respondents (79%,
n = 180) were more likely to explain their occupation in more detail and thus to provide codable answers than respondents who were employed (67%,
n = 186) or in atypical working situations (65%,
n = 186) (see Table 65; χ2(4,792) = 21.164, p < .001).
Little surprisingly, the mean number of words was higher in response to Q6, which required respondents to describe the main tasks of their (main) job, than to Q5, which asked them to name their job title. Moreover, responses that could be coded at the 4-digit ISCO-08 level were significantly longer than responses that could not be coded at this level (see Table 66; Q5: T(763) = -3.774, p < .001; Q6: T(763) = -7.497, p < .001). However, even codable responses to Q5 merely averaged slightly over two words, meaning that many responses of only one to two words length could be successfully assigned an ISCO code. Minimum requirements on response length cannot be recommended.
Findings Cognitive Interviews:
Did the answers to the open-ended questions allow for classifying respondents’ occupations at the 4-digit ISCO-08 level?
As shown in Table 67, all 16 German respondents and 11 Polish respondents provided sufficient information in their answers to questions 5 and 6 to allow their occupation to be classified at the 4-digit ISCO-08 level.
Did respondents read the instructions in both questions (Q5 and Q6) and how understandable did they find them?
With the exception of two Polish respondents (PL01, PL06), who argued that they found no need to read the instructions because the questions were clear, all participants indicated that they had read the instructions when answering the two questions. Most found both instructions “very” or “rather understandable”. Two problems arose regarding the instruction for question 5 (job title).
Firstly, participant DExx rated the instruction “rather not understandable”, because he was not sure whether he had answered in the desired level of detail. In his opinion, the instruction was not clear, especially because he felt that the two terms in the first example (forwarding merchant / commercial employee) were very similar:
- “I don't know, this ‘forwarding merchant and not commercial employee’? Yes, ok, I can see now that one term is a little bit more specific. In the case of ‘machine fitter instead of worker’ I think it's clearer because I think it's easier to see what's intended. But I'm not sure whether I should have written more, maybe. I could have gone on to write, for example, ‘project planning engineer in the electricity grid’ or something like that. So, I was kind of puzzling over it. The explanations didn't really help me.” (DExx)
Secondly, participant DExx pointed out that the example for civil servants was not well chosen, as the title “Studienrat/rätin” (study council) was rather unspecific (and thus the opposite of “exact” as requested by the instruction):
- “At first, I would probably have typed in ‘teacher’. It says here that if you are a civil servant, you should enter the official title. In detail, I would have written ‘Gymnasiallehrer’ (grammar school teacher), whereas ‘Studienräte’ (study council) can be all kinds of teachers. It's not so clear to me. That's why I wrote in both. Because I'm not sure.” (DExx)
Did respondents find it difficult to answer both questions in an open-ended format?
Three Polish respondents, who were older and less educated, reported problems with writing down their professional activities (Q6) on their own. The other respondents (in both countries) had no difficulties answering the two questions in an open-ended format.
However, many of the Polish respondents found it difficult to answer both questions due to three reasons. Firstly, eleven participants said it was unclear whether the questions referred to their profession by education or their current occupation, which in four cases were not related at all:
- “It is not clear for me. My occupation by education is a manager. I graduated from marketing. But the instruction to the question I read define what I should really consider regarding my present position and occupation practiced. That’s why I typed this one. But in fact I am a manager and this is not really related to my current position.” (PL)
- “I wonder whether this could be a profession by education or the one I do? The description here refers to the job I do, so that’s what I write here.“ (PL)
Secondly, five respondents reported difficulties because of the variety, flexibility, or complexity of their tasks:
- “This was rather difficult. Sometimes I work as a researcher and technology engineer, sometimes as a scientist and then as a lecturer.” (PL)
- “When you are self-employed and your business has flexible profiles, it is hard to define occupation clearly.“ (PL)
Thirdly, three self-employed respondents reported difficulties in defining their occupation and describing their professional activities because they were unsure whether to focus on activities related to company management (reporting, controlling, advertising) or their area of market activities (e.g., making furniture,; fitting appliances).
Three participants (two German, one Polish) indicated that they had difficulty deciding how much text they were supposed to enter in the text box(es):
- “This question is quite broad. I have a lot of essential tasks, so it might be good if the question mentioned a specific number of activities you are supposed to type in. […] I don’t know how much I should write in here.” (DE)
- “In both questions, I felt I wrote in more text than I was supposed to. I practically wrote a whole line in each row and I wasn’t quite sure if that was okay or if you should do it very concisely.” (DE)
- “It would be more comfortable, if we have one box for each activity.” (PL)
One German participant stated that she did not know the exact title of her (unskilled) job and therefore simply entered “jobber”:
- “To be honest, I don't know what it's called in my case. But I definitely understand the question. […] I'd have to look it up in my employment contract. It was some English term. I don't know.”
Did respondents prefer other question formats (e.g., with closed answer options) to indicate their occupation?
Participants were presented with the “search tree” implemented in the Survey Codings Occupation Database and asked to select their occupation from the options displayed. Most respondents had difficulty finding their occupation among the categories presented and it took them a long time to select an option. With the exception of one participant, who immediately identified his occupation (“Transport, logistics, port, airpoirt” → “Driver” → “Courier”), none of the German respondents could find their actual occupation in the search tree. In the Polish sample, another 12 participants indicated that they could not find their correct occupation in the search tree.
Most respondents (
n = 19) selected the occupation closest to their actual one, one German respondent only selected a job at the top level of the search tree (“Care, children, welfare, social work”), and four respondents only selected a job at the second level of the search tree (e.g., “Construction, fittings, housing” → “Fitting”). Four German respondents did not select any category because they felt that none of the categories fitted their jobs. In the following, we present examples of two of the problems mentioned above:
- Occupation closest to actual one selected: “I’m looking for ‘geriatric nurse’ under the tab ‘Care, children, welfare, social work’, but it seems it doesn’t belong here. Then I’ll look under ‘Health care, paramedics, laboratory’. Then I would select ‘Nursing’ and then… there is no ‘geriatric nursing’ here. Then I would probably select ‘Certified nurse’ because everything else doesn’t fit.”
- No fitting category found: “I looked under ‘education, research, training’, then ‘Secondary teaching’, because I assume to find my occupation ‘grammar school teacher’ here. But I can’t find my profession. Or am I blind? But I don’t know under which other category I should look. [Respondent reads through all categories]. My profession must be under ‘Education’. It is not listed under ‘Secondary teaching’ and it is not ‘Primary teaching’. There are teachers for vocational education listed, but there is just nothing for general schools. It all refers to vocational teachers. That doesn’t fit. I can’t find anything.”
Three German respondents not only had problems finding the right occupation in the search tree, but also found the functionality of the search tree unintuitive and did not realise that they could click on the categories to go to more specific subcategories.
When asked which question format (open-ended questions vs. search tree) respondents would prefer to indicate their occupation, the majority of participants (
n = 26) stated that they would prefer to answer two-open ended questions. They justified this preference firstly because they could not find their occupation in the search tree, and secondly because they could answer more quickly and accurately if they typed in their job title and job description rather than having to read an extensive list of options. One respondent had no preference as to the question format and the five respondents who preferred the search tree (DE:
n = 2; PL:
n = 3) argued that – even if they had difficulty finding their occupation – this format could in principle avoid typing errors and guaranteed that respondents answered in the way intended by the researchers and with the desired level of detail.
Summary:
- Most respondents in the cognitive interviews, and the majority of respondents in web probing provided enough detail in their answers to Q5 and Q6 to allow for classifying their occupations at the 4-digit ISCO-08 level (even when their answers were limited to very few words).
- Most respondents found the instruction clear and helpful, but some suggestions were made on how to re-word or substitute the examples given.
- Many Polish respondents had difficulties answering Q5 and Q6 because a) they found it unclear whether the questions ask about their learned profession or the one they currently perform (which were sometimes very different), b) they had several equally important jobs and were unsure on which to report, and c) some self-employed participants were unsure whether to report activities related to company management (reporting, controlling, advertising) or related to their area of market activities (e.g., making furniture).
- Most respondents had difficulty finding their correct occupation in the search tree provided by the Survey Codings Occupation Database of the project Social Sciences & Humanities Open Cloud (SSHOC) and therefore preferred answering open-ended questions on their occupation.
