Findings Web Probing:
Despite the fact that the instruction in question 13 asked respondents to describe the sector of business or industry in as much detail as possible, most of the answers in all three countries were very short (UK: M = 2.3 words, DE: M = 2.2 words, PL: M = 2.2 words). Even when additionally taking the respondents’ answers on their occupation in Q5 and Q6 into account, it was only possible to assign the 3-digit NACE code to approximately 60% of the respondents (see Table 68).
There were no significant differences in codability depending on country (see Table 68; χ2(4,792) = 9.098,
p = .059), gender or age. However, there were significant differences in the codability of responses based on the working situation of respondents (see Table 69; χ2(4,792) = 12.872, p < .05). Respondents in atypical working situations were most likely to not provide any form of substantive answer. Importantly, however, respondents who were employed were significantly more likely to provide substantive, but non-codable responses (39%,
n = 108) than respondents who were self-employed (30%,
n = 69) or in atypical working situations (31%,
n = 89). This finding, which is consistent with the pattern of response quality to Q5 and Q6, indicates that respondents who are employed are least likely to find their occupation and sector worth a detailed explanation.
Also mirroring the findings from the questions on occupation, responses that could be coded at the three-digit NACE 2.0 level were significantly longer than responses that could not be coded at this level (see Table 70; T(746) = -4.568, p < .001). However, with codable responses to Q13 averaging between two and three words for codable responses, minimum requirements on response length cannot be recommended to improve response quality.
The low share of codable responses suggests that most web respondents either did not read the instruction (thoroughly) or were not motivated to spend much time on typing in more detailed answers. The findings from the cognitive interviews should shed light on this question.
Findings Cognitive Interviews:
Did the answers to the open-ended question allow for classifying the sector of business or industry in which the respondents work in at the 3-digit level of NACE 2.0?
As shown in Table 71, ten of the 16 German respondents provided sufficient information in their answers to question 13 to classify the sector of business or industry at the 3-digit level of NACE 2.0. If we additionally considered the respondents’ answers to questions 5 and 6, it was possible for all but one German respondent to classify the sector at the 3-digit level.
Regarding the Polish respondents, only eight provided sufficient information to classify the sector of business or industry at the 3-digit level of NACE 2.0. If we additionally considered the respondents’ answers to questions 5 and 6, it was possible for a total of twelve Polish respondents to classify the sector at the 3-digit level.
All in all, it was possible to determine the NACE 2.0 code for 84% (
n = 27) of the cognitive interview participants.
Did respondents read the instruction below the question text and did they find it understandable?
All but one respondent (PL12) indicated that they had read the instruction when answering question 13 and most found the instruction “very” or “rather understandable”. However, as mentioned above, the answers of more than half of the respondents (
n = 17) were not coherent with the instruction, suggesting that it needs to be revised.
Three German respondents criticised the grammatical structure of the examples given (with the “not:...” in brackets). This would make it unnecessarily complicated to understand what kind of answer is desired:
- “By ‘not’, do you mean not to write it? I understand it to mean that you want me to be more specific. So, I shouldn't just write such a broad term? […] Perhaps it would be better to turn it around and not write it in brackets but integrate it directly into the sentence. So ‘Instead of factory, please write manufacture...’.” (DE03)
Seven respondents (DE:
n = 3, PL:
n = 4) suggested revising some of the examples to make them clearer and adding or substituting some of them:
- “The example ‘management of real estate on a fee or contract basis’ is already very specific, whereas ‘software development’ is still a very general word and not very meaningful in this case. If for ‘factory’ the specification is ‘manufacture of cutlery’, for ‘software development’ I also expect a more precise specification such as ‘software development for computer games/apps’.”
- “The examples all refer to the economy, so it might be helpful to include something that is not purely economic. Education, for example. That the explanation is a little more diverse.”
- “They are helpful, but I wonder whether it would not be worth having a close look at the industry or sectors which are dominant in the economy (in terms of number of people employed in them), because ‘cutlery production’ is rather detailed and few people would use this hint. I understand that this is supposed to show how detailed the answer should be. Simply, it could be better to name more popular industries.”
Four Polish respondents pointed out that the term “market sector” was confusing in this question, because they associated it with the notions of “private sector”, “public sector”, and “not for profit sector”. The term “market sector” was interpreted much broader than “industry” and using both terms together in the question text confused respondents.
Summary:
- Only about 60% of the web probing respondents gave sufficiently detailed answers to Q13 to assign the 3-digit NACE code, even when combining their answers to Q13 with those to Q5 and Q6. In the cognitive interviews, results were markedly better, with 84% of respondents providing sufficient details.
- Respondents found the instruction clear and helpful, however, more than half of them did not answer in accordance with the instruction. Some suggestions were made on how the instruction could be revised.
- Several Polish respondents were confused by the term “market sector” in the question text, which they associated with the private, public or not-for-profit sector. Using it together with the term “industry” made them wonder what kind of information was required.
