Findings Web Probing:
Respondents were randomly assigned to one of the three question versions with clarifications shown on demand (ToolTip), to all respondents (via instruction) or without clarification. In total, 62% (n = 491) of respondents reported that they were employed, 38% (n = 300) reported to be self-employed, and only one respondent answered question Q7 with “Don’t know” (see Table 14). There were no significant differences between question versions (χ2(4,792) = 3.022, p = .554) or countries (χ2(4,792) = 3.236, p = .519).
Based on the response distributions, there is no indication whether or in which way clarifications are presented impacts response behaviour. Cognitive interviews were used to gain insights on possible silent misunderstandings, that is an incorrect understanding and subsequent self-assignment of respondents to the answer categories.
Findings Cognitive Interviews:
In the cognitive interviews, respondents were first shown question version 1 (ToolTip) and after discussing this question version, they were shown question version 2 (instructions displayed directly) on a separate screen.
In Germany, nine respondents declared themselves as employees and seven respondents as self-employed. In Poland eight respondents declared themselves as employees and seven respondents as self-employed. Additionally, one respondent (PL15) could not assign himself to one of the options.
While in Germany there were mainly no problems with the understanding of the two terms “employee” and “self-employed”, in Poland different understandings of “employee” were mentioned. This is because in Poland the term for “employee” (PL: Pracownik najemn) can have negative connotations. Therefore, a question was explicitly asked in Poland whether the term “employee” was actually perceived negatively.
Eight Polish respondents stated that the term "employee" had negative connotations for them.
They mostly understood it to refer to working arrangements based on another type of contract than
employment contract (of mandate, specific work) or simple jobs such as being hired, without any
social security or rights:
- “Hired [PL: najemny] means there is a contract, but I would identify it more with blue collar work, with some simple jobs.” (PL05, employee)
- “For me an ‘employee’ is a person who must get money of course, but who works for some limited time only, when needed.” (PL09, self-employed)
This assessment most likely stems from the characteristics of the Polish labor market with very complex regulations which do not protect the weakest players of the labor market.
The other eight Polish respondents defined “employee” in neutral terms, as a person employed
based on any kind of contract or working for someone else’s business, also as a subcontractor.
Apart from the negative connotation of the term “employee” in Poland, two respondents (PL07, PL15) who owned a company with limited liability had problems classifying themselves correctly:
- “I am one of the owners [of a company], so I wonder whether I am self-employed. A limited
liability company is a kind of business where you are not self-employed, you have to sign an
employment contract with such a person. Limited liability company is a family business,
with 50% for me and 50% for my husband.” (PL07, self-employed)
- “On one hand, I am not an employee and on the other hand a self-employed person makes
me think of a one-man business. And I am employed in my own company, but this is a limited
liability company and I do not have to be employed there.” (PL15, don’t know)
Additionally, in Germany two respondents (DE03, DE05) had problems understanding what was
meant by “family workers”:
- “The explanation of ‘self-employed’ is very informative, it's a longer text. Only the last part,
with the family workers, I didn't understand.” (DE03, employee)
- “For example, ‘members of producer’s cooperatives, family workers decide for themselves
which of the answer options best fit their situation’ […] It is a bit unclear what is meant.”
(DE05, self-employed)
Except for two respondents (DE09, PL04), no one used the ToolTips to answer this question. Most of
the participants explained that they knew the answer to this question without looking up clarifications and that it was easy for them to differentiate between employee and self-employed:
- “I understand what is meant by the question. It's very simple, whether I'm self-employed or
employed.” (DE03, employee)
- “I didn’t check it because the answer was clear.” (PL14, self-employed)
In addition, seven respondents (DE: n = 1, PL: n = 6) explained that they did not use the ToolTips because they either did not know they could use it, did not notice it, or forgot about it:
- “I forgot that there is such an option. But I read that such an option is possible at the beginning.“ (PL05)
- “I did not notice that I can move my mouse over it.“(PL06)
In both countries, the explanations of the terms “employee” and “self-employed” in the ToolTips
were found to be informative. In Poland, seven respondents preferred the version with the ToolTips,
eight preferred the version with the instructions shown directly and one respondent (PL02) found
both versions equally good. In Germany, eleven respondents preferred the version with ToolTips
and five the version with the instructions shown directly.
Respondents preferring ToolTips stated that it is easier to answer the question in this format and
that it is clearly arranged:
- “I think the ToolTips are better. It's clearer. You can click on it when you need it. […] With the
alternative version, I immediately think: ‘Oh God, that's a lot to read’.” (DE01, employee)
- “[Otherwise], I would have to read everything, even if it didn't concern me. I find the version
with the ToolTips clearer; you can answer quickly, and you don’t get lost in the explanation.”
(DE16, self-employed)
- “Although I did not check these instructions in ToolTips, I am a visual person. If there is too
much text for me, I just don’t read it.” (PL10, employee)
Respondents preferring the instruction shown directly explained that they would probably oversee
the ToolTips, because they are hardly intuitive:
- “I have to say, at first I thought the ToolTips were cool, they looked cool. But when I compare
it now, it's true that I saw them but ignored them. Here I at least have to read about it, with
the description below the text I find it better.” (DE06, employee)
- “I have the whole message here. I just forgot that the blue ones, the text fragments with blue
highlights, may lead to some explanation.” (PL05, employee)
One respondent (PL06) explained that there is no consistency in how the information is displayed in
the questionnaire, that’s why he overlooked the ToolTips:
- “Explanation directly below question is clearer and I prefer this option. It is also because
the previous questions came with such an explanation below, so that’s what I expected.“
(PL06, employee)
Summary
- In web probing, question format (ToolTips vs. instructions shown directly vs. no instruc-
tions at all) had no effect on respondents’ answers to Q7.
- In the cognitive interviews, the majority of the respondents did not use the ToolTips and
there was no clear preference between the two versions (ToolTips vs. Instructions shown
directly).
- In Poland, the term “employee” was partly understood in a negative way and was under-
stood in different ways. In addition, respondents who had a company with limited liability
had problems classifying themselves.
- In Germany, the definition for “family worker” was not clear to everyone.
