In order not to influence the later question, the research design provided for the test persons to first read the introductory page and to continue with the first version of the question without any follow-up questions on the part of the interviewer. After reading, answering and discussing the two versions of the question, the test persons read the introductory page again and were asked follow-up questions about it.
Did the test persons express comprehension problems when reading the introduction?
Two problems became apparent when reading the introductory text. The first problem was that the information from the last paragraph was not processed correctly during the first reading. As a result, the core message that the decommissioning of an emission right means that it is no longer available to power plants, and that in this way an active contribution is made to environmental protection, was not received when the introductory text was read. This problem affected four of the six test persons:
One reason for overlooking the last paragraph could have its origin in the second problem. This is because some test persons remained unclear about the connection between their role as a private individual and the industry's emission output:
Consequently, when reading the first question, one test person expressed uncertainty as to whether she could and should actually decide on this issue (“If I were in the position to decide that now?”, TP 02).
Furthermore, one test person believed that the introductory text was about a current innovation in emissions legislation, which would make emissions rights subject to a charge for the first time:
Do the test persons already know the word "emission right" in advance?
Four test persons stated that they had already known the term emissions law prior to the interview (TP 01, 03, 05, 06). Of these, two test persons stated that they could explain both the term emission rights and emissions trading (TP 01, 05). However, only test person 01 easily gave a definition of emission rights or emission trading that correctly defined both terms and related them to each other:
The other test persons gave explanations that showed a partly correct, partly abbreviated or even incorrect understanding of emission rights or emissions trading:
No test persons indicated that the introductory text contained words unfamiliar to them.
The introduction should be formulated in such a way that it clearly conveys to the respondents that a) the decommission of an emission right represents a contribution to climate protection, and b) they, as individuals, may subsequently decide on the decommission of an emission right.
The core message of the introduction is currently summarized in the last paragraph of the introduction page. Some test persons did not pay enough attention to this paragraph. Others did not take from the paragraph that they, as a private individual, are given the opportunity to decommission an emission right, which in turn has implications for companies.
We recommend that the core message that respondents can choose between a contribution to climate protection (for the public good) and a sum of money (for themselves) be prominently placed at the beginning of the experiment. This can be done, for example, on a separate survey page. Then, on a second page, the respondents are given an explanation (varied in complexity) of the principle of emissions trading.
The mention of the average CO2 consumption of a private household is not necessary for understanding the question and could be deleted to reduce complexity. The current placement also led respondents to fear a shortage of supply. However, since this information is to be retained to serve as an order of magnitude to better estimate personal consumption, we recommend that the sentence be placed in a way that does not create uncertainty about whether the decommission of an emission right could lead to a shortage of personal energy supply. This information could be placed accordingly as an introduction to the experiment.
An implementation could look as follows:
First page:
“Through consumption, electricity consumption, heating and mobility, each German produces an average of eight tons of CO2 per year.
In this survey, you are asked to decide whether you would like to decommission a so-called emission right and thus do something for climate protection, or whether you would like to receive 5 EUR instead.
By decommissioning an emission right, private individuals can make an active contribution to climate protection.
Before you decide, you will find additional background information on emission rights on the next page.”
[„Durch Konsum, Stromverbrauch, Heizen und Mobilität verursacht jede/r Deutsche im Durchschnitt acht Tonnen CO2 pro Jahr.
Im Rahmen dieser Umfrage werden Sie gebeten, sich zu entscheiden, ob Sie ein sogenanntes Emissionsrecht stilllegen und damit etwas für den Klimaschutz tun möchten oder ob Sie stattdessen 5 EUR erhalten möchten.
Durch das Stilllegen eines Emissionsrechts kann man als Privatperson einen aktiven Beitrag zum Klimaschutz leisten.
Bevor Sie sich entscheiden, erhalten Sie auf der nächsten Seite zusätzliche Hintergrundinformationen zu Emissionsrechten.“]
Second page:
“The European Union (EU) wants to quickly and significantly reduce emissions of climate-damaging greenhouse gases (e.g. CO2) and be climate-neutral by 2050. In order to effectively control the emission of these pollutants, large power plants and industrial facilities must acquire a so-called emission right for each ton of greenhouse gas emitted. The amount of emission rights issued by the EU is strictly limited. At the beginning of February 2021, an emission allowance cost around EUR 42 (incl. VAT) per ton of CO2.
As a private individual, one can buy such an emission right and decommission it. If you do this, it is no longer available to power plants and industrial facilities. The power plants can therefore emit one ton of CO2 less. By decommissioning, you make an active contribution to climate protection.”
[„Die Europäische Union (EU) möchte den Ausstoß von klimaschädlichen Treibhausgasen (z. B. CO2) schnell und deutlich reduzieren und bis 2050 klimaneutral sein. Um den Ausstoß dieser Schadstoffe wirksam zu kontrollieren, müssen große Kraftwerke und Industrieanlagen für jede ausgestoßene Tonne Treibhausgas ein sogenanntes Emissionsrecht erwerben. Die von der EU ausgegebene Menge dieser Emissionsrechte ist strikt begrenzt. Ein Emissionsrecht kostete Anfang Februar 2021 etwa 42 EUR (inkl. MwSt.) pro Tonne CO2.
Man kann als Privatperson ein solches Emissionsrecht kaufen und stilllegen. Wenn man dies macht, steht es den Kraftwerken und Industrieanlagen nicht mehr zur Verfügung. Die Kraftwerke können somit eine Tonne CO2 weniger ausstoßen. Durch die Stilllegung leistet man einen aktiven Beitrag zum Klimaschutz.“]
Question: Some test persons felt the need to read the explanations again while reading the question text, but only one test person went back to the introduction page to do so. Therefore, we recommend including a clearly visible information button on the question page that can be used to display the explanations.
In addition, we recommend formulating the question text in such a way that both the choice faced by the respondents and the significance of the individual answer options become clear. This could be done, for example, as follows:
“Today, we offer to purchase an emission right for you, which entitles you to emit one ton of CO2. The costs for the procurement of the emission right are borne by us as the client of the study. You can choose whether you want to decommission the emission right now or whether you want to receive 5 EUR instead.
If you decide to decommission the emission right right now, the industry will have to emit one ton of CO2 less from now on than before.
If you choose to receive 5 EUR, it will be credited to your account via [survey company].”
[„Wir bieten Ihnen heute an, für Sie ein Emissionsrecht, das zum Ausstoß einer Tonne CO2 berechtigt, zu kaufen. Die Kosten für die Beschaffung des Emissionsrechts tragen wir als Auftraggeber der Studie. Sie können wählen, ob Sie das Emissionsrecht jetzt stilllegen oder ob Sie stattdessen 5 EUR erhalten möchten.
Wenn Sie sich dafür entscheiden, das Emissionsrecht jetzt stillzulegen, muss die Industrie ab sofort eine Tonne CO2 weniger ausstoßen als vorher.
Wenn Sie sich dafür entscheiden, 5 EUR zu erhalten, werden Ihnen diese über [Umfrageunternehmen] auf Ihrem Konto gutgeschrieben.“]
Answer format: The answer choices should be explained in the question text or in an instruction below, as described above, to prevent confusion about the combination of answer options.
A good half of the respondents chose the answer "don't know/not specified" in the pretest, although this was for different reasons. In order to be able to distinguish between different triggers for refusing to answer in the survey, we recommend asking respondents who choose "don't know/not specified" a question. A combination of closed answer options and the open text field facilitates the evaluation of such a demand and offers respondents who would like to comment on the content of the topic or their decision the opportunity to do so.
This could look like this:
“You answered the previous question with ‘don't know/not specified’. Why did you choose this answer?
[Sie haben die vorangegangene Frage mit ‚weiß nicht/keine Angabe‘ beantwortet. Wieso haben Sie sich für diese Antwort entschieden?
The list of reasons can be extended as needed.
After reading the introductory text for the first time, the test persons first saw and answered the first version of the question (Control Treatment 1), followed by follow-up questions from the interviewer. Subsequently, they were shown the second version of the question (Control Treatment 2) and answered follow-up questions about it.
Control Treatment 1: For what reasons do test persons choose their response option?
In the first version of the question, half of the test persons chose to decommission the emission right now, and the other half chose "don't know/not specified". With the exception of one test person who stated that she did not understand the question (TP 06), the test persons did not seem to have any problems with the question text.
The three test persons who chose to decommission the emissions rights justified their responses by saying they wanted to do the right thing for the environment:
However, one of these test persons was confused by the answer options and had to reassure the interviewer that she had understood the question correctly before committing to an answer:
Control Treatment 1: Why do test persons choose the "don't know/not specified" category?
Similar uncertainty was evident among the three test persons who opted for "don't know/not specified". The test persons were unsure or even irritated by the combination of answer options. They seemed to have difficulty reconciling the alternatives of decommission the emission right or receiving a monetary reward for not decommission it:
The third test persons expressed anger at the choice of making a seemingly environmentally friendly decision (which she did not think was the right one in terms of climate protection) or a selfish, environmentally damaging one:
She wished there was an additional open response field where she could write in her opinion, as this was not possible with the given response options.
Control Treatment 1: Does the introductory text provide enough information to answer the question?
Three test persons stated that they would have liked more detailed information, but only one of them referred to the explanations on emission rights in the introductory text (TP 01). The other two would have liked the implications of the answer options to be explained in more detail, i.e., what the consequences of deciding to decommission the emission right would be or why they should receive five euros for not decommission it (TP 02, 06). These test persons both chose to answer the question with "don't know/not specified." Another indication that these two test persons had not taken the relevant information from the introductory text and the question text was that both answered the question as to who would bear the costs for the emission right that this was the respondent.
Control Treatment 1: How easy or difficult was it for the test persons to answer the question?
All three test persons who had decided to have the emission right decommissioned stated that the question had been "very easy" for them because it was a decision in the interest of climate protection:
The three test persons who answered the question with "don't know/not specified", on the other hand, stated that they found the answer "rather easy" (TP 05) or "rather difficult" (TP 02, 06). Unknown or unexplained terms did not play a role here, but rather the content of the question or the combination of answer options:
Do test persons find it easier to understand Control Treatment 1 or 2?
Three test persons found Control Treatment 2, including the time dimension, easier to answer than the first version. One test person reasoned that the additional option of giving industry another year reinforced her decision to have the emission right decommissioned immediately ("It's definitely easy, but I think [the second version] is better because you can make more decisions. If I had a year now to decommission the [emission right] and give industry more opportunities to emit CO2, I think the answer of decommissioning the [emission right] immediately is better." , TP 01). However, another test person found the second version easier only because she had not read the first version completely (TP 06).
Two test persons found Control Treatment 1, i.e., without the time dimension, easier to answer because they had to think more in Control Treatment 2 (TP 04) or because they were confused by the additional answer option whose meaning was not explained (TP 03). One test person found both versions equally good (TP 05).