(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Systematic mapping of climate and environmental framing experiments and re-analysis with computational methods points to omitted interaction bias [1]

['Lukas Fesenfeld', 'Department Of Social Sciences', 'University Of Bern', 'Bern', 'Department Of Humanities', 'Social', 'Political Sciences', 'Eth Zürich', 'Zürich', 'Liam Beiser-Mcgrath']

Date: 2024-02

Ambitious climate policy requires acceptance by millions of people whose daily lives would be affected in costly ways. In turn, this requires an understanding of how to get the mass public on board and prevent a political backlash against costly climate policies. Many scholars regard ‘framing’, specially tailored messages emphasizing specific subsets of political arguments to certain population subgroups, as an effective communication strategy for changing climate beliefs, attitudes, and behaviors. In contrast, other scholars argue that people hold relatively stable opinions and doubt that framing can alter public opinion on salient issues like climate change. We contribute to this debate in two ways: First, we conduct a systematic mapping of 121 experimental studies on climate and environmental policy framing, published in 46 peer-reviewed journals and present results of a survey with authors of these studies. Second, we illustrate the use of novel computational methods to check for the robustness of subgroup effects and identify omitted interaction bias. We find that most experiments report significant main and subgroup effects but rarely use advanced methods to account for potential omitted interaction bias. Moreover, only a few studies make their data publicly available to easily replicate them. Our survey of framing researchers suggests that when scholars successfully publish non-significant effects, these were typically bundled together with other, significant effects to increase publication chances. Finally, using a Bayesian computational sparse regression technique, we offer an illustrative re-analysis of 10 studies focusing on subgroup framing differences by partisanship (a key driver of climate change attitudes) and show that these effects are often not robust when accounting for omitted interaction bias.

Funding: This work was supported by the Swiss National Science Foundation as part of the project "Same, same but different? Multiplex networks in Swiss and German Climate Mitigation Policy" (100017_188950 to LF). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Building on previous maps of the environmental framing and communication literature more broadly [ 2 , 3 , 6 , 40 – 43 ], we provide a systematic mapping of the experimental framing literature in this field and a critical appraisal of framing effects across population subgroups reported in existing studies. The lack of publicly available replication data and the large variation of experimental designs and outcome variables present in the literature do not allow for conducting a meaningful meta-analysis that compares standardized effect sizes across different framing experiments. Thus, instead of conducting a meta-analysis, the key contributions of this paper are twofold: First, we provide a systematic mapping of existing framing experimental research on climate and environmental framing and present results of a survey with authors of these studies. Here, we show that most studies report significant subgroup effects but do not use advanced methods to check for the robustness of subgroup effects. Second, we thus illustrate the use of Bayesian sparse regression and computational methods for assessing the robustness of heterogeneous framing effects and preventing potential omitted interaction bias. Given the prominence of discussions about the robustness of partisan framing effects across ideological subgroups, we re-analyze data from a set of published studies using (different from most published work to date) LASSOplus, a machine-learning-based Bayesian sparse regression method. This more advanced computational method reduces potential overfitting of statistical models and the omitted interaction bias that can lead to non-robust and misleading subgroup framing effects in classical OLS regressions [ 38 , 44 , 45 ].

Empirical evidence for the effect of framing is often generated through experiments embedded in survey-, field-, or lab studies. Such experiments, in which study participants are randomly assigned to treatments and control conditions, are seen as a gold standard for assessing the effectiveness of frames in altering public opinion [ 2 , 3 ]. Typically, in such experiments, study participants are randomly confronted with differently framed messages. The aim is to assess how these different framing treatments alter respondents’ climate beliefs, attitudes, and behaviors, particularly in comparison across population subgroups. For example, Bernauer and McGrath [ 5 ] as well as Bain et al. [ 10 , 29 ] randomly assigned individuals to different messages that either emphasized the risks of failing to combat climate change (control frame) or highlighted different co-benefits of climate mitigation, such as economic, community building, and health benefits (treatment frames), to study if framing climate mitigation policy around co-benefits instead of risks increases public support. Others, such as Hung and Bayrak [ 1 ], have randomly varied specific terminology (e.g., climate change vs. climate crisis) used in climate communication to assess how such re-labeling affects citizens’ perceptions and attitudes across different population subgroups. While many researchers presume such framing to be an effective communication technique for altering mass public opinion and behavior concerning climate change [ 2 , 10 , 12 , 29 – 31 ], some scholars have expressed doubts [ 5 , 21 , 32 – 36 ]. These scholars argue that on salient and contested issues, such as climate change, people are likely to hold relatively stable, consciously formed preferences and cannot be easily manipulated through simple framing [ 5 , 21 , 32 – 36 ]. Some also suspect a bias against reporting non-significant effects in the current framing literature [ 22 , 36 ]. They point to the use of experimental designs and statistical methods that involve risks of producing noisy effects with low external validity and omitted interaction bias, especially when studying heterogeneous framing effects across population subgroups [ 22 , 37 – 39 ].

Most framing studies on climate and environmental communication look at framing effects across population subgroups (i.e., heterogeneous framing effects). According to directional-motivated reasoning models [ 2 ], framing political messages around prior beliefs and values can reduce cognitive dissonance [ 24 , 25 ] and lead to stronger framing effects on individuals’ attitudes. For example, empirical studies have shown that individuals perceive frames tailored to their ideological core beliefs as less threatening. Accordingly, many studies (especially in polarized political contexts such as the United States) assume that frames aligned with citizens’ ideologies and party identification are more effective at altering climate policy attitudes [ 2 , 26 – 28 ].

Climate and environmental communication is an essential lever for building public understanding of the problem’s severity and increasing support for climate change (policy) solutions [ 1 – 3 ]. One widespread climate and environmental communication technique is framing [ 4 – 6 ]. Framing is an inherent part of communication and occurs when actors use messages to alter people’s preferences by changing the presentation of an issue or an event [ 2 , 7 , 8 ]. In climate policy, politicians or other stakeholders may emphasize specific subsets of preexisting arguments–such as economic or health-benefits of climate change mitigation [ 5 , 9 , 10 ]–in an attempt to influence public opinion in favor of (or against) climate action (so-called emphasis framing). Communicators may also use different, but logically equivalent phrases to describe climate change mitigation (so-called equivalence framing). Yet, is framing an effective communication technique to alter public opinion about climate change–especially across different subgroups of a population? Many studies on climate and environmental communication suggest that framing can effectively influence public opinion across population subgroups as it safeguards individuals’ identities by appealing to their existing values and prior beliefs [ 11 – 16 ]. Framing theory holds that the effectiveness of framing in altering people’s attitudes varies according to whether the related information is available in individuals’ memories, is accessible, and is evaluated as applicable in a given situation [ 7 , 17 ]. The framing literature also builds on a bounded rationality model [ 18 ] and often assumes that citizens have limited capacity to process information systematically [ 7 , 19 – 21 ]. From this perspective, individuals use frames as heuristics to minimize cognitive effort when forming policy attitudes [ 17 , 22 , 23 ].

Results

Our systematic mapping is based on the PRISMA identification standard [46] and best-practice guidelines for systematic mappings [40, 47] (for details, see Methods). After a scoping analysis of the existing experimental framing literature in climate and environmental communication studies (for details, see Methods), we find that there exists a clear lack of experimental research on equivalence framing (i.e., logically equivalent but different descriptions of the same issue) and thus focus our review here on experimental emphasis framing studies (i.e., frames varying specific subsets of preexisting arguments). In essence, while equivalence framing could in principle be a powerful communication technique for altering people’s climate and environmental attitudes and policy preferences [2, 48, 49] (e.g., by altering messages on whether a person has a 10% risk of dying or a 90% chance of surviving due to climate-induced extreme weather), in the context of real-world climate and environmental communication and respective research appears to be a less prominent and studied framing strategy. We thus focus our mapping and critical appraisal on emphasis-framing experiments.

In total, we identified 121 emphasis-framing experimental studies published in 46 peer-reviewed journals between 2007 and 6/2020. All studies use an experimental design to assess the effects of different types of emphasis framing treatments on an individual’s climate and environmental beliefs, attitudes, and behaviors (see Methods Table B in S1 Text for the complete list of studies). While most studies considered in our mapping relate specifically to climate change, some studies also include treatment groups and dependent variables related to other environmental issues, such as air pollution. We decided to include all these studies to increase the scope of our findings. According to the experimental stimuli used in these 121 studies, we classified them into six climate and environmental emphasis framing research categories. These are issue and solution frames, value- and norm-based frames, re-labeling frames, psychological distance frames, consensus and uncertainty frames, and source cue frames (for further details, see below and in Table A in S1 Text).

Potential risk of over-reporting significant framing effects Our first goal is to systematically map existing emphasis framing experiments on climate and environmental issues. Fig 1 provides an overview of our mapping (for further details, see Methods and Table B in S1 Text). Approximately 92% (n = 111) of the framing studies we reviewed report significant main framing effects. Only 7% (n = 9) report non-significant main effects, and 1% (n = 1) do not report any main effects. Around 20% (n = 24) of all studies do not report and discuss any heterogeneous treatment effects (e.g., interactions between participants’ characteristics, such as party ideology, and framing treatments). In contrast, 70% (n = 85) of all reviewed studies identify at least one significant subgroup effect, while 10% (n = 12) report no significant subgroup effects at all. In other words, 88% (n = 85) of all studies that report on heterogeneous treatment effects (n = 97) find some significant subgroup effects. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 1. Overview of 121 emphasis framing experimental studies in the field of climate and environmental politics, economics and psychology published between 2007 and 6/2020. Note: Panel designs (i.e., repeated measurements for the same study participants at two or more points in time) are used to study whether framing effects vary over time (e.g. how long the effect of a one-time exposure lasts). Competing frames are used to emphasize competing arguments in a debate (e.g., pro and contra climate mitigation messages). https://doi.org/10.1371/journal.pclm.0000297.g001 Besides the overview provided in Fig 1, we also observe some temporal trends in the data (for further details, see Table B in S1 Text and Fig B-F in S1 Text). First, the total number of published emphasis-framing experimental studies per year has substantially increased since 2007. While in 2007 there was only one published framing experimental study, in 2019 we identified 18 framing experimental studies in our mapping. Over the review period, the share of studies that employ large-n (n>1000), non-convenience and population-representative samples has increased. However, while the size and representativeness of study samples has grown over time, studies have rarely reported statistical power calculations, especially for estimation of sub-group effects. Thus, while an increasing number of studies report significant subgroup effects (see Fig E in S1 Text), there is a risk that the estimation of these effects is underpowered. Moreover, the vast majority (120 out of 121) reviewed studies use classical linear or logistic regression models to estimate main and sub-group effects. Only one of the reviewed articles has used more advanced statistical methods, such as Bayesian sparse regression methods, to control for potential omitted interaction biases and double-check the robustness of estimated heterogenous effects. Below, we discuss the purpose and application of these methods in more depth (see “Critical appraisal and re-analysis of framing studies”). Survey-based experiments have experienced the largest growth rate over the review period–from 0 studies using this study design type to 16 studies using survey experiments in 2019. In contrast, field- and lab-based experiments or the combination of field and lab-based experiments with survey experiments stays at a very low level (around 0–1 studies per year). While US-focused studies have the largest overall share of all experiments, they also experienced the strongest average growth rate per year between 2007 and 2020. Yet, over time also the number of (internationally) comparative framing experiments increased–but at a lower average growth rate per year. Most of these comparative framing experiments compared the US with another country, often from Europe. Even though in recent years some framing experiments have been conducted in developing and emerging economies, such as Brazil, China, or India, we still see a substantial lack of framing experiments in the developing country context. Moreover, we can identify some trends in terms of the framing types being studied (see Table A in S1 Text and Fig B in S1 Text). In the first period from 2008 to 2013, psychological distance frames belong to the most widely studied framing types. For example, some of these earlier framing studies [50, 51] varied the spatial, social, and temporal distance of climate and environmental impacts to assess whether people support ambitious mitigation more when they perceive climate and environmental change as a proximate problem. From 2013 onwards, issue and solution frames were the most widely studied frames, with a peak of up to 10 of these experimental studies published in 2016. Issue and solution frames often emphasize environmental risks and co-benefits of environmental protection or climate mitigation. For example, some studies [10, 29] in this category highlight that emphasizing co-benefits of climate mitigation (such as technological innovation, green jobs, community building, or health improvements) could foster public support for ambitious mitigation policies. The second most widely studied framing type in our sample of reviewed studies, especially since 2013, are value- and normative-based frames that emphasize values and social norms, and attribute responsibility for environmental problems and solutions. In recent years, also source-cue frames, varying the sender of a message, and re-labeling frames, changing specific words (e.g., global warming vs. climate change) in a message, have been studied more widely. In contrast, research interest in frames varying the degree of consensus or uncertainty about climate change existence and impacts has decreased over time. Finally, the number of articles including published replication data has increased since 2007 but none of these studies were preregistered. In the last three years, the majority of published framing experiments makes replication data publicly available. The number of articles that reported non-significant main or heterogeneous treatment effects has also increased, especially since 2013. For example, before 2013 none of the reviewed studies reported any non-significant heterogeneous framing effects, while in 2019 at least four studies did so. However, the number of reported non-significant effects has increased only very slowly, and most published studies still report only significant framing effects.

Author survey suggests that bundling non-significant and significant effects eases publication One concern that arises in view of the large proportion of studies finding statistically significant framing effects is that there may be a file-drawer problem, where mainly significant effects are published, and non-significant ones not [22]. We implemented an online survey to assess how the authors of published framing experiments experienced the publishing process and dealt with non-significant framing effects they encountered (for further details, see Section VI in S1 Text). We contacted all 173 authors of the 121 publications via email and received a total of 63 responses (a response rate of 36% of all authors and around 52% of all reviewed studies; most often the lead author of each study responded to our survey). We find that around 80% (n = 50) of all respondents also identified non-significant effects in their framing experiments. Around 76% (n = 38) of this subset of authors tried to publish their results, including non-significant effects in peer-reviewed journals. And again, only 63% (n = 24) of this sub-subset of authors who tried to publish non-significant effects (or 48% of survey respondents that identified them) were able to successfully publish studies with non-significant effects. However, according to these authors, in most cases, publishing their findings was only possible when non-significant results were bundled together with other significant effects (for further details, see Methods). Therefore, the observed gap between the small number of published non-significant framing effects (see Fig 1 above) and the substantially larger number of identified non-significant framing effects reported by the surveyed authors strongly suggests a potential publication bias towards significant treatment results.

Lack of publicly available data makes it difficult to formally assess file-drawer problem Formally assessing the potential existence and magnitude of a ‘file-drawer problem’ would require public access to the data and re-analyses of the original study results as part of a meta-analysis. However, only 23% (n = 28) of the 121 articles we reviewed made their data publicly available. In addition, out of those 93 reviewed articles whose data was not published, we obtained data for 29 studies by contacting authors via email (i.e., overall, we could not get access to the data of more than 53% (n = 64) of all reviewed studies). The large number of experiments that report significant framing effects without publishing data or making replication data available on request thus raises significant barriers for researchers attempting to assess the robustness of published results. For example, extra and often unsuccessful efforts to obtain access to data increase the costs of systematically re-analyzing existing studies, assessing their results’ robustness, and estimating the size of the potential file-drawer problem. Finally, the lack of publicly available data and large variance in experimental designs prevents meaningful meta-analyses on the distribution and magnitude of average framing effects. For example, while some experimental designs use a (placebo) control group, others only compare effects for different framing groups. Also, the manifold larger and smaller variations in treatment wording and design make a proper comparison of effect sizes in a meta-analysis very difficult. Besides systematic mapping of the work in this area, we thus focus our illustrative critical appraisal on one key area of interest to many researchers: heterogeneous framing effects.

Making inferences about framing effects by sub-group As mentioned above, climate and environmental communication researchers are often interested in how framing effects vary across population subgroups. Druckman and McGrath [2], for instance, note that “rather than continually testing the impact of one frame after another, the literature would benefit from […] investigating which types of messages resonate in light of motivations and particular prior beliefs, values and identities.” For example, in view of the possibility of directional-motivated reasoning, one prominent argument in the climate and environmental communication literature is that frames aligning with peoples’ prior beliefs reduce cognitive dissonance [24, 25] and are thus more effective at shifting public opinion about climate change. Researchers, therefore, typically split their sample into groups based upon respondent characteristics and then re-estimate their statistical models to assess, for example, whether the framing effect is more (or less) significant for Democrats or Republicans in the United States. While this is a valid approach for generating descriptive insights regarding variation in treatment effects, researchers occasionally slip into the use of causal language. For instance, some of the reviewed studies state that “issue frames can lead Republicans and those on the political right to view climate change policy as less important” [52] or highlight that “Republicans […] increase their support if Republican politicians take leadership roles in supporting proposed bills, and if air quality benefits are emphasized” [53]. Crucially, however, random assignment of treatment does not guarantee the identification of heterogeneous causal effects [54]. In this section, we therefore outline descriptive and causal inference in the exploration of sub-group effects and the fundamental challenges to inference through the example of omitted interaction bias. We also examine how incorporating additional interaction effects in a manner suggested by previous research [38, 44, 45, 55] affects the estimates of previously published subgroup effects. Finally, we reflect on these challenges in the pursuit of causal inference for sub-group effects, while also highlighting recent work that allows for more principled descriptive examination of sub-group effects. One example of a threat to causal inference from considering sub-group effects in isolation is omitted interaction bias [38, 44, 45, 55]. Omitted interaction bias occurs where differences between the sub-groups on other characteristics, such as age, education, and income, also result in heterogeneous treatment effects that are left unmodelled, which are absorbed by the included interaction effect. While such sub-group analysis is valid for generating descriptive inferences about how treatment effects vary across these individuals, the typical approach of researchers is to interact the treatment with the sub-group of interest, which is vulnerable to omitted interaction bias. The lack of randomized experimental manipulation and/or adjustment for other potential heterogeneous treatment effects diminish the ability of researchers to draw causal inferences about how treatment effects vary across subgroups. In the existing literature, however, researchers often draw causal inferences from the results of sub-group analyses, where the sub-group membership is not randomized and/or other sub-group effects are left unmodelled. For example, some studies conduct sub-group analyses where they estimate treatment effects separately for Democrats and Republicans. However, party identification is not a trait that can be randomly assigned. Thus, not accounting for other characteristics that may also moderate the effect of treatments associated with party identification (e.g., age, education, gender) could lead to biased estimates. However, instead of making a descriptive statement about significant differences in framing effects between Democrats and Republicans, many studies make causal inferences and policy recommendations about which frames most successfully shift support across different partisan groups [53]. As many sub-group characteristics of interest, such as partisan identification, are difficult if not impossible to manipulate experimentally, researchers likely need to adjust for other heterogeneous framing effects in order to make causal claims. Yet, several studies [37–39, 56–58] have shown that standard specification choices and statistical methods (e.g., ordinary least squares [OLS] regressions) can run the risk of producing non-robust and noisy heterogeneous framing results because of overfitted models, even in perfectly randomized experiments [37, 38, 54]. If researchers wish to exclude this potential risk to their causal inferences, then they need to assess how sensitive published heterogeneous framing effects are to model misspecification. Note however, that these may not be all potential threats to causal inference, see for instance Bansak [54] for a generalized framework for estimating causal moderation effects and possible methods for conducting sensitivity analysis in this context. Recent research points towards the use of machine learning in estimating such heterogeneous framing effects across population sub-group effects and prevent omitted interaction bias [38, 44, 45, 55] (for further details, see Methods). With uncertainty about the true data generation process, the number of relevant heterogeneous treatment effects and conditional effects amongst relevant covariates results in large numbers of parameters to be estimated. In such circumstances, OLS estimation faces problems of statistical efficiency. Machine learning methods for variable selection, such as Lasso, overcome this problem by setting parameters with little predictive power to zero. This improves statistical efficiency by excluding non-meaningful heterogeneous effects, thereby reducing the set of heterogeneous effects estimated. As our systematic mapping (see above) shows, the vast majority (120 out of 121) reviewed studies used classical linear or logistic regression models to estimate subgroup effects and does not make use of more advanced methods to assess or mitigate omitted interaction bias, such as LASSOplus [38]. To re-assess published framing effects along these lines, we employed one such method, LASSOplus [38]. This estimator was chosen as it is explicitly tailored to estimating heterogeneous treatment effects, as is often the goal of framing studies, with substantial simulation evidence showing its improvement upon OLS for this task. LASSOplus has also been shown to be a superior approach compared to other sparse regression techniques that seek to overcome overfitting and omitted interaction bias [38]. LASSOplus allows for simultaneous estimation of sub-group effects for all included pretreatment covariates (e.g., age, education, income, and gender) and regularizing insignificant effects to avoid overfitting (for further details, see Methods). Many studies covered by our mapping and review focus on the politically polarized country case of the United States (see Fig 1) and examine how framing effects vary by respondents’ partisanship. Hence, we decided to illustrate the sensitivity of heterogeneous framing effects by assessing the robustness of partisan subgroup effects for studies with publicly available data. From the 28 studies with publicly available data, 10 studies focus on partisan subgroup effects in the US context. We thus concentrated our re-analysis of partisan subgroup effects on these 10 available studies.

[END]
---
[1] Url: https://journals.plos.org/climate/article?id=10.1371/journal.pclm.0000297

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.