Proceed with caution: challenges in embedding the precautionary principle in Paris-aligned benchmarks

By Malgorzata Olesiewicz, data scientist, SI research and Jaakko Kooroshy, global head of SI research 

Climate regulations now require the precautionary principle to be respected. But what does it mean? 

“Principle 15: Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation.”

This precautionary principle meant to ensure that measures which could prevent serious environmental damage are not dismissed only because the damage itself is uncertain.[1] The principle was first put into practice in the 1970s within German environmental legislation[2] and later popularised through a number of international agreements[3] in particular the United Nations 1992 Rio Declaration.[4] Three decades on, it has been incorporated into a host of regulatory frameworks, overseeing everything from genetically modified crops, X-rays, and asbestos to the pharmaceutical approval process.[5]

In 2020, the European Commission introduced the principle into the sustainable finance regulatory framework including for climate indexes, as labelled ‘Paris-aligned’ or ‘climate transition’ benchmarks (PABs and CTBs)[6]. To apply these labels, among other criteria, PABs and CTBs Regulation prescribes that benchmark administrators “shall formalise, document and make public” how the precautionary principle has informed estimation methodologies for any input data used in index construction.[7] In this blog, we attempt to shed some light on the complex questions faced by index providers in interpreting and implementing the principle in the context of benchmarks. 

The precautionary principle in the context of corporate carbon emissions

The principle is particularly relevant for corporate emissions estimates because steady emissions reductions (-7% on an annual basis) are key to meeting the regulatory requirements of these climate benchmarks, and a significant share of corporates do not disclose emission data.[8]

But the EU legislative arsenal (e.g., Article 13 in the EU Commission Delegated Regulation) doesn’t specify what a reasonable application of the precautionary principle looks like in this context or for other regulatory use cases, leaving practitioners to devise their methodology and criteria.[9] This ambiguity creates challenges: how can carbon emissions estimations reflect the precautionary principle and, more importantly, what are appropriate modelling assumptions to make? 

Reducing underestimation and underreporting

A prominent approach suggested in the literature is the 95^th quantile estimation,[10] which effectively assumes that the non-reporting companies have worst-in-class emissions. With disclosure currently still voluntary in most jurisdictions, gaps might not be random since companies themselves decide whether to disclose their data (i.e., self-select).[11],[12] The approach, therefore, assigns the 95^th percentile of the carbon intensity distribution of reporting peer companies to non-reporting, rather than the usually preferred average or median (50^th percentile) to ensure that corporate emissions data is not underestimated. In other words, the estimation method aims to incentivize companies to report their emissions and to report them accurately.

This approach seems straightforward, but it implicitly assumes a relatively orderly distribution of the underlying data. In the case of corporate carbon emissions, however, investors must contend with large volumes of unreported data and very high variances within peer groups producing statistical distributions with very long tails. For example, for the FTSE All World, in 2020 the 95^th quantile of reporting companies by ICB Subsector was on average 10 times[13] higher than the median.

In this set-up, choosing the 95^th percentile replaces large volumes of unreported data with extreme values. While this does ensure that values are rarely underestimated, it can come with unintended consequences, including on average significantly overestimating emissions which often end up being an order of magnitude higher than actual emissions or peer group averages.

Moreover, to provide more reliable estimates, carbon estimation models often have to trade off larger sample sizes for more homogenous peer groups with much smaller sample sizes (choosing the Sector rather than Subsector for example creates a larger, but also much more heterogenous peer group with the 95^th quantile on average being pushed to 21 times the median). More granular peer groups indeed help the accuracy of the estimates, but also makes calculations of the 95^th percentile (and the estimates based on it) very volatile year-on-year as they are often driven by disclosures from just one or two firms. 

In a nutshell, the 95^th percentile approach turns out to be impractical for corporate emissions estimates, given the large number of non-reporting companies, high degrees of heterogeneity in reported data and sample size limitations, resulting in unrealistically high estimates for non-reporting companies that tend to be volatile year-on-year. This may be especially problematic for several investment use cases where total emissions of the universe are of interest or where the ranking of the companies depends on the distribution of the entire universe.

An alternative, but ultimately similar approach suggests using – in addition to the 95^th percentile – a range of other percentiles to generate a distribution of estimated values.[14] The authors of this alternative approach suggest that the mean of the distribution should be considered the most accurate estimate but offer investors to choose higher quantiles of the distribution to comply with the precautionary principle at their discretion.

While this approach may be preferable to a more simplistic 95^th percentile approach, it ultimately side-steps the question of what an appropriate implementation of the precautionary principle looks like. Effectively this forces data users, rather than data providers to determine how the precautionary principle should be implemented.

Addressing some of these challenges, FTSE Russell’s approach for implementing the precautionary principle for emissions estimates relies on three premises.

Firstly, the principle is implemented at each step of the dataset curation. This means that its application should not be limited only to the actual estimation methods, but also the sourcing and screening of the input data and guard not only against under-estimating of non-reported data, but also against underreporting by disclosing companies.  In practice, we detect underreporting by first checking if all the required data has been disclosed (i.e., materiality) and then by checking their quality using proxy data such as production data as a benchmark. Finally, we replace the outliers with respective quantiles of the carbon intensities distribution within respective peer groups. An overview of our approach is presented in Figure 2. 

Secondly, we rely on a multi-model approach for emissions estimation. Limited trust should be placed in any individual estimation model or data source. The approach uses two models based on reported data (median and interpolation or regression model) and one model based on macro-level data (input-output model), which reduces the dependency on the accuracy and representativeness of reported carbon data (for details see our latest report on carbon emissions). In this approach, the biases of one estimation technique can be attenuated by the introduction of alternative methods, such as different source data, peer group classifications, and statistical methods. Moreover, the aggregation of multiple models stabilises prediction by reducing annual variation, which is crucial for quantitative investment strategies.

Finally, we aim for a pragmatic and balanced implementation of the precautionary principle. In practice, this means minimizing the risk of underestimating data while at the same time avoiding excessive volatility or distortion in the overall data distribution, which would make this data unsuitable to use in sustainable investment solutions.

For more on sustainable investment, subscribe to the blog. 

[1] Report of the United Nations Conference on Environment and Development A/CONF.151/26, United Nations General Assembly, (1992).

[2] Bourguignon, D., The precautionary principal Definition, applications, and governance, European Parliamentary Research Service, (2015).

[3] e.g., Vienna Convention for the Protection of the Ozone Layer (1987)

[4] [ibid.] Report of the United Nations Conference on Environment and Development A/CONF.151/26, United Nations General Assembly, (1992).

[5] [ibid.] Bourguignon, D., The precautionary principal Definition, applications, and governance, European Parliamentary Research Service, (2015).

[6] The principle has been also discussed in the recent taxonomy regulation, for details see: https://finance.ec.europa.eu/system/files/2022-10/221011-sustainable-finance-platform-finance-report-usability_en_1.pdf

[7] See Article 13 of the EU Commission Delegated Regulation for minimum standards for EU Climate Transition and Paris-Aligned benchmarks, 2020/1818 of 17 July 2020; https://eur-lex.europa.eu/eli/reg_del/2020/1818/oj

[8] Simmons J., Kooroshy J., Bourne E., Jain M., Clements L., Mind the gaps: clarifying corporate carbon, FTSE Russell, (2022).

[9] Bourguignon, Didier. The precautionary principal Definition, applications, and governance (European Parliamentary Research Service, 2015).

[10] Hoepner, A., and Joeri R., Emissions estimations should embed a precautionary principle, Nature Climate Change 11.8 (2021): 638-640.

[11] Olesiewicz et al., (2022), Navigating the corporate disclosure gap: Modelling of Missing Not at Random Carbon Data. 

[12] Allen, E., Lyons, K., Tavares, R.,The application of machine learning to sustainable finance. Journal of Environmental Investing 8, (2017):115–129.

[13] For 2019 (FTSE All World), 95th percentile was on average 8 times higher than median on a subsector level and 19 times higher on sector level

[14] Han et al., (2021), Estimation of Corporate Greenhouse Gas Emissions via Machine Learning, Bloomberg Quant Research.