Illustration of a Methodology to Characterize Some Preference Uncertainties in Multicriteria Decision Analysis

doi:10.21203/rs.3.rs-2082082/v1

Download PDF

Research Article

Illustration of a Methodology to Characterize Some Preference Uncertainties in Multicriteria Decision Analysis

https://doi.org/10.21203/rs.3.rs-2082082/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Any multicriteria decision analysis (MCDA) method should make sense to its users and be practical in helping make decisions. Using selected civil and environmental engineering examples in life cycle sustainability assessment and in prioritizing environmental assessment and remediation, the presented methodology strives to meet this objective by emphasizing the importance of visualizing some of the preference uncertainties associated with choice and ranking decision analyses. The emphasis on visualization should compel stakeholders to discuss MCDA results and deliberate conclusions, which is a need in any MCDA practice. The methodology employs simple yet well-founded equations derived from partial-order theory for uncertainty visualization. These equations can be easily programmed using a spreadsheet software or used via a publicly available software highlighted in the article. In addition to its simplicity, the methodology avoids the requirement to quantify stakeholder preferences or to aggregate attribute scores in all situations. As shown through the examples, reasonable decisions can often be made by visualizing the information as it is.

multicriteria decision analysis

life cycle assessment

uncertainty

partial order

preference

prioritization

The practices of civil and environmental engineering often involve the evaluation of potentially sustainable solutions to challenges of societal interest (Engineers Canada, 2016; 2019), and tools such as multicriteria decision analysis (MCDA) are sometimes used in these evaluations. Cinelli et al. (2020; 2022) recently compiled a list of 205 MCDA methods and developed a software to guide an analyst in selecting appropriate methods based first on the type of decision to be made: ranking, choice, sorting, or clustering. Ranking is described by Roy (1996) as prioritization or putting objects (i.e., elements) in a rational order based on one or more criteria (i.e., attributes) deemed important to decision-makers. Roy (1996) described choice as a decision problem that involves selecting the best element from a group of elements based on important attributes. Sorting is placing elements into a priori groups based on attributes that are important to decision-makers, and clustering is placing elements into a posteriori groups based on similar attributes that naturally emerge among some of the elements (Roy, 1996; Cinelli, Kadziński, Miebs, Gonzalez, & Słowiński, 2022). Roy (1996) proposed an element–attribute matrix, depicted in Table 1, to organize the relationships among the elements being evaluated by decision-makers and the attributes used in an evaluation.

Table 1

*Element–Attribute Matrix Used to Organize Information in Multicriteria Decision Analysis*
Element	Attributes,\(A\)
Element	\({a}_{1}\)	\({a}_{2}\)	\({a}_{3}\)	\(\cdots\)	\({a}_{n}\)
\({e}_{1}\)	\({a}_{1}\left({e}_{1}\right)\)	\({a}_{2}\left({e}_{1}\right)\)	\({a}_{3}\left({e}_{1}\right)\)	\(\cdots\)	\({a}_{n}\left({e}_{1}\right)\)
\({e}_{2}\)	\({a}_{1}\left({e}_{2}\right)\)	\({a}_{2}\left({e}_{2}\right)\)	\({a}_{3}\left({e}_{2}\right)\)	\(\cdots\)	\({a}_{n}\left({e}_{2}\right)\)
\({e}_{3}\)	\({a}_{1}\left({e}_{3}\right)\)	\({a}_{2}\left({e}_{3}\right)\)	\({a}_{3}\left({e}_{3}\right)\)	\(\cdots\)	\({a}_{n}\left({e}_{3}\right)\)
\(⋮\)	\(⋮\)	\(⋮\)	\(⋮\)	\(\ddots\)	\(⋮\)
\({e}_{m}\)	\({a}_{1}\left({e}_{m}\right)\)	\({a}_{2}\left({e}_{m}\right)\)	\({a}_{3}\left({e}_{m}\right)\)	\(\cdots\)	\({a}_{n}\left({e}_{m}\right)\)
Note. \(A\) is the set of attributes, \({a}_{j}\) is an attribute, \(E\) is the set of elements, \({e}_{i}\) is an element, and \({a}_{j}\left({e}_{i}\right)\) a score for the \({i}^{\text{th}}\) element against the \({j}^{\text{th}}\) attribute (Roy, 1996).

Beyond decision type, Cinelli et al. (2022) used decision-makers’ preferences to further refine the selection of appropriate MCDA methods. Referring to Table 1, decision preferences consist of the implicit or explicit rules that decision-makers use (a) to score an element against a given attribute by using ordinal, interval, or ratio measures; (b) when two elements have equivalent scores against a given attribute; or (c) when an element does not have better or worse scores for all attributes when compared to the attribute scores for another element (i.e., a lack of dominance or subordinance, respectively).

MCDA is inherently subjective. First, subjectivity is present because of the decision modelling uncertainties, denoted as ontological uncertainties by Lane and Maxfield (2005) and Paasche et al. (2020). Such subjectivities are introduced when framing and populating the element–attribute matrix. A unique and complete set of attributes for a given decision context does not always exist because the assigned attributes depend on the preferences of the people involved in decision-making. Brüggemann and Halfon (2000) devised a stability index to characterize the potential for changes in element ranks when an attribute is added or removed (Pavan & Todeschini, 2004), and de Farias Aires and Ferreira (2018) recently performed a literature review on this rank reversal phenomenon in MCDA. Another form of ontological uncertainty is there often exists several ways to score an element against an attribute, and the selected measure depend on the decision-makers' preferences. Second, once an attribute measure is assigned by decision-makers, epistemic uncertainties in the score exist due to the incomplete information in the referenced data set. This form of uncertainty can be characterized via possibility theory, such as fuzzy logic (Ross, 2010) or by characterizing the data’s reliability, completeness, and context correlation via a data pedigree matrix (Weidema & Wesnæs, 1996; Ciroth, Muller, Weidema, & Lesage, 2016; Maister, Di Noi, Ciroth, & Srocka, 2020). Third, semantic uncertainty describes the commmunication ambiguity that may emerge among decision-makers (Beven, 2016; Paasche, Paasche, & Dietrich, 2020) when they must rely on qualitative score descriptions to assign attribute scores to elements. For example, each decision-maker may have a different perspective on whether two elements are equivalent when compared to a given attribute. However, these differences can be resolved through discussion among decision-makers. Fourth, Riedener (2021) recently explored the uncertainty in what is important to people (i.e., axiological uncertainty) in the context of decision-making, which is directly related to ranking and choice decisions when element dominance is absent. Decision-makers’ preferences become crucial in weighing the importance of some attributes over others. Nevertheless, the subjectivity in MCDA that is introduced via axiological, epistemic, ontological, and semantic uncertainties should not be something to be avoided because it is the natural result of the interpretivist or constructivist nature of MCDA: there are multiple truths, values, and preferences held by the people involved in an MCDA process (Lincoln, Lynham, & Guba, 2018). Instead, these forms of uncertainties should be acknowledged and modelled whenever reasonable to do so.

Professor Ronald A. Howard, an early thought leader in MCDA, understood its subjectivity yet recognized people’s aspiration to be reasonable. He described decision analysis as “no more than a procedure for applying logic. The ultimate limitation to its applicability lies not in its ability to cope with problems but in [people’s] desire to be logical” (Howard, 1966, p. 113). This desire aligns with an intent to be accountability to others in decision-making. He detailed this purpose with the following definition that acknowledges the subjective and multidimensional yet organized nature of decision analysis:

Decision analysis is a logical procedure for the balancing of the factors that influence a decision. The procedure incorporates uncertainties, values, and preferences in a basic structure that models the decision. Typically, it includes technical, marketing, competitive, and environmental factors. The essence of the procedure is the construction of a structural model of the decision in a form suitable for computation and manipulation; the realization of this model is often a set of computer programs. (Howard, 1966, p. 98)

The last sentence in the quotation above conveys a sense that Howard’s definition reduces decision analysis to merely a quantitative procedure, but elsewhere he emphasized decision analysis as a catalyst for collaboration among people amid subjectivity:

One of the most important advantages of decision analysis lies in the way it encourages meaningful communication among the members of the enterprise because it provides a common language in which to discuss decision problems. Thus, engineers and marketing planners with quite different jargons can appreciate on another’s contributions to a decision. Both can use the decision-analysis language to convey their feelings to management quickly and effectively. (Howard, 1966, p. 103)

Brüggemann and Carlsen (2012) are well-known researchers in the field of applying partial order theory to MCDA, and they humorously illustrated the subjectivity of ranking methods: “In plain words the process is to add apples and oranges to get bananas, eventually being ordered by their size!” (p. 293). Is there a single way, if any way, to derive bananas from other fruit? The inherently subjective nature of MCDA means collaboration among decision stakeholders is needed to make practical sense of MDCA results.

Baltussen et al. (2019) performed a systematic literature review of the use of MCDA in healthcare decision-making, and they examined 36 research articles on this topic representing at least two dozen countries. They found the MCDA methods in two thirds of the examined articles omitted the opportunity for stakeholders to discuss or deliberate the MCDA results and to apply human nuance to quantitative results. They recommended stakeholder deliberation should be part of all MCDA methods so that quantitative results are always critically evaluated for merit within a given context (i.e., do not just trust the numbers). DiStefano and Krubiner (2020) agreed with the recommendation provided by Baltussen et al. (2019) but asserted that quantitative MCDA, with its attribute scores and weights, be avoided when evaluating possible healthcare options:

We take a stronger position regarding the flaws of quantitative MCDA and, building on [Baltussen’s et al. (2019)] own critical assessment, strongly caution against it. … Quantitative MCDA may mask the complex trade-offs that exist within and between decision criteria and remain generally inaccessible to those who are not well-versed in its technical methods of appraisal. We advocate for a predominantly qualitative approach to MCDA appraisal centred around deliberation and supplemented with decision aids. (DiStefano & Krubiner, 2020, p. 292)

Shedding quantitative MCDA in engineering decision-making is likely unrealistic because of engineering’s innate nature, but the essence of DiStefano and Krubiner’s (2020) assertion that stakeholder deliberation is crucial should be heeded. In an engineering application, for example, Awuah-Offei et al. (2021) applied DiStefano and Krubiner’s (2020) recommendation by incorporating qualitative community expectations in the selection of mine design alternatives. Furthermore, DiStefano and Krubiner (2020) emphasized that each decision stakeholder should deem any applied quantitative MCDA method as understandable and reasonable for the decision context. An MCDA method must make sense; it should be practical.

Thus, the purpose of this article is to propose a simple methodology to help make multicriteria decision analysis results practically relevant to stakeholders by approximating some of the preference uncertainty in those results in a way that encourages discussion. The methodology focuses on choice or ranking decisions and specifically on the axiological uncertainty that arises in the common situation when one element does not dominate another element across all attributes. Decision-makers’ preferences about the relative importance of some attributes compared to others (e.g., attribute weightings) does influence the resolution of this situation. The epistemic, ontological, and semantic uncertainties outlined previously are equally important considerations during MCDA deliberations but are only discussed peripherally herein.

Referring to Table 1, the prerequisites of the proposed methodology are (a) the element–attribute matrix is fully populated with elements, attributes, and scores; (b) each score within an attribute column is greater than, less than, or equal to other scores within the same column as determined through a logic deemed acceptable to stakeholders; and (c) for every attribute, larger scores mean greater element preference (e.g., an element with an attribute score of 5 is preferred over an element with a score of 4). Fulfilling the first prerequisite means stakeholders have agreed on a set of elements to be ranked or that form the possible choices. They have also agreed on the attributes that are relevant to the decision. Completing the second prerequisite means stakeholders have developed a defensible and attribute-specific scheme to quantify and denote practical differences or similarities between elements (e.g., scores are rounded to context-specific significant figures). The last prerequisite means the natural way of scoring an attribute may need to be transformed to implement the proposed methodology.

Once the three prerequisites are satisfied but before applying any context-specific scheme to aggregate the attribute scores (e.g., adding scores or using another method to combine scores across attributes), the axiological uncertainty in element choice or element ranks can be approximated using an approach derived from the work of Brüggemann et al. (2004) and Carlsen (2008), which was summarized by Mauri and Ballabio (2008). The approach is based on partial order theory, which is described elsewhere (e.g., Epp, 2020) and has been used previously by Thiessen and Achari (2012; 2014) to rank contaminated sites. Based on a completed element–attribute matrix, the first step is to generate a Hasse matrix that relates every element to every other element in terms of dominance, subordinance, or incomparability according to the relations set expressed in Eq. 1 (Mauri & Ballabio, 2008). A Hasse matrix is the tabular representation of a directed graph called a Hasse diagram, and both the matrix and diagram describe the partial ordering of elements in a set, called a poset (Epp, 2020).

\({H}_{{e}_{b}\perp {e}_{c}}\left({e}_{b}\right)=\left\{\begin{array}{c}\text{If} {a}_{j}\left({e}_{b}\right)\ge {a}_{j}\left({e}_{c}\right) \forall a\in A⟶1, \text{else}\\ \text{If}\text{ }{a}_{j}\left({e}_{b}\right)<{a}_{j}\left({e}_{c}\right) \forall a\in A⟶-1, \text{then}\\ 0\end{array}\right\}\) 1

As previously described in a conference paper by Thiessen and Achari (2020), the relation of Element \(b\) compared to Element \(c\), denoted as \({H}_{{e}_{b}\perp {e}_{c}}\left({e}_{b}\right)\) in Eq. 1, is equal to 1 if all attribute scores for Element \(b\), denoted as \({a}_{j}\left({e}_{b}\right)\), are greater than or equal to the corresponding attribute scores for Element \(c\), denoted as \({a}_{j}\left({e}_{c}\right)\). This relation means Element \(b\) is preferred or equivalent when compared to Element \(c\). If this first statement is not true, the second statement in Eq. 1 is evaluated. If the second statement is true, \({H}_{{e}_{b}\perp {e}_{c}}\left({e}_{b}\right)=-1\), meaning Element \(b\) is not preferred nor equivalent when compared to Element \(c\). Both statements express comparability, which is denoted by the relational operator \(\perp\). If both statements are false, \({H}_{{e}_{b}\perp {e}_{c}}\left({e}_{b}\right)=0\), and the relative preference of Element \(b\) to Element \(c\) cannot be determined; the elements are incomparable and is denoted by the relational operator \(\parallel\). The pairwise comparisons codified in Eq. 1 are analogous to alternative pairwise comparison in the analytic hierarchy process presented by Saaty (1980). A free software, Decision Analysis by Ranking Techniques (DART) (Talete Srl, 2018a; 2018b) can be used to generate the Hasse matrix from the element–attribute matrix.

Using the values in the Hasse matrix, Brüggemann et al. (2004) and Carlsen (2008) proposed the three equations denoted as Eq. 2 to quantify the minimum possible rank, average rank, and maximum possible rank of an element, respectively. The variables \(n\), \({n}_{\le {e}_{b}}\), and \({n}_{\parallel {e}_{b}}\) are the number of elements, the number of 1-entries, and the number of 0-entries in a row in the Hasse matrix, respectively. The difference between the minimum and maximum possible ranks for an element is the axiological uncertainty band for that element. The element’s rank can vary within this uncertainty band and depends on decision-makers’ preferences about an attribute’s importance. Focusing on the right equation in Eq. 2, the range of the uncertainty band increases as \({n}_{\parallel {e}_{b}}\) increases in proportion relative to \({n}_{\le {e}_{b}}\). The centre equation can be used to characterize a poset as a totally ordered set (toset) with an unbiased measure of central tendency.

\({rk}_{min}\left({e}_{b}\right)={n}_{\le {e}_{b}}+1\)	\({rk}_{av}\left({e}_{b}\right)=\frac{\left({n}_{\le {e}_{b}}+1\right)\left(n+1\right)}{n+1-{n}_{\parallel {e}_{b}}}\)	\({rk}_{max}\left({e}_{b}\right)={n}_{\le {e}_{b}}+{n}_{\parallel {e}_{b}}+1\)	2
These equations are used to quantify ranking or choice uncertainties, as described with the illustrations in the next section.

Illustrations

The methodology presented herein is illustrated with three examples. The first and second examples are based on life cycle sustainability assessment (LCSA) results published in research articles and show how the methodology can be applied to decisions centred on choosing an alternative. Articles by Visentin et al. (2021) and by Balasbaneh and Marsono (2020) were selected because they are recently published LCSAs focused on environmental engineering and civil engineering topics, respectively, and have well-presented element–attribute matrices. The third example focuses on ranking contaminated sites in India for further investigation or remediation and is based on a conference paper presented by Thiessen and Achari (2020).

Choice Of Zero-valent Iron Production Method

Zero-valent iron is a nanomaterial (nZVI) used in soil and groundwater remediation to chemically reduce contaminants, such as chlorinated solvents, to mitigate or remove their biological toxicity. Visentin et al. (2021) performed an LCSA of four methods to produce nZVI: (a) milling—iron particles are physically ground down to a size of 20 nm; (b) reduction with sodium borohydride (NaBH₄)—aqueous solutions of ferric chloride and of NaBH₄ react to form nZVI; (c) reduction with hydrogen gas (H₂) with goethite and hematite synthesis (G/H)—G/H particles are synthesized and subsequently reduced via heat in an H₂ atmosphere; and (d) reduction with H₂ without G/H synthesis—G/H particles are procured from elsewhere and subsequently reduced via heat in an H₂ atmosphere. (Visentin, da Silva Trentin, Braun, & Thomé, 2021)

Visentin’s et al. (2021) LCSA results are summarized in Table 2. To satisfy the third prerequisite described in the Methodology, the scores were translated to ranks as shown in the table as italicized numbers in parentheses.

Table 2

*Results of Life Cycle Sustainability Assessment of Zero-Valent Iron Production Methods*
Zero-valent iron production methods	Life cycle sustainability assessment dimensions, endpoint categories, and results
	Environment				Economy		Society
	Human health (mPt)	Ecosystem quality (mPt)	Climate change (mPt)	Resources (mPt)	Internal costs (USD/kg)	External costs (USD/kg)	Human resource management (index)	Community development (index)	Societal development (index)	Corporate social responsibility (index)
Milling	7.67 (3)	0.181 (4)	2.38 (4)	2.58 (4)	139.68 (4)	4.75 (4)	0.666 (4)	0.754 (2)	0.896 (4)	0.825 (4)
Reduction with sodium borohydride	15.36 (1)	2.44 (1)	9.05 (2)	10.32 (1)	1011.09 (3)	37.99 (1)	0.633 (1)	0.754 (2)	0.896 (4)	0.825 (4)
Reduction with hydrogen and G/H synthesis	8.90 (2)	1.51 (2)	9.74 (1)	8.86 (2)	1211.45 (1)	32.29 (2)	0.635 (3)	0.833 (4)	0.820 (2)	0.802 (2)
Reduction with hydrogen and no G/H synthesis	3.76 (4)	0.51 (3)	3.75 (3)	3.47 (3)	1109.61 (2)	10.78 (3)	0.635 (3)	0.833 (4)	0.820 (2)	0.802 (2)
Note. The information in this table is from Table 1 in the research article by Visentin et al. (2021) but has been significantly restructured to follow the typical format of an element–attribute table. Also, the italicized numbers in parentheses have been added and are ranks, where larger ranks are preferred. The units are as follows: mPt is millionth of points, which is a unit of environmental load caused by the method; G/H is goethite/hematite; and USD/kg is US dollars per kilogram. Refer to the article by Visentin et al. (2021) for details.

Based on the ranks in Table 2, the Hasse matrix in Table 3 was prepared using Eq. 1 via the DART software (Talete Srl, 2018a). Focusing on the first row in the Hasse matrix, the iron milling method is preferred over or is equivalent to the nZVI production method using sodium borohydride because the value in the intersecting cell is 1. Iron milling is incomparable to the other two methods because pairwise comparison of the ranks in Table 2 reveal some of milling’s ranks are higher or lower than the corresponding ranks for the other two options (i.e., values are 0).

Table 3

*Hasse Matrix of Life Cycle Sustainability Assessment Results for Zero-Valent Iron Production Methods*
Zero-valent iron production method	Zero-valent iron production methods and relation set results
Zero-valent iron production method	Milling	Reduction with sodium borohydride	Reduction with hydrogen and G/H synthesis	Reduction with hydrogen and no G/H synthesis
Milling	—	1	0	0
Reduction with sodium borohydride	−1	—	0	0
Reduction with hydrogen and G/H synthesis	0	0	—	−1
Reduction with hydrogen and no G/H synthesis	0	0	1	—
Note. The results in this table are based on the ranks associated with the zero-valent iron production methods (i.e., elements) and endpoint impact categories (i.e., attributes) in Table 2 and were calculated using Eq. 1.

Figure 1 illustrates the axiological uncertainty across the four nZVI production methods based on the Hasse matrix in Table 3. In both figure panels, the possible rank ranges for each production method are shown as grey vertical bands (i.e., axiological uncertainty bands), as determined by the left and right equations in Eq. 2. The vertical placement of the uncertainty bands for nZVI milling and reduction by H₂ without G/H synthesis supports a conclusion that either of these two nZVI production method could be chosen as the most sustainable method. This result is consistent with Visentin’s et al. (2021) conclusion: “The LCSA indicated that the milling method and the hydrogen gas method (without G/H synthesis approach) were the most sustainable” (§ 4). Visentin et al. (2021) sought to establish a toset of the nZVI production methods based on relative sustainability index values listed in their article (§ 3.2.3), and the resulting ranks are shown as triangles in the left panel of the figure. The right panel depicts another toset, using the Hasse average ranks calculated according to the centre equation in Eq. 2, and appear to be better single-point estimates of the associated uncertainty. However, this discrepancy does not affect the chosen alternatives in this case.

Choice Of Retaining Wall Design

Retaining walls are geotechnical structures installed to hold back soil in hilly terrain. Balasbaneh and Marsono (2020) performed an LCSA of five retaining wall designs commonly used in Malaysia for residential buildings: crib, keystone, cantilever, gabion, and masonry retaining walls. Their LCSA results are reproduced in Table 4 (i.e., the element–attribute matrix), and the scores have been translated to ranks, like what was done in the previous example. Similarly, Table 5 is the Hasse matrix that was developed from Table 4.

Table 4

*Results of Life Cycle Sustainability Assessment of Retaining Wall Designs*
Retaining wall design	Life cycle sustainability assessment criteria and results
Retaining wall design	Global warming potential, kg CO₂ eq	Ozone layer depletion, CFC-11 eq	Cost present value, Malaysian ringgit	Social impacts, index ^a
Crib	23,525 (3)	328 (1)	921,100 (2)	0.501 (1)
Keystone	28,680 (2)	255 (3)	951,500 (1)	0.328 (4)
Cantilever	34,810 (1)	296 (2)	595,600 (5)	0.420 (2)
Gabion	12,784 (5)	179 (5)	826,300 (3)	0.387 (3)
Masonry	20,020 (4)	225 (4)	689,000 (4)	0.311 (5)
Note. The information in this table is from Table 9 in the research article by Balasbaneh and Marsono (2020, p. 2150) but has been significantly restructured to follow the typical format of an element–attribute table. Also, the italicized numbers in parentheses have been added and are ranks, where larger ranks are preferred. CO₂ eq is carbon dioxide equivalent and CFC-11 eq is chlorofluorocarbon-11 equivalent. ^a There appeared to be a typographical error in Table 9 in the research article, and thus the social impact values listed on p. 2149 and depicted in Fig. 6 of Balasbaneh and Marsono’s (2020) were used instead.

Table 5

*Hasse Matrix of Life Cycle Sustainability Assessment Results for Retaining Wall Designs*
Retaining wall design	Retaining wall design and relation set results
Retaining wall design	Crib	Keystone	Cantilever	Gabion	Masonry
Crib	—	0	0	−1	−1
Keystone	0	—	0	0	−1
Cantilever	0	0	—	0	0
Gabion	1	0	0	—	0
Masonry	1	1	0	0	—
Note. The results in this table are based on the scores associated with the retaining wall designs (i.e., elements) and life cycle sustainability assessment criteria (i.e., attributes) in Table 4 and were calculated using Eq. 1.

The axiological uncertainty across retaining wall designs is illustrated in Fig. 2. The left panel shows Balasbaneh and Marsono’s (2020) design order with the gabion wall being the recommended design from an LCSA perspective. One can observe that the uncertainty band for the gabion wall is wider than the adjacent band associated with a masonry wall. In contrast, the right panel shows the five designs ordered according to Hasse average rank. This presentation of the results places the masonry wall as the preferred design instead, which is intuitive base on the narrower breadth and high vertical position of its uncertainty band. This example emphasizes the perspective that uncertainties in MCDA should be quantified to inform stakeholder discussions and decision-making.

Ranking Contaminated Sites For Assessment Or Remediation

In 2015, India’s Ministry of Environment, Forest, and Climate Change (MoEFCC) published its report titled Inventory and Mapping of Probably Contaminated Sites in India (COWI, 2015a) to support the ongoing site assessment and remediation of 320 contaminated sites in the country. In the report, the MoEFCC’s consultant, COWI (2015d), described a screening-level contaminated site ranking scheme, called Stage I prioritization, to guide further investigations and remediation activities. Stage I prioritization is based on a priority score for each contaminated site that is the summation of scores ranging from 4 to 20 for each of the following five site attributes: industry profile (IP), land use (LU), population at risk (P), groundwater system at risk (GW), and surface water system at risk (SW) (COWI, 2015d). Larger priority scores are associated with contaminated site requiring greater attention by decision-makers.

Thiessen and Achari (2020) compiled the scores for the IP, LU, P, GW, and SW attributes for all 320 sites according to the guidance provided by COWI (2015b; 2015c; 2015d). The resulting element–attribute matrix is in the Supplementary Information because of its size; however, an excerpt is shown in Table 6 to show its structure. Likewise, the associated Hasse matrix is not provided because of its size, but it can be derived by using the element–attribute matrix in the Supplementary Information and the DART software (Talete Srl, 2018a).

Table 6

*Structure of Stage I Prioritization Attributes Table*
Contaminated site	Stage I prioritization attributes and scores
Contaminated site	Industry profile	Land use	Population at risk	Groundwater system at risk	Surface water system at risk
S1	16	16	20	20	20
S2	16	16	20	20	12
S3	16	4	12	20	20
\(⋮\)	\(⋮\)	\(⋮\)	\(⋮\)	\(⋮\)	\(⋮\)
S320	16	4	20	20	16
Note. This excerpt illustrates the structure of element–attribute information in the Supplementary Information, which was compiled based on guidance by COWI (2015b; 2015c; 2015d).

The rank uncertainty across all 320 contaminated sites, ordered by Stage I priority scores, is presented in Fig. 3. The vertically wide grey band illustrates the large axiological uncertainty in contaminated site ranks if one removes the assumption that IP-, LU-, P-, GW-, and SW-scores must be added together, as proposed by COWI (2015d). This figure emphasizes the need for decision-makers to first understand and discuss information uncertainties before fixating on ways to aggregate attribute scores in MCDA. For example, Fig. 3 may help decision-makers cluster sites (i.e., group sites a posteriori) according to uncertainty band widths and positions relative to the vertical axis. Sites with narrower uncertainty bands positioned higher on the vertical axis may be higher-priority sites. Nevertheless, the desire for an unbiased single-point estimator of rank is reasonable. As illustrated with the previous two choice examples, the Hasse average rank is a suitable single-point estimator if a toset is desired, and Fig. 4 depicts the contaminated sites ordered according to it. Although the overall site order is different, what is notable across both figures is the higher priority sites, positioned on the right side of the figures, largely remain the same, which may mean their relative position in the site order is insensitive to the attribute aggregation function (e.g., priority score or Hasse average rank). Again, perhaps adequate decisions can be made with only the uncertainty bands.

The presented contaminated site ranking example and the two life cycle sustainability assessment choice examples illustrate the appreciable uncertainty that exists in multicriteria decision analysis results. Specifically, an element’s position in a toset, used for ranking or for choice decisions, is dependent upon the algorithm applied by decision-makers to create the total ordering. The applied algorithm reflects the decision-makers’ preferences about which attributes are more important than others in creating a total order out of the natural and partial ordering that is intrinsic to a populated element–attribute matrix. There is not a mathematically unique toset algorithm to combine attribute scores, and the selection depends on the group of people striving to make a logical decision. This nonunique phenomenon was denoted as axiological uncertainty because the toset algorithm depends on decision-makers’ preferences—their beliefs about which attributes most important or least important. The phenomenon was illustrated with the retaining wall and contaminated site examples where the element order changed depending on the toset algorithm. However, the phenomenon is not always consequential, as illustrated by the stability of the preferred nanoscale zero-valent iron production methods and the stable order of the higher-priority contaminated sites. Nevertheless, decision-makers should critically examine MCDA results before implementing decisions.

Given the potential for different MCDA results depending on the applied toset algorithm, decision-makers should defer the effort to articulate, deliberate, and document attribute preferences. Instead, the recommendation is to first characterize the range of possible element orders to determine if the resulting axiological uncertainty bands provide the needed decision insight without the effort of devising a toset algorithm. This parsimonious approach is aligned with the article’s purpose of providing a practically relevant methodology, and the associated calculations are simple and can be conducted with commonly available or publicly accessible software. Referring to the illustrations, the preferred nanoscale zero-valent iron production methods, retaining wall type, and highest priority contaminated sites are obvious by the position and breadth of the associated uncertainty bands. Also, the act of reviewing and discussing the uncertainty bands as a decision-making group fits well with an original intent of MCDA. Depending on the outcome of deliberations, decision-makers may still desire a toset algorithm to communicate a logical decision process to those affected by the decision. The Hasse average rank equation is an unbiased measure of central tendency that may satisfy this desire without the need to solicit attribute preferences from decision-makers. If the decision context requires that explicit attribute preference be applied to generate a toset, the Hasse average rank results can be used as a reference case for transparency in communication decision rationale. For example, why was the gabion wall deemed to be the most sustainable wall type by the researchers when the Hasse average rank for the masonry wall was higher?

The article’s focus is on axiological uncertainty and multicriteria decision analysis, but some discussion on the connected epistemic, ontological, and semantic uncertainties is warranted. As noted previously, the examples illustrate that axiological uncertainty in multicriteria decision analysis can be large. This large uncertainty is magnified when the ontological uncertainties of framing an element–attribute matrix and choosing an attribute measure, the epistemic uncertainties of the input data, and the semantic uncertainties associated with interpersonal communication are all incorporated. If all four uncertainty forms could be accurately quantified, the overall uncertainty in the MCDA results may be so large that one could think that the MCDA process is pointless. It is not; effective MCDA gets people talking about the practical meaning of all these uncertainties with the intent of making logical and practical decisions.

Any multicriteria decision analysis method should make sense to its users and be practical in helping make decisions. Using civil and environmental engineering examples, the illustrated methodology strives to meet this objective by emphasizing the importance of visualizing the uncertainty in preferences associated with choice and ranking decision analyses. The emphasis on visualization should compel stakeholders to discuss MCDA results and deliberate conclusions, which was a desire of one of the originators of quantitative MCDA and remains a current need. The methodology employs simple yet well-founded equations for uncertainty visualization. These equations can be easily programmed using spreadsheet software (e.g., MS-Excel®) or used via DART, a publicly available software. In addition to its simplicity, the methodology avoids the requirement to quantify stakeholder preferences or combine attribute scores (i.e., aggregation) in all situations. As shown through the examples, reasonable decisions can often be made by visualizing the information as it is.

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this article.

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Author Contributions

Author contributions are defined using the Contributor Roles Taxonomy (CRediT). R. J. T. contributed to this article by fulfilling the following roles: conceptualization, methodological development, investigation, data curation, formal analysis, and writing (i.e., original draft, review, and editing). G. A. contributed to this article by reviewing, providing input, and editing the article.

Ethics Approval

Ethics approval was not required for the research described in this article because neither human nor animal subjects were involved.

Consent to Participate

Consent to participate was not required for the research described in this article because human subjects were not involved.

Consent to Publish

Consent to publish was not required for the research described in this article because human subjects were not involved.

Data Availability

Data generated or analyzed during this study are provided in full within the article and its supplementary materials.

Awuah-Offei, K., Que, S., & Rehman, A. U. (2021). Evaluating mine design alternatives for social risk using discrete choice analysis. Sustainability, 13(16), 8700. Retrieved from https://doi.org/10.3390/su13168700
Balasbaneh, A. T., & Marsono, A. K. (2020). Applying multi-criteria decision-making on alternatives for earth-retaining walls: LCA, LCC, and S-LCA. The International Journal of Life Cycle Assessment, 25, 2140–2153. Retrieved from https://doi.org/10.1007/s11367-020-01825-6
Baltussen, R., Marsh, K., Thokala, P., Diaby, V., Castro, H., Cleemput, I.,.. . Broekhuizen, H. (2019). Multicriteria decision analysis to support health technology assessment agencies: Benefits, limitations, and the way forward. Value in Health, 22(11), 1283–1288. Retrieved from https://doi.org/10.1016/j.jval.2019.06.014
Beven, K. (2016). Facets of uncertainty: Epistemic uncertainty, non-stationarity, likelihood, hypothesis testing, and communication. Hydrological Sciences Journal, 61(9), 1652–1665. Retrieved from https://doi.org/10.1080/02626667.2015.1031761
Brüggemann, R., & Carlsen, L. (2012). Multi-criteria decision analyses. Viewing MCDA in terms of both process and aggregation methods: Some thoughts, motivated by the paper of Huang, Keisler and Linkov. Science of The Total Environment, 425, 293–295. Retrieved from https://doi.org/10.1016/j.scitotenv.2012.02.062
Brüggemann, R., & Halfon, E. (2000, May). Introduction to the general principles of the partial order ranking theory. In L. Carlsen (Ed.), Order theoretical tools in environmental sciences. Proceedings of the second workshop October 21, 1999 in Roskilde, Denmark, (pp. 7–43). Retrieved from https://www2.dmu.dk/1_viden/2_publikationer/3_fagrapporter/rapporter/fr318.pdf
Brüggemann, R., Sørensen, P. B., Lerche, D., & Carlsen, L. (2004). Estimation of averaged ranks by a local partial order model. Journal of Chemical Information and Computer Sciences, 44(2), 618–625. Retrieved from https://doi.org/10.1021/ci034214m
Carlsen, L. (2008). Hierarchical partial order ranking. Environmental Pollution, 155(2), 247–253. Retrieved from https://doi.org/10.1016/j.envpol.2007.11.023
Cinelli, M., Kadziński, M., Gonzalez, M., & Słowiński, R. (2020). How to support the application of multiple criteria decision analysis? Let us start with a comprehensive taxonomy. Omega, 96, 102261. Retrieved from https://doi.org/10.1016/j.omega.2020.102261
Cinelli, M., Kadziński, M., Miebs, G., Gonzalez, M., & Słowiński, R. (2022). Recommending multiple criteria decision analysis methods with a new taxonomy-based decision support system. European Journal of Operational Research, 302(2), 633–651. Retrieved from https://doi.org/10.1016/j.ejor.2022.01.011
Ciroth, A., Muller, S., Weidema, B., & Lesage, P. (2016). Empirically based uncertainty factors for the pedigree matrix in ecoinvent. The International Journal of Life Cycle Assessment, 21, 1338–1348. Retrieved from https://doi.org/10.1007/s11367-013-0670-5
COWI. (2015a, December 30). Inventory and mapping of probably contaminated sites in India. Ministry of Environment, Forest and Climate Change. Retrieved from http://moef.gov.in/wp-content/uploads/2019/11/ilovepdf_merged.pdf
COWI. (2015b, December 30). Task 1: Existing data and general information on contaminated sites—Final report. In Inventory and mapping of probably contaminated sites in India (pp. 1–150 of 959). Ministry of Environment, Forest and Climate Change. Retrieved from http://moef.gov.in/wp-content/uploads/2019/11/ilovepdf_merged.pdf
COWI. (2015c, December 30). Task 4: Site inspections—Final report. In Inventory and mapping of probably contaminated sites in India (pp. 466–593 of 959). Minstry of Environment, Forest and Climate Change. Retrieved from http://moef.gov.in/wp-content/uploads/2019/11/ilovepdf_merged.pdf
COWI. (2015d, December 30). Task 5: Prioritization of sites—Final report. In Inventory and mapping of probably contaminated sites in India (pp. 596–691 of 959). Ministry of Environment, Forest & Climate Change. Retrieved from http://moef.gov.in/wp-content/uploads/2019/11/ilovepdf_merged.pdf
de Farias Aires, R. F., & Ferreira, L. (2018). The rank reversal problem in mult-criteria decision making: A literature review. Pesquisa Operacional, 38(2), 331–362. Retrieved from https://doi.org/10.1590/0101-7438.2018.038.02.0331
DiStefano, M. J., & Krubiner, C. B. (2020). Beyond the numbers: A critique of quantitative multi-criteria decision analysis. International Journal of Technology Assessment in Health Care, 36(4), 292–296. Retrieved from https://doi.org/10.1017/S0266462320000410
Engineers Canada. (2016, September). National guideline on sustainable development and environmental stewardship for professional engineers. Retrieved from https://engineerscanada.ca/national-guideline-on-sustainable-development-and-environmental-stewardship-for-professional-engineers
Engineers Canada. (2019, October). Engineers Canada paper on environmental engineering. Retrieved from https://engineerscanada.ca/engineers-canada-paper-on-environmental-engineering
Epp, S. S. (2020). Discrete mathematics with applications (5th ed.). Cengage Learning.
Howard, R. A. (1966). Decision analysis: Applied decision theory. In D. B. Hertz, & J. Melese (Ed.), Proceedings of the Fourth International Conference on Operational Research (pp. 97–113). Wiley. Retrieved from https://sdg.com/wp-content/uploads/2015/06/Decision-Analysis-Applied-Decision-Theory.pdf
Lane, D. A., & Maxfield, R. R. (2005). Ontological uncertainty and innovation. Journal of Evolutionary Economics, 3–50. Retrieved from https://doi.org/10.1007/s00191-004-0227-7
Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2018). Paradigmatic controversies, contradictions, and emerging confluences, revisited. In N. K. Denzin, & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (5th ed., pp. 108–150). SAGE Publications.
Maister, K., Di Noi, C., Ciroth, A., & Srocka, M. (2020, June). A product social impact life cycle assessment database: PSILCA database v.3 documentation. GreenDelta. Retrieved from https://psilca.net/wp-content/uploads/2020/06/PSILCA_documentation_v3.pdf
Mauri, A., & Ballabio, D. (2008). Chapter 5—Similarity/diversity measure for sequential data based on Hasse matrices: Theory and applications. In M. Pavan, & R. Todeschini (Eds.), Data handling in science and technology—Scientific data ranking methods: Theory and applications (Vol. 27, pp. 111–138). Elsevier. Retrieved from https://doi.org/10.1016/S0922-3487(08)10005-3
Paasche, H., Paasche, K., & Dietrich, P. (2020). Uncertainty as a driving force for geoscientific development. Nature and Culture, 15(1), 1–18. Retrieved from https://doi.org/10.3167/nc.2020.150101
Pavan, M., & Todeschini, R. (2004). New indices for analysing partial ranking diagrams. Analytica Chimica Acta, 515(1), 167–181. Retrieved from https://doi.org/10.1016/j.aca.2003.11.019
Riedener, S. (2021). Uncertain values: An axiomatic approach to axiological uncertainty. De Gruyter. Retrieved from https://doi.org/10.1515/9783110736199
Ross, T. J. (2010). Fuzzy logic with engineering applications (3rd ed.). Wiley. Retrieved from https://doi.org/10.1002/9781119994374
Roy, B. (1996). Multicriteria methodology for decision aiding. Kluwer Academic Publishers.
Saaty, T. L. (1980). The analytic hierarchy process: Planning, priority setting, resource allocation. McGraw-Hill.
Talete Srl. (2018a, December). DART (Decision analysis by ranking techniques) [Software]. Joint Research Centre, European Commission. Retrieved from https://joint-research-centre.ec.europa.eu/system/files/2018-12/dart2_05setup.zip
Talete Srl. (2018b, December). Decision analysis by ranking techniques user manual. Joint Research Centre, European Commission. Retrieved from https://joint-research-centre.ec.europa.eu/system/files/2018-12/dart-user_manual.pdf
Thiessen, R. J., & Achari, G. (2012). Can the National Classification System for Contaminated Sites be used to rank sites? Canadian Journal of Civil Engineering, 39(4), 415–431. Retrieved from https://doi.org/10.1139/l2012-015
Thiessen, R. J., & Achari, G. (2014). Application of partial orders and Hasse matrices in ranking contaminated sites. In R. Bruggemann, L. Carlsen, & J. Wittmann (Eds.), Multi-indicator systems and modelling in partial order (pp. 253–274). Springer. Retrieved from https://doi.org/10.1007/978-1-4614-8223-9_12
Thiessen, R. J., & Achari, G. (2020). Prioritization of contaminated sites using sparse data (preliminary). Proceedings of Geoenvironment-2020: International Seminar on Contaminated Sites, (pp. 36–44). Retrieved from http://www.nercs.in/download/Proceedings_InternationalSeminar_on_Contaminated_Sites_Geo2020.pdf
Visentin, C., da Silva Trentin, A. W., Braun, A. B., & Thomé, A. (2021). Life cycle sustainability assessment of the nanoscale zero-valent iron synthesis process for application in contaminated site remediation. Environmental Pollution, 268, 115915. Retrieved from https://doi.org/10.1016/j.envpol.2020.115915
Weidema, B. P., & Wesnæs, M. S. (1996). Data quality management for life cycle inventories—an example of using data quality indicators. Journal of Cleaner Production, 4(3–4), 167–174. Retrieved from https://doi.org/10.1016/S0959-6526(96)00043-1

No competing interests reported.

MethodologytoCharacterizePreferenceUncertaintyinMCDASuppInfoRev0.xlsx

Download PDF

Version 1

posted

You are reading this latest preprint version

Illustration of a Methodology to Characterize Some Preference Uncertainties in Multicriteria Decision Analysis

Status:

Version 1

Abstract

Figures

Background

Methodology

Illustrations

Choice Of Zero-valent Iron Production Method

Choice Of Retaining Wall Design

Ranking Contaminated Sites For Assessment Or Remediation

Discussion

Conclusion

Statements and Declarations

Funding

Competing Interests

Author Contributions

Ethics Approval

Consent to Participate

Consent to Publish

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1