A VIKOR-based group decision-making approach to software reliability evaluation

Software reliability evaluation is important to attribute of software quality. A new group decision-making is provided to evaluate software reliability, where the model is based on an extended VIsekriterijumska optimizacija i KOmpromisno Resenje technique. The individual utility and individual regret in general VIKOR technique are extended to group utility and group regret. A specific regret matrix is provided in new VIKOR-based GDM method. A new ranking method is provided in a static environment. Another new ranking method is provided in a dynamic environment. To implement a user-based evaluation for software reliability, the picture fuzzy set is used for handling the questionnaire information. It is noted that the existing projection measure is not always reasonable in picture fuzzy setting. To solve this problem, a new normalization projection measure is provided in picture fuzzy setting. And a new GDM method is established for software reliability evaluation. The feasibility and practicability developed method in this work are illustrated by an experimental analysis.


Introduction
Software reliability is one of the most important factors in the software development process, which directly impacts on software quality (Utkin and Coolen 2018). It is defined as the probability of failure-free software operation for a specified period of time in a specified environment (Lyu 1996). Software reliability is important to attribute of software quality, together with functionality, usability, performance, serviceability, capability, installability, maintainability, and documentation. Software reliability is hard to achieve, because the complexity of software tends to be high. Many software reliability researches have contributed to literature. Assessing software reliability is an important issue in the modern software development process. However, to the best of the author's knowledge, no good quantitative methods were developed to represent software reliability without excessive B Chuan Yue yuechuan-1988@163.com 1

Contribution 4 For the Question 4, an extended VIKOR method is developed, where the ideal decision is a decision matrix. The individual utility vector is extended to a group utility matrix; the individual regret vector is extended to a group regret matrix in the developed VIKOR method. A specific regret matrix is provided in the new VIKOR-based GDM method. A new ranking method under a static environment and a new ranking method in a dynamic environment are provided in this work. And the new GDM method is applied to software reliability evaluation.
A detailed introduction of this work is listed as follows. Section 2 introduces the related work. Section 3 introduces the preliminary knowledge. Section 4 presents a normalization projection measure. Section 5 develops a VIKOR-based GDM method, where the algorithmic pseudocodes are also attached in detail. Section 6 provides a detailed assessment process and an experimental analysis of software reliability. Finally, Sect. 7 draws conclusions and future researches.

Related work
Three related researches are introduced in this section. One of related researches is software reliability assessment; another work is VIKOR-based GDM methods; the third work is projection measures. The main weaknesses of current work and research motivations of this work are also mentioned.
There are many researches on software reliability (Zhao et al. 2021;Rahman et al. 2021;Mohammadzadeh et al. 2021;Mahmoudi et al. 2021;Islam et al. 2021). For example, Kang et al. (2018) developed a Bayesian belief network model for software reliability in nuclear power plants. Sinha et al. (2019) introduced an early prediction method of hardwaresoftware reliability based on functional failures. Cho et al. (2019) proposed an exhaustive test cases for the software reliability of safety-critical digital systems. Huo and Li (2019) modeled a cost-effective-based software defect prediction method. Kaliraj and Bharathi (2019) explored a path testing based reliability analysis framework. Bertsatos et al. (2019) described a testing of 3D-ID software reliability in sex and ancestry estimation. Lanna et al. (2018) focused a feature-family-based reliability analysis of software product line. Greisberger et al. (2019) suggested an interrater reliability of angular measures using TEMPLO two-dimensional motion analysis software. Levitin et al. (2019) focused an optimization of partial software rejuvenation policy. Shan et al. (2019) developed a software structure characteristic measurement method based on weighted network. Guo et al. (2018) presented a software reliability demonstration for nuclear safety-critical Digital Instrumentation and Control system. Moazzeni et al. (2018) proposed a reliability improvement of software-defined networks. Dohi et al. (2018) described an optimal periodic software rejuvenation policies based on interval reliability criteria. Zou et al. (2018) proposed a software reliability hierarchical structure modeling in I&C system Software Life Cycle. Abuta and Tian (2018) addressed the reliability over consecutive releases of a semiconductor Optical Endpoint Detection software system. Using a deep learning model, Wang and Zhang (2018) developed a software reliability prediction based on the RNN encoder-decoder.
These studies have greatly contributed to software reliability researches. However, this work finds that the software reliability assessment is multi-dimensional. MADM method and GDM method able to deal with it better. The software reliability assessments are less based on decision science.
The VIKOR is one of the common decision-making methods, which is often used in MADM (Curiel-Esparza et al. 2019;Štirbanović et al. 2019;Chen 2016). A part of VIKOR-based MADM is used in GDM. For example, some VIKOR-based decision support systems are introduced in fuzzy environments (Ploskas and Papathanasiou 2019;Ren et al. 2017). The VIKOR-based GDM methods (Büyüközkan et al. 2019;Çalı and Balaman 2019) are introduced by some scholars under intuitionistic fuzzy environment.  proposed a VIKOR-based GDM approach under interval type-2 fuzzy environment. The VIKOR-based GDM technologies with Pythagorean fuzzy information (Chen 2018;Liang et al. 2019; are shared by scholars. The scholars Büyüközkan and Güler 2020;Wu et al. 2016;You et al. 2015) developed some VIKOR-based GDM methods with linguistic information. The researcher (Gupta et al. 2016) modeled two VIKOR methods for GDM problems with intuitionistic fuzzy information. Yue (2020b) suggested a VIKOR approach to software reliability assessment in GDM setting.
Above-mentioned methods have made great contributions to GDM. However, as mentioned in Question 4 in Introduction section, the group utility and group regret are not presented in the traditional VIKOR-based GMD method. This research attempts to solve this question.
Projection measures have been attracted by scholars. Some of them are used in MADM problems. Another work is used in GDM problems. For example, Tsao and Chen (2016) addressed a projection-based compromising method for MADM with interval-valued intuitionistic fuzzy information.  explored a projection-based regret theory method for MADM under interval type-2 fuzzy sets environment. Li et al. (2018) presented a grey correlation projection-based MADM approach to economic emission dispatch.  offered a bi-projection MADM model with linguistic terms. Wu et al. (2018) performed a hesitant fuzzy linguistic projection model to a MADM problem. Tang et al. (2020) contributed a projection-based MADM approaches for evaluating the service quality of Chinese commercial banks.
Some projection-based GDM researches also attracted the attention of researchers. For example, Liao et al. (2018) focused a projection-based distance measure for GDM in intuitionistic fuzzy setting. Xu and Liu (2013) conducted a projection-based GDM method with interval fuzzy information. The researches (Liu and You 2019;Ye 2017) surveyed two bidirectional projection measures for GDM with neutrosophic numbers. Ju and Wang (2013) introduced a projection method for GDM with incomplete weight information in linguistic setting.
In recent years, some normalized projection measures have contributed to decision science by scholars. For example, Wan et al. (2018) observed a normalized projectionbased GDM method with Pythagorean fuzzy information. Yue and Jia (2017) proposed a direct projection-based GDM methodology with crisp values and interval data. Yue (2017bYue ( , a, 2019aYue ( , b, c, 2020 developed some normalized projection measures to GDM problems with different decision information.
These studies have greatly enriched and strengthened the MADM and GDM problems. However, as mentioned in Question 3 in Introduction section, this research finds that the projection measure is not always reasonable in picture fuzzy setting. To solve this problem, this research attempts to develop a new normalized projection measure.

Preliminaries
Definition 1 A picture fuzzy set (PFS) (Cuong 2014) on a universe X is interpreted as Especially, if π A (x) = 0, then the PFS A returns to an intuitionistic fuzzy set (Atanassov 1986). If π A (x) = 0 and η A (x) = 0, then the PFS A returns to a fuzzy set (Zadeh 1965).
For convenience, Wei (2017) called the array of three numbers A PFN can be explained by the following a real-world example.
Example 1 In a democratic election station, the council issues 100 voting papers for a candidate. The voting results are divided into four groups accompanied with the number of papers that are "vote for" (70), "abstain" (6), "vote against" (20) and "refusal of voting" (4). Group "abstain" means that the voting paper is a white paper rejecting both "agree" and "disagree" for the candidate but still takes the vote. Group "refusal of voting" is either invalid voting papers or did not take the vote. The voting results can be written as a PFN α = (0.7, 0.06, 0.2), where π α = 1 − 0.7 − 0.06 − 0.2 = 0.04. Some operations of PFNs are provided by scholars as follows.
Definition 3 Let X = (x i j ) m×n be a matrix. If all the x i j are PFNs, then X is called a picture fuzzy matrix (PFM). Especially, if m = 1 or n = 1, then X is called a picture fuzzy vector (PFV).
The projection measure Jia 2015, 2017) is an important tool for measuring the closeness between two evaluation matrices. A projection measure of a PFV on another was introduced by Wei et al. (2018), as shown in the following Definition 4.

Presented projection measure
As mentioned in the Introduction, the projection is a useful and comprehensive measure. However, this research finds that Eq. (4) does not always conform to the Criterion 1, as shown in the following Example 2.  That leads to the following research objectives (ROs). Establish a projection measure of a PFVα on anotherβ such that -RO 1: 0 ≤ Projβ (α) ≤ 1 for all the PFVsα andβ. To achieve these ROs, a new measure is developed as follows.

Criterion 2
The closer the N Pr ojβ (α) is to 1, the closer the vectorα is to theβ.
is the normalization projection of matrix X onto Y , where For two decision matrices X and Y , their closeness is based on the following Criterion 3.

Criterion 3
The closer the N Pr oj Y (X ) is to 1, the closer the matrix X is to the Y .

Research methodology and algorithm
Based on the new projection measure between two PFMs and existing VIKOR methods, a new GDM approach is developed under picture fuzzy environment, which will be applied to the software reliability evaluation.

Research methodology
In this section, an assessment method based on VIKOR method and the normalization projection measure is elaborated in order to evaluate software reliability.
The ith software product with respect to n attributes is evaluated by t DMs, which can be expressed by the following group utility matrices: where expresses the positive (agreeing, approving) degree, η i k j expresses the neutral degree and ν i k j expresses the negative (disagree, disapproving) degree of software product A i with respect to attribute u j . And expresses the refusal degree. For the weight vector w = (w 1 , w 2 , . . . , w n ) of attributes, the weighted group utility matrices are determined by: where by Definition 2. In VIKOR method, the ideal decision is a reference point. The largest group utility is composed of positive ideal decision as follows: where Here it is defined as: where To calculate the closeness between Y i and Y + , a projection of Y i onto Y + is based on Eq. (6) as follows: where . For the closeness between Y i and Y + based on Eq. (12), similar to Criterion 3, we have the following Criterion 4.

Criterion 4 The closer the N Pr oj Y + (Y i ) is to 1, the closer the Y i is to the Y + , the better the alternative A i is.
For measuring the closeness between G i onto G − , similar to (12), we have the following equation: where Similar to Criterion 4, we have the following Criterion 5.

Criterion 5 The closer the N Pr oj G − (G i ) is to 1, the closer the matrix G i is to the G − , the larger the group regret with regard to alternative A i is, the worse the alternative A i is.
The group utility measure of alternative A i is based on the following equation: where It is obvious that 0 ≤ S i ≤ 1. Similar to Criterion 5, we have the following Criterion 6.

Criterion 6
The closer the S i is to 1, the closer the Y i is to Y + , the larger the group utility is, then the better the alternative A i is.
as the largest group utility, the S − is called as the smallest group utility.
Inspired by literature (Zeng et al. 2019), the normalized group utility is defined as: In order to avoid the case that denominator S + − S − is zero in the traditional VIKOR method (Opricovic and Tzeng 2007), Eq. (15) sets N GU i = 1 when S + = S − .

Criterion 7 The closer the N GU i is to 1, the better the alternative A i is.
The group regret measure of alternative A i is determined by: where is same as in (13). Equation (16) satisfies the condition 0 ≤ R i ≤ 1, and we have the following Criterion 8.

Criterion 8 The closer the R i is to 1, the larger the group regret with regard to alternative A i is, the worse the alternative A i is.
The R + is called as the smallest group regret, the R − is called as the largest group regret. The normalized group regret of A i is defined as: where, in order to avoid the case that denominator R + − R − is zero, we let N G R i = 0 when R + = R − .

Criterion 9 The closer the N G R i is to 1, the better the alternative A i is.
Thus, a comprehensive VIKOR measure of alternative A i can be obtained by the following relation: where the λ is referred to as a compromise coefficient, and λ ∈ [0, 1]. The values of λ and 1 − λ are the weight of the normalized group utility N GU i and the normalized group regret N G R i , respectively. If λ > 0.5, it is indicated that DMs tend to make decisions based on the group utility; if λ < 0.5, it is indicated that DMs tend to make decisions based on the group regret; if λ = 0.5, it is indicated that DMs adopt a balanced and compromised way to make decisions. In general, the value ν = 0.5 is adopted. The grades of alternatives are affected by VIKOR indexes Q i , S i and R i collectively. The relative importance of three VIKOR indexes is that Q i S i and Q i R i , where the " " indicates "superior to".

Criterion 10 The larger the value Q i , the better the alternative A i is.
Let Q(A (h) i ) denote that the alternative A i is ranked in hth position by Q. A compromise solution should satisfy the following two conditions: where the m is the number of alternatives, then the alternative A i 1 is regarded as a compromise solution.
Note: The Q in Eq. (18) is compromised by N GU and N G R. Hence, the compromise solution should be consistent with N GU and/or N G R. However, the solution generated by Q may be inconsistent with the solutions generated by S and/or R (see an example in Table 4). So a supplementary condition is introduced as follows.
Condition 2. Acceptable stability in decision making: The alternative A i 1 must also be ranked first by S or/and R.
If the alternative A i 1 satisfies the only one condition, then the compromise solutions comprise a set based on the following rules (Opricovic and Tzeng 2007;Ren et al. 2017): -The compromise solutions are comprised of the alterna- for maximum M (the positions of these alternatives are "in closeness").

Assessment algorithm
The assessment algorithm of software reliability is based on above-mentioned VIKOR method and normalized projection measure with picture fuzzy information, which are involved into the following steps. The algorithmic pseudocode used to describe algorithms in this text is also attached in detail. The pseudocode can help us to understand the algorithms easily. It can serve as an intermediate step in the construction of programs implementing algorithms in one of a variety of different programming languages.
Step 1. Establish the group utility matrices. The group utility matrix X i of alternative A i is established by Eq. (5), where the X i (i ∈ M) are PFMs. A pseudocode for Step 1 can be described as follows: Function: Establish the set of group utility matrices X 1 , X 2 , · · · , X m : Step 2. Construct the weighted group utility matrices. For a given weight vector w = (w 1 , w 2 , . . . , w n ) of attributes, the weighted group utility matrices Y i (i ∈ M) are constructed by Eq. (8). The algorithmic in Step 2 can be described using an easily understood form of pseudocode as follows: Step 3. Determine the largest group utility. The largest group utility Y + is determined by Eq. (9). A pseudocode in Step 3 is given as follows: Step 4. Establish the group regret matrices. The group regret matrices G i (i ∈ M) based on Y + and Y i (i ∈ M) are established by Eq. (10) . The group regret matrices can be implemented by the following pseudocode: Step 5. Determine the largest group regret matrix.
The largest group regret matrix is determined by Eq. (11). A simple pseudocode example is provided as follows: Function: Determine the largest group regret matrix G − ; Step 6. Calculate the closeness between Y i and Y + . The normalization projection of Y i (i ∈ M) onto Y + are determined by Eq. (12).
A programming pseudocode is shown as follows: for i := 1 to m for k := 1 to t for j := 1 to n π i k j : Step 7. Measure the closeness between each group regret matrix and the largest group regret matrix based on the normalized projection.
The normalized projection of each group regret matrix G i (i ∈ M) onto the largest group regret matrix G − are determined by Eq. (13). A programming pseudocode is shown as follows: Function: Calculate the normalized projections N Proj G − (G i ), for i = 1, 2, · · · , m; Input: G i , G − , for i = 1, 2, · · · , m; Output: N Proj G − (G i ), for i = 1, 2, · · · , m; for i := 1 to m for k := 1 to t for j := 1 to n Step 8. Construct the group utility measurement. The group utility measure S i of alternative A i is determined by Eq. (14). The VIKOR index S i can be implemented by the following pseudocode: Function: Calculate the VIKOR index S i , for i = 1, 2, · · · , m; Input: N Proj Y + (Y i ), for i = 1, 2, · · · , m; Output: S i , for i = 1, 2, · · · , m; Step 9. Determine the normalized group utilities. The normalized group utilities are determined by Eq. (15). The normalized group utility can be implemented by the following pseudocode: Function: Determine the normalized group utility N GU i , for i = 1, 2, · · · , m; Input: S i , for i = 1, 2, · · · , m; Output: N GU i , for i = 1, 2, · · · , m; Step 10. Construct the group regret measurement. The group regret measurement of alternative A i is established by Eq. (16). The group regret measurement of alternative A i can be expressed by the following pseudocode: Function: Construct the group regret measurement R i , for i = 1, 2, · · · , m; Input: N Proj G − (G i ), for i = 1, 2, · · · , m; Output: R i , for i = 1, 2, · · · , m; Step 11. Construct the normalized group regret measurement. The normalized group regret measurements are constructed by Eq. (17). The algorithms used in Step 11 can be implemented by the following pseudocode: Function: Determine the normalized group regret N G R i , for i = 1, 2, . . . , m; Input: R i , for i = 1, 2, · · · , m; Output: N G R i , for i = 1, 2, · · · , m; Step 12. Construct the comprehensive VIKOR measurement.

C. Yue
The comprehensive VIKOR measure Q i of alternative A i is calculated by Eq. (18). The comprehensive VIKOR measure Q i can be expressed in an easily understood form of structured pseudocode: Function: Construct the comprehensive VIKOR measure Q i , for i = 1, 2, . . . , m; Input: N GU i , N G R i , for i = 1, 2, . . . , m; Output: Q i , for i = 1, 2, . . . , m; Step 13. Rank the preference order of alternatives. The alternatives are ranked in descending order in accordance with the VIKOR measures S i , R i and Q i . The compromise solutions are based on above Condition 1 and Condition 2 with two rules. The preference order of alternative A i can be based on the following pseudocode: Function: Rank the preference order of alternative A i , for i = 1, 2, . . . , m; i 1 ) denotes the software A i 1 is ranked in the first position in the ranking list by R.}; Output: The preference order of alternative A i , for i = 1, 2, · · · , m; . . , A i h+l } is tied for the hth position in the ranking list; end end return: is in decreasing order of software products A 1 , A 2 , · · · , A m with clas-sification}.

Experimental analysis
A software reliability assessment is shown in the following illustrative example section. The superiorities developed methodology in this paper are shown through some experimental analyses.

Illustrative example
This section provides a real evaluation of software reliability. Four software products are evaluated here, which are used in a university, Guangdong, China. For convenience, the evaluated software as alternatives comprise a set denoted by The DMs are users, who are from three different colleges in this university. They are written as D = {d 1 , d 2 , d 3 }, where each d k (k = 1, 2, 3) is a group of users from a college. More specifically, d 1 is the users from college of mathematics and computer science; d 2 is the users from college of mechanical engineering; d 3 is the users from college of ocean and engineering.
The assessment criteria/attributes are determined by DMs collectively. Four assessment attributes comprise a set denoted by U = {u 1 , u 2 , u 3 , u 4 }= {program complexity, program categories, programming language, design methodologies}, where the program size is used as a measure of program complexity; the program categories indicate system complexity; different programming languages have different complexity and structure, therefore the possibility for different languages to introduce errors are different; different design methodologies for the same software may have different impact on the quality of the final software products (Zhu and Pham 2017). Each software is evaluated by three DMs, whose evaluation values are integrally arranged in an evaluation matrix. By Step 1, four evaluation matrices are written as {X 1 , X 2 , X 3 , X 4 }, which are shown in Table 1.
In Table 1, the assessment values x i k j in X i are characterized by (μ i k j , η i k j , ν i k j )(i, j = 1, 2, 3, 4; j, k = 1, 2, 3), where the μ i k j is the ratio that the voters vote for ith software with respect to jth attribute in kth college; η i k j is the ratio that the voters abstain on ith software with respect to jth attribute in kth college; ν i k j is the ratio that the voters are against on ith software with respect to jth attribute in kth college.
The largest group utility Y + can be obtained by Step 3, which is also shown in Table 2.
Then the group regret matrices G i (i ∈ M) based on Y i (i ∈ M) and Y + are obtained by Step 4; the largest group regret matrix G − is obtained by Step 5. The group regret matrices and the largest group regret matrix are shown in Table 3.
To measure the closeness of each Y i to the largest group utility Y + , the normalized projections N Proj Y + (Y i ) (i ∈ M) are calculated by Step 6.
To measure the closeness between each G i and the largest group regret G − , the normalized projections N Proj G − (G i ) (i ∈ M) are calculated by Step 7, which are shown in Table  4. The group utility measures S i (i ∈ M) are obtained by Step 8, which are also shown in Table 4. The group regret index R i (i ∈ M) are obtained from N Proj G − (G i ), which are also shown in Table 4.
From Table 4 we can see that the largest group utility S + is 0.9965; the smallest group utility S − is 0.9717; the largest group regret R − is 0.7749; the smallest group regret R + is 0.7544. Therefore we have comprehensive VIKOR measurements Q i (i ∈ M) based on Step 12, which are also shown in Table 4. And the rankings of four products based on S i , R i , Q i , respectively, are summarized in Table 4. Table 4 shows that A 3 is ranked by Q i in the first position in the ranking list; {A 1 , A 2 } are ranked by Q i in the second position in the ranking list. And we can see that Q(A Condition 1 is satisfied. However, Condition 2 is not satisfied because A 2 is ranked by S i in the first position in the ranking list, and A 1 is ranked by R i in the first position in the ranking list. Thus, the compromise solutions are comprised of the alternatives {A 1 , A 2 , A 3 }. That is, the preference order of four products ranked with classification is {A 1 , A 2 , A 3 } A 4 ; the preference order of four products ranked in detail is

Comparison with a VIKOR-based GDM method
This section provides a comparison with a VIKOR-based GDM method (Yue 2020b). The data are based on the same illustrative example in Sect. 6.1. According to the idea in (Yue 2020b), a negative ideal decision from Y i in (8) is shown as follows: where Similar to (12), the normalization projection of Y i onto Y − , based on Eq. (6), are shown in the following equation: where is also measure for ranking alternatives, which is shown in the following criterion.

Criterion 11 The smaller value N Pr oj Y − (Y i ) means the better alternative A i (Yue 2020).
A group utility measure of alternative A i is provided by Yue (2020b) as follows: Criterion 12 The larger S 2 i means the better alternative A i (Yue 2020b).
A group regret measure of alternative A i is provided by Yue (2020b) as follows: are same as in Eq. (14).
If the group utility measure is replaced by Eq. (21), the group regret measure is replaced by Eq. (22), then the comprehensive VIKOR measure of software A i is shown as follows: where the N GU 2 The S 2 i is the same as in Eq. (21); R 2 i is the same as in Eq. (22). λ is a compromise coefficient, and λ ∈ [0, 1].
The VIKOR indexes S 2 i , N GU 2 i , N G R 2 i and rankings of four software products A i are shown in Table 5. Let λ = 0.5. The VIKOR comprehensive index Q 2 i and the ranking are also shown in Table 5. Table 5 shows that Q 2 (A (1) 2 ) = 1, so the A 2 and A 3 should be in the same classification based on the measure Q 2 . We further verify the difference 1 ) = 0.1238 − 0.0000 = 0.1238 < 0.3333, we know that A 1 and A 4 should be in the same classification. So we have {A 2 , A 3 } {A 1 , A 4 } based on the measure Q 2 . This ranking is different than the ranking {A 1 , A 2 , A 3 } A 4 with classification based on the measure Q in Table 4. The primary causes of difference are that there is not the specific regret information in the model (Yue 2020b). The group regret of software A i is measured directly by the complement of group utility 1 − S i . This is an imperfection.

Rankings based on different measures
In above experimental analysis, the measures S, N GU , R, N G R are used in current model. If each measure is used separately to rank the software A i , similar to the ranking of comprehensive VIKOR measure Q, we provide the following criterion.

Criterion 13 Let Mea(A (h)
i h ) denote that the alternative A i h is ranked in hth position by measure Mea ∈ {S, N GU , R, N G R}. Beginning by A i 1 , the software products {A i 1 , A i 2 , · · · , A i m } are ranked with classification in phases.
. . , A i h } are tied for the first place in the ranking list.

Phase 2. Beginning by
. . , A i l } are tied for the second place in the ranking list. In this case, we have that . . . . . . Phase end. According to the thought in Phase 2, we can end the procedure until A i m is graded in the ranking list.
According to Criterion 4, the group utility S i in (14) is also a measure. We search the ranking based on S i in Table  4. According to Criterion 13, We can see that S (A (1) 2 ) = 0.9965, S(A From Table 4, we can see that N GU (A (1) Table 4 shows that R( 2 ) = 0.7749. We determine the ranking in ascending order because the R is a cost index. From the difference R (A (4) 2 ) − R(A (1) 1 ) = 0.0205 < 1/3, we know that the four software products {A 1 , A 2 , A 3 , A 4 } is also one grade according to measure R.
If the regret measure R is normalized, then the N G R in Table 4 shows that N G R(A (1) The rankings with classification based on the measures S, N GU , R, N G R, Q in Table 4 are summarized in Table  6.
According to Criterion 13, the VIKOR indexes S 2 , N GU 2 , N G R 2 , Q 2 are also some measures to rank the order of four software products. Their results have been shown in Table 5. Similar to above analysis, we note that S 2 (A   Table 6 Rankings with classification based on the measures 1 ) = 0.1144 < 1/3, we can conclude that {A 2 , A 3 } {A 1 , A 4 } according to the measure N GU 2 . Similarly, we can verify and conclude that {A 2 , A 3 } {A 1 , A 4 } according to measure N G R 2 . These rankings, including the ranking based on the measure Q 2 , are also shown in Table 6.

Comparison with the classical projection
Equation (4) provided a classical projection, which has been used for years. As mentioned in Question 3 in the Introduction, this projection measure is not always reasonable in picture fuzzy setting. This section shows an experimental comparison with it. The data are based on the same illustrative example in Sect. 6.1.
In this subsection, the basic procedure is the same as the algorithm in subsection 5.2, but the projections are based on the classical projection measure. Specifically, (12) is replaced by where |Y + | = |Y + | 2 , |Y + | 2 and Y i Y + are the same as in (12).
(13) is replaced by where |G − | = |G − | 2 , G i G − and |G − | 2 are same as in (13). Eq. (14) is replaced by where Proj Y + (Y i ) is the same as in Eq. (24). Eq. (16) is replaced by where Proj G − (G i ) is the same as in Eq. (25). The comprehensive VIKOR index Q i is the same as in Eq. (15) and let λ = 0.5. Three VIKOR indexes and rankings of four products based on the classical projection are summarized in Table 7. Table 7 shows that Q(A 2 ) = 0.3825 > 1/3. Such Condition 1 is not satisfied. Only Condition 2 is satisfied. So the ranking with classification is {A 1 , A 4 } A 3 A 2 . This ranking with classification is different than the order based on Q from the Table 4, which is caused by the different projection measures.

Dynamic experiments
This section provides some experiments under dynamic environment. The experimental data are based on the data in Sect. 6.1.
First, let the compromise coefficient λ of Q i in (18) as a dynamic parameter. In order to show a clear figure, let λ = α/100, where α ∈ [0, 100]. Let α increases from 0 to 100. Then the rankings of four software products A 1 , A 2 , A 3 , A 4 based on Q i in (18) are shown in Fig. 1.
From Fig. 1 [90,100]. How to define the order of four software products in whole process? We define the criterion of ranking with intersection cases as follows.

Criterion 14
If the curves of rankings of software products A i and A j are intersecting, then the A i and A j are graded into the same classification.
According to Criterion 14, the ranking with classification of four software products based on Q i in (18) Second, the experimental data are tested in another dynamic environment. Now let us observe the x 1 11 = (μ 1 11 , η 1 11 , ν 1 11 ) = (0.49, 0.21, 0.20) in X 1 in Table 1. Let μ 1 11 = δ/100, η 1 11 = 0.8 − δ/100, where the δ ∈ [0, 79] is a dynamic parameter. Other values are the same as in Table 1. Let λ = 0.5 in (18), and let δ increases from 0 to 79. Then the rankings of four products A 1 , A 2 , A 3 , A 4 based on Q i are shown in Fig. 2. Figure 2 shows that A 3 is ranked in the first as δ increases/overlaps from 0 to 79; the curves A 2 and A 4 are intersected by the curve of A 1 . So we can say that A 3 {A 1 , A 2 , A 4 } according to Criterion 14.
In order to examine the influence of the compromise coefficient λ, we let λ be 0.3, 0.4, 0.6 respectively, the rankings of four products {A 1 , A 2 , A 3 , A 4 } based on Q i under the same  Figure 3 shows that the curves of four products {A 1 , A 2 , A 3 , A 4 } are intersecting. So we can say that four software products {A 1 , A 2 , A 3 , A 4 } are divided into one classification according to Criterion 14. Figure 4 shows that the curve of A 3 is located above the A 1 , A 2 , A 4 , and the curves of three products A 1 , A 2 , A 4 are intersecting. So we can conclude that A 3 {A 1 , A 2 , A 4 } according to Criterion 14.  (18) with λ = 0.6 and parameter δ such that x 1 11 = (μ 1 11 , η 1 11 , ν 1 11 ) = (δ/100, 0.8 − δ/100, 0.20) Figure 5 shows that the curve of A 2 , A 3 are located above the A 1 and A 4 , and the curves of A 1 and A 4 are intersecting. So we can conclude that A 3 A 2 {A 1 , A 4 } according to Criterion 14.
If let compromise coefficient λ be 1, then the Q i in (18) is degenerated into the largest group utility N GU i . If let compromise coefficient λ be 0, then the Q i in (18) is degenerated into the largest group regret N G R i . For the dynamic parameter δ in Fig. 2, the rankings of four products A 1 , A 2 , A 3 , A 4 based on N GU i in Eq. (15) are shown in Fig. 6. Figure 6 shows that four curves are disjoint. In this case, we can say that the ranking of four software products is A 2 A 3 A 4 A 1 based on N GU i in (15) under this dynamic sense.
For the dynamic parameter δ in Fig. 2, the rankings of four products A 1 , A 2 , A 3 , A 4 based on N G R i in Eq. (17) are shown in Fig. 7.  17) with parameter δ such that x 1 11 = (μ 1 11 , η 1 11 , ν 1 11 ) = (δ/100, 0.8 − δ/100, 0.20) Figure 7 shows that A 1 is intersecting to A 2 , A 3 , A 4 . Therefore, the four software products {A 1 , A 2 , A 3 , A 4 } are divided into one classification based on N G R i in Eq. (17) with parameter δ. Now we examine the changes based on the group utility S i in the dynamic sense. Let parameter δ be same as in Fig. 2, the rankings of four products based on S i in (14) are shown in Fig. 8. Figure 8 shows that four curves are disjoint. Therefore, the ranking based on S i in (14) with parameter δ is A 2 A 3 A 4 A 1 according to Criterion 14. Now we examine the ranking based on the VIKOR index R i . From Criterion 8, we known that the smaller the R i , the better the product A i is. In order to logical consistency, the R i is transformed as:  Fig. 9 Rankings of four products A 1 , A 2 , A 3 , A 4 based on R i in (28) with parameter δ such that x 1 11 = (μ 1 11 , η 1 11 , ν 1 11 ) = (δ/100, 0.8 − δ/ 100, 0.20) where it is obvious that the larger the value R i , the better the software A i is.
The rankings of four products A 1 , A 2 , A 3 , A 4 based on (28) are shown in Fig. 9, where the dynamic environment is same as in Fig. 2. Figure 9 shows that A 1 is intersecting to A 2 , A 3 , A 4 . Therefore, the four software products {A 1 , A 2 , A 3 , A 4 } are divided into one kind based on R i in (28) with parameter δ.
From above experimental analysis, we can see that (1) the different model may lead to different result; (2) the different experimental condition may lead to different result. We know that there are no the best model for decision science. A good model should be relative to other model. To show the most preferred ranking, a natural idea is that the ranking that occurs more frequently should be considered as a higher acceptance solution. The rankings based on above experimental results, including the results in Table 6, are shown in Table 8.   Table 8 shows that the software A 3 is the most preferred product, which appears 16 times in 18 experiments. So this solution should be believable. In this sense, A 3 is ideal, followed by A 2 , A 1 and A 4 .

Conclusion
Regarding the Questions 1 and 4 in Introduction section, this work has provided a VIKOR-based GDM method, in which the users of software are experts. The users' satisfaction and expectations are reflected in assessment criteria, which are focused on the concerns of the users of software. Regarding the Question 2 in Introduction section, an evaluation method of software reliability has been developed by the aid of a novel VIKOR-based GDM method with picture fuzzy information. Regarding the Question 3 in Introduction section, a new normalization projection measure has been developed in picture fuzzy setting.
These are some main innovations and differences in the new VIKOR-based GDM method, which are listed as follows: 1. The group utility of each alternative A i is shown by a decision matrix X i in the new method; whereas the group utility of each alternative A i is shown by a decision vector in the traditional VIKOR method.
2. The largest group utility is composed by a decision matrix Y + (see (9) ) in the new method; whereas the largest group utility (or the best/positive criterion (Opricovic and Tzeng 2007), or ideal solution (Büyüközkan et al. 2019)) is composed by a vector f + in the traditional method. These is no the worst (or negative) criterion f − in the new method. 3. A group regret matrix G i is provided in the new method.
The largest group regret matrix is composed by the maximum matrix G − of G i (i ∈ M) in the new method; whereas there is no group regret in the traditional method. 4. The group utility measurement is based on the normalization projection of Y i on Y + in the new method; whereas the group utility measurement is based on the positive and negative ideal solutions shown by two vectors f + and f − in the traditional method. 5. The group regret measurement is based on the normalization projection of G i on G − in the new method; whereas the group regret measurement is based on the positive and negative ideal solutions shown by two vectors f + and f − in the traditional method. 6. A new ranking method under a static environment and a new ranking method in a dynamic environment are provided in this work; whereas only a compromise solution can be obtained in the traditional method. 7. The closeness between two decision matrices is based on a normalization projection in the new method; whereas the closeness between two decision matrices is based on the Euclidean distance or the Hamming distance in traditional method (Çalı and Balaman 2019). 8. There is no the collective decision aggregated by all the individual decisions in decision process in the new method; There is a collective decision aggregated by all the individual decisions in decision process in the traditional method.
The above-mentioned differences and innovations have led to a technical promotion of VIKOR-based GDM method. The technical promotion in this work can greatly improve the evaluation effect of software reliability.
We know that any model has its limitations. The new model provided in this paper is no exception. First, the evaluation information is expressed by PFN. This is a limitation. The future research should extend to other information, for example, interval data, intuitionistic fuzzy number, intervalvalued intuitionistic fuzzy number, and so on. Second, these is only a single decision information to characterize attribute values in this work. However, in practical problems, if the attribute values can be characterized in different information representations, some evaluation questions may be more convenient to handle. The future research should extend the evaluation information to with hybrid information representations. Third, the distance between two decision matrices is employed by a projection measure. This is a limitation. The future research should extend to other measures. For example, the Euclidean distance or the Hamming distance. Fourth, the proposed model is only implemented in software reliability evaluation. This is a limitation. The future research may be implemented in a real time datasets, like UCI Machine Learning Repository data to evaluate the performance of some software qualities. And the time and space complexity will also be considered in future researches, where the algorithm will be specified with more details for the software reliability evaluation. Fifth, the effectiveness of the suggested method is verified in some static experiments and some dynamic experiments. This is a limitation. The effectiveness of the suggested model should also be demonstrated by more methods in future researches, like normal simulation, real-time simulation, and so on.