The implementation of creating the software testing assessment tools for the Integration-Based Model (I-BM) framework includes the following steps:
4.1 Problem identification
In this stage, problem identification will be conducted, including:
a. The absence of a software testing framework to detect errors or fault outputs by testers.
b. Creating the Model-Based Testing (MBT) called I-BM framework as a software testing application.
c. Assessing the complexity level of the I-BM framework.
4.2 Problem analysis
For this stage, the use cases, activities, and database design of the software testing assessment tools to detect errors or fault outputs by testers will be depicted.
4.2.1 Architecture of the Integration-Based Model (I-BM) Framework software testing application
Figure 2 Architecture of the Integration-Based Model (I-BM) framework software testing application.
In Fig. 2, the architecture of the I-BM framework software testing application is depicted, which will be used by both the I-BM admin and tester. The application consists of five tables, namely:
- Tester master: This table is used for managing tester data, with menu options including save, edit, delete, and search.
- Fault master: This table is used for managing fault data, which consists of three levels of faults: low, middle, and high. The menu options include save, edit, delete, and search.
- Spread tester: This table is used for distributing software to testers. The menu options include save, edit, delete, and search.
- Software master: This table is used for managing software data, with menu options including save, edit, delete, and search.
- I-BM framework: This table is the result of combining the tester master, fault master, spread tester, and software master tables. The menu options include assessment and assessment check.
4.2.1 Use Case Diagram
Figure 3 Use Case tools assessment pengujian perangkat lunak framework Integration-Based Model (I-BM).
Based on the above use case, it can be described as follows[25]:
- The I-BM admin actor will input master fault data, tester data, software data, and tester distribution.
- The I-BM admin actor can access the results of software testing using the I-BM framework.
- The I-BM tester actor can input and access the results of software testing using the I-BM framework.
4.3 Design interface Integration-based(I-BM) model framework
For the design of the I-BM framework assessment tools to be used by the I-BM admin and I-BM tester, there are several interface designs. Here are some interface designs for the Integration-Based Model (I-BM) framework[26]:
Modules accessible to the I-BM admin include[27]:
- Master Fault module: This module processes fault level data, consisting of several fields such as fault ID, fault level, and fault value.
- Master Tester module: The purpose of this module is to manage information about testers, with fields including tester ID and tester name.
- Master Software module: This module manages data of the software programs to be tested, with fields such as software ID, software name, software owner, and software description.
- Tester Distribution module: This module processes the distribution of testers to software programs.
- Software Testing Results module: This module displays the results of software testing conducted by the testers.
- I-BM Faults Output Prediction module: This module predicts fault outputs using the neural network algorithm.
Modules accessible to the I-BM tester include:
- I-BM Framework Assessment module: This module is used by the tester to evaluate software testing.
- I-BM Framework Assessment Dataset module: This module allows the tester to view the dataset of software testing results.
4.4 Practical Complexity Application of the I-BM Framework Software Testing
In this stage, the effort estimation is calculated using the use case point (UCP) method. The effort estimation serves as the basis for estimating time, human resources, and costs. The effort calculation starts with the value of unadjusted actor weight (UAW). UAW is calculated based on the actors involved in the system, which are mapped to actor categories to obtain the weight of each actor.
After obtaining the actor weights and use case weights, the calculation of unadjusted actor weight (UAW) and unadjusted use case weight (UUCW) is performed[28]:
Table 1
Calculating the value of UAW
No
|
Actor Type
|
Weight
|
Total Actor
|
Total
|
1
|
Simple
|
1
|
0
|
0
|
2
|
Average
|
2
|
1
|
2
|
3
|
Complex
|
3
|
1
|
3
|
Total UAW
|
5
|
Table 2
Calculating the value of UUCW
No
|
Actor Type
|
Weight
|
Total Actor
|
Total
|
1
|
Simple
|
5
|
0
|
0
|
2
|
Average
|
10
|
4
|
40
|
3
|
Complex
|
15
|
2
|
30
|
Total UAW
|
70
|
The next step is to calculate the magnitude of the value in the unadjusted use case points (UUCP), which is obtained by adding the total value of UAW and the total value of UUCW.
The next step is to calculate the technical complexity factor (TCF) and the environment complexity factor (ECF).
Table 3
Calculating the value of TCP
No
|
Technical Factor
|
Weight
|
Score (0–5)
|
T1
|
Distributed sistem required
|
2
|
4
|
T2
|
Response time
|
2
|
4
|
T3
|
End user efficiency
|
1
|
4
|
T4
|
Complex internal processing required
|
1
|
4
|
T5
|
Reusable code
|
1
|
4
|
T6
|
Easy to Install
|
0.5
|
5
|
T7
|
Easy to use
|
0.5
|
5
|
T8
|
Portable
|
2
|
5
|
T9
|
Easy to Change
|
1
|
5
|
T10
|
Concurrent
|
1
|
3
|
T11
|
Security features
|
1
|
4
|
T12
|
Access for Third Parties
|
1
|
4
|
T13
|
Special training required
|
1
|
4
|
Total TF
|
63
|
Total TCF (0.6+(0.01*TF))
|
1.23
|
Table 4
Calculating the value of ECF.
No
|
Environment Factor
|
Weight
|
Score (0–5)
|
B * S
|
E1
|
Familiarity with the project
|
1.5
|
4
|
6
|
E2
|
Application experience
|
0.5
|
5
|
2.5
|
E3
|
Object oriented programming experience
|
1
|
2
|
2
|
E4
|
Lead analyst capability
|
0.5
|
5
|
2.5
|
E5
|
Motivation
|
1
|
5
|
5
|
E6
|
Stable requirements
|
2
|
5
|
10
|
E7
|
Part time staff
|
-1
|
3
|
-3
|
E8
|
Difficult programming language
|
-1
|
5
|
-5
|
Total EF
|
20
|
Total ECF (1.4 + (-0.03 * EF))
|
0.8
|
After obtaining the TCF and ECF values, the next step is to calculate the use case points (UCP) and hours of effort.
$$UCP = UUCP * TCF * ECF$$
$$UCP = 75 * 1.25 * 0.8 = 73.8$$
$$Hours of effort = UCP * 20 = 1476 (man-hours)$$
The results of UCP and hours of effort have been obtained, which are 73.8 and 1476 (man-hours) respectively. The next step is to calculate the time and cost estimation that will be used in the development of the I-BM framework application.
Table 5
Calculating estimated time and costs with PF = 4
Effort
|
1476
|
Productivity Factor
|
Productivity Factor(Karner, 1993), PF
|
28
|
20
|
10
|
5
|
4
|
2
|
Duration (hours)
|
41328
|
29520
|
14760
|
7380
|
5904
|
2952
|
Duration (week)
|
861
|
615
|
307.5
|
153.75
|
123
|
61.5
|
Duration (month)
|
112.9180328
|
80.6557377
|
40.32786885
|
20.16393443
|
16.13114754
|
8.06557377
|
Table 6
Calculating estimated time and costs with PF = 5
No
|
Position
|
Total
|
Workload (hours/
day)
|
Workload (hours/
week)
|
Total (hours/
week)
|
Workload Total
|
Salary
(Per Hour)
|
Sum
Salary
|
1
|
System Analyst
|
1
|
4
|
20
|
20
|
2,460
|
20,000
|
49,200,000
|
2
|
Programmer
|
1
|
8
|
40
|
40
|
4,920
|
20,000
|
98,400,000
|
3
|
Tester
|
1
|
4
|
20
|
20
|
2,460
|
20,000
|
49,200,000
|
Total
|
3
|
|
196,800,000
|
Table 7
Calculating estimated time and costs with PF = 10
No
|
Position
|
Total
|
Workload (hours/
day)
|
Workload (hours/
week)
|
Total (hours/
week)
|
Workload Total
|
Salary
(Per Hour)
|
Sum
Salary
|
1
|
System Analyst
|
1
|
4
|
20
|
20
|
3,075
|
20,000
|
61,500,000
|
2
|
Programmer
|
1
|
8
|
40
|
40
|
6,150
|
20,000
|
123,000,000
|
3
|
Tester
|
1
|
4
|
20
|
20
|
3,075
|
20,000
|
61,500,000
|
Total
|
3
|
|
246,000,000
|
No
|
Position
|
Total
|
Workload (hours/
day)
|
Workload (hours/
week)
|
Total (hours/
week)
|
Workload Total
|
Salary
(Per Hour)
|
Sum
Salary
|
1
|
System Analyst
|
1
|
4
|
20
|
20
|
6,150
|
20,000
|
123,000,000
|
2
|
Programmer
|
1
|
8
|
40
|
40
|
12,300
|
20,000
|
246,000,000
|
3
|
Tester
|
1
|
4
|
20
|
20
|
6,150
|
20,000
|
123,000,000
|
Total
|
3
|
|
492,000,000
|
Based on the calculation results of time and cost estimation with PF = 4, 5, and 10, it indicates that the development of the application until the software testing phase shows a high cost, requires a significant amount of time, and demands a considerable amount of human resources. The practical complexity/application of the I-BM framework for the best cost, time estimation, and human resources is as follows: a cost of Rp.196,800,000, with a time estimation of 2,460 man-hours, 4,920 man-hours, and a team of 3 people.
4.5 Model Training Methodology
The Integration-Based Model (I-BM) framework utilizes an optimized neural network trained with the Adam optimizer to improve the performance of each layer. The training process includes parameters such as a batch size of 6, 100 epochs, and a validation split of 0.2, which are applied to the entire dataset[29]. Additionally, the SVM algorithm is employed with parameters such as C = 1, kernel = rbf, degree = 3, and gamma='scale'[30]. Table 8 presents the results of the neural network algorithm testing. Figure 8 and Fig. 9 illustrate the outcomes of configuring the neural network model in relation to the I-BM fault output and manual fault output, respectively.
Table 8
General Performance of The Neural Network Classifier
Epoch
|
Learning Rate
|
Hidden Layers
|
Inputs Layers
|
Output Layers
|
Error Per Epoch
|
500
|
0.3
|
4
|
16
|
2
|
0.0023469
|
500
|
0.2
|
4
|
16
|
2
|
0.0023304
|
1000
|
0.3
|
4
|
16
|
2
|
0.0023271
|
Figure 8 shows the testing results of the training and testing losses, both of which have a value of 0.044. Additionally, the training and testing accuracies have the same value of 0.991 or 99%. Figure 9 displays the configuration results of the neural network algorithm model for the manual fault output.
Figure 9 shows the results of testing the neural network algorithm with Adam optimization using a batch size of 6, epochs = 100, and validation split = 0.2. From the configuration, we obtained the following results: the Test loss for both training and testing is 0.0558, and the Test accuracy for both training and testing is 0.9876/98%. Table 9 shows the results of testing the algorithm and SVM.
Table 9
General Performance of The SVM Classifier
Model
|
Parameter
|
Accuracy
|
SVM
|
(rbf; C = 1; degree = 3; scale)
|
0.98360
|
For the SVM method, we can note the results shown in Table 9, which consider the SVM kernel and parameters. The results displayed in Table 9 show that with the rbf kernel; C = 1; degree = 3; and gamma = scale, the best result was obtained with an accuracy score of 0.98360. Table 10 and Table 11 show the results of the performance evaluation of the neural network algorithm and SVM.
Table 10
The performance evaluation results of the neural network algorithm model with batch size 300, learning rate 0.3, momentum 0.2, training time 1000, and validation threshold 20.
|
Precision
|
Recall
|
F-Measure
|
ROC Area
|
Class
|
0.997
|
1.000
|
0.998
|
0.999
|
Validated
|
1.000
|
0.667
|
0.800
|
0.999
|
Invalidated
|
Weighted Avg.
|
0.997
|
0.997
|
0.996
|
0.999
|
|
Table 11
The performance evaluation results of the SVM model with C = 1, kernel = rbf, degree = 3, and gamma='scale'.
|
Precision
|
Recall
|
F-Measure
|
ROC Area
|
Class
|
0.998
|
1.000
|
0.998
|
0.999
|
Validated
|
1.000
|
0.687
|
0.800
|
0.999
|
Invalidated
|
Weighted Avg.
|
0.998
|
0.997
|
0.996
|
0.999
|
|
4.6 The Comparison Results of Fault Output between Neural Network Algorithm, SVM, and I-BM with Actual Expected Fault Output
Table 12 shows the comparison results of accuracy levels between the Neural Network Algorithm, SVM, and I-BM in relation to the actual expected fault output.
Table 12
The result of the accuracy comparison between neural network, SVM, and I-BM algorithms against the actual expected.
NN Accuracy
|
NN Status
|
SVM Accuracy
|
SVM Status
|
I-BM Accuracy
|
I-BM Status
|
0.840351
|
Validated
|
0.83165
|
Validated
|
1
|
Validated
|
0.738451
|
Validated
|
0.880988
|
Validated
|
0
|
Invalidated
|
0.708488
|
Validated
|
0.34892
|
Validated
|
0
|
Invalidated
|
0.953266
|
Validated
|
0.814968
|
Validated
|
1
|
Validated
|
0.917027
|
Validated
|
0.857487
|
Validated
|
1
|
Validated
|
0.900248
|
Validated
|
0.849822
|
Validated
|
1
|
Validated
|
0.928153
|
Validated
|
0.882471
|
Validated
|
1
|
Validated
|
0.509445
|
Validated
|
0.710596
|
Validated
|
1
|
Validated
|
0.934147
|
Validated
|
0.863378
|
Validated
|
1
|
Validated
|
0.565435
|
Validated
|
0.717161
|
Validated
|
1
|
Validated
|
0.937718
|
Validated
|
0.909719
|
Validated
|
1
|
Validated
|
0.919425
|
Validated
|
0.874257
|
Validated
|
1
|
Validated
|
0.939598
|
Validated
|
0.889272
|
Validated
|
1
|
Validated
|
0.919287
|
Validated
|
0.878019
|
Validated
|
1
|
Validated
|
0.972155
|
Validated
|
0.856022
|
Validated
|
1
|
Validated
|
0.939598
|
Validated
|
0.817152
|
Validated
|
1
|
Validated
|
0.919287
|
Validated
|
0.876974
|
Validated
|
1
|
Validated
|
0.972745
|
Validated
|
0.881912
|
Validated
|
0
|
Invalidated
|
0.912635
|
Validated
|
0.834156
|
Validated
|
1
|
Validated
|
0.971153
|
Validated
|
0.856057
|
Validated
|
1
|
Validated
|