Automating the mapping of course learning outcomes to program learning outcomes using natural language processing for accurate educational program evaluation

Quality control and assurance plays a fundamental role within higher education contexts. One means by which quality control can be performed is by mapping the course learning outcomes (CLOs) to the program learning outcomes (PLO). This paper describes a system by which this mapping process can be automated and validated. The proposed AI-based system automates the mapping process through the use of natural language processing. The framework underwent testing using two actual datasets from two educational programs, and the findings were promising. A testament to the potential of the suggested framework was the precision of the mapping detected (83.1% and 88.1% for the two programs, respectively) compared to the mapping performed by the domain experts. A web-based tool was created to help teachers and administrators execute automatic mappings (https://dsaluaeu.github.io/mapper.html). The data and software used in this research project can be found at the following URL: https://github.com/nzaki02/CLO-PLO.


Introduction
The current educational system is far from ideal.Numerous issues arise that need to be continuously identified and addressed.However, educational outcomes can be enhanced by implementing quality assurance and adhering to accreditation procedures.The process of ensuring that academic objectives and standards are followed is part of what is known as quality assurance (QA).A robust QA system is essential in higher education contexts to ensure that students are subject to the highest standard of education.Accreditation signifies that a recognized body has certified that a program offered by a given institution meets its required standards and criteria.As such, it is imperative that higher education institutions (HEIs) are structured in a manner that includes QA as a crucial component.This can be seen as a prerequisite that must be met when HEIs apply for a program or institutional accreditation from a national or international accrediting body.Specific to educational programs, QA procedures ensure that the objectives of a course, program, or related service unit are clearly specified, planned, implemented, and verified.Identified weaknesses and gaps through the implementation of QA need to be addressed to ensure continuous feedback and improvements.QA procedures are designed to verify that high-quality educational programs are being delivered to satisfy the HEI goals.This implies implementing QA at the various levels of a program starting from defining its objectives and goals, its learning outcomes and at a more granular level the course learning outcome (CLOs) for every course being offered by the program.In doing so, an alignment with the HEI goals, mission, and vision need to be considered while continuously making sure of any changing requirements.Additionally, adopting QA will result in the graduates of a program securing better job placements (Immerstein et al., 2020).
Although the idea of QA is not new, it can be improved with the recent advancements in innovative fields such as Artificial Intelligence (AI).For instance, Ujkani et al. (2021) employed natural language processing (NLP) to verify consistency among program syllabi and their associated learning outcomes.
A HEI needs to obtain a permission/license to operate and offer programs of studies.In addition, in certain countries, it is required to obtain accreditation from local or external bodies for the offered degrees to be recognized.Accreditation aims to establish uniform standards across all programs and guarantee that students can exhibit particular competency levels after completing such programs.To have faith that the program outcomes are founded on global standards and best practices based on a peer-reviewed process of a specific program's criteria.Accreditors develop the specific applied program criteria as part of a set of guidelines and operational manuals.Implementing a worldwide benchmarking methodology promotes legitimacy and greater employability for the graduate of such programs.
The program curricula, faculty qualifications, admission standards, facilities, research, community participation, institutional support, and program-specific outcomes are all examined as part of the accreditation process and as required by the accrediting standards and criteria.Thus, prospective students and faculty 1 3 Education and Information Technologies (2023) 28:16723-16742 interested in joining, as well as companies and other HEIs looking for graduates of such programs, view accredited programs high-quality programs.It is for this reason that HEIs may view accreditation as a key priority (Shafi et al., 2019).
One of the main criteria in the evaluation any offered program is making sure that the Program learning Outcomes (PLOs), also known as Student Outcomes (SOs), are regularly and properly assessed.PLOs refer to what students are supposed to be able to achieve upon graduating from a program of study… PLOs are designed to capture knowledge, competencies, and attitudes required to execute a successful program.Clear and measurable PLOs should be defined when designing an educational program to enable proper assessment and continuous improvement of the program.The success level of a program delivery greatly depends on the successful attainment of its PLOs.The attainment of PLOs is usually measured using various direct and indirect assessment tools.While CLOs is the most widely direct tool used, indirect tools may include surveys and other subjective measurements.CLOs represent what students are supposed to achieve upon completing a particular course or module.Since CLOs are defined per course and for every course, they provide a reflection of the overall program curriculum coverage.To measure the attainment of PLOs using CLOs proper mapping between the two is needed where certain CLOs are mapped to each PLO based on relevance.Thus, the achievement levels of associated CLOs constitute a significant performance indicator of the attainment of PLOs and how well the program curriculum is designed to cover such PLOs.
The process of mapping CLOs to their associated PLOS is time-consuming and can be subjective.A two-dimensional matrix that expresses the correlation between PLOs and associated CLOs is frequently employed.However, this mapping task is difficult, even for experienced program educators and directors.The complexity is derived from the fact that mistakes are likely to occur throughout the mapping process, and program directors need to be well aware of proper techniques for mapping while having a good understanding of the overall program curriculum.Since the CLOs-PLOs mapping is usually done by faculty members and due to having courses often offered across programs, it is difficult to identify inconsistencies in such mappings (Alshanqiti et al., 2020).Given the significance of using CLOs to measure the attainment of PLOs, i.e., evaluating the efficacy of a program curriculum, its crucial to maintain consistency and accuracy in doing CLOs-PLOs mapping.
Therefore, the research question this research work is intended to answer is "Can Artificial Intelligence and in particular, Natural Language Processing be employed to automate and provides accurate and consistent mapping of Course Learning Outcomes (CLOs) to Program Learning Outcomes (PLOs), so educational programs are effectively and efficiently evaluated?".
To the best of our knowledge, no previously published work has aimed to automate or enhance the precision of the CLOs to PLOs mapping.There are, however, several initiatives regarding curricular mapping in general.By using a curriculum mapping technique, educators can better understand what has been taught in a class, how it has been taught, and how the PLOs and CLOs were evaluated.
In a previous effort, Plaza et al. (2007) aimed to illustrate the application of curriculum mapping in program evaluation and assessment.The authors adopted a descriptive cross-sectional study approach based on a document outlining learning outcomes and numerous additional student and curriculum data sets that were already available.However, the primary objective was to compare the graphical curriculum maps created by students and professors.Comparing the maps' relative rankings of each domain's emphasis revealed that the intended/delivered and received curricula agreed with one another.
A method for gathering, analyzing, and presenting information on teaching and the evaluation of graduate competencies was introduced by Spencer et al. (2012).The suggested discursive technique encourages reflection-based practice in curriculum design, and the resulting heat maps offer diagrammatic representations of current practices and pointers as to where the curriculum should be redesigned.
Another early work by Veltri et al. (2012) presented a comprehensive and structured curriculum-mapping framework, which was then applied to examine an Information Systems (IS) program.Such frameworks continue to assist in curriculum improvement as well as accreditation to evaluate how intentionally IS curricula advances expected PLOs, and thus learning outcomes are effectively achieved.Lam and Tsui (2014) examined the alignment between subject learning outcomes (SLOs) and the course curricula documents.The authors conducted content analysis to map the SLOs in the curricula documents of a set of compulsory courses.
To ease the complexity of the assessment process, Ibrahim et al. (2015) developed a web-based course assessment tool to manage the collection, aggregation, and analysis of the assessment data.Supraja et al. (2017) automated the labeling of course questions to improve the alignment with the CLOs intended by the course designers.The authors used term frequency-inverse document frequency (TF-IDF) and latent Dirichlet allocation (LDA) to transform the course questions from text to word weightages before employing support vector machine (SVM) and extreme learning machine (ELM) to train and automatically label the questions.
Moreover, the effects of curriculum ideas in higher education were discussed by Linden et al. (2017).The authors of this study concentrated on the intellectual and historical foundations of approaches to curricular theory.In higher education environments, competency-based and outcome-focused contexts were employed to avoid separating the normative and critical roles of curriculum frameworks.They recommended that everyone concentrate on the curriculum's educational value and update it in accordance with higher education norms.
A novel approach to curriculum mapping, known as the web-based learning opportunities, goals, and outcome platform (LOOOP) method, was proposed by Treadwell et al. (2019).The authors conducted a questionnaire survey with a fourpoint Likert scale to ascertain how the instructors perceived the projected benefits of curricular mapping.The authors concluded that instructors' comments on LOOOP's worth and usability were favorable.
For the objective of creating a four-dimensional typology for curricular maps that outlines features relating to their purpose, product, process, and display, Watson et al. (2020) undertook a thorough assessment of the higher education literature.They sought to verify the framework by comparing the parameters with six curriculum maps from medical schools around Australia.Educators who specialize in the health profession are 1 3 Education and Information Technologies (2023) 28:16723-16742 anticipated to use the proposed typology to guide crucial choices regarding the available curricular map possibilities.
A rule-based strategy was recently created by Alshanqiti et al. (2020) that aims to automate the evaluations of academic curriculum mapping.The goal of this method is to make it possible to examine the CLOs-PLOs mappings, identify inconsistencies, and offer recommendations for enhancing the curriculum mapping.The authors recommended a rule-based approach for curricular matrix assessments.The authors also used curriculum mapping specialists to create a web interface tool that leveraged user-based experiments to automate evaluations of their academic programs.However, the CLO to PLO mapping was also manually executed.
Unlike earlier research, our goal here is to use Natural language processing (NLP) to automate, simplify, and remove subjectivity from the CLOs-PLOs mapping process.To the best of our knowledge, this study is the first to use this technique.NLP is a subfield of artificial intelligence (AI) that studies how computers and human (natural) languages interact.It can be viewed as the research and development of tools that enable computers to process linguistic information in a manner comparable to that of humans.NLP has been around for a long time, but due to recent developments in machine learning, big data, data science, and deep learning, it has recently gained popularity as a subject of study.The field spans a wide range of useful applications, including sentiment analyzers, chatbots, search engines, online translators, automatic summarizers, and recommendation systems.The primary objective of this study is to identify the semantic connections between the PLOs and the CLOs to enable more precise mappings and, ultimately, meaningful evaluation of the educational program.A web interface tool is also created to aid administrators and teachers in quickly and automatically performing AI-based mapping.

Proposed framework
This section presents an overview of the proposed framework and the methods employed in this study.The framework takes two sets of PLOs and CLOs inputs and perform text preprocessing (tokenization, remove punctuation and stop words, stemming, and lemmatization).This step is followed by the text representation module in which unique words are extracted and converted into vectors.The cosine similarity algorithm is then employed to facilitate the measurement of the similarity of vectors which indicate similarity between a particular CLO to the PLOs.Finally, we evaluate the outcome of the proposed framework by comparing its performance to what was achieved by the domain expert.The details of the proposed framework are explained in the proceeding sections and illustrated in Fig. 1.

Data
The Department of Computer Science and Software Engineering at the College of Information Technology, United Arab Emirates University (UAEU), offers a Bachelor of Computer Science approved by ABET and is the basis for the CLOs and PLOs data utilized in this study.To graduate, students must complete 42 courses, which equates to 130 credits.For the purposes of accreditation, 26 college and programrequired courses, totaling 121 CLOs, were manually mapped to the six PLOs of the programs.Each course coordinator independently generated the mapping, which was subsequently approved by the department council.
An additional dataset based on its ABET-accredited bachelor's degree program in Information Security delivered by the Department of Information Systems and Security at the same college was also taken into account.Thirty-four courses from the 130 credits hours program were considered (including 173 CLOs).The CLOs were manually mapped to six PLOs, and the department council validated the mapping.The Supplementary Materials files can be accessed online (https:// github.com/ nzaki 02/ CLO-PLO ) and contain the course list, related CLOs, PLOs for the two programs, and manual mappings.

Input module
The first step involves presenting the PLOs and CLOs in a table format that can undergo further text preprocessing.The number of the CLO collections (i.e., the number of the courses in the program) is represented by n and the number of the CLOs in each course, i , is j i where j i = 1, 2, … , n .Consequently, each table is ( j i × 2 ) with each row containing the ID and corresponding CLO.The final table is developed by combining the PLOs with the CLOs; i.e., every PLO is inserted as the first row of each CLO table.Consequently, if the quantity of PLOs is m , we have n × m output tables.This is referred to as the "PLO-and-CLOs" tables and is denoted by

Text preprocessing module
In the next step, each CLO and PLO were converted into a list of their constituent words (word tokenization).All text was presented in lowercase because there is no distinction between lowercase and uppercase terms.Punctuation and stop words-commonly used English terms like "a," "an," "the," "in," "of," etc.-were eliminated.
The last stage in this module involved stemming, which entails eliminating suffixes and reducing a word to a base form so that all of its variations may be represented by the same form, or lemmatization, which is mapping all of a word's variations to its root word, or lemma (Vajjala et al., 2020).For instance, "work" replaces words like "works", "worked", and "working".These procedures were crucial in the current study because they enabled the construction of dense word vectors, known as bag of words (BOW), which is a collection (list) of words that disregard context and order.These also supported the accurate counting of the number of words sharing the same base or stem.
Finally, we constructed the multiset of words (i.e., set of words that may have multiple occurrences) for each CLO and PLO in every table T i .The set of words corresponding to the k th (where k = 1 refers to the PLO in the current table) learn- ing outcome in table T i was represented by t i,k , where i = 1, 2, … , n × m and k = 1, 2, … , q i denote the set of words.

Text representation module
The local vocabularies were defined as V i , i = 1, 2, … , n × m .The words obtained from the table T i (see the input module) was represented by vocabulary, V i .We mapped each word, w , in vocabulary, V i , to a unique integer ID between 1 and | denotes the number of words in V i .This mapping resulted in the genera- tion of a list of unique words w 1 , w 2 , … , w p i where | dimensions, called count-vector of words, where the j th component was the frequency of the word, w j , occurring in t i,k .Consequently, we obtained the matrix (2D array) B i for each T i , i = 1, 2, … , n × m , where the size of each B i is j i + 1 × p i .

CLO-to-PLO mapping module
In this module, the obtained matrices B i , i = 1, 2, … , n × m , were used to accu- rately map each CLO to PLO(s).For this purpose, we employed the cosine similarity, which facilitates the measurement of the similarity of vectors b i,k and where b i,k l and b i,1 l are the l th components of vectors b i,k and b i,1 , respectively.If the cosine value of the vectors b i,k and b i,1 is close to 1, they are deemed to be similar.The possibility of mapping CLO i k , k = 2, 3, … , j i + 1 , in each CLOs group i , i = 1, 2, … , n × m , into the PLO j , j = 1, 2, … , m , for which the cosine similarity is computed, is delineated by establishing a specific dynamic threshold i for each group of CLOs and PLO based on the minimum and maximum values of the cosine similarities computed: Further, we constructed the n "CLOs-to-PLOs" mapping tables.These tables took the form of Boolean matrices (i.e., matrices with 0 and 1 entries), the cosine similarity of the corresponding vectors b i,k l and b i,1 l was greater than or equal to the established threshold , In its turn, m kl = 1 in matrix M i entailed that CLO i k was positively mapped to PLO l , otherwise, i.e., m kl = 0 , it was not mapped

Evaluation module
The purpose of this module is to analyze the accuracy of the CLO-to-PLO mappings produced by the model.This was achieved by comparing the model's CLO-to-PLO mappings to those presented by the human expert(s), as described in Section 2.1.The aggregated Boolean matrices, H i , of sizes j i × m Were based on the table of expert mappings we constructed, where each entry is defined by selecting the maximum of the numbers of 0 and 1 s in the corresponding entries of the expert tables.We then defined the evaluation matrix, S i = s kp i , for each pair of matrices M i and H i .An entry was recorded as 1 if the corresponding entries of M i and H i were of the same value, and 0 otherwise.
We defined the model accuracies with respect to each PLO, each CLO, each course CLOs, and the total accuracy using the evaluation matrices S i as follows: Education and Information Technologies (2023) 28:16723-16742 "PLO" accuracy was defined for each PLO, p , in the program as the ratio of the sum of the sums of the values of the column p in all matrices S i by the total number of CLOs in the program; i.e., "CLO" accuracy was defined for each CLO, k , in the course, i , as the ratio of the sum of values of the row representing the CLO by the total number of PLOs in the program, i.e., "CLOs-to-PLOs" (course) accuracy was defined for all CLOs in the course, i , as the division of the sum of all entries of matrix S i by the total number of the entries in the matrix; i.e., The model (program) accuracy was defined for all CLOs and PLOs in the program as the ratio of the sum of the sums of the entries of all matrices, S i , by the total number of all entries in the matrices S i ; i.e., To illustrate the process, assume we have the following lists of four PLOs and four CLOs along with the corresponding expert(s) mapping presented in Table 1.A value of "1" denotes positive mapping; otherwise, a value of "0" is recorded.
The stop words are removed, lowercased, and then tokenized during the preprocessing stage.Tokenization in this context refers to the division of a phrase or sentence (PLO, CLO) into tokens.The retrieved tokens were extracted and mapped to each CLO based on the data displayed in Table 1, as shown in Table 2:   The cosine similarity is then used to calculate how similar the vectors are, after which the appropriate threshold values are applied.In this instance, the threshold values were computed as 0.0833, 0.1854, 0., and 0. PLO1 and CLO3 were both considered to represent favorable maps when the similarity score was at least 0.0833 for PLO1 (PLO2 and CLO2).No mapping was performed for PLO3 and PLO4.Table 3 displays the overall mappings that were produced.
A comparison of the obtained mapping to the original mapping revealed an accuracy of 100%.

Experimental work and results
By mapping CLOs to PLOs, instructors can ensure that courses are aligned with program goals and help students develop the skills and competencies they need to succeed in their chosen field.However, the mapping process is extremely tedious and highly subjective.In this experimental work, we employed Artificial Intelligence and in particular, Natural Language Processing to automate the process and identify the semantic connections between the PLOs and the CLOs to enable more precise mappings and, ultimately, meaningful evaluation of the educational program.
The 121 CLOs and all 6 PLOs were stored in a single file representing the input file.The input was next transformed into distinct DataFrame tables using pandas, a Python data analysis tool (pandas.DataFrame -pandas 1.4.3 documentation, n.d.).Each table contained all 121 CLOs and a single PLO.Word tokenization was performed after the PLO-CLOs tables were built using the "CountVectorizer," a quick and effective method of counting features in a dataset.The Scikit-learn Python library's (scikit-learn Machine Learning in Python, n.d.) CountVectorizer was imported.This counts how frequently each feature appears in the supplied data.The vectorization process was performed by breaking the input into distinct words and subsequently counting the word frequency.Additionally, preprocessing operations, like lowercasing, removing stop words, and lemmatization were carried out.
After performing the text representation phase, we determined how similar the vectors were by importing "cosine similarity" modules from the Scikitlearn Python library.A 6121 matrix, consisting of 6 PLOs by 121 CLOs, was created as a result of this process.Each member of the matrix represented the cosine similarity score between a CLO and the related PLO.Dynamic thresholds were calculated for each PLO to produce the final mapping (for example, 0.1961, Table 3 Comparison between the manual and calculated mappings 221, 0.3162, 0.1667, 0.2236, and 0.2697, respectively).Each cosine similarity greater than or equal to the appropriate threshold value was denoted by "1" (positive mapping).Cosine similarity below the threshold was denoted as "0" (no mapping).Figure 2 presents an overview of the mappings based on the data from three courses-CSBP421 Smart Computer Graphics, CSBP320 Data Mining, and CSBP499 Special Topics in Computer Science.The data presented in Fig. 2 reveals that all the mappings between the CSBP421 and the corresponding PLOs were detected correctly, apart from the C3-P2, and C4-P2 mappings.The overall accuracy of the course mapping, in this case, was 91.67%.Similarly, the accuracy of the mapping of CSBP320 to the corresponding PLO was 90%, and 88.89% for CSBP499.
Additionally, we evaluated the proposed framework's performance in Table 4 with several well-known, commonly used NLP models, including Bidirectional Encoder Representations from Transformers (BERT) developed by Devlin et al. (2018), SpaCy (Honnibal et al., 2020), Levenshtein (1966), and Winkler (1990).The outcomes revealed that the proposed framework outperformed these models.The overall accuracy obtained was 83.06% compared to the results obtained using BERT, SpaCy, Levenshtein distance, and Jaro-Winkler distance, which were 45. 9%, 52.8, 33.8, and 55.4%, respectively.Additionally, Table 5 compares the proposed framework's overall accuracy to the aforementioned NLP models by mapping each PLO against the 121 CLOs.All PLOs performed better under the suggested framework.
Moreover, given that the positive mapping (as denoted by a "1" value) only makes up 16.8% of the 6 × 121 matrix, it is crucial to use metrics like precision (PR), recall (RE), and F score (F) (Classification: Precision and Recall, n.d.) to ensure an in-depth examination of the detections of the positive and negative mappings:  Table 6 compares the performance of the suggested framework with the aforementioned NLP models.In this instance, the suggested framework was able to accurately map three PLOs (PLO3, PLO4, and PLO5), outperforming the available approaches.SpaCy achieved the best performance for PLO6, BERT performed best for PLO6, and Jaro-Winkler performed best for PLO2.
We evaluated the framework using data from the Bachelor of Information Security curriculum to ensure the proposed approach was reliable and universal.We obtained an overall accuracy of 88.1%.The mapping detection accuracy values for PLO1 to PLO6 were 81.5%, 78.6%, 94.8%, 91.33%, 97.11%, and 84.97%.Table 7 presents the mapping accuracy based on the 34 courses from the Bachelor of Information Security degree.The mapping of courses like CSBP119, CSBP219, and CSBP221 was 100% accurate.In this instance, every course was mapped with an accuracy of at least 75%.This demonstrates unequivocally that the performance of the suggested framework is reliable and consistent.
The following link will take you to a web-based application that was created to help teachers and administrators execute automatic mappings: https:// bidac-uaeu.github.io/ mapper.htm.This tool is depicted in Fig. 3.The tool also features a function that enables users to connect through API, allowing it to be used by members of the general public.Users can construct programs, courses, and the associated CLOs, PLOs, and CLOs.Additionally, the user can use and export the mappings.
In this section, we show that our proposed framework which utilizes NLP is capable of automating and providing accurate and consistent mapping of CLOs to PLOs.A python program which takes a few seconds can save many hours of meetings and discussions on course-program mappings not to mention the consistency and eliminating human errors and possible subjectivities.The performance of our proposed framework was comparable to that of the domain experts.Besides, it was able to outperform several state-of-the-art NLP techniques such as BERT and spaCy making it a reliable tool to be used by academicians, administrators, and educators.

Discussion
The development of accurate evaluation results for the CLOs, which are utilized as a direct measure for assessing the PLOs, depends on effective CLOs-PLOs mapping.Therefore, it is important to choose action verbs carefully so that they are a good match for the PLOs' more general prospects.Subjectivity in determining how to map certain CLOs to PLOs cannot be avoided because educators typically manually map between CLOs and PLOs.However, consistency can be expected when mapping is performed automatically using NLP and AI, assuming that the right action verbs are selected when defining the CLOs.As various techniques can be employed for diversification and more precise measurement, a PLO is typically measured using more than only the outcomes of an assessment of CLOs.Results from surveys and other indirect assessment tools might be used as examples.Additionally, other direct evaluation procedures, including pre-and post-course examinations or projects, are frequently utilized to supplement the results of the CLO assessment.
Based on the outputs for the Computer Science program provided in Table 4, we note that the results generated by the framework proposed in this paper, while generally yielding better accuracy than previous methods, have less than 75% accuracy for t CSBP316, CSBP483, and CSBP461 courses.The results for every other course were at least 75%.
As can be observed in Fig. 4, we examined the CLOs of these courses and how they were mapped to the related PLOs.Wrongly identified mappings are highlighted in red cells in Fig. 4, which also presents the original manual mappings.The cell is marked in red, for instance, if the manual mapping is "1" and the framework mapping is "0" or if the manual mapping is "0" and the framework mapping is "1".For the CSBP316 course, for instance, the mapping between CLO2 and PLO5 was detected as "1" due to the use of the keyword "suitable," but the mapping between the same CLO and PLO6 was overlooked because there is no word-vector matching.Similarly, the framework's decision to map CLO3 to PLO4 was based on the word "principles."In general, CLO2 and CLO4 that are transferred to PLO6 use indirect verbs, such as "select and build," which results in a lower level of matching given what the underlying intention of PLO6.
On the contrary, the output of the mapping performed by the framework may occasionally be more accurate than manual mapping.For instance, the CLO3 in the course CSBP316 should be mapped to PLO1.This was missed during the human mapping process; however, it was detected by the proposed framework.As a result, the proposed framework can be effectively used by accreditation/program assessors to ensure that the findings of the quality assurance are correct and meaningful.It can also be used to validate manual mapping.
The two verbs "design" and "use," which are employed in CLO 1, are mapped to PLO6 via CSBP483.These verbs contribute to the low matching accuracy since they do not exactly correspond to the acts that PLO6 intends.If the best threshold value is found, certain situations, such as the mappings of the CLO2 to PLO6 and CLO4 to PLO6, for example, could be discovered.In these two 1 3 examples, the proposed framework did detect similarity scores; however, they were missed because they fell just shy of the cutoff point.
By comparing the action verbs used in each mapped CLO in CSBP461 to PLO6, it is possible to understand why the accuracy score was so low.For instance, the action verb "develop" employed by CLO2 does not quite match the action verbs utilized by PLO2 and PLO6.CLO3 and CLO4 mappings can be compared in the same way.The accuracy of automated procedures can be undermined if the wrong definitions (rules) are utilized or the wrong action verbs are used to correspond to those PLOs.
When looking at the performance results for the PLOs measurement accuracy, the results for PLO2 and PLO6 for the Computer Science program were lower than 75%, as indicated in Table 5.These PLOs are as follows: • PLO2: Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program's discipline.• PLO6: Apply computer science theory and software development fundamentals to produce computing-based solutions.
In contrast to the CS PLO6 previously described, the PLO6 for the Information Security program reads, "Apply security principles and practices to maintain To better understand why these results are so low, we can see that PLO2 and PLO6 are made up of a variety of requirements described by more than one action verb or a wide range of necessary skills, which suggests that CLOs from a good number of courses must be mapped to cover the wide range of requirements characterized by these PLOs, which is, in fact, the case (PLO2-86 out of the possible 121 mappings were detected correctly, similarly for PLO6-82 were detected accurately).However, the lower level of accuracy may be caused by the usage of "implied" CLOs-based action verbs that are either wholly incorrect or not an exact match to those used in the PLOs.For instance, multiple CLOs map to PLO2, which employs design, implement, and evaluate, and these CLOs use action verbs like "compare", "develop", and "apply".A similar analysis can be presented for PLO6.For instance, PLO6-mapped CLOs utilize verbs like "use," "write," and "translate," even though they don't quite fulfill the demands of this PLO as stated by its action verbs and context.
While the overall accuracy for the information security program is 88.05%, compared to 83.06% for the computer science program, none of the measured PLOs for that program were below 75% (the minimum was 81.5% -PLO1); thus, the latter program has a marginally better F score.The Information Security program has a total F score of 0.37 compared to the Computer Science program's 0.41.

Conclusion
Mapping course outcomes to program outcomes is an important process that helps ensure that individual courses within a program are aligned with the overall program goals and objectives.There are several reasons why we map course outcomes to program outcomes: • Alignment: Mapping course outcomes to program outcomes helps to ensure that there is alignment between the individual courses and the overall program goals.This ensures that students are gaining the knowledge, skills, and competencies that are needed to be successful in their field.• Coherence: When courses are mapped to program outcomes, it ensures that there is a coherent progression of learning across the program.• Assessment: Mapping course outcomes to program outcomes provides a framework for assessing student learning.It ensures that assessment methods are aligned with the program goals and objectives and that students are being assessed on the knowledge and skills that are important for success in their field.• Quality assurance: Mapping course outcomes to program outcomes is an important quality assurance measure.It helps to ensure that courses are meeting the standards set by the program and that students are receiving a high-quality education that prepares them for their careers.Overall, mapping course outcomes to program outcomes is a critical process that helps to ensure that programs are meeting their goals and that students are receiving the education they need to be successful in their chosen fields.In this paper, we introduce an AI-based framework (NLP) for the automatic and precise mapping of CLOs to PLOs.As educational program evaluations are based on these mapping processes, it is important they are accurate and reliable.To the best of our knowledge, this is the first time NLP has been used to solve an issue of this magnitude.Although NLP has demonstrated excellent results in several disciplines, it has yet to be fully embraced within the educational sector.The proposed framework was evaluated against two actual datasets, yielding positive results.The outcomes of the current study could inform future research in this area.The suggested framework performed noticeably better than several well-known NLP methods, like BERT (Devlin et al., 2018) and SpaCy (Honnibal et al., 2020).
Nevertheless, despite its strong performance, the framework has two significant limitations.The first is the threshold optimization, and the second is the absence of semantic connections between verbs like "implement," "build," "develop," "apply," etc.We discovered that certain incorrect mappings were caused by word similarity rather than the overall semantic meaning.As a result, we intend to emphasize the bloom taxonomy in the future by finding the connections between the verbs using a rule-based technique and giving them more weight.Additionally, methods like generic algorithms can be used to enhance the dynamic threshold values applied in the study.

Fig. 2 3
Fig.2Demonstration of detected mappings based on three courses namely CSBP421 Smart Computer Graphics, CSBP320 Data Mining, and CSBP499 Special Topics in Computer Science.* "CS" represents the "Cosine Similarity" score, the "OM" represents the original mapping (1 positive mapping) and the "CM" represents the calculated mapping.The idle case is when the "CMs" are similar to the "OMs"

Fig. 4
Fig. 4 The 3 courses with poorly detected mappings.The wrongly detected mappings are highlighted in red cells

Abbreviations
Abbreviations CLO: Course learning outcome; PLO: Program learning outcome; NLP: Natural language processing; AI: Artificial Intelligence; QA: Quality assurance; HEI: Higher education institutional; ILO: Institutional learning outcomes; ABET: Accreditation Board for Engineering and Technology; BERT: Bidirectional Encoder Representations from Transformers

Table 1
Hypothetical PLOs and CLOs manual mappings

Table 2
Tokens extracted from the Hypothetical PLOs and CLOs

Table 4
Performance comparisons of the proposed framework in detecting correct mappings with other state-of-the-art NLP models

Table 6
Performance comparisons of the proposed framework with state-of-the-art NLP models in terms of precision, recall, and F score

Table 7
Mapping accuracy based on the 34 courses from the Bachelor Degree in Information Security program