An Ontology-Based Medical Information Management System for Electronic Claim Processing Systems

Abstract

Background

Due to the necessity for comprehensive medical knowledge, health insurance companies (HICs) generally have an independent unit for health insurance studies. The variety of insurers policies and contracts signed with health care providers (HCP) generate extremely detailed healthcare insurance processes and payment rules that are very difficult to manage.

The electronic claim processing (ECP) systems, the core component of health insurance information systems, carry out claim processing procedures by integrating relevant information systems, such as Policy Management System, HCP Contract Management System, and Damage Management System, as depicted in Figure 1. In addition to the relevant legislation and laws, ECP systems are responsible for processing claims according to two primary reference documents: the policy of the insured and the contract signed between the HIC and HCP, as shown in Figure 2 [1].

ECP Systems and Components

ECP systems have several components, including rule repository, rule engine, medical information data bank, etc.  The ECP system used in our study, like many other typical ECP systems, has several components as depicted in Figure 3.

Generally, when a claim is submitted to the ECP system, the rule engine decides whether or not the claim may be approved by applying the ruleset’s payment rules (Please refer to the patent document of “healthcare claims navigator” [2] to get a more comprehensive and detailed explanation of a conventional ECP).

The rules have at least one condition phrase and a single rule action (APPROVE or DECLINE), which is triggered automatically when any of the conditions are not satisfied.Most rule engines are designed around conditions that would trigger a DECLINE rule action.  Since the number of healthcare services and conditions that are covered is higher than the number not covered, designing rules around DECLINE triggering conditions minimızes the number of rules and simplifies the structure.  

Sometimes, claims needs to be evaluated by a domain expert, i.e., physician, nurse, insurance expert, etc. because the ECP system cannot make a determination.  In these cases, the decision must branch out into two actions: DECLINE and REFER TO AN EXPERT.

Since there are hundreds of payment rules in a conventional ECP system, managing payment rules is a challenge for domain experts who are primarily responsible for managing the rule management process. In an ordinary payment rule, there may be dozens of different conditions regarding the diagnosis, symptoms, branches, services, etc. which are connected by if-then satements and boolean operators. In most information systems, the rules are only visible and comprehensible to the software developers who write them.

 One approach to addressing these challenges involves specifying precise payment rules in Access Control Lists (ACLs) [3], [4].  With ACLs, action criteria (DECLINE and REFER TO AN EXPERT) are listed as lines that the rule engine checks one by one to see if the claim meets the criteria. If any of the criteria are unmet, the rule action is executed. The ACL approach is widespread and beneficial in different domains, such as network proxy servers [5], file systems [6], [7], and database systems [8]. However, although they are successful on a per-object basis, they have disadvantages when managing per-subject basis authorizations [4]. Consequently, they are not suitable for managing the complex and numerous payment rules found in the health insurance domain.

Another approach is to write the payment rules as plain text which can be read by domain experts and permits them to interfere with the rule management process. One of the most enhanced solutions for this is the Oracle Policy Automation ^TM (OPA) [9] application. Another solution is Claimer ECP ^TM [10] which has a rule engine capable of interpreting plain text rules written by domain experts via simple writing rules. Although writing payment rules in plain text has an essential advantage for rules management with respect to other methods, such as ACL, etc., there are still three critical challenges that these payment rules pose, as follows.

1. Comprehensibility: The rules may be too long and complicated for domain experts to comprehend and interpret.
2. In-rule Consistency: Some healthcare services may be included in more than one rules set. The rules sets can be used under the same rule using different conditions. If one condition results in approval, while another one results in decline, then the payment rule yields an in-rule inconsistency problem.
3. Inter-rule Consistency: Some rules may be inconsistent with others. Such inconsistencies are especially challenging to detect when there are many rules and/or more than one domain expert managing the rules.

Rule Types in Healthcare ECP Systems

There are different rule types in the health insurance domain. The rule may be a common medical rule which is based on well-known medical rules, procedures, or common sense, for example, checking that prostate related services are only given to males. General conditions rules are rules defined by the regulatory authority of each country; for example, services related to a pregnancy termination or to the treatment of a suicide or homicide attempt may or may not be covered according to national regulations.

Special condition rules relate to institutional policies (as opposed to individual insurance policies) and refer to conditions related to services that are covered for institution’s employees for a number of reasons; for example an institution may choose to cover international medical evacuation if their employees work in remote locations where healthcare may not be available. Exclusion rules apply to individual policies and exclude services related to pre-existing conditions or conditions for which the individual is at risk (this is permitted in some, but not all countries). Finally, common insurance rules are used by HICs to manage the claim process more precisely. Such rules are mostly defined to detect fraud or irregularities or to have some claims evaluated by experts instead of automatic processing by the ECP.

The Structure of the Payment Rules

The basic structure of the Claimer ECP payment rules used in this study is presented in Table 1.

Table 1. The basic structure of the payment rules used in this study

Domain experts select and combine the appropriate phrases from Table 1 to form a  plain text payment rule. The phrases in columns A through G are used to construct a single condition of the payment rule. Column H is used to join two conditions when a new condition is added. When all conditions have been defined, Columns I and J are executed to trigger the action as defined by the payment rule sentence.

Notice that Column D contains many parameters describing an attribute of the patient, claim, branch, policy, HCP, etc. Columns B through D construct the first operand to be used in the condition of the payment rule. Column G contains the second operand of the payment rule sentence, which can be a number, date, or a set of diagnoses, health services, drugs, materials, branches, etc. Constructing sets of diagnoses, services, materials, drugs, etc. requires a comprehensive medical information management study. For example, a set of diagnoses of, for example, Diabetes Mellitus, must contain all relevant diagnosis codes from a standard diagnosis list, such as the International Statistical Classification of Diseases and Related Health Problems 10 (ICD 10) [11] as well as a set of all possible health services provided to diagnose, monitor, and treat Diabetes Mellitus.

Finally, Column F contains a condition function to check whether the first operand (the combination of columns B, C, and D) is satisfied with the second operand (Column G). The action in Column J is processed by the ECP system if and only if the claim satisfies all the requisite conditions.

A rule sentence may be structured as described above by selecting relevant phrases from the columns in order, for example:

(A, B, C, D, E, F, G), H, (A, B, C, D, E, F, G), H, (A, B, C, D, E, F, G), H, …, I, J.

The Difficulties in Medical Information Management

Because policy types are so diverse, ECP systems generally have to process hundreds of payment rules. Moreover, payment rules will include any of the sets of items given in Column G of Table 1 as a parameter, which means all such sets of items must be prepared by the domain experts beforehand. The most common way of preparing those sets is by grouping the items (diagnosis, services, branches, laboratory exams, reimbursements, etc.). For example, the following rule requires preparing a couple of sets for diagnosis, services and branches which are related with “Diabetes Mellitus” from the standard diagnosis, services, and branch codes:

If AT LEAST ONE of the DIAGNOSIS is in “DIABETES MELLITUS DIAGNOSIS” OR if AT LEAST ONE of the SERVICES is in “DIABETES MELLITUS SERVICES” AND if EXACTLY ONE of the BRANCH is EQUAL TO “DIABETES MELLITUS BRANCHES” then DECLINE!

Related Works

Ontologies have been widely used in healthcare to facilite use of a common vocabulary. The Systematized Nomenclature of Medicine-Clinical Term (SNOMED-CT) [12] and RadLex [13] are widely used examples. Ontology has also been used by healthcare providers for different purposes, such as evidence-based medicine [14], modeling anatomy [15], clinical decision support systems [16], semantic descriptions of computer tomography (CT) images [17], modeling genes [18], national health data dictionaries [19], creating terminologies [20], [21], knowledge engineering [22], [23], biomedical informatics [15], [24], [25], language engineering [26]–[28], and information systems [29]–[31]. Since these ontologies enable us to define terms that describe concepts and how they relate to one another, they can be useful to the field of information management [32], [33].

Despite its potential, there are only a few studies combining ontology with healthcare insurance. One study [34] proposes the use of semantic Web techniques to minimize information asymmetry between clients and providers in the Indian healthcare insurance domain. Another study developed a patented method  [35] using ontology to convert non-standardized billing codes to standard codes. To date, no studies using ontology in medical data banks used for claim processing have been found in the literature.

Method

This paper aims to propose an ontology-based medical information management system for ECP systems named ONTology-based Medical Information Management System  (ONTMIMS), whereby ontological concepts and relationships support payment rules, so that domain experts can write more straightforward payment rule sentences that are more comprehensible and may improve both in-rule and inter-rule consistencies. The scope of this study does not include performance benchmarking or improvements related to the execution time of the rules since we had not experienced any problems with rules execution in the previous system.

Methods of Ontology Development

Although there is no standard method for ontology development, referred to as “ontology engineering,” there were several frameworks [36] that informed this study.  METHONTOLOGY, proposed by Fernández et al. in 1997, is one of the earliest ontology building methodologies [39].  The On-To Knowledge is another process oriented methodology for ontology based knowledge management systems proposed by Sure et al. [38] in 2002. The Unified Process ONtology (UPON), was proposed by Nicola et al. [37] in 2005.  This “use-case driven, iterative and incremental” approach starts with the creation of a domain “map,” categorizing the main concepts in a hierarchy or taxonomy. Subsequently, the map is extended to include additional concepts and relations needed for the ontology. Finally, through iteration, concepts and relations are formalized as an ontology. UPON builds upon the advantages of the Unified Process (UP) approach which is a widely accepted standard in software engineering.  While these methodologies were helpful in this study, the four-stage approach for healthcare specific ontology development method proposed by Kuziemsky and Lau in 2010 was used as the dominant methodology in this study.  The four-stages are outlined as follows: i) specification and conceptualization, ii) formalization, iii) implementation, and iv) evaluation and maintenance [40]. Since this approach is healthcare specific, it was selected as the most suitable methodology for this study.  The medical data bank used in this study contained thousands of medical lists used by Claimer ECP as a reference in text-based payment rules which were used as a starting point in developing an ontology using Kuzemsky and Lau’s methodology as outlined in the following sections.

Specification and Conceptualization (Stage 1)

To validate the concepts and their specifications, the researcher began by collecting the sets of items used to represent diagnosis and health services groups found in the Claimer ECP payment rules, as indicated in Table 1, Column G. In total, 1,312 diagnosis and health service sets were identified. Open, axial and comparison Grounded Theory (GT) coding approaches were considered for data analysis [41], [42]. The open coding approach was selected because the study was using existing sets of diagnoses and health services. This analysis yielded the attributes presented in Table 2. Those attributes are applicable to the concepts. These attributes help domain experts to develop an understanding of the sets that is needed to define relationships and relevancy among concepts.

Table 2. Attributes and values of the diagnosis and health services item sets

Formalization (Stage 2)

Description Logics (DL) was chosen as the language to describe the entities and their relationships in the ONTMIMS [43]. There are three types of entities in the model developed in this study: concepts, roles, and individual names. Rules are not expressed in DL because they are text-based.  Concepts are sets made up of individuals and their roles. Individuals are defined items, such as sets of diagnosis and health services like those shown in Table 1, Column G. Roles describe the relationships between individuals, and are represented by unique names from the relevant standard classification lists, such as ICD-10 [11], and the national classifications lists used for pricing of healthcare services in Turkey [44], [45], etc. For example, the concept for Diabetes will include individuals made up of all the sets related to diabetes mellitus, including but not limited to a set of diagnosis codes and a set or sets of healthcare services and roles such as drug use for neuropathic pain for diabetics with a physician’s report of necessity.    It will be better to indicate that, the rules, since they are text-based as explained before, are not expressed in DL.

Analysis of the 1,312 sets of items involved in the existing payment rules, revealed two different types of concepts:  medical concepts (like organ, system, or pathology) and domain concepts (like, anemia, or diabetes). A distinction between these concept types is required since payment rules may refer to either a medical concept or a domain concept as a condition for payment because insurance policies may define exclusions, special conditions, and general conditions using the name of a disease (diabetes mellitus) or all diseases related to an organ or system (like autoimmune dieases or endocrine system disorders). Thus, defining medical and domain concepts and the relations between them enables the ONTMIMS to handle more flexible payment rule sentences. The medical concepts used in this study are depicted in Table 3.

Table 3. Medical concepts in ONTMIMS

Most of the domain concepts used in this study are related to the 1.467 sets of items (or individuals of DL) and listed in Table 4.

Table 4. Domain concepts in ONTMIMS

Both medical and domain concepts were mapped with their relations among 1,312 individuals in order to make the payment rules more flexable. As an example, a basic description of the Diabetes Mellitus concept is given in Figure 4.

Although SNOMED CT has an enhanced infrastructure, including thousands of medical terms, findings, diagnoses and relations between the terms, the claim processing system Claimer ECP was used in this study.  Claimer ECP is compatible with some international standards, such as ICD 10 and offered an existing medical data bank structured according to national requirements and standards. The aim of this study was to improve the existing payment rule structure by adding an ontology layer and not to change the whole infrastructure.

Implementation (Stage 3)

Since there are medical and domain experts in the HIC who conduct the design and implementation studies, a rapid system development approach [46] was used to develop the ONTMIMS framework. Protégé [47] was used to define domain ontology, generate knowledge-acquisition, and to define the mappings [48]. In order to query the ontology, the Apache Jena [49] library was incorporated to create an environment where semantic web technologies can be used in Claimer ECP.

Since ONTMIMS inserts an ontology layer between the payment rule repository and the existing medical information data bank in the ECP system, the architecture of the ECP system given in Figure 3 was revised to the architecture shown in Figure 5. In the new structure, the rule engine queries the ontology directly using the Jena library, instead of referring to the 1,312 different sets of items in the medical information data bank as it did in the previous rule sentence structure shown in Table 1.

ONTMIMS also enables domain experts to write new payment rules using concepts, instead of listing all of the sets of diagnosis, services, materials, etc. Naturally, this new layer changes some of the parameters depicted in Table 1 resulting in the structure shown in Table 5.

Table 5. The proposed rule structure after adding ontology concepts

A simple example is given below of a payment rule that checks whether or not the claim is related to diabetes. Previously, the rule would consist of several lines of conditions regarding diagnosis, services, materials, and branches as shown in the example below:

The new rule generated using ONTMIMS becomes much simpler as shown below:

ONTMIMS includes several nomenclatures to address lists mapped within the ontology, i.e., ICD 10 for diagnosis and multiple national health classification lists for the pricing of healthcare services, materials, etc.

Experiments (Stage 4)

Although many studies measure the semantic accuracy and quality of ontologies [50]–[52], the focus of this study is to compare the ONTMIMS with the previous system in terms of the comprehensibility and in-rule and inter-rule consistency of payment rules from domain experts’ perspectives. The experiments were conducted in three stages: i) comparing existing payment rules with their equivalence rules using ontology in terms of comprehensibility; ii) finding in-rule inconsistencies of a set of payment rules with and without ONTMIMS; iii) finding inter-rule inconsistencies of a set of payment rules with and without ONTMIMS . In order to minimize bias, domain experts were selected from outside of the current insurance company so that they could not easily realize which payment rule was written by which approach. Four domain experts were selected for experimentation stage.  The domain experts have graduate degrees from

Comprehensibility

Domain experts were selected with the above requirements because, although there are several methods available to measure the comprehensibility of text [53], [54], evaluating the comprehensibility of payment rules requires experience and knowledge about the domain.

The domain experts were given 40 payment rules to evaluate.  Twenty payment rules were written using the previous approach.  Then, each payment rule was rewritten with precisely the same purpose and meaning using the ONTMIMS approach so that the two groups could be compared.  The diabetes rules provided in the previous Implemention section are an example. There are 5 rules from each rule type given in “Rule Types in Healthcare ECP System”, excluding General Medical Rules, which are very specific to clinical cases and would have required clinical expertise during the experiments.

At this stage, the binary pair-wise comparison method [55], [56] of analytic hierarchical processing was conducted to determine the level of comprehensibility. The domain experts were asked to fill out a 40x40 comparison matrix. The comparison matrix included randomly ordered 40 payment rules in its rows and columns, symmetrically. The domain experts were asked to compare all rule pairs (one from row, and one from column) in the matrix and set 1 if the payment rule in the row is more understandable, otherwise, set 0.  The matrix was symmetric and designed to allow domain experts to compare each pair one at a time. The domain experts made their decisions alone and without interference from others. Domain experts were given a limited duration to complete the task that was sufficient for them to to read and understand each  sentence.

The average comprehensibility level was calculated using the rank of each ruleset given by each of the four domain experts.  The rank of each rule was calculated separately, then, the normalized rank values were assigned as the weights of the comprehensibility scale for payment rules.

In-rule Consistency

At this stage, the same 40 payment rules were modified to create inconsistencies among the rule conditions. For example, two conditions of a single payment rule were designed to conflict with on another; or a single payment rule may have presented a case of tautology or contradiction. The experts were requested to evaluate whether or not the payment rules were consistent. They were warned that the consistency check should only be made by considering the subject rule.

Inter-rule Consistency

As in the previous stage, 20 payment rules were generated with the previous and ONTMIMS approaches so that a total of 40 payment rules were presented to the domain experts. For each approach, half of the rules were changed to create an inconsistency with other rules within the same 20 rulesets. Unlike the previous stage, this time the domain experts were asked to evaluate each rule as to whether or not it has an inconsistency with other rules in the same ruleset. This evaluation was repeated for each rule in the rule sets of both previous and ONTMIMS approaches. All consistency checks were made based on the evaluations of the domain experts individually without getting any help from the DL functions. For example the experts are requested to compare two rule sentences (conventional rule and ONTMIMS rule) and mark if there is an inconsistency between this rule and any of the previous rules.

Results

The average normalized ranks of comprehensibility of each of the rules calculated at the end of the experiments are shown in Table 6 (note that the rules cannot be given explicitly due to proprietary commercial concerns).

Table 6. Ranks of comprehensibility of the payment rules

As seen in Table 6, the average normalized rank was 65.1% for ONTMIMS rulesand 35.3% for conventional rules. This result shows that the ONTMIMS approach increased the domain experts’ comprehension of the rules.

With regard to in-rule inconsistencies, the average number of the true-positive (TP), true-negative (TN), false-positive (FP), and false-negative (FN) decisions provided by the four domain experts and the consequent accuracy of the experiment are shown in Table 7.

Table 7. Accuracy table of finding in-rule inconsistencies of the rules

As seen in Table 7, inconsistencies within the rules are more easily distinguishable by domain experts when using ONTMIMS.

With regard to inter-rule inconsistencies, the average number of TP, TN, FP, and FN decisions of four domain experts and the consequent accuracy of the experiment are shown in Table 8.

Table 8. Accuracy table of finding inter-rule inconsistencies of the rules

Table 8 shows that ONTMIMS provides slightly better results than the conventional approach.

Discussion

In this study, an ontology (ONTMIMS) was developed that covers the concepts referenced in the human-readable payment rules processed by ECP systems. The primary motivation for the introduction of ontology was to simplify the payment rules by using ontology concepts rather than the traditional set of items (diagnosis, healthcare services, etc.) to improve the comprehensibility, in-rule consistency, and inter-rule consistency of the rules.

Experiments were conducted with domain experts to demonstrate the differences between the previous approach and ONTMIMS. By using a pair-wise comparison method, domain experts were asked to rank the comprehensibility of comparable payment rules and then identify in-rule and inter-rule inconsistencies. The experiments showed that ONTMIMS increased comprehensibility from 35.3% to 65.1%. Additionally, the in-rule inconsistencies of the payment rules can be distinguished at a rate of 82.5% with ONTMIMS, and only 65% using the traditional approach. Similarly, the intra-rule inconsistencies between payment rules can be distinguished at a rate of 85% with ONTMIMS, and 78.8% with the traditional approach.

Limitations

Since using ontology in an ECP system may tend to lose granularity and nuance, domain experts should consider using it just to simplify complex rules. The scope of this study is closely related to the payment rules used in the ECP system used in the study. The ontology will not be public for some time due to proprietory commercial considerations.  Another limitation of ONTMIMS is internationally transferrability because the ontology is designed around the Turkish system.

Conclusion

ONTMIMS is the first ontology used in an ECP system in the healthcare insurance domain. This study demonstrates that it can lead to more comprehensible and consistent payment rules. It is planned that the study will continue with a focus on extending the ontology and measuring its accuracy and semantic quality. While this study is a good starting point, a more comprehensive and extensive ontology that covers more diagnosis and healthcare services is needed.

Declarations

Ethics approval and consent to participate

Not applicable 

Consent for publication

Not applicable 

Availability of data and materials

The datasets generated and analyzed during the current study are not publicly available due the commercial reasons of the TPA company. 

Competing interests

The author declares that he has no competing interests 

Funding

This work was supported in part by the Scientific and Technological Research Council of Turkey (TUBITAK) under project (grant) number of 3120489. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the TUBITAK. 

Authors' contributions

IK contributed the proposal of this R&D project to get the grant from TUBITAK, managed the R&D team of this project from the beginning to the end, and contributed to all work packages. IK was the only contributor in writing the manuscript.

Acknowledgements

The author thanks Dr. Pinar YOLUM for her valuable consultancy on the study, Fatma ZAKARYAN, M.D., General Manager of CGM Turkey, the medical and insurance experts Omur SEZER, Ulas OZCITAK, M.D. and Adil KARASOY, M.D. for their medical and insurance expertise and Alp Timurhan CEVIK, Serkan KOYUN, Tukan SENEL and Pinar HORUZ for their valuable technical expertise and efforts. The author also thanks Dr. Osman ABUL for his valuable comments during the project development phases.

References

[1]      I. Kose, M. Gokturk, and K. Kilic, “An interactive machine-learning-based electronic fraud and abuse detection system in healthcare insurance,” Appl. Soft Comput. J., vol. 36, pp. 283–299, 2015.

[2]      E. Shanmugam, “Healthcare claims navigator,” US8682696B1, 30-Nov-2007.

[3]      Network Working Group and R. Shirey, “RFC 4949, Internet Security Glossary, Version 2.” The IETF Trust, p. 494, 2007.

[4]      R. S. Sandhu and P. Samarati, “Access control: principle and practice,” IEEE Commun. Mag., vol. 32, no. 9, pp. 40–48, Sep. 1994.

[5]      B. C. Neuman, “Proxy-based authorization and accounting for distributed systems,” in Proceedings. The 13th International Conference on Distributed Computing Systems, 1993, pp. 283–291.

[6]      Microsoft, “Managing Authorization and Access Control,” 2005. [Online]. Available: https://technet.microsoft.com/en-us/library/bb457115.aspx. [Accessed: 03-Nov-2016].

[7]      A. Grünbacher, “POSIX Access Control Lists on Linux,” in USENIX Technical Conference, 2003, pp. 259–272.

[8]      T. F. Lunt, “Access Control Policies for Database Systems,” DBSec, pp. 41–52, 1988.

[9]      Oracle, “Oracle Policy Automation (OPA).” [Online]. Available: http://www.oracle.com/us/products/applications/oracle-policy-automation/overview/index.html. [Accessed: 16-Oct-2016].

[10]    CompuGroup Medical, “Claimer ECP - Claim Management System.” [Online]. Available: https://www.cgm.com/tr/products___solutions_3/health_payers/cgm_claimer_ecp/CGM_CLAIMER_ECP.en.jsp. [Accessed: 16-Oct-2016].

[11]    World Health Organization, “The International Classification o Diseases (ICD),” 1992. [Online]. Available: http://www.who.int/classifications/icd/en/. [Accessed: 26-Nov-2016].

[12]    S. Schulz, B. Suntisrivaraporn, F. Baader, and M. Boeker, “SNOMED reaching its adolescence: Ontologists? and logicians? health check,” Int. J. Med. Inform., vol. 78, pp. 86–94, Apr. 2009.

[13]    S. Kundu et al., “The IR Radlex Project: An Interventional Radiology Lexicon—A Collaborative Project of the Radiological Society of North America and the Society of Interventional Radiology,” JVIR, vol. 20, pp. 433–435, 2009.

[14]    K. Zheng, R. Padman, M. P. Johnson, and S. Hasan, “Guideline Representation Ontologies for Evidence-Based Medicine Practice,” in Handbook of Research on Advances in Health Informatics and Electronic Healthcare Applications, IGI Global, 2010, pp. 234–254.

[15]    C. Rosse and J. L. V Mejino, “A reference ontology for biomedical informatics: The Foundational Model of Anatomy,” J. Biomed. Inform., vol. 36, no. 6, pp. 478–500, 2003.

[16]    J. Akaichi and L. Mhadhbi, “A Clinical Decision Support System: Ontology-Driven Approach for Effective Emergency Management,” in Improving Health Management through Clinical Decision Support Systems, Hershey, PA: IGI Global, 2016, pp. 270–294.

[17]    N. Kökciyan, R. Türkay, S. Üsküdarli, P. Yolum, B. Bakir, and B. Acar, “Semantic description of liver CT images: An ontological approach,” IEEE J. Biomed. Heal. Informatics, vol. 18, no. 4, pp. 1363–1369, Jul. 2014.

[18]    Gene Ontology Consortium, “The Gene Ontology (GO) project in 2006,” Nucleic Acids Res., vol. 34, pp. 322–326, 2006.

[19]    Y. Yüksek and M. O. Unalir, “Ontology Based Metadata Management for National Healthcare Data Dictionary,” Pamukkale Üniversitesi Mühendislik Bilim. Derg., vol. 18, no. 2, pp. 133–144, 2012.

[20]    D. L. Rubin, “Creating and curating a terminology for radiology: Ontology modeling and analysis,” J. Digit. Imaging, vol. 21, no. 4, pp. 355–362, 2008.

[21]    B. Smith, J. Flanagan, and W. Ceusters, “Ontology and Medical Terminology: Why Description Logics Are Not Enough,” Boston, MA Med. Rec. Inst. (CD-ROM Publ., vol. 10, no. May, p. 14, 2003.

[22]    T. R. Gruber, “A Translation Approach to Portable Ontology Specifications by A Translation Approach to Portable Ontology Specifications,” 1993.

[23]    M. Uschold et al., “Ontologies: principles, methods and applications,” Knowl. Eng. Rev., vol. 11, no. 02, p. 93, Jun. 1996.

[24]    J. J. Cimino and X. Zhu, “The practical impact of ontologies on biomedical informatics.,” Yearb. Med. Inform., vol. 15, no. 1, pp. 124–135, 2006.

[25]    O. Bodenreider and R. Stevens, “Bio-ontologies: Current trends and future directions,” Brief. Bioinform., vol. 7, no. 3, pp. 256–274, 2006.

[26]    J. A. Bateman, “On the relationship between ontology construction and natural language: a socio-semiotic view,” Int. J. - Hum. Comput. Stud., vol. 43, pp. 929–944, 1995.

[27]    E. Klein and S. Potter, “An ontology for NLP services,” Edinburgh, Scotland, 2004.

[28]    P. Cimiano, P. Haase, M. Herold, M. Mantel, and P. Buitelaar, “LexOnto: A Model for Ontology Lexicons for Ontology-based NLP,” in International Semantic Web Conference, 2007.

[29]    N. Guarino, “Formal Ontology and Information Systems,” Form. Ontol. Inf. Syst., vol. 3, no. 15, pp. 3–15, 1998.

[30]    B. Blobel and F. Oemig, “Ontology-driven health information systems architectures,” Stud. Health Technol. Inform., vol. 150, pp. 195–199, 2009.

[31]    B. Blobel, “Ontology driven health information systems architectures enable pHealth for empowered patients,” Int. J. Med. Inform., vol. 80, no. 2, pp. e17–e25, 2011.

[32]    I. Jurisica, J. Mylopoulos, and E. Yu, “Ontologies for Knowledge Management: An Information Systems Perspective,” Knowl. Inf. Syst., vol. 6, no. 4, pp. 380–401, 2004.

[33]    T. Q. Dung and W. Kameyama, “A proposal of Ontology-based health care information extraction system: VnHIES,” 2007 IEEE Int. Conf. Res. Innov. Vis. Futur. RIVF 2007, pp. 1–7, 2007.

[34]    V. K. Sreekanth and D. Biswas, “Information asymmetry minimization system for potential clients of healthcare insurance in Indian context using semantic web,” IEEE TechSym 2014 - 2014 IEEE Students’ Technol. Symp., pp. 282–285, 2014.

[35]    L. Cousineau, P. Cherpes, R. Brewster, and H. Young, “Ontology based method for automatically generating healthcare billing codes from a patient encounter,” US20060020493 A1, 2005.

[36]    O. Corcho, M. Fernández-López, and A. Gómez-Pérez, “Methodologies, tools and languages for building ontologies. Where is their meeting point?,” Data Knowl. Eng., vol. 46, no. 1, pp. 41–64, 2003.

[37]    A. De Nicola, M. Missikoff, and R. Navigli, “A software engineering approach to ontology building,” Inf. Syst., vol. 34, no. 2, pp. 258–275, 2009.

[38]    Y. Sure, S. Staab, and R. Studer, “On-To-Knowledge Methodology (OTKM),” in Handbook on Ontologies, Springer Berlin Heidelberg, 2004, pp. 117–132.

[39]    M. Fernández-López, A. Gómez-Pérez, and N. Juristo, “METHONTOLOGY: From Ontological Art Towards Ontological Engineering,” Proc. Ontol. Eng. AAAI-97 Spring Symp. Ser. | AAAI-97 Spring Symp. Ser. | 24-26 March 1997 | Stanford Univ. EEUU, 1997.

[40]    C. E. Kuziemsky and F. Lau, “A four stage approach for ontology-based health information system design,” Artif. Intell. Med., vol. 50, no. 3, pp. 133–148, 2010.

[41]    S. Anselm and J. Corbin, Basics of qualitative research, 2nd Editio. Newbury Park: SAGE Publications, 1990.

[42]    J. Corbin and S. Anselm, “Grounded theory methodology: an overview,” in Handbook of qualitative research, D. NK and L. YS, Eds. Thousand Oaks: SAGE Publications, 1994, pp. 273–285.

[43]    F. Baader, D. Calvanese, D. Nardi, and D. McGuinness, The description logic handbook; theory, implementation, and applications. Cambridge University Press, 2003.

[44]    SGK and SGK Sağlık Uygulama Tebliği, “Sağlık Uygulama Tebliği (SUT),” 2005. [Online]. Available: http://www.sgk.gov.tr/wps/portal/sgk/tr/saglik/saglik_hizmetleri. [Accessed: 28-Nov-2016].

[45]    TTB, “TTB Asgari Ücret Tarifesi,” Turkish Medical Association, 2016. [Online]. Available: http://www.ttb.org.tr/en/. [Accessed: 26-Nov-2016].

[46]    P. Beynon-Davies, C. Carne, H. Mackay, and D. Tudhope, “Rapid application development (RAD): an empirical review,” Eur. J. Inf. Syst., vol. 8, no. 3, pp. 211–223, Sep. 1999.

[47]    Stanford Center for Biomedical Informatics Research, “Protege.” [Online]. Available: http://protege.stanford.edu/. [Accessed: 16-Oct-2016].

[48]    A. Abu-Hanna, R. Cornet, N. de Keizer, M. Crubézy, and S. Tu, “protégé as a vehicle for developing medical terminological systems,” Int. J. Hum. Comput. Stud., vol. 62, no. 5, pp. 639–663, 2005.

[49]    The Apache Software Foundation, “Apache Jena.” [Online]. Available: http://jena.apache.org/. [Accessed: 16-Oct-2016].

[50]    D. Vrandečić, “Ontology Evaluation,” in Handbook on Ontologies, Berlin, Heidelberg: Springer, 2009, pp. 293–313.

[51]    A. Lozano-Tello and A. Gomez-Perez, “ONTOMETRIC,” J. Database Manag., vol. 15, no. 2, pp. 1–18, Apr. 2004.

[52]    D. Sánchez, M. Batet, S. Martínez, and J. Domingo-Ferrer, “Semantic variance: An intuitive measure for ontology accuracy evaluation,” Eng. Appl. Artif. Intell., vol. 39, pp. 89–99, Mar. 2015.

[53]    R. Taffler, “Readability and Understandability: Different Measures of the Textual Complexity of Accounting Narrative,” Accounting, Audit. Account. J., vol. 5, no. 4, pp. 84–98, Dec. 1992.

[54]    M. Jones and M. Smith, “Traditional and alternative methods of measuring the understandability of accounting narratives,” Accounting, Audit. Account. J., vol. 27, no. 1, pp. 183–208, 2014.

[55]    H. Taira, Y. Fan, K. Yoshiya, and H. Miyagi, “A method of constructing pairwise comparison matrix in decision making,” in 1996 IEEE International Conference on Systems, Man and Cybernetics. Information Intelligence and Systems (Cat. No.96CH35929), 1996, vol. 4, pp. 2511–2516.

[56]    G. Kou, D. Ergu, C. Lin, and Y. Chen, “Pairwise comparison matrix in multiple criteria decision making,” Technol. Econ. Dev. Econ., vol. 22, no. 5, pp. 738–765, 2016.