The present study was conducted in two parts, i.e. designing technical requirements for the HIS and its evaluation based on these requirements.
First Part: Designing Technical HIS Requirements
This part determined technical requirements for the HIS in three stages: Focus Group Discussion (FGD), Modified Delphi Technique and Classical Delphi Technique.
Focus Group Discussion
Ten experts with at least a master's degree in software, networking or IT and seven years of work experience in IT departments of hospitals or medical science universities were first selected from faculty members and the staff. Before holding FGD, a panel of experts was presented with a primary checklist extracted from the comprehensive outline published by the Iranian Ministry of Health and Medical Education regarding HIS assessment criteria [21].
The challenges and technical obstacles to implementing the HIS were discussed in four 4-hour sessions. The FGD was held to identify the major technical requirements from expert perspectives and determine all the technical and implementation problems that had ever faced the experts. At this stage, the experts freely exchanged ideas and discussed technical requirements for the HIS through brainstorming. The expert comments were recorded and then transcribed during the meetings. A list of technical requirements was ultimately developed and they were classified into four domains, i.e. communication service, system architecture, security service and system response time according to the expert comments. The primary questionnaire was then designed as the framework of technical requirements for the HIS, which included the study introduction and objectives and a list of technical requirements for the HIS with an open-ended question for including future expert comments.
Modified Delphi Technique
The Modified Delphi Technique was employed at this stage to reach consensus on the requirements by going through a multi-stage process. As an open-ended round Delphi guided by focus groups or one-to-one interviews, this technique is generally used as a Group Delphi Technique to reach consensus using expert comments and structured questionnaires [25]. Experts present at this stage were those who participated in the FGD. At the end of the focus group, the initial technical requirements for the HIS were individually scored by the experts on a five-point Likert scale ranging from 0:strongly disagree to 4:strongly agree. The requirements receiving a total mean score of over 3 were approved, those with a total score of below 2 were eliminated and those with a mean score of 2-3 were reassessed by the experts to be either approved or removed from the questionnaire. The experts were encouraged to freely express their ideas and propose potential technical requirements for the HIS. They unanimously finalized the HIS technical questionnaire after holding four focus group rounds. Four groups of technical requirements for the HIS were ultimately accepted as a 72-item questionnaire with a mean score of 3.4.
Classical Delphi Technique
At this stage, the 72-item questionnaire approved in the Modified Delphi Technique was distributed among 40 IT experts and IT managers of hospitals and medical science universities across the country with at least a master’s degree and a minimum of 5 years of work experience in the HIS field. This questionnaire included demographic information, 72 closed-ended items and an open-ended question to explore the possibility of including additional technical requirements (Additional file 1). The respondents scored the items on a five-point Likert scale ranging from 0: strongly disagree to 4: strongly agree. At all rounds of the Delphi technique, the participants were anonymously provided with feedback. To meet all the Delphi requirements, including anonymity and feedback, research collaborators were introduced to the study hospitals as another focal point. They were in charge of briefing the IT experts and managers on the study objectives and providing them with feedback on the data obtained at individual rounds. Thirty eight of the 40 questionnaires distributed were completed. The mean scores of the individual requirements calculated in SPSS Statistics for Windows, version 18.0 (SPSS Inc., Chicago, Ill., USA) were employed in statistical analyses. The Classical Delphi Technique was performed in two rounds. The ultimately-approved 73-item questionnaire included one additional item and four domains of technical requirements for the HIS with a mean score of 3.6.
Second Part: Evaluating HISs
After inquiring of the Iranian Ministry of Health and Medical Education about the number of companies supplying the HIS, the software was found to be provided by 19 software suppliers throughout the country, 16 of which consented to have their HIS evaluated and provided a list of hospitals in which their system was being run for at least five years. A total of 16 hospitals, each corresponding to one of the suppliers, were then randomly selected to evaluate the HIS program. Running all the software modules was the only inclusion criterion for selecting a hospital. After completing the HIS technical checklists containing Yes/No items designed in the first part through individually asking from the IT users and experts working at the IT units of the selected hospitals as well as observing the implemented system, the data collected were analyzed in SPSS-18. The hospital HIS received a score of 1 if it met the checklist criteria; otherwise, it received a score of 0. The system performance was evaluated in individual domains as a score in percentage, with 0%–20% denoting a very poor performance, 20%-40% a poor performance, 40%-60% an average performance, 60%-80% a good performance and 80%-100% a very good performance.
Ethical considerations
Before beginning the FGD, all the participants signed informed consent forms, in which they were briefed on the study objectives and their voluntary participation in the FGD and confidentiality of their information were ensured. Before completing the checklist in the second part of the study, the participants were briefed on the study objectives. Their voluntary participation and confidentiality of their information were also ensured.