Group Concept Mapping (GCM) provides a structured approach for consensus building, using quantitative and qualitative methods, allowing for the integration of input from multiple sources into a visual representation of a conceptual framework, and is described in detail by Kane and Trochim . GCM leads to a visual representation of composite thinking of participants and stakeholder groups with the ability to engage in and represent complexity. An online platform supports the collection, management and analysis of data . Stakeholders are engaged to generate ideas, sort the ideas into groups, and rate ideas according to value. The statistical techniques of multi-dimensional scaling and hierarchical cluster analysis aggregate data to reveal patterns through visualisation, allowing for interpretation to support further utility of the model. GCM is a structured applied social research methodology, to connect theory to observation and research to practice, has been widely used in the health-care sector for policy and planning for health services [33, 34, 35, 36, 37] and increasingly in the medical education sector to understand educational processes and outcomes [38, 39, 40].
The GCM process consisted of four key phases: 1) Idea generation, review and synthesis ; 2) Sorting and rating facilitated by the online platform ; 3) Analysis of data using quantitative and qualitative techniques to produce a visual concept map; 4) Confirmation and further interpretation of results using logic model transformation . This study was approved by the Human Research Ethics Committee of Tasmania (reference number H0015769).
Participants were identified using a purposive sampling strategy which aimed to ensure representation from the following stakeholder groups: patients and carers, clinicians from a variety of disciplines, health-care managers, educators and professional association representatives. Potential participants were invited to participate in one or more stages of the project. Junior doctors were recruited for the rating component. Approaches were made both directly by researchers, and through third parties who distributed the invitation via email. This study was conducted between October 2017 and October 2019 with participants across five of eight Australian states and territories.
Participants were invited via email to contribute responses using an online platform . They were asked to complete the focus statement “An attribute or non-clinical competency required of doctors for effective practice in modern health-care systems is . . . ” as many times as they liked. Participants were provided with the following definitions:
- Attribute: A quality or feature regarded as a characteristic or inherent part of someone or something and does not depend on acquired knowledge; and
- Non-Clinical Competency: Transferable, generic professional skills which are not rooted in the medical profession. They may be carried out in a clinical or non-clinical environment by health-care workers but are not uniquely clinical in nature (e.g. communication related skills).
Statements were iteratively reviewed, refined and synthesised with duplicates and irrelevant ideas removed, and similar ideas combined. Guidelines for this review process included determining whether statements needed to be split into more than one idea, elimination of repeated ideas, elimination of statements which were not relevant to the focus statements (e.g. health-care specific clinical skills), and clarification of content if required to ensure ideas were concise and understandable . We determined data saturation through iterative synthesis and comparison of ideas as they were generated onto the online platform. Once we were satisfied that the point of saturation had been reached a research advisory group convened for the project which composed of five clinicians from the disciplines of nursing, medicine and psychology, and one consumer, reviewed the statement list and provided feedback with regards to relevance of the statements to the research, clarity of statements, and completeness of the statement list to confirm saturation. A final set of statements detailing attributes and non-clinical competencies was generated.
Sorting of statements
Participants were invited to sort the statements into groups in a way that made sense to them [31, p.72], and provide a relevant name for each group. This activity occurred online using the Concept Systems Global Maxtm platform . We set a minimum target of 30 sorters with representation from all stakeholder groups, which is in line with the recommended number (20-30) to provide reliable results while acknowledging that larger number of sorters yields higher inter-rater reliability estimates .
Data analysis for cluster map
A cluster map was built and labels determined using the online Concept Systems Global Max analysis program [31, 32] which integrates qualitative and quantitative methods [43, 44], in addition to a qualitative sense-making process.
A similarity matrix was created to identify how often statements were sorted together. Through the process of multidimensional scaling , this similarity matrix was then used to create a two-dimensional ‘point map’ of each statement to visually represent the sorting data, with statements sorted together more often placed closer on the map. A stress value statistic was generated as an indicator of how well the point map represented raw sorting data .
Hierarchical agglomerative cluster analysis using Ward’s algorithm  was used to group statements into clusters. A bridging value was identified for each statement, indicating whether it was anchoring - sorted primarily with others close by, or bridging – sorted with others across a larger area of the map. The option of imposing a filter on the analysis which would require statements to be sorted together more than one time was explored but did not significantly change the outcome and therefore was not utilised.
Determining the number of clusters relied on qualitative review by researchers  using interpretive analysis . Statements in each cluster were examined from maps with five through to 15 clusters, and using expertise in medical education and clinical medicine, the optimal cluster solution was determined . This process was undertaken by one author (KO) and reviewed and confirmed by other authors and the research advisory group. Examination of statements was then made to determine whether there were any statements placed on a cluster boundary which were deemed to better fit in an adjacent cluster and if so the boundary was changed.
Cluster labels were determined using three sources of information: GCM software provides list of 10 best fit labels provided by participants ; the statement bridging values provided information about which statements are the most central to the cluster; and researchers read and synthesised their understanding of the statements in each cluster.
One author (KO) proposed cluster names, the other authors and research advisory group reviewed the decision and made alternative suggestions until agreement was reached. All participants in the GCM process were provided with a provisional set of results and invited to make comment over a 2-week period. A further seven clinicians were interviewed and their feedback on the relevance and utility of the model sought (not reported here). Feedback was considered by the research team for incorporation into the models.
Data analysis for logic model
Subsequently we developed a logic model as a tool to further operationalise the data incorporating inputs, processes and activities, and outputs [49, 50]. Impacts and outcomes are not incorporated in the model as they were not included as part of the initial concept mapping process, rather the logic model focuses on strategies . Each statement was examined to determine whether it related to input, process or activity, or output elements. Statements which incorporated more than one of these categories were split into individual elements and re-worded to ensure that they were understandable. Each element was then grouped according to thematic similarity, starting with elements within the same cluster but incorporating those from other clusters if appropriate. Groupings were then examined for causal linkages between inputs, processes and activities, and outputs, including feedback loops. This process was performed by one author (KO) and the logic model reviewed by all other authors and the research advisory committee to provide input and ultimately confirm the model.
Junior doctors were invited to rate each of the statements generated in the above process using Likert scales according to the following two prompts:
- Relatively how important is this attribute or competency to your role as a doctor? (1=Relatively less important to 5=Relatively more important)
- How well prepared were you when you graduated? (1=not prepared; 2=somewhat prepared; 3=reasonably prepared; 4=well prepared; 5=very well prepared)
Data were entered directly onto a web-based platform. Ratings for each statement were averaged, to provide indicative representation of the relative importance and preparedness as reported by respondents for each statement. The nature of the scale and significantly skewed data warrants caution in further analyses, however these averages were used to produce visual tools to enable a ‘birds-eye’ view of the data. Importance and Preparedness ratings were graphed against each other for all data and for each cluster to produce ‘go-zones’. Go-zones also allow for the identification of statements into one of four quadrants using the average of all statements to determine the distinction between high and low (Figure 1).
Figure 1. Go-zone template
Averages were calculated for illustrative purposes for each cluster and clusters ranked according to least-most important, and least-most prepared. A visual ‘pattern match’ was produced which demonstrates for each cluster, relative importance and preparedness, allowing the easy identification of clusters which are perceived as more or less important, and how this relates to perception of preparedness.