Initial Experience of 10 Imaging Vendors With the IHE SHARAZONE: a New Distributed Peer to Peer Test Service for DICOM Objects

doi:10.21203/rs.3.rs-2832827/v1

The cross-vendor exchange of DICOM (Digital Imaging and Communications in Medicine) images and objects has been identified as a crucial factor in improving the robustness of product interoperability in the healthcare industry. This, in turn, accelerates innovation and advances the field of medical imaging. Despite multiple attempts to establish a sustainable model for the sharing of DICOM samples, the industry has yet to see a successful implementation. To address this issue, Integrating the Healthcare Enterprise (IHE) has created the IHE SHARAZONE, a continuous cross-vendor DICOM data sharing test service.

IHE is a well-respected organization in the healthcare industry, known for its ability to turn standards such as DICOM, HL7 V2, HL7 CDA and HL7 FHIR into practical, real-world solutions for clinical practice. The IHE SHARAZONE aims to provide a solution for the need for a reliable and consistent cross-vendor DICOM data sharing system.

In order to evaluate the effectiveness of the IHE SHARAZONE, a five-month pilot was conducted with ten imaging vendors. The pilot concluded with a participant survey, the results of which are reported in this publication. The findings of the survey provide valuable insights into the initial experience with the IHE SHARAZONE, and can inform future improvements and developments to this important service.

The healthcare industry is constantly evolving, and it is essential that medical imaging technology keeps pace with these advancements. The IHE SHARAZONE provides a crucial step towards achieving this goal, by enabling cross-vendor exchange of DICOM images and objects, and improving the robustness of product interoperability. This, in turn, will accelerate innovation and advance the field of medical imaging, ultimately leading to better patient outcomes.

Nuclear Medicine & Medical Imaging

IHE

DICOM

informatics

imaging

AI

Interoperability is the ability of a digital system to exchange information with other digital systems in order to achieve specific goals, delivering added value to the clinician, and ultimately, the patient. Interoperability standards have been established to ensure that digital health information originating from Creators of such information can be meaningfully consumed by every recipient.

Standards and products have inherent variations in: clinical application, information created, algorithms employed, workflow, and technology[i][ii]^{^[iii]}. Optional and conditional elements are designed into medical informatics standards to accommodate these variations. While IHE profiling constrains standards to address specific clinical needs and minimize variation, misalignment still occurs. Misaligned products lack interoperability, impeding clinical workflow, potentially impacting the safety and effectiveness of the health information technology ecosystem^{^[iv]}.

IHE Connectathons provide a detailed implementation and testing process to enable the adoption of standards-based interoperability by vendors and users of healthcare information systems. During a Connectathon systems exchange information with corresponding systems in a structured and supervised peer-to-peer testing environment, performing transactions required for the roles (IHE Actors) they have selected to perform in carefully defined interoperability use cases (IHE Profiles)^{^[v]}.

IHE SHARAZONE adds a layer of specificity with an object sharing framework that enables open source and commercial organizations developing applications to ensure they can consume and display DICOM objects coming from a variety of sources. Likewise, Creators can use object sharing to ensure DICOM objects can be consumed by a variety of recipients. IHE SHARAZONE closely reflects the reality of Healthcare Delivery Organizations with a variety of different imaging products that need to interoperate by creating and consuming hundreds of different DICOM objects.

The IHE SHARAZONE is built on the existing IHE Gazelle^{^[vi]} test platform. A dedicated Gazelle instance has been provisioned and made available globally, 24/7 to registered vendors for selecting shared objects of interest, testing their consumption and reporting test results.

Participants are required to execute a contract that establishes terms for data usage, non-disclosure, obligations and fees. Any entity developing DICOM commercial or open source software that is willing to accept the IHE SHARAZONE terms and conditions is eligible to participate. Participants self-declare as Creators and/or Consumers:

Creators author and upload shared test input that contains DICOM objects and a test suite (test procedure).
Consumers download shared test input of interest, executes the test suite, and reports results to Creators.

To protect confidentiality, non-patient, artificially created samples are exchanged and report visibility is limited to paired Creators and Consumers for a given test instance. If necessary, a GDPR-compliant (General Data Protection Regulation)^{^[vii]} chat channel, Rocket Chat^{^[viii]}, is available for real-time dialogue between test participants. A dedicated IHE Moderator oversees the testing and is available to guide the participants in the IHE SHARAZONE process.

This overall process is depicted in Figure 1, while Figure 2 presents an example of a report of an executed test produced by a Consumer for the benefit of a Creator.

Is IHE SHARAZONE different from the IHE Connectathon?

While the participants of an IHE Connectathon and the IHE SHARAZONE are similar, the scope of each is different.

Vendors participating in an IHE Connectathon follow a common test process for profiled standards to suit a particular business use case documented in the IHE Technical Framework.
Vendors participating in the IHE SHARAZONE take part to evaluate the ability of Consumer products to properly handle DICOM object content provided by Creator products along with their test suite.

Test process

IHE Connectathon lasts one week, during which systems on the same network exchange information in a controlled and monitored environment, performing IHE Profile transactions and workflows required for the Actor roles that they claim to support. IHE provides the test suites to execute and the pass/fail results are controlled and determined by IHE Monitors.
IHE SHARAZONE service is open 24/7. Creators provide test suites for DICOM objects that often exceed IHE content profile specifications, addressing aspects such as usability and clinical utility. Consumers then complete a test report and may also offer unsolicited feedback. In most cases, a dialogue ensues, in which both participants receive valuable input. This feedback loop establishes an opportunity for both participants to improve implementations. The evaluation of the results is strictly between the Creator and the Consumer participant. An IHE Moderator is available to address administrative and quality matters, and does not grant pass/fail verification of test instances.

Key benefits of IHE SHARAZONE

An IHE SHARAZONE participant seeks to anticipate issues with new or modified DICOM objects prior to introducing a product to the marketplace. Examples include: modifications to existing DICOM objects, application of new DICOM modules (e.g. CT multi energy) or implementation of complex DICOM objects (e.g. Ophthalmic, Breast Tomosynthesis, Structured Reports, Microscopy).
IHE SHARAZONE hosts datasets from multiple generations of products. They remain available to all participants for testing with newer products. This better reflects the mix of product generation present in imaging departments.
IHE SHARAZONE service fosters collaboration between participants. For example a Consumer Vendor may request specific DICOM datasets from Creator vendors.
Consumers ensure that Presentation States properly control the display of source images; providing the desired transformations, annotations, overlays and greyscale or color pipeline. Consumers also ensure that AI results, such as Structured Reports, Segmentation Objects and Parametric Maps are associated with the original study, properly appear in the PACS worklist, and correctly apply overlays, color, and real-world values. Consumers also make certain that AI results do not create conflicts with existing hanging protocols.

Note that the IHE SHARAZONE is not intended as a repository for AI inferencing inputs. AI accelerators, such as the Medical Imaging and Data Resource Center (MIDRC) or the ACR AI-LAB™, address this need .

IHE SHARAZONE is not intended to evaluate strict DICOM conformance. Objects containing errors may be shared, as these sometimes exist in the real world. This could be intentional or discovered during testing. When non-conformance prevents the DICOM object from being handled properly by the Consumer product, the Creator is incentivized to provide a remedy. Early detection of conformance errors by Consumers in the IHE SHARAZONE helps to improve DICOM conformance.

All SHARAZONE participants are encouraged to use other continuous services, such as Gazelle External Validation Service (EVS) to check conformity throughout phases of product development.

The two services are complementary. Vendors typically participate in the Connectathon to evaluate interface robustness in later development phases, after DICOM encoding and network services are established, have been modified, or when additional claims are added to the IHE Integration Statement. Throughout the product development lifecycle (i.e. prior to and between Connectathon events), vendors leverage the IHE SHARAZONE to evaluate prototype encoding of images and objects, released product modifications, or product enhancements that include adoption of DICOM modules, macros, or controlled terminology.

Evaluation of IHE SHARAZONE

Compared to the IHE Connectathon that was established in 1998, the IHE SHARAZONE is in its early stages of development. To test our hypothesis regarding the utility of the IHE SHARAZONE, scrutinize its contract, and evaluate the adaptation of Gazelle, we elected to hold a small-scale test of the IHE SHARAZONE tooling and procedures.

In June of 2021, an initial small-scale implementation of 10 vendors was convened for a 5-month pilot to prove the viability of workflow, tooling and the acceptability of contract terms. Each of the 10 participants signed a common contract that includes terms of use, data usage policy, antitrust rules and non-disclosure terms. Seventeen systems were registered: 13 Creator systems and 12 Consumer systems. Ten sets of test packages were posted, resulting in 23 test reports. Test packages included visible light, visible light video, dose reports, CT multi energy, Grayscale Softcopy Presentation State (GSPS) and ophthalmetry. Most test packages contained one image, one contained a series of 255 images (median = 1, min = 1, max = 255). Participants met every other week as a group with the IHE Moderator to review test progression, tooling, documentation conventions and communication channels.

At the end of the pilot period, all participants agreed that the IHE SHARAZONE was ready for public launch. A brief, anonymous survey was executed to obtain feedback from participants of their pilot experience. To create a larger sample size, multiple individuals from each participating vendor were encouraged to participate.

Table 1: IHE SHARAZONE pilot survey.

#	Question	Targeted Participants	Response Constraint
Q1	What is your IHE SHARAZONE participant role	All	Creator, Consumer or Both
Q2	The IHE SHARAZONE adds benefits to my company's product development process.	All	Scale of 1-5 1 = Strongly Agree, 5 = Strongly Disagree
Q3	What is the percentage of test instances in which you engaged in a dialogue with your test partner?	All	0-100, increments of 10
Q4	Did you require support from the IHE SHARAZONE moderator?	All	Yes, No
Q5	The Shared Test Input test instructions from Creators that I executed were clear and effective.	Consumers	Scale of 1-5 1 = Strongly Agree, 5 = Strongly Disagree
Q6	What could be improved with Shared Test Input instructions?	Consumers	free text response
Q7	On average, in hours, how long did it take you to execute a test report?	Consumers	Number (decimal)
Q8	The test reports that I received from Consumers were clear and effective.	Creators	Scale of 1-5 1 = Strongly Agree, 5 = Strongly Disagree
Q9	What could be improved with test reports?	Creators	free text response
Q10	On average, in hours, how long did it take you to prepare a shared test input ?	Creators	Number (decimal)

Table 2: IHE SHARAZONE pilot survey Results.

Question #	Respondents	Results
Q1	13	Both = 7, Creator = 3, Consumer = 3
Q2	13	Median = 1, Min = 1, Max = 3
Q3	13	Median = 80%, Min = 10%, Max = 100%
Q4	13	Yes = 3, No = 10
Q5	10	Median = 1, Min = 1, Max = 3
Q6	4	4 responses received
Q7	10	Median = 1 hour, Min = 0.5 hour, Max = 4 hours
Q8	9	Median = 1, Min = 1, Max = 4
Q9	3	3 responses received
Q10	7	Median = 2.25, Min = 1, Max = 3, Average = 2.6

Analysis

Due to the small sample size and categorical data types, the ability to provide a detailed statistical analysis is limited. When feasible, Chi-Square Goodness-of-Fit Test was performed to identify differences in groups of responses by role (Minitab 20.4 Statistical Software, State College, PA).

All participants indicated that the IHE SHARAZONE benefits the product development process (median = 1 = Strongly Agree, see Q2 and Figure 3). Further segmentation by role of Creator (n=3), Consumer (n=3) or Both Creator and Consumer (n=7), indicates that the Both Creator and Consumer group provided different responses (p=0.041), due to a single neutral (neither Agree nor Disagree) response. Participants indicated that they engaged in dialogue with their test partners 80% of the time (median = 80%). The Both Creator and Consumer group engaged in dialogue less frequently than participants registered as only Creators, or only Consumers (p=0.0, median = 70% and median = 95% respectively). Few participants (n=3), regardless of role, required moderator assistance (p=0.154).

Consumers (n=10) spent approximately one hour testing downloaded test packages (Q7 and Figure 3) and indicated that test instructions from Creators were clear and effective (median = 2 = Agree, see Q5 and Figure 3).

Creators (n=9) spent over two hours to prepare test packages (median=2.25, see Q10 and Figure 3) and indicated that test reports from Creators were clear and effective (median = 1 = Strongly Agree, see Q10 and Figure 3).

Overall, participants expressed positive feedback to the IHE SHARAZONE pilot. A strong majority agreed that the service offers benefits to the development process by providing: “...an opportunity to test established as well as novel DICOM features”, “... quick and profound feedback” and an “always-available, collaborative environment…”^{^[i]}.

The survey helps to show that the moderator and chat features facilitate the process. The reliance on peer-to-peer dialogue is a foundation of interoperability testing, as all levels of interoperability involve connecting stakeholders prior to connecting machines^{^[ii]}. Conversely, respondents did not indicate a strong reliance on the Moderator, suggesting that tooling and documentation is found to be intuitive. The Moderator reported supporting administrative activities such as uploading test packages, and reminding participants to complete open test instances.

Consumers spent between 0.5 and 4 hours testing, suggesting that the IHE SHARAZONE poses a low burden. Three Consumers indicated that Creators could improve test procedures by including screenshots of expected results; one indicated more specific instructions (vs relying on the template) should be provided.

Creators reported spending between 1 and 3 hours on test package creation, this includes the time to anonymize images and author the test suite (test procedure).

Future

The Survey has demonstrated a good level of satisfaction from the current participants with an environment that meets their needs and provides a clear return on their investments. The likely expansion of IHE SHARAZONE will involve:

New participants. The IHE SHARAZONE's participation is on the rise, aiming to reach 30 participants by the conclusion of 2023. In order to align the participant profile more closely with the imaging market, IHE is seeking to attract participants from small and mid-sized companies.
Additional systems. New participants will register new systems both as Creators and Consumers, and contribute their shared test input. But existing participants will also expand the number of their participating systems, both with newer systems and older systems that need to be verified for interoperability with newer systems.

Such an exponential growth will require improvement in the IHE SHARAZONE infrastructure tooling offered, to address better ways to navigate a much larger choice of shared test inputs as well as a larger set of test reports being produced by Consumers that need to be processed by the Creators.

The survey also identified that the creation of test suites and their fulfillment in creating test reports could be further facilitated by a choice of test suite forms definition and test report creation tools that are semi-automatically derived from the test suite they respond to.

In addition to engaging a broader set of participants and systems which is the natural growth expected, IHE SHARAZONE is exploring the engagements of other communities than the community of product and solutions developers. The standards and profiles development activities are a natural area of growth. As new standards and profiles are introduced, the IHE imaging related domains such as Radiology, Radiation Oncology, Cardiology, Endoscopy, or Pathology could leverage the IHE SHARAZONE to offer “sandboxes'' where early version of these new standardized or profiled objects could be made accessible to early implementers. Through their testing they could identify areas where the standard or profile could be further improved before being released in final form. IHE SHARAZONE has already extended its contract to offer to Profile/Standards Development Organizations such as IHE Radiology and DICOM to become Creators of these new standardized objects and use the IHE SHARAZONE testing process to allow their communities to report on the implementation progress of these new standards.

As previously stated, the Imaging AI community remains keen on utilizing the IHE SHARAZONE, not only to enhance their access to imaging data, but also to ensure seamless integration of AI results into the imaging enterprise, in a manner that supports day-to-day clinical practice[iii]. Multiple Shared Test Inputs with AI results have been added to the IHE SHARAZONE, making them some of the most thoroughly tested packages.

Participating in the IHE SHARAZONE

Healthcare Delivery Organizations should encourage their vendors to participate in the IHE SHARAZONE. Due to the NDA (non-disclosure agreement) obligations of the IHE SHARAZONE contract, vendors are not able to reveal test details, however, they are permitted to provide the names of participating products and the number of test instances each product has participated in. Details pertaining to test peers and test results cannot be disclosed to ensure that the candid engagement of participants is not curtailed.

Healthcare Delivery Organizations that are developing commercially available or open source software intended to Create or Consume DICOM images and other objects are invited to participate in the IHE SHARAZONE^{^[iv]}. Joining involves: review and signing the IHE SHARAZONE NDA contract, and paying the participation fee. Participant fees are structured based on organization size:

Small <250 full time employees (FTE),
Mid-sized 250-2500 FTE, and
Large (>2500 FTE)

As of March 20, 2023, thirty-six products have participated in more than eighty tests in the IHE SHARAZONE^{^[v]}. These thirty-six systems are distributed among twelve vendors as shown in Table 3.

Table 3: IHE SHARAZONE participant profile at time of draft of this manuscript.

	Creator Systems	Consumer Systems
Small participant	4	3
Mid-sized participant	3	1
Large participant	5	3

Interoperability continues to require mobilization and representation from Users and Vendors. Users have a powerful role in influencing Vendor behavior by insisting on participation in IHE interoperability improvement initiatives. This has been clearly demonstrated through tenders or RFPs (request for proposal) that require proof of participation in the IHE Connectathon, in addition to the submission of IHE Integration Statements. As the IHE SHARAZONE adds a meaningful layer of specificity to the IHE Connectathon testing; the IHE User community can continue to drive interoperability by motivating Vendors to participate in the IHE SHARAZONE and request proof of participation.

Consent to participate / Consent to publish All survey participants were informed of the purpose of the study and provided verbal consent for their responses to be used in the analysis and reporting of findings.

Ethics Approval

Ethics approval was not required for this study as it did not involve human subjects. The research was conducted in accordance with all applicable laws and regulations.

Consent to participate / Consent to publish

Not applicable

Funding

This research was not supported by any funding. Administrative support for this research was provided by IHE Europe.

Competing Interests

Steven Nichols is an Employee of GE HealthCare, a member of the IHE Europe Steering Committee, member-at-large of IHE International, and a member of the DICOM Executive Committee
Bruno Laffin is an Employee of Agfa HealthCare, a member of the IHE Europe Steering Committee, and a member of the DICOM Executive Committee
Charles Parisot is the Principal and Chief Executive of InteropEhealth, a member of the IHE Europe Steering Committee, member-at-large of IHE International, a member of the General Assembly of IHE-Catalyst, and a member of the DICOM Executive Committee

Patents and Intellectual Property

There are no patents to disclose

Clunie DA, Dennison DK, Cram D, et al. Technical Challenges of Enterprise Imaging: HIMSS-SIIM Collaborative White Paper. J Digit Imaging. 2016 Oct;29(5):583-614. doi: 10.1007/s10278-016-9899-4. PMID: 27576909; PMCID: PMC5023533.
Boufahja A, Nichols S, Pangon V. Quantitative Evaluation of PACS Query/Retrieve Capabilities. J Digit Imaging. 2021 Oct;34(5):1302-1315. doi: 10.1007/s10278-021-00511-9. Epub 2021 Sep 28. PMID: 34581930; PMCID: PMC8554935.
Design Considerations and Premarket Submission Recommendations for Interoperable Medical Devices, U.S. Food and Drug Administration, https://www.fda.gov/media/95636/download. Accessed February 13, 2023.
Connectathon, IHE International, https://www.ihe.net/participate/connectathon/. Accessed February 13, 2023.
IHE SHARAZONE Gazelle, IHE Catalyst, https://sharazone.ihe-europe.net/gazelle/home.seam. Accessed February 13, 2023.
General Data Protection Regulation (GDPR) Compliance Guidelines, Complete Guide to GDPR Compliance, Proton AG, 2022, https://gdpr.eu/. Accessed February 13, 2023.
Rocket.chat Is GDPR Compliant, Rocket.chat, https://docs.rocket.chat/legal/gdpr. Accessed February 13, 2023.
IHE SHARAZONE, IHE Europe, https://www.ihe-europe.net/IHE_SHARAZONE. Accessed February 13, 2023.
Interoperability in Healthcare, HIMSS, https://www.himss.org/resources/interoperability-healthcare. Accessed February 13, 2023.
Pierce JD, Rosipko B, Youngblood L, Gilkeson RC, Gupta A, Bittencourt LK. Seamless Integration of Artificial Intelligence Into the Clinical Environment: Our Experience With a Novel Pneumothorax Detection Artificial Intelligence Algorithm. J Am Coll Radiol. 2021 Nov;18(11):1497-1505. doi: 10.1016/j.jacr.2021.08.023. Epub 2021 Sep 28. PMID: 34597622.
IHE SHARAZONE Statistics (DICOM), IHE Catalyst, https://sharazone.ihe-europe.net/statistics.html. Accessed March 13, 2023.

Initial Experience of 10 Imaging Vendors With the IHE SHARAZONE: a New Distributed Peer to Peer Test Service for DICOM Objects

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Discussion

Conclusion

Declarations

References

Status:

Journal Publication

Version 1