The aims of this scoping review are (i) to show which possible outcome areas for measuring effectiveness of DNT can be identified, (ii) to depict which areas have been the focal point of research on effectiveness of DNT so far, and which areas have been researched less, and (iii) to show how effectiveness has been measured in previous studies. The discussion section is structured around these research questions Therefore, we divided the discussion section into three main parts: discussion of the framework development (i), critical reflection in relation to the scientific literature (i) and discussion of the quantitative analysis (ii & iii).
Framework Development
The comprehensive DNT outcome framework was developed to show which possible outcome areas can be identified for the evaluation of DNT. This framework can be used by researchers to structure their effectiveness evaluation and to check whether essential outcome areas are considered in their evaluation. Thus, its purpose is thus to encourage researchers to focus on specific outcome areas or include additional outcome areas in their work. It is also intended to promote and structure discussion and reflection on desirable or necessary research objectives of DNT and may help to draw inferences on areas in which undesirable negative effects may emerge. The framework was developed using deductive and inductive methods, and therefore comprises elements that have already been researched specifically for the field of technology as well as a derivation from general nursing contexts to the specific context of DNT. Parts of the framework are therefore generic and could also be used for the evaluation of general nursing care interventions. We have also included both effectiveness and efficiency studies in the development of the Framework. Being aware of the differences of these two concepts, we assume that the incoherent use of the words within the analysed studies was caused by the fact that “efficacy and effectiveness exist on a continuum” [91] and the generalizability depends on the viewpoint of the observer and the observed condition [91]. The incorrect or incoherent classification of the two terms has already been described in the scientific literature [92]. In order to allow researchers to choose from a variety of possible outcomes adapted to the particular circumstances and context of the study, and to decide which outcome areas are to be evaluated, it is necessary to develop the most comprehensive framework possible. This led us to include studies referring to themselves as effectiveness studies and studies that refer to themselves as efficacy studies. We leave it to the judgement of the respective researchers to decide which are the optimal outcome areas and corresponding outcomes for their aspired studies.
Critical reflection of the framework in relation to the scientific literature
When comparing our framework with existing frameworks in the field of digital technologies in health care we can state that to the best of our knowledge there is no other such comprehensive framework with a special focus on nursing. As stated in the methods section, we incorporated other frameworks in the development of the DNT framework. To highlight the specific strengths of the newly developed framework, we shall now briefly describe the differences between the DNT framework and some other frameworks in similar contexts. The most comprehensive framework available (MAST) was incorporated in the design of the DNT framework [18]. MAST provides seven domains, including a focus on patients and a focus on the organisation. Five of the seven domains are highly relevant to the nursing context and were therefore included in the development of the DNT outcome framework (safety, clinical effectiveness, patient perspectives, economic aspects, organisational aspects). Formal caregivers are only rarely considered in MAST, and informal caregivers are not considered at all. These target groups are presented and highlighted in much greater detail in the DNT results framework. As a holistic framework, MAST also contains references to socio-cultural, ethical and legal aspects. These are important areas for the evaluation of DNT in general, but they do not fit into the specific context of an effectiveness evaluation for DNT that we wanted to depict in this study. The analysis of ethical and socio-cultural effects requires different research approaches, which are not reflected in our sample. In order to cover these areas, it would be necessary to include a macro perspective underpinned by scientifically sound data. We have deliberately not focused on the macro level, but it would be a possibility to complement this with further research.
For a second comparison, the Infoway Benefits Evaluation Framework [17] is used. This framework divides the evaluation of health information systems into six main dimensions: system, information, service, use, satisfaction and net benefits. The “net benefits” dimension includes many outcome areas such as patient safety, health outcomes, access to care and productivity that were integrated into the DNT outcome framework. User (in this case patient) satisfaction was also integrated into the DNT outcome Framework, but we added areas relating to formal and informal carers, as they were not taken sufficiently into account in the Infoway Benefits Evaluation Framework. Other attributes such as accuracy, performance or functionality refer primarily to the effectiveness of the respective technology, so we therefore excluded them from the DNT Framework.
For another comparison we refer to the adapted version of the Nursing Care Performance Framework [4], which displays the impact of ICT solutions on nursing care. This framework shows important areas especially for formal caregiving, which can also be found in the DNT Outcome Framework. Informal caregivers are not represented, which is certainly due to the focus. The effects on patients are presented in a very specific way. Our DNT outcome frameworks can be helpful to complement some details on outcome areas, such as psychological health or health behaviour. The comparison of the DNT framework with the systematization in a systematic review of effectiveness studies in the field telemedicine [93], indicates that the DNT Framework covers all important outcome areas on the micro and meso level. Ekelanda et al. also include a few aspects on the macro level, e.g. in the area of politics [93]. This level was excluded in the DNT outcome framework, as it implies a different perspective of analysis, and none of the studies in the scoping review related to the macro level. The exploration of outcome areas, outcome tools and outcome measures on the macro level, however, is an interesting field for future research. In sum, it can be said that the developed DNT outcome framework closes an existing gap in nursing and technology research by including all important outcome areas relevant to nursing.
Quantitative Analysis
We evaluated all outcome tools and measures of the included 123 articles with a view to establishing which outcome areas have so far been focused on by research on DNT effectiveness, and which areas have been researched less. There are considerable differences in the researched outcome areas for the individual outcome target groups. It should be pointed out, however, that no valuation of the significance of an outcome area can be made at present. This could be the topic of further research. The study presented here constitutes a first step towards summarizing existing trends.
Persons in need of care were by far the most frequently researched target group. Psychological health, intensity of service utilization, and organisational patient safety indicators were measured much more frequently than needs assessment/ fulfilled needs, health knowledge & attitudes or intersectoral continuity of care. There might be several reasons for this. On the one hand, it might be more difficult to capture fulfilled needs, intersectoral continuity of care, or health knowledge & attitudes with the existing standardised instruments or outcome measures. On the other hand, it might simply not have been of such profound interest during the evaluation because the respective technologies did not target these areas. It is interesting to note that intersectoral care is an area that has so far largely been neglected. Intersectoral care in form of communication or collaboration between different healthcare sectors (e.g. ambulatory care and inpatient care) [94] could, for example, help to prevent hospitalisations [95]. . One reason for the neglection of intersectoral care might be that while it is already difficult to implement digital technologies in a single sector, sector boundaries possible constitute a major challenge. The decision-making structures of a healthcare system might be seen to be a barrier to change in this context [96].
Overall in our sample, formal and informal caregivers have not been researched frequently in terms of the effectiveness of DNT. Relationship to person in need of care was most frequently measured among formal caregivers, along with guideline compliance and physical/psychological workload. Well-being/quality of life, health-related quality of life, medical condition and caregiver burden were not measured once in our sample. Several other outcome areas were very rarely analysed. This shows that these aspects were neglected in the past, while quality of life and caregiver burden on professional caregivers are still not being evaluated. The reasons for this should be clarified. It is unclear whether this is the case because these outcome areas are generally considered to be less important than others, or whether there are other reasons for non-evaluation. Assuming that an important goal of digital technologies in nursing care is to relieve nursing staff, it seems inappropriate to only evaluate the direct workload (e.g. physical load or psychological stress through direct work) and not evaluate the effects of digital technologies on the general burden on or the quality of life of formal caregivers. Existing instruments such as the Professional Care Team Burden (PCTB) scale harbour the potential to contribute to the evaluation of DNT in this context [97].
The caregiver burden of informal caregivers was the most frequently analysed outcome area, while all other outcome areas were analysed very seldom or not at all. One reason might be that most instruments for measuring caregiver burden have been developed for informal caregivers [97]. On the other hand, there seems to have been a socio-political interest in reducing the burden on family members through technology in the past. At the same time, the medical health condition, knowledge and competences or the use of caregiving support of informal caregivers have not been analysed in a single study. Medical health may not have been recorded for reasons of personal data protection. Specialised nursing knowledge and skills do not seem to play such a large role in research on technological support for informal carers. We expect enhanced support for informal caregivers to play a more important role in the future, and, hence, those research areas that are seldom examined now to become more important.
The most frequently analysed outcome areas for healthcare organisations were care process quality, operational efficiency and communication/social interaction. Little research has been done on financial performance, access to care and working conditions, while recruitment and staffing areas were not evaluated at all. The frequently researched areas correspond with the potential goal of technologies to improve efficiency while maintaining a high quality of care [98]. Aspects of working conditions or effects on the recruitment or staffing processes from an organisational perspective have apparently never been analysed. If it is assumed that, from an organisational point of view, the main purpose of the technologies is to increase efficiency while maintaining or improving the quality of care, factors such as working conditions for carers play a minor role. This is consistent with the results on formal caregivers. Nevertheless, it is interesting that the impact of existing technologies on recruitment has not been investigated, as technologies are often claimed to be beneficial in terms of enhancing the attractiveness of a healthcare organisation for prospective and current employees [99], which could be expressed in an evaluation of the effects of a technology on recruitment figures. In the light of the current lack of skilled workers, proof of such effects might be an interesting finding.
The third research question was addressed by showing which outcome areas were evaluated with outcome tools and which areas were more likely to be covered by other measures. At the same time, the variety and range of the tools used were presented. The wide range of outcome tools – especially in the field of psychological health - makes it difficult to compare the studies’ results, and a common set of standards for using outcome tools shared by several studies would considerably help improve comparability To this end, further research is needed to assess and evaluate existing outcome tools.
Limitations
Limitations that refer to the underlying scoping review are described in Krick et al. 2019. Especially important for this additional methodological analysis are the following aspects:
Publication bias in particular should be considered here. Studies without positive results are often not published in journals. Outcome areas for which effects are difficult to prove may be underrepresented in this article due to this publication bias, whereby, there is a possibility that studies on certain technologies may have been over-represented, under-represented or not presented at all due to negative results. This may indirectly affect the presented outcome tools and measures.
The included outcome areas and outcome tools presented could also be influenced by the fact that we included both effectiveness and efficiency studies in the development of the framework. It could be criticised that a further breakdown of a framework for efficiency studies and a framework for effectiveness studies is necessary because they may differ in their outcomes and tools. However, we decided to combine these concepts for the reasons of applicability and comprehensiveness, as described in the discussion section. Based on the included publication period of seven years, the question needs to be considered whether outcome tools or outcome measures are only mapped for the indicated period, and therefore relevant measures of the effectiveness of DNT applied outside that period are missing. There is also the possibility that outcome areas overlap or might have been composed differently by other researchers. Overall, we have tried to ensure the highest possible standard for the outcome framework, by combining literature and expert knowledge. The current version is very comprehensive, but the field of research on DNT is very dynamic, and future adaptions should be included.