This systematic review will consists of two phases. In the Phase 1, we will descriptively report the demographics and characteristics of research performed in each country to date (research landscapes). In the Phase 2, we will assess the quality of the research based on the published reports in journals (research quality) (Figure 1).
Inclusion criteria and search strategy
All clinical and biomedical research conducted in Malaysia or Indonesia from January 1962 (Malaysia after Singapore independence) to December 2019 will be identified from the following databases: PubMed, EMBASE, CINAHL and PsycINFO. We will include all published peer-reviewed papers of health and biomedical research done in each country (Malaysia or Indonesia) or by citizen of each country (Malaysian or Indonesian) with an affiliation in one of the institution in each country (Malaysian or Indonesian). We will also search for additional literature from MyMedR (http://mymedr.afpm.org.my/) database as it specifically compiles published papers in health and biomedical research conducted in Malaysia or by authors who has a Malaysian affiliation. MyMedR also draws from MyJurnal, an online system used by Malaysia Citation Centre (MCC), Ministry of Higher Education Malaysia to collect and index all the Malaysian journals. Search results will be compiled into Endnote reference management software where duplicates will be removed. If necessary, authors and institutions will be contacted. A medical librarian and a science officer at the Faculty of Medicine and Health Sciences Universiti Putra Malaysia will assist in these tasks. The review work will be completed by two separate teams with each is based in Malaysia and Indonesia, respectively.
Study selection and data extraction
All reviewers will independently screen identified articles by title and abstract. Full text of eligible article will be retrieved and independently extracted using a standard data extraction template. This template has been pilot-tested on 10 articles among all the reviewers for clarity, and modification of the template was done accordingly. Any discrepancy will be solved by consensus between three or more reviewers. To ensure the data quality, a reviewer (BHC) will reassess 10-20% of the articles. The final piloted template is available as Additional file 1.
In the event of duplicate publications or multiple reports of a research study, we will use the most complete data set aggregated across all known publications. Duplicate publications are defined as two or more published articles that report on the same research question.
1. Research landscapes
The Phase 1 of the project will describe the characteristics of the reported research project such as team members and the journal that publishes the article. The following lists the research characteristics of interest (see Additional file 1).
- Institution and qualification of the corresponding author/s
- Numbers of authors, institutions and specialties
- Numbers of oversea collaborating authors and institutions
- Numbers of study site
- Journal type: local vs. regional vs. international, open access vs. traditional subscription-based, general vs. discipline-specific
- Setting- healthcare facility (hospital, clinic, etc.) or community
- Type of study- audit vs. research- secondary (reviews) or primary (diagnostic, prognostic, etiologic or interventional), clinical vs. non-clinical (laboratory, public health, health service, etc.)
- Data collection designs
- Years when the study conducted, completed and published
- Health conditions studied or organ systems that are involved
- Drugs, devices/tools, surgical, psychological, or health services
2. Research quality
In Phase 2 of the study, the research quality will be assessed based on the following criteria in three domains: relevance, credibility and usefulness (Table 1). All reviewers will learn about the principles of clinical epidemiology through a workshop and reach consensual understanding on the terms used to represent research quality in this project. During the workshop, we will implement a training session for all reviewers in which all reviewers will read and score the same articles. This will be followed by discussion on any similarity or difference in the quality assessment and scores. This will help to ensure uniformity in the understanding of the quality domains when applied on the papers. We will also determine the inter-rater reliability agreement using Cohen’s kappa κ and intra-class correlation (ICC).The kappa κ is a measure of agreement between different observers beyond chance agreement [29]. The κ statistic will be computed separately for each domain’s item (0 or 1). The ICC will be used to assess the domains’ subtotal (3, 4 and 3) and the grand total score of the tool (Table 1).
The Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01- 0.20 as none to slight, 0.21-0.40 as fair, 0.4 - 0.60 as moderate, 0.61- 0.80 as substantial, and 0.81-1.00 as almost perfect agreement [30, 31]. For the ICC, values < 0.40 is poor, 0.40 - 0.59 is fair, 0.60 - 0.74 good, and 0.75 - 1.0 is excellent [32, 33]. We specify an a priori level of κ > 0.60 and ICC > 0.75 must be achieved before the second phase of this study begins. Retraining and reassessment of the reviewers on different articles will be conducted until the inter-rater agreement reach the desirable levels. The expected lower bound of a 95 % confidence limit for κ is no less than 0.60, with an assumed same marginal prevalence of zero score of 30%. Using alpha and beta error rates of 0.05 and 0.2, respectively, a pair of two reviewers will rate 20 papers each [32, 33], with five pairs of reviewers and 100 samples for the subtotal and total ICC estimation [31].
2.1 Relevance
The relevance of a research will be assessed from three perspectives: scientific relevance, the composition of the research team and societal relevance. A research is being scientifically relevant if it addresses a true and real scientific problem and provides the needed knowledge to understand an existing phenomena. Scientific relevance also denotes that the research sets out on justified scientific foundation and informed of existing evidence. Thus, a scientifically relevant research is usually a globally relevant research due to its highly generalizable topic and subjects of research.
Societal relevance refers to the research that addresses a true and real problem in the society. This relevancy may exist at a smaller and wider population such as it may relevant for all the human population in the world or it may be relevant to a particular group of condition or disease in a unique population. These two domains of scientific and societal relevance relate to having a novelty in the research.
The last domain in the relevance category is about the research team of comprising investigators and experts of relevant professional qualifications. This may include patients and public people in certain research area when opinion of the end-users are considered important such as intervention or experience of the patients or family members.
2.2 Credibility
This category is further assessed after it is judged that the research is relevant. Four essential features that are considered the very minimums in a research for it to be credible and its results to inform or contribute to practice change are data collection design, precision, important sample (external validity) and internal validity.
The design of the data collection of a research is to be appropriate to its objective or research question. The approach used in the data collection depends on whether it is a causal or non-causal research, and then experimental or non-experimental conduct of the research would provide better data. The time feature or characteristic of the variables involved in the research should be collected in their intended phases or stages such as a risk factor in the asymptomatic phase, or symptoms or biomarkers in the latent period.
Sampling and samples are the next important credibility domain. The sample of the participants is to be right group of the population for the research. They represent the important population to which the results could be generalised to later. However, in causal or experimental research, comparability between groups in the research take precedence over representativeness because confounding or prognostic factors between groups results in valid outcomes as of the exposure.
Quantitative research is essentially about measurement, measuring tools and process. The measurement of the variables is to be done by validated tools, through a standardised process, and if necessary by trained and blinded assessors. Any query or suspicion on the methods of measurement in the research will cause internal non-validity.
A credible research provides an appropriate and rational sample size estimation. This bases on the research question and its primary objective, and a similar earlier research. Adequate sample size is required for sufficient precision in a research. The achievement or non-achievement of the desired sample size should be reported or justified and discussed, respectively.
2.3 Usefulness
The research that is credible worth its results a good attention. Usefulness of the research results consists of it being important outcomes, providing meaningful estimates and fair conclusion as supported by the research designs.
Important outcomes are that of high priority and concern to the end-users. These generally refer to the hard outcomes or strong correlates or intermediate markers of the hard outcomes to the exposure in the research. Examples of important outcomes include the diagnoses of the conditions, and the examples of the surrogates are blood or serum markers.
Results of a research are meaningful when they are easily understood in the context of clinical practice or daily life of patients. The meaningful estimates are usually the direct results of the study such as the actual numbers of occurrence, incidence and prevalence rates, and risk ratios. Indirect outcome measures such as plasma glucose excursion and transformed estimates such as standardised or log of the unit of measurement will need reverse transformation of the units or they would complicate translation and interpretation of the results.
Lastly, conclusion of the research bears the second testimony to that of the readers’ own judgement of the research. As the final interpretation and remarks by the authors and investigators of the research, it is important to put the results of the research as an evidence in the right context and applicability taken into consideration of the constraint in the research designs and limitations encountered along the whole research process.
Table 1 Research quality domains used in the screening tool
Relevance (3 domains)
|
Credibility (4 domains)
|
Usefulness (3 domains)
|
[1] Scientific relevance
o Indicating this with an acceptable literature review or citing systematic reviews*
[2] Societal relevance (area researched or involvement of end user eg. patient)
[3] Research team / experts
o The research is led by expert in the relevant field or conducted with relevant experts
*Set the right research priorities; clear research question/hypothesis
|
[1] Data collection design- appropriate for the research question*
o Experimental, non-experimental, time feature of variables considered
[2] Important samples (external validity)- representative of or generalizability to an important and relevant population; comparability between groups in randomised control trials
[3] Internal validity – validated instrument, measurement process and by trained or blinded assessors
[4] Precision- appropriate sample size estimation and achievement
* Ethical conduct & patient safety/rights/priorities included
|
[1] Important outcome used and reported*
[2] Meaningful estimates- practical numerical results taking into consideration response rate, missing data, proper statistical test and analysis**
[3] Conclusion based on results
o Take into consideration the study limitations***
* Outcomes that truly matter to patients
** The study provides useful data for the intended end-users; unusual or unexpected analysis is explained and justified
*** No over-claimed or misleading conclusion
|
Subtotal score = 3
|
Subtotal score = 4
|
Subtotal score = 3
|
Total score = 10
|
Data analysis
The principal investigator has the overall responsibility for compilation, maintenance and management of the review database. The database is stored on a password-protected computer.
Every eligible and included journal article will be assessed according to two main areas – the research characteristics and quality of the research as reported in the article. Data will be checked for any missing data and errors. The data will be reported descriptively, with frequency and percentage for categorical data while mean and standard deviation (median and interquartile range) for normally distributed (and not normally distributed) continuous data. Time series plot will be conducted to investigate the trends and patterns of the research characteristics, health conditions studied and quality of research over the years. Geographic information system (GIS) may also be plotted to evaluate the locations and areas of research conducted. Longitudinal trends of certain research characteristics, health conditions or areas in different settings, by different clinical or biomedical disciplines will be explored.
Associations between characteristics of the research and quality will be explored, and the independent effect of each of the determinants will be quantified in multiple linear regression analysis. Additionally, the research quality as a categorical outcome will be explored as tertiles. The highest tertile will be compared to the lowest tertile, and the determinants will be assessed in multiple logistic regression. Longitudinal trends of the research quality will be explored. A calculated 95% confidence interval and two-sided α of 0.05 will be used to test significance. Model checking will be conducted in order to get the best and parsimony final model that meet statistical assumptions. Estimates will be obtained with PASW 25.0 (SPSS, Chicago, IL) and MLwiN version 3.02 (Centre for Multilevel Modelling, University of Bristol).