Estimating research beneficiaries to reduce bias in public health research using direct and indirect production efficiencies

doi:10.21203/rs.3.rs-3246822/v1

Download PDF

Article

Estimating research beneficiaries to reduce bias in public health research using direct and indirect production efficiencies

https://doi.org/10.21203/rs.3.rs-3246822/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Research and development produce academic publications that benefit industries and regions, both directly and indirectly. Direct effects are often realized as revenue, profits, taxes, or the quantity of publications. However, indirect effects, or spillover effects, such as improvements to an area because of newly obtained knowledge, are much harder to measure. Calculating the productivity efficiency of a region, for R&D publications, by research location is one way spillover effects can be measured. Furthermore, compiling both main effects and spillover effects into an aggregated productivity measurement will facilitate the true R&D beneficiaries and non-beneficiaries by region. Productivity measurements are sensitive to regional and industrial factors. Total Factor Productivity (TFP) takes these into account in the form of capital and labor. Measuring TFP using the sum of direct and indirect effects makes it possible to identify regions of the true R&D beneficiaries and non-beneficiaries through estimates of productivity, innovation, and competitiveness. Applying a quantitative analysis to identify these beneficiaries and non-beneficiaries will demonstrate whether interpretations regarding the knowledge distribution system are warranted or not and provide maps of possible research gaps and biases. This analysis focuses on childhood elevated blood lead level research publications, and it identified a significant cluster of high production in the Northeast and a significant cluster of low production in the northern Midwest of the U.S. for articles published between 1990 and 2019.

Social science/Economics

Social science/Geography

Complex systems have a way of muddying the water for the individuals that participate within them. This is because complexity is represented by large networks that are constantly changing, which leads to the deep organization we call chaos (Richards, 2021). As a result, the more complex a system is, the more chaos it contains. Furthermore, participating inside of a system among systems adds additional layers of chaos. Academia is a complex system that sits atop many other complex systems. One of the subsystems within academia that acts as a measuring stick is the research and development (R&D) knowledge distribution system. This system disseminates R&D knowledge through publishing scientific journals, books, and other media. Impact factors, citations, journal prestige, and other attributes are sometimes used as measures of individual or academic achievements (Li et al., 2019). However, arguments from within academia suggest that the academic R&D distribution system is dysfunctional. One critique focuses on over-production of academic articles by journals, which the authors argue diminishes the quality of R&D (Altbach and de Wit, 2018). Another critique suggests that the unusual structure of the academic publishers’ business model created an affordability crisis that prevents academic libraries from affording to purchase the research conducted by their own faculty (McGuigun et al., 2008). And another claim, that the academic publishing industry earns too much profit compared to other industries, reduces both accessibility and quality of R&D (Hagve and Martin, 2020). If there is legitimacy to these claims, it indicates that the academic R&D system is inefficient, but it must be questioned whether this is a biased perspective from within the academic system.

Echoing many of the criticisms mentioned, Michael Eisen, the co-founder of one of the largest open-access journals, advocates for open-access journals as a solution (Van Noorden, 2013). The benefits of open-access journals are enticing, particularly the fact that they are free for everyone, except funders and authors. However, the intangible benefits of traditional journals are higher. This is not unlike the intangible costs, which tend to be more than the tangible costs when publishing in a traditional journal (Johnson et al., 2018). The intangible benefits of traditional journals are associated with price and selectivity, which shows that traditional journals have a function beyond just publishing articles (Van Noorden, 2013). This intangible aspect is referred to as filtering and it occurs before publication. To clarify, traditional journals are filtered by the cost and process of selectivity. The filtering process can be broken down into sorting and assessing quality, content aggregation, distilling evidence, consensus seeking to create standards, granularization, system integration, enforcing standards of disclosure, and developing workflow analytics (Johnson et al., 2018). This process costs money and improves the quality of the journal that a publisher represents. The quality of a journal contributes to its brand which is a representation of the quality expectation from consumers (e.g., academia and the science community). The desire for this qualifying process is apparent in the slow recognition given to open-access journals as a viable way to publish research (Van Noorden, 2013). Yet, there has been a significant growth in the use of open access journals in the last two decades, with the top-performing universities publishing 80%-90% of their research using this method (Huang et al, 2020). This is most likely attributed to the benefits outlined above. Considering the two-sided arguments that have been made, the question of whether biased perspectives from within the R&D system are again relevant.

Regardless of how R&D has been published, it has a positive effect on economic growth because it leads to innovation (Blanco et al., 2016). Improvements to productivity create private benefits, along with spillover benefits, within and among firms, industries, and geographic regions. Some of these positive benefits include a rising standard of living, intermediate goods, and new ideas. These new ideas feed innovation. As innovation improves, the competitiveness of firms, industries, and geographical regions improve (Veselica, 2019). Therefore, measures of R&D production efficiency can be used as an indicator of innovation and competitiveness for an economy, by industry, region, or both. In this case, a high quantity of research in an area does not represent the need for research in an area or industry, but instead it might represent elevated levels of innovation and competitiveness. Likewise, high profits by the R&D publication industry might be a result of elevated levels of innovation and competitiveness coupled with efficient business practices. As R&D production efficiency increases due to innovation and competitiveness, a cyclical process begins. As the current system meets its capacity to match demand it evolves, and new outlets for distribution become available, such as open access journals. Therefore, the arguments of too much research production, high costs of journals, high profits, and debates between traditional and alternative publication methods potentially indicate higher levels of R&D production, innovation, and competitiveness.

R&D production efficiency measures will then represent innovation and competitiveness measures. Yet, care must be taken when measuring production efficiency because there are a variety of methods for measuring it. Also, there is significant dispersion in R&D across geographies (Blanco et al., 2016). In fact, productivity diverges most by geographic region and industry (Altomonte and di Maruo, 2022). Hence, selecting a productivity measurement that allows for the effects that are relevant to the output measurement across industry and geography are necessary. A strong option for this is the Total factor productivity (TFP). The TFP measurement is an index of output efficiency given a series of inputs (i.e., labor and capital) (Altomonte and di Maruo, 2022). TFP has been used to measure granularity, or the proportion of low producers to high producers (Altomonte and di Maruo, 2022). It has also been used to measure private R&D productivity across the U.S. (Blanco et al., 2016). TFP can be used to account for industrial and geographic heterogeneities and idiosyncratic shocks and can be used to bridge the gap between micro- and macro-economic levels (Altomonte and di Maruo, 2022). TFP is used by the U.S. Bureau of Labor Statistics, the U.S. Congressional Budget Office, and the International Monetary Fund as a measure of long-run productivity (). Therefore, TFP is a proven method that can be used to understand industry and geographic distributions and fluctuations of R&D production efficiency, competitiveness, and innovation.

A novel method used to assess R&D production was performed in 2020. Millard et al used text-mining tools to analyze thousands of research articles by research location and taxonomy (Millard et al., 2020). Their analysis effectively showed that more than 50% of the publications analyzed had research locations in five countries, 20% of the publications analyzed were specific to the bumble bee (i.e., Bombus), and 25% of the publications analyzed were specific to the honeybee (i.e., Apis). On a global scale, these statistics show a potential bias to study two of 200,000 animal pollinators (Marks and Pauline, 2005) in five of 193 countries (United Nations, 2022). However, these discrepancies are influenced by economic decision making, in addition to the biases suggested by the authors. This is evidenced by the fact that the top five countries identified (i.e., United States of America, Brazil, Australia, Canada, and China) represent the largest GDPs by region in 2020 (World Economic Forum, 2020). Considering this, the use of research location opposed to publication location is a novel concept that can aid in capturing spillover effects while assessing R&D productivity.

This study aims to refine the discussion surrounding the potential biases in the R&D knowledge distribution system by producing maps displaying R&D production efficiencies, innovation, and competitiveness throughout the U.S, and identify the primary beneficiaries and non-beneficiaries for this topic. Furthermore, combining publication location productivity with research location productivity to calculate efficiencies will demonstrate the true production efficiencies for R&D publications. Hence, it will provide output maps of the total production efficiency, which will identify regions of total competitiveness and innovation. The total production efficiency values can be used to cluster regions based on their production, innovation, and competitiveness. The clustered outputs will allow R&D decision makers to identify geographical research gaps based on R&D production, competitiveness, and innovation. All of which are important factors needed to aid policy makers in effective policy making decisions (Altomonte and di Mauro, 2022). This analysis can be reproduced for any topic where research location is relevant, and it can be used to assess the distribution of R&D production by research location, publication location, or combined productivity. The result of this analysis is relevant for independent researchers, R&D publishers, R&D decision makers, and public policy decision makers.

For this analysis the topic of childhood elevated blood lead levels (EBLL) was selected because of its historical record of research, it will help public policy decision makers to better understand the R&D production in this field, and because data on this topic is accessible. Also, spatiotemporal analysis an appropriate methodology for public health topics (Gui-Quan Sun et al., 2016). The data was extracted from a body of public health research articles on the topic of childhood EBLL. This corpus was curated by the librarians from the U.S. Environmental Protection Agency as a representative list of all available childhood EBLL research between 1990 and 2019. All the research articles identified were conducted in the U.S. or Canada and written between 1990 and 2019. Three information content aggregators were used to extract the body of research — PubMed, Web of Science, and ProQuest. Each of the information content aggregators were filtered for English language, scholarly articles, and peer-reviewed articles. The search terms used for PubMed included: “child” or “child, preschool” or “infant” or “infant, newborn” or “adolescent” and “lead/blood.” The search terms used for Web of Science included: “blood lead” or “lead level*” or “lead exposure” or “exposure to lead” or “Pb exposure” or “Pb level*” or “environmental lead” and “children” or “pediatric.” And the search terms used for ProQuest included: “blood lead” or “lead levels” or “lead exposure” or “exposure to lead” or “Pb exposure” or “Pb levels” or “environmental lead” and “child*” or “pediatric.” The output from these three content aggregators, on the topic of childhood EBLL, in the U.S. and Canada, and between 1990 and 2019 was 1,073 scientific journal articles. These articles were used to extract data for this analysis.

The first step for data extraction was to identify the geographic aggregation unit that would be used for the analysis. The criteria for selecting a geographic aggregation unit were 1) a unit that could easily be extracted from all the journal articles, and 2) a unit that is commonly used to aggregate other data. The U.S. state level geographic unit was selected. The decision to use this aggregation unit excluded Canada from the analysis. Next, the geography data was manually extracted from the 1,073 childhood EBLL research articles. The articles’ titles and abstracts were manually reviewed, and the U.S. state where the research was conducted was extracted and documented; this is referred to as the “research location.” Study location does not necessarily align with the location of the authors’ affiliation. Articles that did not have a study location were excluded from the analysis.

In addition, article subtopics were manually extracted and identified as either “environmental” or “non-environmental” studies. Environmental studies were divided into subcategories: “indoor”, “outdoor”, or “both”. “Indoor” refers to potential indoor Pb sources and/or measurements inside a residence or structure, such as paint or dust concentrations. “Outdoor” refers to potential outdoor Pb sources and/or measurements, such as ambient air or soil measures. And “both” refers to studies that included both “indoor” and “outdoor” measurement and/or modeling components. Articles classified as “non-environmental” typically focused on blood Pb and health outcomes without any consideration of environmental exposure sources. This process was conducted by two separate researchers, independent of one another, and then validated for quality assurance and control. This data and manual data extraction process is the same used in a previous study which demonstrated the use of research location for a knowledge synthesis methodology (Grokhowsky, 2023).

The publication years, publication geographies, and journal names were provided by the content aggregators, and stored in the metadata file included with the literature search. To access this data, the research articles’ metadata was parsed, and the publication years were stored, publication zip codes were stored, and journal names were stored. To identify the publication location at the U.S. state-level, each zip code was cross-referenced with the U.S. Census Bureau’s zip code shapefiles and the corresponding state was recorded. The output of this process provided a column of the years the articles were published, U.S. states where the articles were published, and the journal names that published the articles. The U.S. state where the articles were published is referred to as the “publication location.”

Using the data extracted from the 1,073 childhood EBLL research articles, quantities by geography were calculated for 1) the observed number of journals that published childhood EBLL research in each state; 2) the observed number of articles published in each state; 3) the observed number of articles researched in each state; 4) the observed number of articles published each year in each state; and 5) the number of articles for each subtopic in each state. Correlations for these data points were calculated. The publication year and subtopic frequencies were then derived into binary indicator variables. This was done by replacing the frequency values equal to 0 with a 0 and frequency values greater than 0 with a 1 (e.g., a state with 0 articles written about a subtopic would be recorded as 0 while a state with 1 or more articles written about a subtopic would be recorded as 1). This was conducted for both the quantity of articles published each year and the quantity of articles for each subtopic. Then the derived indicator variables were encoded as categorical values.

Time series analysis was performed by grouping the quantity of childhood EBLL research articles by publication year. The quantity of articles was adjusted for overall research growth equal to 4% per annum (Johnson et al., 2018). An exponential decay formula was used to make this adjustment. The decayed article quantities were evaluated using a time series plot resulting in three distinct periods (1990–1995, 1996–2015, and 2016–2019) and confirmed using Chow tests. These same time periods were found in a previous analysis of this data (Grokhowsky, 2023). Chow tests were conducted by creating three binary indicator variables for each time period and regressing them to the decayed quantity of research in each location using ordinary least squares regression. Thus, three additional indicator variables were derived to represent the time periods published.

In addition to extracting and grouping data from the research articles, feature variables were identified and obtained at the U.S. state level. Sociodemographic variables and structure ages have previously been correlated with EBLL (Xue et al, 2022; Teye SO et al., 2021; Hauptman et al, 2021). Therefore, sociodemographic and structure age data were collected as potential feature variables. Additionally, economic and environmental variables were obtained, as these may also influence research production on childhood EBLL. All feature variables were collected for the timespan between 1990 and 2019, and the mean values per U.S. state were used. In some cases, the timespan was averaged by the available dates between 1990 and 2019 (i.e., mean of data collected every three years or mean of data collected every decade). The primary data sources included the U.S. Census Bureau, the U.S. Bureau of Economic Analysis, the U.S. Bureau of Labor and Statistics, the U.S Environmental Protection Agency, the U.S. Department of Agriculture, the U.S. Geological Survey, and the National Science Foundation. Each feature variable was downloaded using either the governing body’s data portal or an application portal interface (API). All available data for the time period between 1990 and 2019 was downloaded and averaged to create the feature variable dataset. This resulted in 89 feature variables (Table S-1). For the sociodemographic variables that were reported as total units, the per capita value was calculated to account for the total population. Also, feature variables with 5% or fewer missing data were imputed using simulations or mean values from the surrounding U.S. states. If more than 5% of the data was missing the variable was removed from the analysis. Imputation was used for social and economic factors and neighborhood averaging was used for physical factors. For example, a simulated imputation procedure was completed for Wyoming’s “Management Earnings” variable, and the average values were calculated using Maryland’s and Virginia’s values for the District of Columbia’s “Soil Pb Mean,” and “Soil Pb Median” variables. It was done this way because Maryland and Virginia are the two U.S. states surrounding the District of Columbia. In this case we assumed that near things are more alike than far things (O’Sullivan and Unwin, 2010).

Next, education and age of structure data were separately aggregated into derived feature variables. Sociodemographic factors, including maternal and paternal education level, may be associated with higher EBLL for children living in the home, therefore the variables representing the population based on education level were combined into two separate, continuous variables that measure the population of: 1) “High School Education and Lower” and 2) “Some College Education and Higher.” Furthermore, the variables for the quantity of structures based on their year built were aggregated. The aggregation thresholds were based on the dates Pb-based paints and leaded gasoline were historically used. The variables that represent the quantity of structures built between 2005 or later, 2000-04, 1990-99, 1980-89, 1970-79, 1960-69, 1950-59, 1940-49, and 1939 or older, were combined into four derived variables: 1) “Structures Built After 1970”; 2) “Structures Built Before 1970”; 3) “Structures Built Before 1960”; and 4) “Structures Built Before 1950.” Lastly, all continuous, feature variables were standardized to account for the vastly different scales between them.

Before estimating the population of the quantity of EBLL research articles by publication location, research location, and the quantity of EBLL research publishers, exploratory analysis of the observed data was conducted. This included summary statistics and bar charts to display the observed quantity of articles written by geography and subtopic. Next, TFP_OBS was calculated using the observed quantity of childhood EBLL research per publication location and the observed quantity of childhood EBLL research per research location. The Cobb-Douglas production function was used to calculate the Solow residuals as the TFP_OBS. The Cobb-Douglas production function: log y = log A + α log K + β log L;y = observed values, K = capital, and L = labor was used (Vinood, 2008). The observed values (y) equaled the quantity of research published in each region summed with the quantity of research conducted in each region, the capital measurement (K) equaled the sum of state and federal R&D obligations, and the labor measurement (L) equaled the observed number of childhood EBLL publishers per state. The observed number of childhood EBLL publishers per state was used to represent the labor measurement because of strong correlations between the number of these publishers with both the number of articles per research location and the number of science and engineering articles published per 1,000 graduate students, quantity of observed research articles by research location, and quantity of observed research articles by publication location. Hence, we assumed the number of publishers in geographic region is a strong representation of the scientific research publication labor in that geographic region. The observed quantities of research articles by publication location and research location were decayed by 4% per annum (Johnson et al., 2018). The Solow residuals were stored for the model as the TFP_OBS. TFP_OBS residuals were assessed, and the estimates were visualized by the 10th and 90th percentiles of the TFP_OBS value on a map.

The number of childhood EBLL journals per state, the number of childhood EBLL articles written per publication location, and the number of childhood EBLL articles written per research location were estimated to an infinite population with three separate random forest regression models. Each random forest regression model was calculated, and the variables were extracted in order of importance. The top 5 most important feature variables were then used as the only features for estimation of the revised random forest regression models, for each of the three observed quantities. After the models were estimated, a repeated cross validation was calculated for each model. The original models were then tuned for the maximum number of trees and re-run. These final models were used to capture estimates of the population of the quantity of EBLL journals per state, the quantity of EBLL articles written by publication location, and the quantity of EBLL articles written by research location. These estimated values were then used to calculate the TFP_EST with the Cobb-Douglas production function using the total quantity of EBLL articles (i.e., publication location and research location) and EBLL journals. For the TFP_EST measurements, the dependent variable (y) equaled the estimated quantity of research published in each region summed with the estimated quantity of research conducted in each region, the capital measurement (K) equaled the sum of state and federal R&D obligations, and the labor measurement (L) equaled the estimated number of childhood EBLL journals per state. The Solow residuals were stored for the model. Residual analysis, calculated from the TFP_OBS minus the TFP_EST was visualized and checked for normality, and the estimates were visualized by quantile bins. Next, global and local spatial autocorrelation was calculated, using the queen’s neighborhood, for the residuals (i.e., TFP_OBS minus the TFP_EST) and the TFP_EST to assess spatial independence of the residuals and local clusters of the estimated values. Spatial autocorrelation was only assessed for the 48 continental U.S. because Hawaii and Alaska are not directly connected to the mainland, which would cause incorrect results in the cluster analysis. Significant (i.e., p < 0.05) Local Indicators of Spatial Association (LISA) were mapped.

All analysis was produced using R 4.1.2 (2021-11-01) statistical computing and graphics language. The integrated development environment (IDE) used was Rstudio 2022.12.0. R Markdown 2.20 was used to format code chunks and produce dynamic code and commentary documents.

The literature search returned 1,073 scientific journal articles on childhood EBLLs. After extracting research locations, and discarding articles missing research locations, there were 489 articles remaining. Eight articles did not have a publication year recorded, which were removed, leaving 481 articles that comprised the observed sample dataset. The range of the quantity of childhood EBLL journals per state was 0 to 57 with a mean of 10.71; the range of articles by publication year per state was 0 to 82 with a mean of 13.00; and the range of articles by research location per state was 0 to 50 with a mean of 9.43. Ten states do not have observed childhood EBLL journals (Fig. 1). The top five states with the most observed childhood EBLL journals are New York, California, Massachusetts, Pennsylvania, and Maryland (Fig. 1). There were no observed articles on childhood EBLL by publication location in sixteen of the states (Fig. 2). The top five states with an observed publication location of this research are Massachusetts, New York, California, Louisiana, and Indiana (Fig. 2). There were no observed articles by research location in seven states (Fig. 3). The top five states with an observed research location were New York, Michigan, California, Louisiana, and Massachusetts (Fig. 3). The proportion of states where environmental subtopics were written is 34% for publication location and 35% for research location (Fig. 2 and Fig. 3).

Next, the time trends for the decayed publication year were plotted and analyzed. These plots showed two peaks of research on childhood EBLLs in 1995 and 2016 (Fig. 4). Chow tests were used to confirm these results by comparing the differences between the coefficients of ordinary least squares regression of the quantity of articles to the time period indicator variables (p-values < 0.001 for time periods between 1990-95, 1996–2015, and 2016-19). As there is typically a lag between research being conducted and a journal article being produced, these peaks were interpreted as the change in the CDC’s level of concern for children’s blood lead levels to < 10 µg/dL in 1991 (U.S. Center for Disease Control, 2022), and the Flint water crisis in 2014 (U.S. Center for Disease Control, 2022). The proportion of states and the District of Columbia with articles published between 1991–1995, 1996–2015, and 2016–2019 was 41%, 84%, and 65% respectively.

The observed TFP values, referred to as TFP_OBS, were calculated using the Solow residuals from the Cobb-Douglas production function for the publication locations, research locations, and the average TFP values. The TFP_OBS were mapped by the 10th and 90th percentiles of the TFP_OBS values to display high versus low values. These values were visualized for all articles, all-environmental articles, and all-environmental articles written in each of the time intervals identified by time-series analysis (i.e., 1990-95, 1996–2014, 2015-19). Michigan and Louisiana are consistently observed in the 90% TFP among all groups. Connecticut is consistently observed in the 10% TFP among all groups. All the Cobb-Douglas production models significantly varied from random chance (p-values < 0.001). Additionally, the amount of variability explained by each model was greater than 87% (R²-adjusted > 0.87).

Three random forest regression models estimated the number of childhood EBLL journals per state, number of EBLL articles by published state, and the number of EBLL articles by research state. The Root Mean Squared Error (RMSE) for each model was 7.67 for journal quantities, 12.10 for publication locations, and 8.15 for research locations. Also, the percentage of variance explained was 67.55%, 54.98%, and 47.11%, respectively. Cross validation returned the respective RMSE values of 7.03, 10.86, and 6.84. All cross validated percent of variance explained was higher than the modeled data which provides evidence that the modeled data is not overfit (Westfall and Arias, 2020). The percentage of variance equaled 71% for publication locations, 69% for research locations, and 79% for the number of journals. The features selected by the machine learning model included structure age, education, race, contamination, and income equality. All of these were chosen by the model specifically for estimation purposes and were not interpreted as hypothesis tests.

The estimates taken from the three random forest models were used to calculate estimated TFP values using the same procedure as the observed data, referred to as TFP_EST. Like the observed Cobb-Douglas production models, all the models (i.e., all-article model, all-environmental article model, environmental article between 1990-95 model, environmental article between 1996–2014 model, and environmental article between 2015-19 model) significantly varied from random chance (p-values < 0.001). Additionally, the percentage of variation explained by each model was greater than or equal to 93%. The TFP_EST were mapped by the 10th and 90th percentiles of the TFP values to display high versus low values (i.e., beneficiaries versus non-beneficiaries). All the model estimates returned the same U.S. states or territories within the 90th percentile and 10th percentile, except for the model of environmental articles published between 1990–1995. For all but the environmental article model with publications between 1990–1995, the locations with TFP_EST in the 90th percentile was California, Tennessee, Illinois, Pennsylvania, Delaware, and the District of Columbia, and the locations with TFP_EST in the 10th percentile were Oregon, Arizona, Colorado, Louisiana, Georgia, and Kentucky (Fig. 5). For the environmental article model with publications between 1990–1995, the locations with the TFP_EST in the 90th percentile was the same as the 90th percentile of the other models, but the locations with TFP_EST in the 10th percentile were Oregon, Arizona, Colorado, Kentucky, Alabama, and Georgia (Fig. 6). The granularity was equal to 1.0 for all estimates and was calculated by dividing the number of states in the 10th percentile by the number of states in the 90th percentile, for each model (Altomonte and di Mauro, 2022). The granularity values represent an even distribution of beneficiaries and non-beneficiaries. All residual plots confirmed a near normal distribution.

Lastly, local spatial autocorrelation was calculated for the average TFP_EST values, and LISA clusters were mapped. The LISA clusters represent states that significantly cluster compared to random chance (p-value < 0.05). The most notable comparisons between the all-article model clusters and all-environmental article model clusters are a high production cluster surrounded by high production states in the northeastern U.S., three high production clusters surrounded by low production in California, Tennessee, and Florida, and a low production cluster surrounded by high production in Utah (i.e., Utah is a low EBLL R&D producer nearby California which is a high EBLL R&D producer) (Fig. 7). The most notable contrast is a low production cluster surrounded by low production in Michigan, which is seen in the all-environmental article model and not the all-article model (Fig. 8). Comparing and contrasting the environmental article models by time-period identified a difference with the 1990–1995 time-period compared with the all-environmental article model and the other two time-period environmental article models (Fig. 9). The environmental article model with publications between 1990–1995 includes the same high production clusters minus Massachusetts, Connecticut, New Jersey, and Vermont but also includes Idaho as a high producer surrounded by low producers. Also, New Jersey stands out as a low producer surrounded by low producers. The low producers are assumed to be states on the southern border because New York is a high producer. These southern border states include Maryland, Pennsylvania, and Delaware.

Debates surrounding the R&D knowledge distribution system stem from the perspective framed by the R&D productivity, innovation, and competitiveness in a location or industry. In other words, critics of the system can be biased by the segment of the system they are closest to. For instance, a researcher attempting to publish a scientific article in the Northeastern U.S., a traditionally competitive geographic location, in public health, a traditionally competitive field, will have a much different perspective than a researcher publishing an article in Michigan in the liberal arts field. Furthermore, this is relevant to the time-period of research being published. This is clearly seen in the LISA cluster maps over the selected time periods where, for the topic of environmental EBLL R&D, Michigan transitions from a neutral producer to a low producer, New Jersey transitions from a low producer to a high producer, Massachusetts, Connecticut, and Vermont transition from neutral to high producers, and Idaho transitions from a high producer to a neutral producer. Conversely, the overall body of EBLL R&D (i.e., environment and non-environment subtopics) between 1990–2019, is highly competitive in the Northeast U.S., and neutral in the northern Midwest. These results show a shift in R&D where a significant increase of EBLL R&D publications occurs after the 1990-95 timeframe in Michigan, Idaho, and much of the Northeast U.S. Again, this is likely due to nationally recognized events (i.e., CDC’s reduction of the BLL threshold and the Flint Water Crisis). To this purpose, quantitative analysis of R&D distributions by location and industry, using methods that account for geographical and industry variations, are needed for researchers and research decision makers to understand the evolution of the R&D knowledge distribution system.

Awareness of the geographic distribution of R&D is becoming more important for effective policy making (Altomonte and di Mauro, 2022). In fact, it is known that heterogeneous productivity boosts aggregate competition. This makes intuitive sense because a homogenous distribution of productivity would mean competition is approaching an equilibrium among geographies. Hence, heterogeneity is better for competition. This can be measured by granularity. Other analyses of firm productivity using TFP have measured granularities of 1.92. These higher granularity values suggest that production can be imbalanced by almost 2-fold due to a small proportion of regions or industries that produce most of a product. Imbalances like this spark competitiveness between firms and regions. The regional granularity measurements calculated from this analysis remained at 1.0 for all observed and estimated values. This demonstrates that the regional competition for EBLL R&D by U.S. state is not competitive. Moreso, it is likely that a higher resolution geographic analysis will identify a different distribution, and it is possible that because of the low-resolution aggregation in our analysis, the granularity value is diluted. If higher resolution analysis identifies a diverse distribution, then perceptions on R&D by researchers will vary too. And variations of perspectives will have different influences on how policies are decided.

Additionally, this analysis can provide a revised method for identifying research bias for location dependent research. Research bias can come from data collection, analysis, interpretation, and publication (Simundic, 2013). All these biases can cause a skewed number of articles per region or industry. In fact, Millard et al.’s research on animal pollinator R&D publications throughout the world demonstrates a potential data collection and/or publication bias. However, because the quantities of publications were not controlled by labor or capital it is possible that the distribution of publications, they arrived at, are a representation of the international GDP distribution. This was shown previously where the top five countries that published animal pollinator research were the same countries with the top GDPs for their region. With this knowledge, the map of research location TFP_EST can provide insight on research location bias while accounting for labor and capital in each region. Yet, it is important to note that if this map is to be used in this way it should be interpreted by domain experts because what might appear to be research bias might not be. Examples of how this can occur are a researcher or university having a long-standing study and/or community relationship in a particular area, or a persistent contamination issue in a region.

Finally, there are limitations to this analysis. The first limitation is that this methodology is relevant to location dependent research. It can be applied to publication locations only, but that would exclude analysis of spillover effects. Furthermore, location dependency affects interpretation when data is aggregated. This means that interpretations should only be made at the geographic scale analyzed due to the ecology fallacy (O’Sullivan and Unwin, 2010). An example of this is our speculation that higher resolution data will have higher variances of TFP. We cannot say whether higher or lower variances will occur at a different resolution, and it is only speculation until an analysis at a higher resolution scale is conducted. Next, the manual extraction of subtopics is a subjective process. This was encountered when one researcher felt the subtopic should have been designated as ‘environmental’ for an article while another felt the subtopic should have been designated ‘non-environmental’. To resolve these issues, the entire article would need to be read, and that too can leave room for interpretation. Another limitation is that the TFP_EST relies on generalized estimates from a machine learning algorithm (i.e., random forest) and the features available for this time-period and geography. It would be better if we had access to the full population of research articles by topic for each state but obtaining them all would be infeasible, if not impossible. And lastly, the Cobb-Douglas production function relies on aggregated capital and labor values. The state and federal research and development obligations are used as the capital variable and the estimated number of EBLL journals per state are used as the labor variable. Furthermore, the use of the number of journals that publish research on a specific topic is an assumption of the labor force for the topic of interest. All these limitations must be considered if this analysis is to be reproduced.

To refine the discussion surrounding potential biased viewpoints regarding the R&D knowledge distribution system, and identify research beneficiaries, a quantitative analysis of R&D distributions by location and industry, using methods that account for geographical and industry variations was demonstrated. The purpose of this quantitative analysis is for researchers and research decision makers to understand the overall picture of R&D for a particular field, by geography, time, and subtopic and identify true research beneficiaries. TFP using the combined main effects and spillover effects accounts for geographic and industry variations. The mapped TFP values were used to identify innovation and competitiveness throughout the U.S. by state. The 10th and 90th percentiles of childhood EBLL research and clusters of high and low research regions were visualized and interpreted. Also, granularity was calculated using the quantities of high and low producers at the 10th and 90th percentiles. The Northeast U.S. was consistently the highest EBLL research production cluster surrounded by high EBLL research production for all analyses, and the primary research beneficiary for this topic. California, Tennessee, and Florida were consistently high producing states surrounded by low producing states. The environmental subtopic maps were the same except for Michigan, which was calculated as a low producer surrounded by low producers, and Utah, which was calculated as a low producer surrounded by high producers (i.e., California). Michigan was the primary non-beneficiary for this research. The time-period maps for environmental subtopics showed that the Northeast began producing a significant amount of environmental EBLL research after the CDC’s reduction of the BLL threshold, while Michigan and Idaho reduced their environmental EBLL research production. These values remained the same through the Flint Water Crisis. Granularity remained at 1.0 for all subtopics, environmental subtopics, and all time periods. This indicates a lack of competition for EBLL R&D across the U.S at the state scale, but competition will probably be different at higher resolutions. It is at either higher-resolution, local variations or regional scales that account for biased perceptions that might cause researchers, R&D decision makers, and critics to diverge from the reality of the R&D distribution system. Quantitative analyses like the one presented are needed to promote awareness of the correct geographic distribution of R&D for effective policy making. Additionally, analyses like this can be used to determine geographic research bias during the R&D planning phase. Consequently, a quantitative analysis of R&D distributions by location and industry, using methods that account for geographical and industry variations, should be considered for geographically dependent fields, to inform a stronger decision-making process.

Acknowledgments

Anonymous

Competing Interests

The authors declare there are no competing interests.

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.

Informed Consent

This article does not contain any studies with human participants performed by any of the authors.

Funding Source

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Altbach, P. G., & de Wit, H. (2018). Too Much Academic Research Is Being Published. International Higher Education, (96), 2-3. https://doi.org/10.6017/ihe.2019.96.10767
Altomonte, C., & Di Mauro, F. (2022). The Economics of Firm Productivity: Concepts, Tools and Evidence. Cambridge: Cambridge University Press. doi:10.1017/9781108774277
Blanco, L.R., Gu, J. and Prieger, J.E. (2016), The Impact of Research and Development on Economic Growth and Productivity in the U.S. States. Southern Economic Journal, 82: 914-934. https://doi.org/10.1002/soej.12107
Carayannis, Elias & Grigoroudis, Evangelos. (2014). Linking innovation, productivity, and competitiveness: Implications for policy and practice. The Journal of Technology Transfer. 39. 10.1007/s10961-012-9295-2.
Grokhowsky, Nicholas. (2023). An inferential spatiotemporal approach for knowledge synthesis to identify trends in public health research. Heliyon. (Manuscript submitted for publication)
Gui-Quan Sun, Marko Jusup, Zhen Jin, Yi Wang, Zhen Wang, Pattern transitions in spatial epidemics: Mechanisms and emergent properties, Physics of Life Reviews, Volume 19, 2016, Pages 43-73, ISSN 1571-0645, https://doi.org/10.1016/j.plrev.2016.08.002.
Hagve, Martin. (2020). The money behind academic publishing. Tidsskrift for den Norske Laegeforening, ISSN: 0807-7096, Vol: 140, Issue: 11, Page: 1-5
Hauptman M, Niles JK, Gudin J, Kaufman HW. Individual- and Community-Level Factors Associated With Detectable and Elevated Blood Lead Levels in US Children: Results From a National Clinical Laboratory. JAMA Pediatr. 2021 Dec 1;175(12):1252-1260. doi: 10.1001/jamapediatrics.2021.3518. PMID: 34570188; PMCID: PMC8477303.
Hemisphere, Dept., International Monetary Fund. Western. United States : Selected Issues, International Monetary Fund, 2014. ProQuest Ebook Central, https://ebookcentral.proquest.com/lib/ncsu/detail.action?docID=1779677.
Huang CK, Neylon C, Hosking R, Montgomery L, Wilson KS, Ozaygen A, Brookes-Kenworthy C. Evaluating the impact of open access policies on research institutions. Elife. 2020 Sep 14;9:e57067. doi: 10.7554/eLife.57067. PMID: 32924933; PMCID: PMC7536542.
Johnson R, Watkinson A, & Mabe, M. (2018). The STM Report: An overview of scientific and scholarly journal publishing.
Marks R and Pauline R. Native Pollinators. Fish and Wildlife Habitat Management Leaflet. 2005 May; Number 34, https://www.nrcs.usda.gov/Internet/FSE_DOCUMENTS/stelprdb1048334.pdf, [accessed 30 August 2022].
McGuigan, G. S. (2008). The business of academic publishing: A strategic analysis of the academic journal publishing industry and its impact on the future of scholarly publishing. Electronic Journal of Academic and Special Librarianship, 9(3). http://southernlibrarianship.icaap.org/content/v09n03/mcguigan_g01.html
Millard, Joseph & Freeman, Robin & Newbold, Tim. (2019). Text‐analysis reveals taxonomic and geographic disparities in animal pollination literature. Ecography. 43. 10.1111/ecog.04532.
O’Sullivan, David, and David J. Unwin. Geographic Information Analysis. John Wiley & Sons, Inc., 2010
Simundić AM. Bias in research. Biochem Med (Zagreb). 2013;23(1):12-5. doi: 10.11613/bm.2013.003. PMID: 23457761; PMCID: PMC3900086.
Teye SO, Yanosky JD, Cuffee Y, Weng X, Luquis R, Farace E, Wang L. Exploring persistent racial/ethnic disparities in lead exposure among American children aged 1-5 years: results from NHANES 1999-2016. Int Arch Occup Environ Health. 2021 May;94(4):723-730. doi: 10.1007/s00420-020-01616-4. Epub 2021 Jan 4. PMID: 33394180.
United Nations. “About Us.” https://www.un.org/en/about-us#:~:text=The%20United%20Nations%20is%20an,contained%20in%20its%20founding%20Charter. [accessed 30 August 2022].
US Center for Disease Control. “Childhood Lead Poisoning Prevention.” 1990s-200s CLPPP Timeline | Lead | CDC, U.S. Center for Disease Control, https://www.cdc.gov/nceh/lead/about/2000s.html, [accessed 7 November 2022].
Van Noorden, R. Open access: The true cost of science publishing. Nature 495, 426–429 (2013). https://doi.org/10.1038/495426a
Veselica, R. (2019). Linking Innovation And National Competitiveness. Varazdin: Varazdin Development and Entrepreneurship Agency (VADEA). Retrieved from https://proxying.lib.ncsu.edu/index.php/login?url=https://www.proquest.com/conference-papers-proceedings/linking-innovation-national-competitiveness/docview/2188520733/se-2
Vinod, H. D. (2008). Hands-on intermediate econometrics using R: Templates for extending dozens of practical examples. Hackensack, NJ: World Scientific.
Westfall, Peter H. And Andrea L. Arias. Understanding Regression Analysis: A Conditional Distribution Approach. CRC Press, 2020
World Economic Forum. “Chart of the Day: This is how the $88 trillion global economy is spread around the world ". October 13, 2020. https://www.weforum.org/agenda/2020/10/global-economy-gdp-growth-covid-19/. [accessed 23 March 2023].
Xue J, Zartarian V, Tornero-Velez R, Stanek LW, Poulakos A, Walts A, Triantafillou K, Suero M, Grokhowsky N. A Generalizable Evaluated Approach, Applying Advanced Geospatial Statistical Methods, to Identify High Lead Exposure Locations at Census Tract Scale: Michigan Case Study. Environ Health Perspect. 2022 Jul;130(7):77004. doi: 10.1289/EHP9705. Epub 2022 Jul 27. PMID: 35894594; PMCID: PMC9327739.

(Not answered)

EfficienciesSupplement.docx
EfficienciesDataRepo.docx
Data Repository
statefeaturesfinal.csv
Feature Data Set as CSV file for anonymization
efficienciesData.csv
Primary Data Set as CSV file for anonymization

Download PDF

Version 1

posted

You are reading this latest preprint version

Estimating research beneficiaries to reduce bias in public health research using direct and indirect production efficiencies

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1