Boundary Negotiating Artifacts to Envision the Desired Research Data Infrastructure toward Reproducibility in Data-Intensive Science

doi:10.21203/rs.3.rs-3193818/v1

Download PDF

Research Article

Boundary Negotiating Artifacts to Envision the Desired Research Data Infrastructure toward Reproducibility in Data-Intensive Science

https://doi.org/10.21203/rs.3.rs-3193818/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

New data-intensive research paradigms and surging volumes of research data have amplified the need for more effective systems of data curation, management, and sharing. This methodological paper focuses on the development and implementation of a workshop protocol aimed at enabling researchers in various sub-disciplines of data-intensive fields to achieve consensus about data-sharing and reproducibility practices. As such, it addresses a key gap in our understanding of how researchers from those sub-disciplines negotiate and reach consensus on disciplinary practices, which is crucially important to the design of research-data infrastructure and data-management support services. Specifically, we first convened a design-thinking workshop, attended by 17 principal investigators from the field of data-intensive health science, which simulated the identification and solution of real-life research problems in a multidisciplinary context. A key plank of our methodology is its use of boundary-negotiating artifacts as a conceptual model to delineate and analyze the artifacts (e.g., Post-its and other notes) that were created and used during the workshop. This approach provided insights into the development and evolution of common practices in the participants’ workplaces. Then, to validate and extend our methodology, we applied the workshop protocol to two additional case studies, respectively involving 30 academic librarians and 13 criminologists. The results confirm the potential applicability and adaptability of our approach to even more data-intensive disciplines, and provide insights that should be valuable to a range of stakeholders including but not limited to national funders, memory institutions, and research establishments.

Boundary Negotiating Artifacts

Research Data Infrastructure

Reproducibility

The data-intensive research paradigm and the closely connected problem of the ‘data deluge’ have profoundly transformed scientific infrastructure since the former emerged two decades ago (Hey & Trefethen, 2003). In response, various proposals for enhancing data curation and management have been made; and recently, many of these have been consolidated into the concept of research-data infrastructure (RDI; Vogl et al., 2019).

This has inspired a variety of user-centered approaches to designing information systems for data sharing and reproducibility, as well as numerous meta-reviews, surveys, and field studies of researchers’ data-sharing and reproducibility practices (e.g., Zuiderwijk et al., 2020). Some previous research has used a profiling method focused on individual researchers’ data-sharing and reproducibility practices (e.g., Jeng & Lyon, 2016). Not unexpectedly, however, this somewhat atomistic approach has left numerous gaps in our understanding of how researchers from various sub-disciplines negotiate and reach consensus on disciplinary practices; and the diversity in the types of data used by the participants in such research renders its outcomes less than optimally useful for libraries and data centers hoping to provide comprehensive and efficient support services for research-data management.

Within any sub-discipline, a process of consensus-building can lead to better decisions, e.g., settle complex multi-party disputes, prioritize the most-needed facilities, and drive more effective RDI implementation and standards. It is therefore reasonable to assume that fostering such consensus-building processes could revolutionize the design of mechanisms and systems for assisting cross-disciplinary collaborative research, and also help relevant entities such as national funding bodies, memory institutions, and research establishments to effectively promote transparent and rigorous research in all areas. To that end, this paper seeks answers to the following research questions:

RQ1: How can a workshop protocol best be designed and implemented to achieve consensus about data-sharing and reproducibility practices among researchers in a particular data-intensive sub-discipline?

RQ2: How applicable is that optimal workshop protocol to other data-intensive sub-disciplines or stakeholders?

Our methodology for answering RQ1 was inspired by a problem-solving technique, the design-thinking workshop, which was created by UK Council (n.d.). We initially used such a workshop to capture data-intensive health science (DIHS) researchers’ day-to-day research-data practices, as well as how they identified and solved problems via negotiation with members of other sub-disciplines. At the time the workshop was held, all 17 participants were principal investigators (PIs) in DIHS, including biomedicine and health data science, and were tasked with reaching a consensus on suitable solutions to specific real problems in their data lifecycles. To simulate cross-lab communication, the participants were also subdivided into groups, each representing one DIHS sub-field.

To further scrutinize how this initial consensus workshop brought together the collective efforts of the research participants in terms of data-sharing and reproducibility practices, we examined our own data through the lens of the boundary-negotiating artifact (BNA) model (Lee, 2007), which is suitable for capturing work/workplace practices that have not yet been developed or are still developing. Through such examination, people can better identify the state of current work: for instance, whether it is pursued individualistically vs. is already subject to common standards that will be difficult to change.

To address RQ2, regarding the potentially wider applicability of our workshop protocol, we conducted two further workshops, one with academic librarians and the other with criminologists.

2.1 Techniques Leading a Consensus: Double Diamond Model

Double Diamond Model. To structure this workshop’s activities, we adopted the Double Diamond model (Design Council, 2005), one of several well-known methods for facilitating design thinking. Initially synthesized from the design processes used by leading companies, it represents such processes visually, to help designers and non-designers process information and create solutions to complex problems, and has been used in various fields, including business, public service, healthcare, and education. The purpose of the first of its four stages, Discover, is to identify actual problems, contextualize them, and thus highlight related opportunities. After enough information for this contextualization process has been gathered and thoroughly discussed, the team can filter it and set a context for development, in the Define stage. In this second stage, the collection of various perspectives that marked the previous stage (Discover) gives way to aligning everyone’s thoughts and building common ground. Then, in the Develop stage, based on the shared context that was built in the first diamond, the team starts to design. In this stage, multi-disciplinary working is critical, because opinions from multiple fields inspire more ideas. The final quadrant of the model is the Deliver stage, in which the solution is tested.

An artifact refers to the interface, usually made by humans, between an internal operating environment and an external usage context. When the internal and external environments are well-bridged, the desired goals or functions can be achieved through the artifact (Simon, 1969). In other words, an artifact is an object that has been deliberately designed to achieve a specific purpose, and it can do so provided that certain design conditions are met. Lee (2007) proposed the use of BNAs to help improve communication among different communities. In that process, artifacts facilitate communication by providing a common point of reference that these disparate groups can all use.

2.2 Boundary Objects and Boundary-negotiating Artifacts

In information science, boundary objects (BOs) are objects used to facilitate communication and coordination among different communities. This concept’s originators, Star and Griesemer (1989), never propounded a strict definition of it, on the grounds that the BO nature of any given thing depends on how it operates in context. Nevertheless, prior literature is generally in agreement that BOs help to align the common goals and activities of different entities (citations needed). Examples of BOs in information science include metaphors, depositories, and standards, as well as their related physical entities such as datasets, music scores, and catalogs, which can be used in information-system design and to foster communication and coordination among researchers, engineers and other stakeholders (Huvia et al. 2017; Vuillemot et al., 2021).

BNAs are derived from BOs. Lee (2007) coined the term BNA for two reasons. First, a BO is a single point of contact during collaboration, rather than a longer developmental process from the two parties “not knowing each other” to “mutual agreement” (pp. 315–316). Second, BOs are closely related to structure and standardization, but often, people start to interact and collaborate before a standard is in place. Accordingly, Lee proposed BNAs not to replace BOs, but to ensure that “non-standard” artifacts would also be valued and given due weight during discussions. Lee divided BNAs into five types: 1) self-explanation artifacts, 2) inclusion artifacts, 3) compilation artifacts, 4) structuring artifacts, and 5) borrowed artifacts. Each is described in detail below.

Self-explanation artifacts are created through a thinking-by-doing process. People usually make this kind of BNA for private use based on their personal experience and expertise, for purposes that include learning, recording, organizing, remembering, and reflecting (Hara & Fichman, 2014; Lee, 2007). In specific cases, such as a researcher’s notebook, this type of BNA can be created to facilitate collaboration. Yet, in the absence of external explanations, it can be challenging to understand these artifacts.

Inclusion artifacts are concepts used to symbolize and develop new ideas. They are usually created from self-explanation artifacts through a negotiation process, which Lee (2007) called “gatekeeping” (p.321). A team can use inclusion artifacts to exhibit their ideas in ways that are, in comparison to self-explanation artifacts, more mutually comprehensible.

Compilation artifacts bring two or more parties into alignment by providing a common understanding of objects. By the same token, the only goal of such alignment is to exchange or coordinate understanding (Chung et al., 2016). In the alignment process, BNAs of this type play a vital role in the evolution of the various elements of the final product.

Structuring artifacts are created to structure the final product. This kind of BNA directs and coordinates collaboration, thus providing ordering principles for a range of activities. Examples include the templates provided for structuring journal articles.

Lastly, borrowed artifacts are imported from other communities and “used in unanticipated ways” (Lee, 2007, p. 331). The communities that created these BNAs are under no obligation to make them meaningful to those who have borrowed them. Hence, the borrowers can use them creatively to further their own projects’ goals.

The BNA model emphasizes the communicative function of the artifact itself. The audience for its information can complete communication and coordination tasks through it, regardless of whether they are partially aware or completely unaware of such information. Importantly, Lee (2007) argued that communication tasks can be achieved between communities with the assistance of a third party, while Star (1989) deemed consensus on things to be the key to building a communication bridge. In other words, BOs are mainly used as key tokens for aligning groups’ respective goals and activities by providing a common understanding of the information being shared, whereas BNAs are mainly used to create a process for the members of heterogeneous groups to aggregate their thoughts and achieve common understanding.

Previous research in the fields of computer-supported cooperative work (CSCW) and human-computer interaction (HCI) has demonstrated the potential of using personal, e.g., medical data to support collaborations intended to improve individuals’ well-being. Chung et al. (2016), for instance, used the BNA model as their framework for interpreting patient-generated data in the context of patient-provider collaboration. Specifically, they found that in the context of patients tracking their own health data, self-explanation artifacts were the least universal, and patients also tended to collect too much data and/or irrelevant data. Such work provides important insights into the use of BNAs to better understand how disparate groups find common ground and achieve collaboration.

More recently, Kolovson et al. (2021) employed the BNA model to guide their analysis of qualitative data on the power asymmetries that occur when personal data is collected and used by collegiate sports teams. Specifically, they mapped the model’s five-part typology of artifacts onto the process of personal-data reporting by National Collegiate Athletics Association (NCAA) athletes. Initially, they found that data remained in the form of a self-explanation artifact until it was collected by a staff member, at which point it was transformed into a borrowing artifact. But upon further review, the data took on the form of inclusion artifacts. That study’s findings emphasize the importance of considering power dynamics and ethical issues when personal data are collected and utilized.

We chose DIHS as a case for three reasons. First, it usually involves human or biological data, which are sensitive and complex. Second, as compared to most fields, it involves more stakeholders, such as patients and external suppliers, which may complicate RDI infrastructure; and third, it is known to require significant sharing of historical data and reproducibility practices. Given their intense requirements for collaboration and cooperation, both concurrently and sequentially, research teams in DIHS often rely on precise divisions of labor. Thus, team members need to make frequent decisions about whether and how data and related artifacts (e.g., protocols, data processing) should be stored and managed.

3.1 Protocol Construction

This section describes our approach to designing a protocol that can capture and align DIHS researchers’ day-to-day research activities in a manner that serves the goal of reproducible science. To answer our research questions, we ran a design-thinking workshop based on the Double Diamond model.

Figure 1, which utilizes the Double Diamond model as a visual guide, connects the four stages of our workshop design with the 19 artifacts that were generated during the workshop itself. In other words, this schematic provides a holistic view of how the workshop unfolded, and how each artifact contributed to our understanding of DIHS researchers’ day-to-day activities.

3.1.1 Stage I: Discover

In this initial stage of the workshop, we focused on grounding participants in our research context and facilitating initial discussions about their data-intensive health science experiences and needs. The sub-stages of this stage, the participants’ workshop activities, the artifacts and the artifacts’ functions are set forth in Table 1.

Table 1

Design-thinking workshop protocol: Stage I
#	Sub-stage	Activities	Artifacts and their functions
Stage I: Discover – Identifying current data-sharing and reuse (DSR) needs
--	--	--	Protocol for the workshop (Artifact #01).
I-1	Reading a DSR case	As a warm-up: participants read a mock case prepared in advance by the research team.	A4 sheet with DSR mock cases (Artifact #02). The mock cases were intended to help the participants reach shared understandings of DSR and reflect on their own DSR experiences, as a means of informing subsequent sessions.
I-2	Sharing DSR experiences	Participants discussed the case with their group and shared their own DSR experiences. Facilitators moderated the discussion using the Objective, Reflective, Interpretive, Decisional framework.	A4 blank sheet for note taking (Artifact #03). The notes that participants took while reading cases helped them to concentrate and to better organize key concepts for sharing.
I-3	Articulating DSR needs	Participants wrote down DSR needs linked to their experiences on Post-its.	Yellow Post-its with DSR needs (Artifact #04). Each Post-it should contain only one DSR need.
I-3	Articulating DSR needs	Those who had difficulty expressing their needs were given a sentence template to follow.	A sentence, as a template, for helping shape a DSR need (Artifact #05): “When... [a scenario], I can/can’t... [a DSR need]”.

As a warm-up activity, each “lab group” of participants read the same sample case prepared by the research team in advance (I-1), as shown in Fig. 2. Profiles of users whose work involved biomedical data were composed based on real DIHS researchers’ data activities and sharing attitudes, as drawn from the Data Curation Profile (DCP) Directory.[1] The participants then discussed the sample case with their fellow group members, and shared their own data-sharing and reuse (DSR) experiences (I-2). Then, based on those experiences, they wrote DSR needs on yellow Post-its (I-3). This part of the design followed the Objective, Reflective, Interpretive, Decisional method (ORID; Stanfield, 2000) of gathering group wisdom via conversation. Participants in ORID try to assess others’ experience objectively, then reflect on their own experience, negotiate with the other group members, and finally reach a consensus.

3.1.2 Stage II: Define

The goal of the second quarter of the Double Diamond model, Define, is to generate a consensus about which problem scenario – out of all those the group members have generated – is the most important. It is then selected as their group’s main scenario. Then, using the Jiro Kawakita (KJ) Method (Scupin, 1997), participants explain their needs or specific terms to their fellow group members, and cluster the yellow Post-its with similar DSR needs together. The clusters are given names on another color of Post-its, e.g., orange ones as shown in Fig. 3. Table 2 provides further details about this stage’s activities and artifacts.

Figure 3. Participants explain their data needs to their group via an KJ method (stage II-1, Artifacts #05 and #06)

Table 2

Design-thinking workshop protocol: Stage II
#	Sub-stage	Activities	Artifacts and their functions
Stage II: Define – Building common ground for a data-sharing and reuse (DSR) scenario
II-1	Grouping DSR needs	Through the Jiro Kawakita’s “KJ” method, participants explained their needs and specific terms to the other group members, and clustered the Post-its from the previous stage based on similarities in the DSR needs they expressed. They were allowed to create multiple levels within any category, though in the event, this did not actually occur.	Orange Post-its with clustering labels, each representing a set of similar DSR needs (Artifact #06).
II-2	Determining the main scenario	The participants selected one DSR need as their group’s main scenario, for which they would design and apply solutions in subsequent sessions. The selection process helped participants identify the DSR scenario they found most representative.	The Post-it receiving the most votes from the participants (Artifact #07).
II-3	Contextualizing the scenario	To enable all participants to gain a better understanding of the selected scenario, the person who wrote it down on the Post-it was interviewed by the others using interview scripts provided by the research team.	Badge in the shape of a human body (Artifact #08), used for ongoing identification of the person who created Artifact #07.
			Interview guide following the ‘who, where, why, when, what, and how’ structure (Artifact #09), intended to facilitate participants’ learning more about the interviewee’s data, her/his willingness and motivation to share data, who it is shared with, and so on.
			A blank sheet for note-taking (Artifact #10).
II-4	Identifying the jobs to be done, the pain, and the gain (JPGs) in the scenario	Participants discussed how the various elements of the scenario fit into the JPG structure, then wrote them down on Post-its in various new colors.	A sheet displaying the JPG structure (Artifact #11) and Post-its with participants’ Jobs (Artifact #12), Pains (Artifact #13), and Gains (Artifact #14). These artifacts helped participants to remain focused on the interviewee’s DSR needs in subsequent sessions.

After exchanging their DSR experiences and relating them to the situation from Stage II-1, the members of each group selected a specific orange sticky note, containing a problem for which they were expected to develop further solutions in the following stages. Additionally, one member with specific data needs was elected as a representative and interviewed by the others using a pre-determined 5W1H (who, where, why, when, what, and how)-based interview script. By doing this, the rest of the participants were able to deepen their understanding of 1) the problem context (II-2) and 2) the representative’s data needs, willingness and motivation to share data and ensure reproducibility, whom it will be shared with, and so forth (II-3). Further details including the interview questions can be found in Table 3.

Table 3

Interview script
	Main topic	Questions
What	Data type	• Please provide me with a brief description of the dataset. • What particular tools/instruments are used to record and yield these data?
When	Sharing occasions	• In which stage of the data lifecycle would you share your data with others? • What types of data that you produce are most frequently shared?
Why	Sharing willingness	• What drives you to share your data with others? To what extent do you share your data? • Who can have access to your data? And what approaches do you consider appropriate for obtaining your data?
How	Efforts at data curation	• How do you prepare for data sharing? What actions are required?
Where	Access	• Where or in which repository do you have your data shared?
Who	Stakeholders	• Who do you share your data with? • Who else would be interested in your data?

Figure 4 shows Artifact #08, a badge used for easy visual identification of the individual who had provided the scenario being discussed, as a means of anchoring such discussion that person. This artifact was added during the second round of the workshop because, during the first round, the absence of any such identifier led to unanchored discussions, with participants often seeming to forget the designated interviewee. Artifact #09, shown in Fig. 5, is an interview guide with a 5W1H structure. This tool was designed to aid those participants who were interviewing, by structuring each interview and recording interviewees’ responses for further reference and analysis.

Figure 6 illustrates an ideation structure aimed at allowing its users to clearly state their thoughts (Artifact #11). We also asked participants to consider the three elements of the JPG model, as prompted by the questions listed in Table 4.

Figure 6. The Job-Pain-Gain framework being used to deepen a data scenario (stage II-4, Artifacts #10-#12)

Table 4

Job-Pain-Gain framework prompts
Category	Prompt
Jobs	What do you need to do to share your research data?
Pains	What difficulties are you likely to encounter in the process of making your data ready for sharing? What difficulties are you likely to encounter in the process of sharing your data?
Gains	What will you gain from the data being shared?

3.1.3 Stage III: Develop

As shown in Table 5, Stage III enabled the participants to develop solutions to the defined problem or challenge. They discussed potential solutions and appended a solution card next to a “pain” Post-it if they considered that solution helpful for dealing with that type of pain. Twelve pre-filled solution cards (Artifact #15) (e.g., “More labor is needed”) and an unlimited number of empty ones (Artifact #16), as well as a “magic” card (Artifact #17), were integral components of this stage. The pre-filled cards’ solutions consisted of ones identified in previous studies (Corti et al., 2014; Lee & Stvilia, 2017; Cox & Verbaan, 2018), and served as initial stimuli to thinking and discussion around established solutions.

Table 5

Design-thinking workshop protocol: Stage III
Stage III: Develop – Brainstorming solutions to the data-sharing and reuse scenario
III	Seeking solutions	Participants discussed potential solutions to the problems in the scenario and placed a solution card relevant to each “pain” Post-it next to it.	Pre-filled solution cards (Artifact #15) and empty solution cards (Artifact #16).
III	Seeking solutions		A “magic” card (Artifact #17), provided so that the participants could fill in their own solution that, unlike those on the empty cards, was completely unconstrained by real-world/feasible considerations.

However, we also aimed to foster creativity and innovation, and thus potentially open the door to transformative solutions. To that end, we provided empty cards on which the participants could write down any new, innovative solutions they conceived of during the workshop, beyond those already listed on the pre-filled cards. Additionally, a “magic” card was included to encourage the participants to imagine solutions without any constraints, even if they seemed far-fetched or impossible. The complete set of pre-filled solution cards’ content is included as Appendix 1.

3.1.4 Stage IV: Deliver

In the final stage, Deliver, the participants took six scenes sketched during the preceding stages of the workshop and joined them together as a storyboard illustrating the scenario, its associated DSR needs, and their solutions. Then, they gave a six-minute presentation on this storyboard to the other groups (Fig. 7). Table 7 provides further details about this stage’s activities and artifacts.

Table 6

Design-thinking workshop protocol: Stage IV
Stage IV: Deliver – Presenting the solutions
IV	Delivering solutions	Participants applied their chosen solutions to the scenario, in the form of a storyboard with six scenes, and gave a six-minute presentation on it to the other groups.	A story frame with a participant’s drawing (Artifact #18).
IV	Delivering solutions		Storyboard (Artifact #19) comprising six previously collected sketches.

The aim of the storyboard (Artifact #19) was to succinctly convey a DSR situation and its context narratively, thus rendering it clearer and more understandable. The purpose of the six-minute presentation about it was to facilitate the sharing of insights and the sparking of new ideas among the participants, leading ideally to a collective understanding of DSR challenges and solutions.

Figure 7. Participants’ storyboard sketches (stage IV, Artifacts #18 [the six sketches individually] and Artifacts #19 [the six sketches collectively])

3.2 Data Collection and Processing

We adopted purposeful sampling and convenience sampling as our participant-recruitment strategies. Two sessions of the data-intensive health science workshop were held, one in September and the other in October 2019. The duration of each session was about 3 hours. The first session was broken down into three “lab” groups, and the second, into two. Each such group consisted of at least three, but not more than four participants, arranged according to the similarity of their research interests – namely, biomedicine, biomedical information, epidemiology, life science, and disease research – as shown in Fig. 6. All sessions were audio-recorded and photographed without identifiable faces. Data collection yielded a total of 6 hours and 19 minutes of recordings for transcription (79,760 Chinese words, approximately equivalent to 61,354 English words), 216 post-its, and 623 photos.

At this stage, our primary focus is not to delve deeply into the specific results or findings derived from the first two workshop sessions, but rather to highlight the development and application of our methodology. The positioning of this paper is methodological, with a particular emphasis on the genesis of our approach and the way we interwove each artifact with BNA theory. This structure allows us to explore how we designed the workshop protocol, how we collected and processed the data, and how we aligned those data with the BNA framework, thus providing a comprehensive overview of our methods. The detailed analysis of the data we gathered and our subsequent findings will be the subject of our next paper.

4.1 Participant overview

Each of the nine female and eight male participants in the data-intensive health science workshop worked at a different institution. The top two types of departments they were affiliated with were biomedical engineering (n = 3) and biomedical sciences (n = 3), followed by computer science (n = 2), information management (n = 2), pharmacy/pharmacology (n = 2), and statistics/data science (n = 2). Among the participants’ Ph.D. degrees, five were in engineering-related fields, five in biomedical sciences, and three in pharmacy. Just over half the participants (n = 9) were assistant professors, six were associate professors, and two were full professors. Most (n = 14) were aged between 31 and 40, and the rest between 41 and 50. Their work experience in their respective fields ranged from 1 to 15 years, with more than three-quarters having 5 years’ experience or less. Each of the participants was given an alias for anonymization purposes, as shown in Appendix 2.

4.2 Artifacts as Boundary-negotiating Artifacts

Table 7 presents a variety of artifacts generated during the workshops, classified by their BNA types. These artifacts were integral to the workshop process, facilitating structured discussions and ideation.

We discovered that during the Divergence phase, the practice of generating and gathering diverse ideas significantly fostered individual inspiration and enhanced mutual understanding of the group members’ perspectives as they grappled with the unfamiliarity of the discussion topic. Their lack of a shared language or context necessitated their heavy use of borrowed and self-explanation artifacts, which helped them to recall, organize, and present their ideas and to form a personal understanding of the topic. This personal understanding, in turn, set the stage for the development of a shared understanding among the team members. For example, a data-sharing case, provided as a borrowed artifact, synchronized the team’s understanding of terms. In other words, in this stage, the team was striving to reach a consensus without a sufficient shared understanding, and borrowed artifacts helped bridge this gap. And in the process of reading the case, note papers were provided to each person so that they could create their own self-explanation artifacts.

By the time the first workshop progressed to the second diamond, a basic understanding and shared context had been established. The Divergence phase in this second diamond was marked by an increased use of compilation artifacts, which allowed the participants to expand their imaginative space and propose solutions, capitalizing on the shared understanding that had been built. The team’s challenge then shifted to articulating its opinion clearly. Inclusion artifacts played a central role in helping the members crystallize that consensus.

On the other hand, the main purpose of the final Convergence stage was to reach a consensus, making it a suitable venue for the use of inclusion artifacts to promote team discussions and concretize the consensus. More specifically, we designed a grouping phase in which everyone categorized the experiences shared by every member. In that phase, those experiences – written on Post-its – were spread out in front of every team member, and they were asked to explain one another’s experiences as fully as possible, as a means of arriving at a common view. In that process, the Post-its with others’ experiences on them were borrowed artifacts, and encouraged the participants to elaborate on their own experience using terms mentioned by others. When sharing their own experiences, despite others not necessarily being familiar with their disciplines, the terms or scenarios that the participants borrowed from the case and applied helped to explain what happened in their own disciplines. And the cards with possible solutions (Artifacts #15, #16, and #17) helped participants elaborate on their experience more thoroughly.

Table 7

How workshop artifacts acted as boundary-negotiating artifacts (BNAs)
	Artifact type	Artifact name	Artifact description	#	Unit	BNA type
--	Artifact #01	Protocol for the workshop	An agenda that allowed the curators to host the workshop and ensure that each participant’s data-sharing experience was presented.	1	counts	Structuring
I-1	Artifact #02	Data sharing and reuse (DSR) mock-cases and personas	Fictitious but realistic cases that aimed at helping the participants reflect their experience.	17	counts	Borrowed
I-2	Artifact #03	Blank A4 sheets	The notes that participants made, e.g., to help themselves concentrate.	17	counts	Self-explanation
I-3	Artifact #04	Yellow Post-its	The notes on which participants write down their DSR needs.	56	counts	Self-explanation
I-3	Artifact #05	Sentence template	For helping shape a given DSR need.	5	counts	Structuring
II-1	Artifact #06	Orange Post-its	To label the categories derived from the participants’ clustering of DSR needs.	22	counts	Compilation
II-2	Artifact #07	The winning Post-it from among the set of Artifact #06	The most-voted-for Post-it was used to frame the workshop’s central scenario.	5	counts	Inclusion
II-3	Artifact #08	A badge with a shape of a human body	A badge put on the interviewee to ensure the discussion stayed focused on that person.	3	counts	Inclusion/ Borrowed
II-3	Artifact #09	Interview guides	To help the participants to ask and answer questions about one another’s experience.	3	sets	Structuring
II-3	Artifact #10	Blank A4 sheets	For interview note-taking.	5	counts	Self-explanation
II-4	Artifact #11	A sheet setting forth the Jobs-Pains-Gains structure	An ideation structure that helped the participants avoid mixing multiple concepts in the same statement.	5	counts	Structuring
II-4	Artifact #12	Post-its with participants’ Jobs	For the participants to clearly state what tasks they performed when sharing data.	36	counts	Self-explanation
II-4	Artifact #13	Post-its with participants’ Pains	For the participants to clearly state their key challenges during data-related activities.	30	counts	Self-explanation
II-4	Artifact #14	Post-its with participants’ Gains	For the participants to clearly state what positives they achieved through data-related activities.	30	counts	Self-explanation
III	Artifact#15	Pre-filled solution cards	Solutions to DSR problems identified in prior literature.	60	cards	Compilation (from solution cards)
III	Artifact #16	Empty cards	An empty card that is provided so the participants can fill in their own solutions	60	cards	Compilation
III	Artifact #17	Magic cards		12	counts	Compilation
IV	Artifact #18	Sketches	A sheet with a participant’s drawing.	25	counts	Inclusion
IV	Artifact #19	Storyboards	Six of Artifact #18 joined together to form a narrative.	5	counts	Inclusion

Our findings from the initial workshop underscored novel user scenarios that highlight the inherent challenges of data-intensive research environments and how practitioners are trying to overcome them. Group 05’s consensus illuminated the importance of meticulous data documentation throughout the research process. In their scenario, the PI’s team had chosen a particular research path, path A. However, upon encountering challenges, the team decided to backtrack to the point of divergence to explore a different path. Unfortunately, due to inadequate documentation of data records, they struggled to accurately identify the initial fork in their research path or to verify the most accurate dataset for a specific date. This underscores the pivotal role of comprehensive data management in maintaining the continuity and potential success of the research process.

Group 02, meanwhile, discussed the inherent difficulties of fostering research reproducibility and systematic data recording within a highly competitive research culture. Participant 09, a molecular biologist, revealed that while the value of research reproducibility and methodical documentation of key steps and decisions was widely acknowledged, there was a prevailing concern about intellectual theft; and such apprehension made people reluctant to share data or early research outputs before formal publication. This revelation underscores the delicate balance required between, on the one hand, promoting transparency and collaboration, and on the other, safeguarding intellectual property, within a competitive academic milieu.

The robust insights from the workshops contribute to the development and refinement of the lifecycle-transparency concept proposed by Jeng et al. (2020), which emphasizes capturing the syntactical contexts of artifacts and stakeholders over the entire lifecycle of a research effort. This includes critical data elements such as timestamps, agreements, or dependency requirements for each research phase. The acquisition of patient consent and institutional review board approvals before conducting a clinical trial are examples of such contexts.

In response to needs such as Group 05’s, lifecycle transparency advocates the keeping of an all-inclusive record of research activities, which can mitigate the risk of losing track of the research pathway. It provides a systematic means of recording data on Blockchain, enabling researchers to return to any point in the research process and accurately identify their project’s status at that time.

Group 02’s workshop experience likewise highlighted how decentralized framework proposed by Jeng et al. (2020) could offer a potential solution. By providing an immutable and verifiable record of the research process on Blockchain, this framework enhances researchers’ confidence about sharing early research products without fear of their ideas being stolen, by ensuring that the origin of ideas and data can be traced.

A Blockchain-based approach addresses the limitations of existing open-science tools that are often dedicated to certain research phases or disciplines, and as such, are insufficient to support comprehensive lifecycle transparency. The adoption of the lifecycle-transparency concept and a decentralized framework, therefore, would therefore directly respond to the data needs and challenges expressed by the present study’s 17 bio-medicine researchers, and potentially revolutionize the way they and their colleagues manage and share their data.

5.1 Additional case study 1: academic librarians

Our first case study involved a modified version of the Double Diamond workshop model, and was held on August 2020. The 30 participants were divided into eight groups based on their respective professions and areas of expertise, each of which worked collaboratively throughout the workshop. There were two groups dedicated to Bio-Medical and Medical fields, as well as two groups focused on fundamental science and engineering. Each of these groups consisted of four participants. Additionally, there were three groups composed of members from the fields of Finance, Law, Humanities, and Social Sciences, consisting of four participants as well. Finally, there was a group comprising two members from the information center. This division was to ensure that each group shared a common stand of knowledge relevant to research data services in a library setting. As in the original workshop described above, this one was divided into four stages: Discover, Define, Develop, and Deliver. However, its focus was tailored to suit its context, research data services (RDSs) in a library setting.

Each “library” group was tasked with identifying and exploring potential data-related services that an academic library could feasibly implement. The goal was not to create a comprehensive plan, but rather to stimulate creative thinking and discussion about such possibilities. The deliverable for each group was a list of potential RDS that a library could offer. This case study demonstrated the versatility of the Double Diamond workshop model, by showing its applicability beyond our original case and its potential to foster creative thinking and problem-solving in diverse contexts.

5.2 Additional case study 2: criminologists

Our second case study[2], which incorporated the protocol established, entailed a further adaptation and extension of the Double Diamond model in May 2021. This study engaged two small focus groups: the first group comprised two female and three male researchers from a criminology research institution, while the second consisted of three female faculty members from the criminology department of a research university. The focus groups' agenda did not adhere strictly to the model; instead, it was customized to suit the context, concentrating specifically on the usage and dissemination of research data by criminology researchers.

During the focus-group interviews, special emphasis was placed on using the JPG model to explore the participants’ DSR experiences, concerns, and willingness. Additionally, the researchers’ expectations and preferences were collected as a means of helping them collectively shape their ideal repository for criminology research data.

This case study not only contributed further to our understanding of data-sensitive disciplines, but also demonstrated the feasibility of adopting a more comprehensive approach to research-data management and sharing in additional disciplinary contexts with unique practices.

The deployment of diverse artifacts throughout the workshops was instrumental in the design process, which played a crucial role in both the original and subsequent workshops. We utilized the boundary-negotiating artifact (BNA) model to analyze the Post-it notes and other artifacts produced during these sessions, providing pivotal insights into the development and evolution of common practices in the participants' workplaces.

In particular, self-explanation artifacts created individually were pivotal, often highlighting potential barriers in understanding among the participants. Nevertheless, they aided participants in reflecting on their practices, expediting the recall, organization, and presentation of their ideas. As these thoughts were articulated through the artifacts, they facilitated a collaborative workshop environment.

Compilation artifacts, on the other hand, played a decisive role in both the divergence and convergence phases of the workshops. These artifacts enabled participants to broaden their imaginative space and propose solutions informed by collective comprehension. This process culminated in a consensus, encapsulating discussions with clear conclusions and solutions.

Inclusion artifacts, typically introduced after compilation artifacts, stimulated further team discussions and solidified the consensus during the convergence phase. Borrowed artifacts acted as valuable external catalysts, introducing helpful external resources to the discourse. These artifacts bridged gaps in shared understanding among the team members, fostering synchronization and comprehension. Meanwhile, structuring artifacts, defined by their capacity to "push and negotiate boundaries themselves", provided a structured outline for participants to follow, enhancing the overall coherence of the discussions. Structuring artifacts provided a structured outline for participants to follow, thereby enhancing the overall coherence of the discussions.

Our methodology involving these artifacts and the BNA model informed the design of subsequent workshops, such as those conducted with academic librarians and criminologists. The former aimed to understand how librarians curate, manage, and share research data, and the latter examined the data sharing practices within a discipline that deals with highly sensitive data. The strategic incorporation of these artifacts in a workshop framework ensured their optimum utility in refining the decision-making process, fostering cohesion and consensus among the participants. This strategy confirmed that our protocol could be successfully applied across various data-intensive disciplines.

In conclusion, our workshops demonstrated the potential of our protocol for fostering consensus on data sharing and reproducibility practices in a variety of data-intensive disciplines. They also highlighted the value of the BNA model for gaining insights into the negotiation and evolution of these practices, and their potential support by libraries and data centers. The consistent effectiveness of the artifacts across different contexts underscored their significance, with implications for the design of research-data infrastructure, data-management support services, and other stakeholders, including national funders, memory institutions, and research establishments. These findings hint at the potential to revolutionize the design of systems and mechanisms for supporting cross-disciplinary collaborative research, thereby promoting transparent and rigorous research across all areas.

Ethical Approval

This research has been reviewed and approved by the Office of Research Ethics (ORB), the equivalent body to the Institutional Review Board, at National Taiwan University (NTU), Taiwan, under ORB approval number 201711HS017 on May 11, 2018.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

J.W.: Conceptualization, Writing – original draft, Writing – review & editing.

Y. Yu-Ju,: Writing – original draft, Writing – review & editing.

C.H. & Y. Yi-Jie: Conceptualization, Protocol Design, Writing – original draft

S. Y.: Lead and conducted the additional case study 2.

Funding

This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan, under NSTC 111-2636-H-002-004-, NSTC 111-2621-M-002-007-, and the NTU112L900202 & 112L9A001 from The Featured Areas Research Center Program by the Ministry of Education (MOE) in Taiwan.

Availability of data and materials

Appendixes are available at the Open Science Framework: https://osf.io/vw67k/

Chen, H. C., Jeng, W., He, D., & Chi, Y. (2017). Social science data repositories in data deluge: A case study of ICPSR’s workflow and practices. The Electronic Library.
Chung, C. F., Dew, K., Cole, A., Zia, J., Fogarty, J., Kientz, J. A., & Munson, S. A. (2016). Boundary negotiating artifacts in personal informatics: Patient-provider collaboration with patient-generated data. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 770-786).
Corti, L., Bishop, L., Van den Eynden, V., & Woollard, M. (2014). Managing and sharing research data: A guide to good practice. Sage Publishing. https://doi.org/10.25607/OBP-1540
Corti, L., Van den Eynden, V., Bishop, L., & Woollard, M. (2014). Managing and sharing research data: A guide to good practice. Sage.
Cox, A. M., & Verbaan, E. (2019). Exploring research data management. Facet publishing.
Design Council. (2005). The 'double diamond' design process model. Design Council.
Hara, N., & Fichman, P. (2014). Frameworks for understanding knowledge sharing in open online communities: Boundaries and boundary crossing. In P. Fichman & H. Rosenbaum (Eds.), Social Informatics: Past, Present and Future (pp. 89-100). Cambridge Scholars Publishing.
Huvila, I., Anderson, T. D., Jansen, E. H., McKenzie, P., & Worrall, A. (2017). Boundary objects in information science. Journal of the Association for Information Science and Technology, 68(8), 1807-1822.
Jeng, W., He, D., & Chi, Y. (2017). Social science data repositories in data deluge: A case study of ICPSR’s workflow and practices. The Electronic Library.
Jeng, W., & Lyon, L. (2016). A report of data-intensive capability, institutional support, and data management practices in social sciences. International Journal of Digital Curation, 11(1), 156–171. https://doi.org/10.2218/ijdc.v0i0.0
Kolovson, S., Liang, C., Munson, S. A., & Starbird, K. (2020). Personal data and power asymmetries in US collegiate sports teams. Proceedings of the ACM on Human-Computer Interaction, 4(GROUP), 1-27.
Lee, C. (2007). Boundary Negotiating Artifacts: Unbinding the Routine of Boundary Objects and Embracing Chaos in Collaborative Work. Computer Supported Cooperative Work, 16, 307-339. https://doi.org/10.1007/s10606-007-9044-5
Lee, D. J., & Stvilia, B. (2017). Practices of research data curation in institutional repositories: A qualitative view from repository staff. PloS one, 12(3), e0173987.
Leigh Star, S. (2010). This is not a boundary object: Reflections on the origin of a concept. Science, Technology, & Human Values, 35(5), 601-617.
Scupin, R. (1997). The KJ method: A technique for analyzing data derived from Japanese ethnology. Human Organization, 56, 233-237.
Simon, H. A. (1969). The sciences of the artificial. The MIT Press.
Shih, Y.-R. & Jeng. (2023). The Mobility of Trust, Contract, and Value: The Practice of Open Research Data and Repositories for Criminology in Taiwan. Journal of Library and Information Studies 21(1), 137-168.
Star, S. L., & Griesemer, J. R. (1989). Institutional Ecology, “Translations,” and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387–420. http://www.jstor.org/stable/285080
Vogl, R., Rudolph, D., & Thoring, A. (2019). Bringing structure to research data management through a pervasive, scalable and sustainable research data infrastructure. In The Art of Structuring (pp. 501-512). Springer, Cham.
Stanfield, B. (2000). The Art of Focused Communication.

https://docs.lib.purdue.edu/dcp/
The complete study of the Additional Case Study 2 was published in Shih & Jeng (2023).

No competing interests reported.

Download PDF

Editorial decision: Revision requested
22 Feb, 2024
Reviews received at journal
16 Dec, 2023
Reviewers agreed at journal
07 Dec, 2023
Reviews received at journal
19 Oct, 2023
Reviewers agreed at journal
29 Aug, 2023
Reviewers invited by journal
21 Aug, 2023
Editor assigned by journal
15 Aug, 2023
Submission checks completed at journal
28 Jul, 2023
First submitted to journal
22 Jul, 2023

You are reading this latest preprint version

Boundary Negotiating Artifacts to Envision the Desired Research Data Infrastructure toward Reproducibility in Data-Intensive Science

Status:

Version 1

Abstract

Figures

1 Introduction

2 BACKGROUND

2.1 Techniques Leading a Consensus: Double Diamond Model

2.2 Boundary Objects and Boundary-negotiating Artifacts

3 RESEARCH DESIGN

3.1 Protocol Construction

3.1.1 Stage I: Discover

3.1.2 Stage II: Define

3.1.3 Stage III: Develop

3.1.4 Stage IV: Deliver

3.2 Data Collection and Processing

4 FINDINGS

4.1 Participant overview

4.2 Artifacts as Boundary-negotiating Artifacts

5. ADDITIONAL CASE STUDY

5.1 Additional case study 1: academic librarians

5.2 Additional case study 2: criminologists

6 DISCUSSION AND IMPLICATIONS

Declarations

References

Footnotes

Additional Declarations

Status:

Version 1