PROTEUS: an immersive tool for traveling across space and time scales

In the context of scientiﬁc visualization, the main challenge is to achieve a good balance between the accurate representation of scientiﬁc data (computed or stored in databases) and the visual quality of the virtual experience itself. Traditional scientiﬁc software relies on custom tools meant to be used on conventional terminals or software developed with mouse and keyboard in mind. However, the true advantage of Virtual Reality is the enhanced ability of ﬁnding connections between apparently heterogeneous information, discovering correlations that would otherwise be harder to detect. In particular, in the ﬁeld of Digital Humanities, it is increasingly necessary to develop and validate virtual reality tools that are capable of combining various scientiﬁc data in a Virtual Reality context with access and consultation of online repositories. This paper reports the main aspects of the implementation of a heterogeneous relational database available online storing 3D models, metadata and chemical analyses related to diﬀerent sectors of digital humanities. The Virtual Reality software, called PROTEUS, allows the user to shrink himself from the macroscopic world of Digital Humanities, down to the microscopic world of molecular sciences. The web platform is connected to the virtual reconstruction software, which dynamically loads the information available in the online repository within the virtual reality application. This application also provides a detailed explanation of the reconstruction of the 3D models for the various objects, and the chemical analyses. The real advantage of a virtual reality application is the real time interaction with data and 3D models, thus allowing the researcher to understand and manipulate objects, to test hypotheses and to seek meaningful results, visualizing the metadata while changing the parameters of the simulation in a dynamic and interactive way. This is a signiﬁcant step forward in the democratization of science, by means of an user friendly and immersive access to advanced scientiﬁc algorithms, and the integration of the Proxima Molecular Perception library, which allows the automatic perception of structural and topological features of the underlying molecular and supra-molecular systems.


Introduction
The need to better integrate technology with archaeology was born as early as the 1960s. Binford and Quimby [1], leading actors of Processual Archeology (also called New Archaeology), have led in an avant-garde way the use of scientific and technological techniques such as quantitative, taxonomic and laboratory analysis within archaeology, establishing a strong interconnection between apparently distant disciplines such as digital humanities and chemistry.
The evolution of graphic engines and the ease of use of software for the creation of 3D models, derived from photogrammetry and 3D modeling, have allowed a rapid proliferation of projects related to 3D reconstructions in the field of digital humanities. Over the years, also thanks to the advancements of display technologies, the focus has mainly been on creating virtual environments with high visual fidelity, at the cost of a less refined user interaction. Interaction is the ability to "immerse" the user in the virtual environment so as to carry out studies and analyses directly from within the reconstructed environment, thus overcoming the distinction between "reconstruction" and "simulation". The simulations are now context-dependent, so that the virtual reconstruction is not just a simple visual feature, but is a reference for the user to better understand, manipulate and test different hypotheses. In fact, it is now possible to interact in a more functional and intuitive way with this type of simulations.
This new approach, called Cyber Archeology, is mainly based on the transition from the concept of "reconstruction" to that of "simulation". In this framework, PROTEUS was born: an application aimed to mark the transition between reconstruction and simulation, that allows real time interaction with the models displayed in the virtual environment providing all users the ability to enter metadata and create interconnections between them. Traditional visualization software is either focused on the microscopic world of molecules or on case-by-case macroscopic scenarios. In this context, there is a lack of a general integrated software for the simultaneous visualization of both kinds of data. In fact, in most cultural-heritage applications, sample points are selected for further chemical analysis in well-defined positions of a macroscopic model (e.g. a statue, a painting, etc.). However, the visualization of the sample data (that is the chemical data associated to the sample points) is currently separated from the visualization of the macroscopic object. The goal of PROTEUS is to completely rethink the visualization process by providing a universal workflow that is easily scalable so as to cover a wide range of possible scenarios. In particular, we started by providing a common infrastructure for loading macroscopic 3D models (e.g. those obtained from photogrammetry) with associated sample points, we then proceeded to the development of molecular-specific visualization tools (e.g. the so-called ball-and-stick representation in Virtual Reality applications) and we finally defined a common file format for the association of the sampling data to the sample points. At this stage, the sampling data could be either a molecule generated from a file containing atom types and coordinates (XYZ file) or taken from the Protein Data Bank (PDB) [2], or it could also be some general text maybe with a description of the sample point. The choice of the PDB file format is motivated by the large amount of available chemical data (obtained either from experimental studies and/or quantum chemical computations) that is compatible with such format. In particular, the Protein Data Bank acts as a huge reservoir of chemical structures obtained by different experimental techniques (e.g. Crystallography, NMR, etc.). While PDB and XYZ file formats are conceptually similar (both of them store types and cartesian coordinates of the atoms forming the considered molecule), the XYZ file format has the advantage of being simpler to parse, lighter and easily obtainable from quantum chemistry software (e.g. Gaussian [3]). The PROTEUS software also includes the Proxima Molecular Perception library [4] for handling chemical data. In particular, the Proxima software is a C++ library (integrated within C# Unity [5] using underlying C functions) that provides methods for the parsing of the XYZ file format, together with the detection and evaluation of chemical bonds and non-covalent interactions. The integration of Proxima within PROTEUS allows for further chemical tools to be provided in future releases. Another advantage of using Proxima is the correct handling of periodic structures such as the ones obtained from X-Ray Crystallography or Molecular Dynamics (MD) simulations. In these systems, a unit cell is repeated along three vectors in order to produce the entire system possibly adding some bonds that might be missing because they involve atoms that are on two opposite sides of the unitary cell. The use of Proxima in a VR application is a huge step forward in the democratization of science, since more and more non-expert users can have access to advanced perception algorithms. The PROTEUS development has therefore been focused on the design and implementation of different levels of visualization, which allow for a seamless transition from the macroscopic object under analysis to its microscopic components (e.g. the chemical compounds detected through spectroscopy), in order to stimulate the search for one of the possible "simulation slices" for that context. [6] Methods Unity 3D Engine Unity [5] is a cross-platform graphic engine developed by Unity Technologies that allows the development of real time applications and games and other interactive content, such as architectural visualizations or 3D animations. Unity game engine launched in 2005, with the aim to "democratize" game development by making it accessible to more developers. It simplify the creation of applications with little programming knowledge required and with a powerful and customized editor, by being accessible and easy to use at the same time.
Thanks to its ease of use and free license for development, unity has been a great success not only in video games, but also in research projects. It is in fact the perfect tool for developing interactive applications to support scientific visualization and has been used in the SMART laboratory since 2012 (the year of its establishment at the Scuola Normale) for most of its real-time applications.

Database Creation
Regarding the transfer of data between the virtual reality application and the web server, the Unity API was used to allow ideal integration with the XAMPP web development multiplatform [7]. XAMPP is composed of a Relational Database (MySql), an Open Source software compliant with the ANSI SQL and ODBC SQL standard licensed GNU General Public License, which supports programming languages such as Java, PHP, Python and many more. The use of MySql is matched with the XAMPP multiplatform software made up of Apache HTTP Server and all programming tools in PHP and Perl.

Environment digitalization
To better reproduce the case study environment, so as to achieve a better user immersion inside the scene, a 3D model of the Sala Stemmi (an historical room inside Scuola Normale Superiore, Pisa) has been acquired through photogrammetry. This acquisition process required to shoot 359 pictures, given as main data information to Metashape [8], a dedicated photogrammetry software. There were 4 sessions of acquisition: two allocated for the wall and two for the ceiling reconstruction. The shot technique expects to take overlapping pictures, using different shot angles also at different heights, maintaining a fixed distance from the subject. If the acquisition workflow is properly executed, the photogrammetry software is able to manage and outcome an effective 3D model. A slightly different workflow has been used to shot the ceiling pictures, because of their hollowed structure. The camera has been placed in a perpendicular position and the shots have been taken perpendicularly, to better acquire the overall structure and shape of the coffered ceiling.
At the end of the acquisition workflow, the outcome is a high-definition model of the room (usually defined as High-Poly 3D object). While these kinds of models are extremely accurate, the amount of geometrical data and the 3D object surface are not suitable and optimized for real time applications. Thanks to specific software it is possible to decimate the geometry of the mesh, with the outcome being a low definition 3D model (referred to as Low-Poly), more suitable for real time rendering. The decimation process reduces the number of vertices, maintaining the original volumes and shapes of the High-Poly mesh. The next step is the retopology process, which consists in the optimization of the mesh, by reconstructing the surfaces with quads, namely polygons with four vertices for each face. As a matter of fact, a 3D object surface with this configuration is optimized for virtual environment and real time applications. The retopology process has been facilitated and fastened in the last few years by means of procedural algorithms, which analyze the surface of a given 3D object, returning a quad mesh. The following step is baking, which consists of transferring high-definition information from the High-Poly to Low-Poly after the creation of a proper UV map. UV mapping consists in creating clusters of the 3D models and organizing them in a 2D space: in this way it is possible to project the 3D object on a plane surface and to store high definition information in images files format. At this point the asset is complete and the Low-Poly model is optimized for real time applications. Careful analysis of the geometrical structure and repetitive pattern of the floor, suggested a different workflow to improve its visualization by means of a specific material creation software, which permits to reproduce the typical herringbone flooring of the Sala Stemmi as a tileable texture (Albedo, Normal and Height maps). After integration of the textured plane in the scene, the original lighting conditions of the room have been baked on the Albedo map. This step is essential for a better realism, hence a better user experience of the environment. The wooden beam has been 3D modeled taking care of the original ceiling volumes and dimensions. The texturing process combined seamless texture for visualization of the actual beam, while the painted surface has been acquired by a high resolution picture shot by a professional reflex camera. The high resolution images have been used for a better visualization and interaction with the target object inside the virtual environment.

Previous Work integration
In a previous work [9], an application that was only limited to the specific case of the "Sala Stemmi" room was implemented with the main aim of testing a multiscale approach that was still under development. In fact, the virtual scenario was implemented as a case study, with direct integration with UnityMol [10], a separate software capable of displaying molecules from the PDB database.
The purpose of this new work is to reconsider the results of the first exploratory study bringing it to a new level of visualization and interaction, by transforming a closed application, related to a single case study, to an open application, dynamically integrated with new types of simulations. The PROTEUS immersive tool for travelling across space and time scales is, in fact, a dynamic tool capable of interfacing with online repositories both at the macroscopic level, with 3D models and complex scenarios (archaeological reconstructions, cultural heritage, art works) and also at the microscopic level thanks to the online integration of a chemical repository, managed internally by the application. It will be also possible to integrate the results of different chemical analyses performed on art works, such as X-Ray (XR), ultraviolet-visible (UV-vis), and infrared (IR) spectroscopic analyses, reflectography, and UV light shooting. PROTEUS is therefore an extremely modular tool, which allows a dynamical integration of external data with model reconstructions, connected online in an open manner with various heterogeneous databases and repositories, which contain extremely different data.

Database Connection
As mentioned above, it will be possible to integrate PROTEUS with a generic web server, thus enhancing the panel of conceivable applications, concerning both organization and implementation. It will be possible also to implement relational databases that are capable of communicating with PROTEUS , providing a more structured access to information. In the specific case study of Sala Stemmi we used a SQL relational database to download the 3D models.
The implementation of assets in Unity permits a dynamical upload and download of 3D models, metadata and results of chemical analysis within the application, using a script that connects actively the web server with the virtual reality application. Within the virtual simulation it is therefore possible to connect to the web server and download digital assets at run time, without the need to write new software whenever new elements need be added to the simulation. This degree of interactivity and flexibility allows for accurate and up-to-date reconstructions.

Multidisciplinary interface
A new interface was developed for uploading the online models in order to permit an user friendly access to people with different professional backgrounds, including cultural-heritage experts, restorers, chemists, and even scholars with different backgrounds. Then, the skills and needs of the user will be satisfied by connection to a web server with variable parameters ensuring the interaction with a specific open online repository. After establishing the connection parameters, it will be possible to connect to a web server and download models in real time within the application via an upload interface. Next, the imported models can be translated, rotated, and inserted in the simulation, creating a virtual scenario with the possibility of attaching metadata to the models such as links to other 3D models or heterogeneous databases. From a chemical perspective it is possible to connect to a generic online repository and download data from a database, with the possibility of dynamically linking molecules to 3D models.
As mentioned above, the PROTEUS tool derives from the prototype of a virtual reality viewer, capable of managing macro-elements (Unity reconstruction) interfaced with a software that manages the micro-elements (UnityMol). The limitation of closed software, restricted to the case study, interfaced with an external tool for chemical management, required a complete reformulation of the entire project. PROTEUS therefore arises from the need to create a modular software, to be updated and implemented easily and which, at the same time, contains a powerful molecular viewer. We therefore started by validating the new PROTEUS tool for the case study of the previous work. The objective of the case study was to add or modify sampling points on the 3D models in the Sala Stemmi (coffered ceiling) by downloading the 3D models of the crest room from the online repository and then entering the points.

Implementation Details
In Fig. 4, the internal organization of the software in Unity is shown. In particular, the MacroScene prefab is responsible for handling all the different Macroscopic objects in the scenario. The MacroScene script allows to add more than one MacroObject by simply dragging the relative GameObject associated in the list of the script. Each MacroObject is another prefab containing all the scripts required for the correct interaction with the MacroObject (such as the bounding box for better hand interactions in scaling etc.). As a matter of fact, we decided to employ the MRTK mixed reality toolkit for handling natural hand interactions so as to simplify the visualization process for the user. The MacroObject script has also a public reference to the 3D model representing the corresponding object. The Sample Points are inserted as child of the MacroObject, with each Sample Point being a prefab with a script indicating whether the metadata associated is of type text or is a molecule (in the latter case, the content of an XYZ file is put in the text field of the script and Proxima handles the correct molecular representation). In order to specify the relative position of the metadata to be displayed with respect to the Sample Point position, a Sample Placement flag is specified that could be of type: • XY • XZ • YZ Together with a distance and angle value they allow to place the metadata in the desired location. In Fig. 5, the metadata position with respect to the Sample Point is shown. The final result, in the case of the Sala Stemmi, are the sample points shown in Fig. 1 where the metadata are the blue labels with the relative text describing the age and the type of sample collected. On the top left, instead, an example of a chemical structure (the calcium carbonate) is shown inside the Sala Stemmi showing the capabilities of the software.

Conclusion
The applications developed and optimized for virtual reality permit an interpretation of the data that is far more immediate and understandable. Through this new visualization approach, the various data can be analyzed in depth; virtual reality offers a unique and highly immersive point of view within the reconstructed context, a factor that makes understanding closer to our perceptive sensitivity.
The possibility of having an open tool, freely interfaceable with other applications and online repositories, permits great flexibility and opens up new methods of visualization and shared interaction between professionals with different skills. Thanks to the dynamic loading of 3D models and molecular data, it is possible to understand and manipulate objects, to test hypotheses and to seek meaningful results, visualizing the metadata and changing the parameters of the simulation in a dynamic and interactive way.
The use of PROTEUS enhances collaboration and knowledge sharing, favoring multi-disciplinarity and effectively creating a strong point of contact between two apparently very different realities such as chemistry and cultural heritage. The ability to manipulate simulation and data in real time allows the users to test hypotheses and interact deeply with the reconstruction. The potential of virtual simulations can be expanded with the possibility of multi-user sessions and offers the possibility for scholars to collaborate, working together in the same virtual space, even if they are physically distant.