O ct 2 02 1 Ultra-chaos : an insurmountable objective obstacle of reproducibility and replicability

It is crucial for scientific progress to be able to replicate scientific findings [1–4], because scientific claims should gain credence due to the reproducibility and replicability of their main supporting evidences. Such kind of reproducibility is a cornerstone of the modern science. However, according to Nature’s survey of 1,576 researchers, more than 70% of the surveyed failed to reproduce another scientist’s experiments, and more than half agreed that there exists a significant “crisis of reproducibility” [2]. Here we reveal a new kind of chaos, i.e. ultra-chaos. Unlike a normal-chaos, statistics of an ultra-chaos are sensitive to small disturbances. Some illustrative examples of ultra-chaos are given here. It is found that statistical non-reproducibility is indeed an inherent property of an ultra-chaos that is at a higher-level of disorder than a normal-chaos. So, it is impossible in practice to replicate experimental/numerical results of an ultra-chaos even in statistical meanings, since random environmental noises always exist and are out of control. Thus, ultra-chaos is indeed an insurmountable objective obstacle of reproducibility and replicability. Similar to Gödel’s incompleteness theorem, such kind of “incompleteness of reproducibility” reveals a limitation of scientific methodologies that can be traced back to Galileo. The ultra-chaos opens a new door and possibility to study reproducibility crisis, statistical significance, computational fluid dynamics (CFD), chaos theory, turbulence theory, and so on.


Introduction
According to Nature's survey of 1,576 researchers in physics & engineering, chemistry, earth & environment, biology, medicine and so on, more than half have failed to reproduce their own experiments and more than half of these surveyed agree that there indeed exists a significant "crisis of reproducibility" [2]. In addition, in a current investigation on evaluating replicability of laboratory experiments in economics, the replication ratio is only 66% on average [3]. The so-called "crisis of reproducibility" widely rises in numerous fields of science and engineering. Currently, computational science has led to exciting advances in many fields, such as the Computational Fluid Dynamics (CFD). The reproducibility and replicability in the CFD serve as a minimum standard to judge scientific claims and discoveries [5]. However, as reported by Mesnard and Barba [6], "completing a full replication study of previously published findings on bluff-body aerodynamics is harder than it looks, even when using good reproducible-research practices and sharing code and data openly". In a survey about the reproducible researches [5], the replication ratio in computational science is only about 25%. There might be lots of reasons that could lead to irreproducible researches, such as poor experimental design, low statistical power, poor analysis, insufficient oversight/mentoring, selective reporting, raw data not available from original lab, code & computational mesh unavailable, insufficient peer review, fraud, and so on [2,6].
Note that all of the above-mentioned reasons are "artificial", more or less. Why is the replication ratio so low in so many fields? Can we completely avoid non-reproducibility if all of the above-mentioned artificial obstacles could be overcome? Is there an insurmountable "objective" obstacle of reproducibility and replicability?
It is well-known that, due to the famous butterfly-effect, trajectories of a chaotic system are rather sensitive to small disturbances so that it is practically impossible to have the reproducibility and replicability of chaotic trajectory. Fortunately, many chaotic systems have reliable statistical properties that are reproducible and replicable. So, statistical reproducibility is more important than trajectory ones. Nowadays, most of scientific claims are supported by statistical results and methods such as the statistical significance, p-values, and so on. Unfortunately, they are based on such a hypothesis that statistical results should be stable, i.e. insensitive to small disturbances. Here we reveal a new kind of chaos, i.e. ultra-chaos. Unlike a normal-chaos, statistics of an ultra-chaos are sensitive to small disturbances. We illustrate below that such kind of ultra-chaos indeed exist and thus is really an insurmountable objective obstacle of reproducibility and replicability.

Some illustrative ultra-chaos
The chaos theory [7][8][9][10][11][12][13][14][15][16] is widely regarded as the third greatest scientific revolution in physics in 20th century, comparable to Einstein's theory of relativity and the quantum mechanics. A chaotic system has at least one positive Lyapunov exponent so that any a small disturbance is enlarged exponentially: this is the mathematical reason of the famous butterfly-effect [8]. So, it is extremely difficult to gain a reproducible/replicable trajectory of a chaos in a long enough interval of time. A new numerical strategy, namely the "Clean Numerical Simulation" (CNS) [17][18][19][20][21][22][23], was proposed to overcome this obstacle. Unlike other algorithms, the CNS can greatly reduce the global numerical noises, i.e. not only the truncation error but also the round-off error, to such a required tiny level that reproducible/replicable computergenerated chaotic simulations can be obtained in a long enough interval of time. This kind of simulations with trajectory reproducibility/replicability in such a long interval of time provide us reliable results, which can be used as benchmark solution to study the influences of artificial/objective small disturbances to statistics of chaotic system.
For example, let us consider here the continuum limit of a chain of pendulums coupled through elastic restoring force and damped friction, governed by the damped driven sine-Gordon equations (2)-(5) [24][25][26] with a Gaussian white noise ǫ(x, t) in case of the standard deviation of σ n = 0, σ n = 10 −18 and σ n = 10 −20 , respectively. Using the reproducible/reliable result in case of σ n = 0 as the benchmark solution, we can accurately investigate the influence of the tiny environmental noises in cases of σ n = 10 −18 and σ n = 10 −20 , respectively. It is found that the tiny environmental noises are equivalent to the numerical noises in essence. Besides, the small disturbances indeed lead to huge and distinct deviations of chaotic trajectories and many physical quantities from the benchmark solution (in case of σ n = 0) when t > 360. For example, the total spectrum energy E s (t) in case of σ n = 10 −18 and σ n = 10 −20 have huge deviations from that of the benchmark solution in case of σ n = 0, with the maximum relative error 10880%, as shown in Fig. 1. It indicates that, due to the existence of the tiny environmental noises, the ultra-chaotic system under consideration might contain much larger energy than that without any environmental noises.
How about their statistic properties? Fig. 2 shows the influence of small disturbances to the spatial statistics, i.e. the spatial mean µ(t) and the spatial root-mean-square (RMS) u rms (t), in case of σ n = 0, σ n = 10 −18 and  Figure 1: Influence of small disturbances to the total spectrum energy E s (t) of an ultra-chaos. The curves are based on the CNS results in t ∈ [0, 3600] of an ultra-chaos governed by the damped driven sine-Gordon equations (2)-(5) (see Appendix) in case of σ n = 0 (red), σ n = 10 −18 (black) and σ n = 10 −20 (blue), respectively, with the maximum relative error 10880%, where σ n denotes the standard deviation of Gaussian white noise. σ n = 10 −20 , respectively, with the maximum absolute error 67.21 for µ(t) and the maximum relative error 675% for u rms (t). Note that, when there exist no environmental noises (σ n = 0), the spatial mean µ(t) is always zero and u rms (t) is always less than 20. However, the tiny environmental noise leads to a gradually increasing deviation of µ(t) from zero and results in u rms (t) > 20 in t ∈ [2200, 3600]. Fig. 3 shows the influence of the small disturbances to the temporal statistics, i.e. the temporal mean µ(x) and the temporal root-mean-square (RMS) u rms (x), in case of σ n = 0, σ n = 10 −18 and σ n = 10 −20 , respectively, with the maximum absolute error 33.98 for µ(x) and the maximum relative error 187% for u rms (x). Note that, when there exist no environmental noises (σ n = 0), we always have |µ(x)| < 10 and u rms (x) < 15. However, due to the tiny environmental noise, it mostly holds |µ(x)| > 10 and u rms (x) > 15. Note that, when there exist no environmental noises (σ n = 0), the global mean µ is always zero, but the tiny environmental noise leads to the huge deviations of the global mean µ = 14.12 in the case of σ n = 10 −18 and µ = −17.64 in the case of σ n = 10 −20 , respectively. functions (PDFs) of the chaotic simulation u(x, t) in case of σ n = 0, σ n = 10 −18 and σ n = 10 −20 , respectively, where the relative error of the PDFs at u = 0 in case of σ n = 0 and σ n = 10 −20 reaches 62%. Note that, when there exist no environmental noises (σ n = 0), the PDF has a kind of symmetry about u = 0. However, such kind of symmetry is lost due to appearance of the environmental noises. In other words, the environmental noises result in the symmetry breaking of the PDFs of u(x, t). Besides, tiny difference between the environmental noises (such as σ n = 10 −18 and σ n = 10 −20 ) leads to huge deviation of their PDFs of u(x, t), as shown in Fig. 4. Fig. 5(a) shows the influence of small disturbances to the autocorrelation functions (ACFs) of the time series u(0, t) in case of σ n = 0, σ n = 10 −18 and σ n = 10 −20 . Obviously, the ACF of u(0, t) has sensitivity dependance on the environmental noises (SDEN). Especially, the ACF of u(0, t) in case of σ n = 10 −18 has huge deviations from those in case of σ n = 10 −20 and σ n = 0. Expand u(x, t) in Fourier series where a k (t) denotes the amplitude and i = √ −1. It is found that the autocorrelation functions (ACFs) of the amplitude a k (t) of the Fourier series of u(x, t) also have sensitivity dependence on the environmental noises, too, as shown in Fig. 5(b) for the comparison of the ACFs of the real part of a 1 (t) in case of σ n = 0, σ n = 10 −18 and σ n = 10 −20 .
Therefore, the objective small disturbances indeed lead to the great deviations of chaotic simulations of the damped driven sine-Gordon equation not only in trajectories but also in statistics. We designate here such kind of chaotic systems as "ultra-chaos". In other words, statistical results of an ultra-chaos are sensitive to small disturbances. In a similar way, we also investigated the influence of the tiny environmental noise to the spatiotemporal chaos governed by one-dimension complex Ginzburg-Landau equation (CGLE) [22], and gained the qualitatively same conclusions as mentioned above, i.e. the statistical results of the chaos governed by the CGLE have the sensitivity dependence on environmental noises (SDEN). So, it is also an ultra-chaos, too. Therefore, the so-called ultra-chaos indeed widely exist and thus has general meanings. This is a surprising result, since it has been widely believed (or assumed) that statistics of a chaos should be "stable" even if its trajectory is sensitive to small disturbances. Mostly, this is indeed true, such as the chaos governed by Lorenz equation [17,20] with one positive Lyapunov exponent, as shown in Fig. 6, and the chaos governed by the four-dimensional Rössler equation with two positive Lyapunov exponents [27], as shown in Fig. 7, respectively. We designate here such kind of chaos as "normal-chaos". Note that the four-dimensional Rössler equation [27] with two positive Lyapunov exponents belongs to the so-called hyper-chaos [10,[27][28][29][30], i.e. a chaos with at least two positive Lyapunov exponents. This reveals that the ultra-chaos is essentially different from the previously reported types of chaos and thus is indeed a new concept.
Note that the environmental noises considered in this article are at a quite tiny level, i.e. 10 −18 and 10 −20 , which is at least two and four orders of magnitude smaller than the round-off error of traditional algorithms in single/double precision. If traditional algorithms in single/double precision are used, the tiny objective environmental noises are submerged by the man-made numerical noises so that we can not investigate the influence of these tiny objective noises at all. For example, the round-off error (and also the truncation error) of the CNS algorithm used here for the sine-Gordon equations (2)-(5) is at the level of 10 −230 , which is at least 210 orders of magnitude smaller than the tiny environmental noises! So, it is the CNS which provides us a powerful tool to accurately investigate the influence of such tiny environmental noises to an ultra-chaos.
In summary, the above-mentioned examples illustrate that small disturbances can indeed lead to the huge deviations of an ultra-chaos, not only in trajectory but also in statistics. In other words, statistical properties of an ultra-chaotic system have sensitivity dependance on small disturbances. Therefore, even in statistical meanings, small disturbances could result in the loss of the reproducibility and replicability of an ultra-chaos, no matter what and how we do. Obviously, an ultra-chaos is at the higher-level of disorder than a normal-chaos. It should be emphasized that small disturbances always exist in practice, and are out of control. Thus, theoretically speaking, the reproducibility and replicability for an ultra-chaos are essentially impossible in practice, forever. Therefore, ultra-chaos is indeed an insurmountable objective obstacle of reproducibility and replicability.

Concluding remarks and discussions
Here we revealed a new classification of chaos. There exist two kinds of chaos. One is the normal-chaos, whose statistics are insensitive to small disturbances. The other is the ultra-chaos, whose statistics are sensitive to small disturbances. Some illustrative examples of ultra-chaos are given here. It is found that, for an ultra-chaotic system, objective tiny environmental noises are essentially equivalent to artificial numerical noises and lead to huge derivations in statistics, and thus should not be neglected carelessly. There- fore, ultra-chaos is an insurmountable objective obstacle of reproducibility and replicability, since tiny environmental noises always exist in the real world and are out of control. Similar to Gödel's incompleteness theorem, such kind of "incompleteness of reproducibility and replicability" reveals a limitation of scientific methodologies that can be traced back to Galileo. Note that scientific findings are based on the reproducibility and replicability of their main supporting evidences: such kind of the reproducibility and replicability is a cornerstone of the modern science. It is an open question whether or not this "incompleteness of reproducibility and replicability" might shake the cornerstone of the modern science. The ultra-chaos as a new concept might open a new door and possibility to study the reproducibility crisis, the statistical significance, computational fluid dynamics (CFD), chaos theory, turbulence theory, and so on.
First, given a chaotic system, we must judge whether it is an ultra-chaos or not. For a normal-chaos, it is even unnecessary to use a very fine grid to gain its numerical simulations, since its statistics are not sensitive to numerical noises. Besides, it is also unnecessary to consider the influence of small environmental noises to a normal-chaos, since environmental noises are essentially equivalent to numerical noises in mathematics. However, for an ultra-chaos, its statistics are sensitive to tiny environmental noises, as illustrated here. Note that every dynamical system has environmental noises, which are random, unavoidable, and out of control. Therefore, from practical viewpoint, it is impossible to replicate an ultra-chaotic system even in the statistical meanings. Theoretically speaking, it is impossible to make any reliable/replicable predictions about such kind of ultra-chaos, forever. Thus, without doubt, it is much more difficult to solve an ultra-chaos than a normal-chaos in mathematics.
Secondly, since turbulence has a close relationship with spatiotemporal chaos [31], it is important to know whether a turbulent flow belongs to ultrachaos or not. If a turbulent flow is a kind of normal-chaos, which we designate as "normal-turbulence", it is unnecessary to use a very fine grid of discretization for numerical simulations, since its statistics are not sensitive to numerical noises. In this case, numerical noise itself is not a trouble at all. However, If a turbulent flow is a kind of ultra-chaos, which we designate as "ultra-turbulence", it should be impossible in practice to gain reproducible/replicable results even in statistics, no matter how fine grid of discretization for numerical simulation or how advanced apparatus with very high precision for experimental measures are used. This also illustrates the importance of ultra-chaos as a new concept, since turbulence is a very important field in science and engineering.
Thirdly, the ultra-chaos as a new concept should be of benefit to deepen our understanding about the so-called "crisis of reproducibility". The nonreproducibility of many scientific researches has leaded to growing worry about the reliability of claims of new discoveries that are based on "statistically significant" findings [32]. As reported by many researchers [33,34], the statistical significance can easily be obtained even from pure noise. Besides, many researchers suggest that p-value is insufficient and that statistical significance should be abandoned [35]. For an ultra-chaos, the derivations caused by either the artificial disturbances or objective environmental noises are macroscopic, and more importantly, the corresponding statistics are sensitive to these disturbances/noises. Note that the statistical significance and p-values are based on reliable/replicable results of statistics. Naturally, for an ultra-chaotic system whose statistics are sensitive to small disturbances, it should be impossible in practice to replicate new discoveries based on "statistically significant" findings. Without doubt, an ultra-chaos is indeed an insurmountable obstacle to reproducibility and replicability, and is really an objective cause to "crisis of reproducibility".
Unlike the so-called hyper-chaos [10,[27][28][29][30], i.e. a chaotic system with at least two positive Lyapunov exponents, an ultra-chaos focuses on the sensitivity dependence of statistics on small disturbance. Note that the hyper-chaos governed by the four-dimensional Rössler equation [27] is just a normalchaos. This reveals that ultra-chaos is essentially different from the previously reported types of chaos and thus is indeed a totally new concept. Note that ultra-chaos is only one possible reason of non-reproducibility. What we would like to emphasize here is that, even if all man-made disturbances are smoothed out, the reproducibility and replicability might be still essentially impossible even in statistical meanings for some dynamic systems such as an ultra-chaos.
Finally, we emphasize that scientific claims should gain credence due to the reproducibility and replicability of their main supporting evidences. Unfortunately, we can not gain such kind of credence for an ultra-chaos! Would the "incompleteness of reproducibility" shake the cornerstone of the modern science? Would the ultra-chaos lead to a crisis of confidence in scientific researches? How should we understand and interpret numerical/experimental results of such kind of ultra-chaotic systems? Should we regard random environmental noises as an inherent part of solution of an ultra-chaos? How should we define the "truth" of an ultra-chaos? What kind of "truth" could an ultra-chaos tell us? Do there exist some ultra-turbulence in practice? Certainly, there are still lots of works to do in future.

Appendix: Supplementary Information
The chaos theory [7][8][9][10][11][12][13][14][15][16] is widely regarded as the third greatest scientific revolution in physics in 20th century, comparable to Einstein's theory of relativity and the quantum mechanics. It is Poincaré [7] who first discovered "the sensitivity dependence on initial conditions" (SDIC) of chaotic systems, which was rediscovered by Lorenz [8] with a more popular name "butterfly-effect": the exact time of formation and the exact path of a tornado might be influenced by tiny disturbances such as a distant butterfly that flapped its wings several weeks earlier. In addition, Lorenz further discovered "the sensitivity dependence on numerical algorithms" (SDNA) of chaotic systems: computer-generated chaotic numerical simulations given by different algorithms in single/double precision quickly depart from each other with distinct deviations [36]. Naturally, such kind of non-replicability of chaotic trajectory leaded to some heated debates on the credence of numerical simulations of chaos, and brought a crisis of confidence: some even made a rather pessimistic conclusion that "for chaotic systems, numerical convergence cannot be guaranteed forever" [37].
In order to gain a reproducible/reliable numerical simulations of chaotic trajectory, Liao [17] suggested a numerical strategy, namely the "Clean Numerical Simulation" (CNS). In the frame of the CNS [17][18][19], the temporal/spatial truncation errors are reduced to a required tiny level by means of a high enough order of Taylor expansion in time and a fine enough spatial discretization with spatial Fourier expansion, respectively. Besides, the round-off error is reduced to a required tiny level by means of reserving a large enough number of significant digits for all physical/numerical variables and parameters in multiple precision [38]. Furthermore, an additional simulation with even smaller numerical noises is needed to determine the so-called "critical predictable time" T c by comparing these two simulations, so that the numerical noise is negligible and thus the computer-generated result is reproducible/reliable within the whole spatial domain in the time interval The CNS has been successfully applied to gain reproducible/reliable simulations of many chaotic systems, such as Lorenz equation [20], Rayleigh-Bénard turbulent flows [21], and some spatiotemporal chaotic systems related to the complex Ginzburg-Landau equation [22], the damped driven sine-Gordon equation [39] and the chaotic motion of a free fall desk [23]. Especially, more than 2000 new periodic orbits of three-body system have been found by means of the CNS [40][41][42], which were reported twice by the popular magazine New Scientist [43,44], because only three families of periodic orbits of the three-body problem have been reported in three hundred years since Newton mentioned it in 1687. All of these illustrate the validity of the CNS for chaos.
It should be emphasized that such a reproducible/reliable result given by the CNS provides us a "true" solution, and more importantly, a kind of reproducibility and replicability of chaotic trajectory. Such kind of trajectory reproducibility and replicability can remain in a long enough interval of time t ∈ [0, T c ] as long as the numerical noises could be globally reduced to a rather tiny level [22,23,39]. This kind of strict reproducibility and replicability of chaotic trajectory provides us a confidence of credence/reliability and especially a benchmark solution. For example, by means of an algorithm based on the CNS, Liao and Wang [20] gained a computer-generated chaotic simulation of Lorenz equation, which is reproducible/reliable in a quite long interval of time 0 ≤ t ≤ 10000 (Lyapunov unit) and thus provides us a true result and a benchmark solution of the Lorenz equation in t ∈ [0, 10000], while numerical simulations given by other algorithms in single/double precision are reproducible/reliable only in a quite smaller interval 0 ≤ t ≤ 40 (Lyapunov unit) and has a distinct deviation from the benchmark solution for t > 40. Having the benchmark solution of the Lorenz equation in such a long enough interval of time t ∈ [0, 10000], it is possible to investigate the influence of numerical noise to statistics of computer-generated chaotic simulations given by other algorithms in single/double precision.
Thus, by means of the CNS, the artificial numerical noises can be globally reduced to a required tiny level much smaller than the objective environmental noises so that we can accurately study the influence of the environmental noises to the statistical reproducibility and replicability of some spatiotemporal ultra-chaos. For example, we consider here the continuum limit of a chain of pendulums coupled through elastic restoring force and damped friction, governed by the damped driven sine-Gordon equation [24][25][26] but with a Gaussian white noise ǫ(x, t): u tt = u xx − sin(u) − αu t + Γ sin(ω t − λ x) + ǫ(x, t) subject to a periodic boundary condition u(x + l, t) = u(x, t), where the subscript denotes the derivative, x and t are the spatial and temporal variables, u(x, t) denotes the angle of pendulum, α is a constant related to the damped friction, Γ is a constant related to the elastic restoring force, ω is the temporal frequency, λ = 2π/l is the spatial frequency with l being the total length of the system, respectively. Here, ǫ(x, t) corresponds to the tiny environmental noises. Without loss of generality, we follow Chacón et al. [24] to choose the following values of these physical parameters ω = 3 5 , α = 1 10 , Γ = 461 500 , l = 500, λ = 2π subject to the zero initial condition u(x, 0) = 0, u t (x, 0) = 0.
As reported by Qin & Liao [39], when there are no environmental noises, i.e. ǫ(x, t) = 0, the above model corresponds to a spatiotemporal chaos whose statistics are sensitive to numerical noises, and thus is an ultra-chaos. Let σ n denote its standard deviation of Gaussian white noise ǫ(x, t). We use the same CNS algorithm [39] to gain reproducible/reliable numerical simulations in the three cases σ n = 0, σ n = 10 −18 and σ n = 10 −20 , respectively, where σ n = 0 corresponds to the benchmark solution given by the CNS without any objective environmental noise, the others correspond to the reproducible/reliable simulations with the influence of the corresponding tiny environmental noises. Unlike Qin & Liao [39] who used a CNS algorithm with N = 16384 and N s = 60 to gain reliable simulations in t ∈ [0, 900], we discretized the spatial domain here much better by N = 2 16 = 65536 equidistant points, used the multiple precision with N s = 230 significant digits for all variables and parameters, and applied the variable stepsize scheme in the temporal dimension with a given allowed tolerance tol = 10 −230 of the governing equations. According to Qin & Liao [39], it holds T c ≈ min {0.0558N + 10.7, 16.5N s − 87.8} .
Thus, we can gain the reproducible/reliable chaotic simulations within the whole spatial domain in a much larger temporal interval t ∈ [0, 3600] in the above-mentioned three cases, which is long enough for statistical analysis.