As already shown in Table 2, we designed the test cases to be based on the rate of occurrence or intensity of small cycle slips (1 to 3 cycles) which are harder to detect. Intensities of the small cycle slips are categorized as high (Test Case 1), moderate (Test Case 2), and low (Test Case 3). Note that additional (1, 1) slips were included in Test Case 2. All the four stations showed similar results for the three test cases, despite the differences among the physical environments of the sites noted in Section 2 above. We start by showing typical results from one of the stations.
In Fig. 4, we show the plots for all three test cases involving station NIST. In the top row, we show the distribution of added cycle slips for each of the test cases (columns) from which the intensities of small cycle slips (1 to 3 cycles) can be inferred. The rate of missed detections (second row) varies based on the intensity of small cycle slips. As expected, it clearly shows that the highest rate of missed detections occurred in Test Case 1. By comparison, the rate of missed detections is much lower in Test Case 3 with low number of small slips (1 to 3 cycles). Small slips are more likely to be a challenge in problematic data with higher noise levels. Therefore, for this study we considered it more important to detect small slips as bigger slips are easier to detect and estimate in general.
As is also evident from Fig. 4, there is a narrow window of RMS scale factor and Ion residual threshold values which results in very high number of false detections. These are identified as RMS scale factors k = 1 and k = 2, and to a lesser extent the Ion residual threshold h = 0.1. It is not surprising that small values for these parameters result in high instances of false positive detections, with the algorithm mis-identifying noise in the time series as cycle slips.
Also apparent in Fig. 4 is the slight tendency for the rate of missed slips to increase with larger values of \(k\) and \(h.\) The larger these values are, the larger the differences in the test statistics need to be before they are flagged as potential cycle slips or outliers. The effect of larger parameter values is most evident for Test Case 1 and Test Case 2 with moderate to high number of small slips.
Despite the different monumentation and fairly wide range of geographic locations and to some extent the time of year represented by the four stations (Table 1), there is similarity of parameter set performances for the stations (Figs. 5 and 6). Given the similarity among the results for different sites, we present the remainder of the analysis by using an aggregate of all the results for each of the stations separately, and similarly for all the stations considered together. Analysis of the aggregated performance should provide a somewhat more general assessment of algorithm performance and optimal parameter settings, although admittedly only for a dataset restricted to four relatively high-quality stations.
Figure 7 shows an aggregate analysis of slip detection for each station for the case of data with high intensity of small slips (Test Case 1). The number in each grid cell of plots in the left column shows the average number of missed slips for every realization in which ten slips are inserted. The number in each grid cell of plots in the right column shows the average number of false detections for every ten inserted slips. We see that no set of parameters yields a detection of all inserted slips, across all 40 realizations (10 slips each). Nevertheless, some sets of parameters obviously perform better than others, on average. The randomness associated with our adopted slip insertion process results in detection performance scores (i.e., average number of correct, missed, and false detections) that behave effectively as random variables. The consistency among similar averaged scores from cell to cell for each station, and the similarity of detection results for parameter sets among stations, suggests that samples of size 400 inserted slips is large enough to characterize DbEdit’s average performance. The occurrence of real cycle slips in nature could similarly be modeled as samples from a random process, although admittedly not necessarily drawn from the same distribution as we have used. Yet, given the multitude of environmental and other factors that may contribute to the occurrence of cycle slips in real data, we might invoke the central limit theorem to justify a similar mean performance given real cycle slip data, an assumption that could be tested through further analysis of real-world cycle slip statistics. We leave such an analysis of cycle slip statistics to future work.
Figure 8 shows an aggregate analysis using data from all of the four stations combined for the case of data with high intensity of small slips (Test Case 1). An optimal solution occurs when both the rate of missed detections and false detections are minimal. Therefore, in each plot, the smallest of the numbers in the grid cells are the most desirable for an optimal solution. The number in each grid cell of Fig. 8(a) shows the average number of missed slips for every ten inserted slips. The number in each grid cell of Fig. 8(b) shows the average number of false detections for every ten inserted slips. In Fig. 8 (c), where we introduce the term parameter optimality score, the numbers in each grid cell are subsequently derived from the numbers in the corresponding grid cells of Figs. 8(a) and 8(b), as a way of combining the two datasets into a final single plot for the overall interpretation of the results. The number in each grid cell of Fig. 8(c) equals the average number of missed slips for every ten inserted slips plus the average number of false detections for every ten inserted slips. For the optimality score, we use a color scale (indicated at the bottom of the plot) such that the smaller the number in a grid cell the more optimal are the parameter values corresponding to that grid cell. It is more problematic for a positioning solution to have missed slips (additional phase biases in an arc of data) than to have false slips, but in this case, we applied equal weighting in computing the optimality score since there would be no noticeable impact of down-weighting false detections for the area of focus (average zero false slips).
Figure 9, which is similar to Fig. 7, shows an aggregate analysis of slip detection for each station for the case of data with moderate intensity of small slips (Test Case 2). Figure 10 is also similar to Fig. 8 but shows an aggregate analysis using data from all of the four stations combined for the case of data with moderate intensity of small slips (Test Case 2). Both Figs. 8(c) and 10(c) helps to focus on the desired range of parameters values which results in low missed detections and low false detections. We see that the most optimal parameter values are k = 6 and h = 0.3.
Additional interpretations can also be made in conjunction with plots (a) and (b) of both Figs. 8 and 10. From the analysis of both Figs. 8 and 10 considered together, it can be seen that for zero false detections, there are at least a few different interpretations that can be made in terms of the desired combinations of values for both parameters (i.e., RMS scale factor k and ionospheric residual threshold h). For instance, 6 ≤ k ≤ 8 and 0.3 ≤ h ≤ 0.5 resulted in at least 95% success rate with zero false detections; 6 ≤ k ≤ 12 and h = 0.3 resulted in at least 97% success rate with zero false detections; k = 6 and 0.3 ≤ h ≤ 0.5 resulted in at least 95% success rate with zero false detections; k = 6 and 0.3 ≤ h ≤ 1.5 resulted in at least 93% success rate with zero false detections; and 6 ≤ k ≤ 8 and 0.3 ≤ h ≤ 1.0 resulted in at least 90% success rate with zero false detections. In all these interpretations which consider both the high and moderate intensity (occurrence) of small cycle slips, the upper limit is 97.5% success rate as defined by the data with high intensity of small cycle slips (Fig. 8(a)).
Fig. 8(a) also defines the range of combinations of the two parameters that begins to negatively impact the success rate for the case of small slips. These include for example, k = 12 with 1.0 ≤ h ≤ 1.5; and 8 ≤ k ≤ 12 with h = 1.5 both of which resulted in 11 to 18% detection failure rate.