In this paper, the system contains a Cooperative Relay node, a source (S) node and a destination (D) node. The source is a satellite and the destination is an earth station. The relay is a half duplex relay, present at the earth. Each of these units are well equipped with a single antenna for downlink transmission. Considering the cost of maintenance, ease to use and gain in the practical cases, single antenna is employed in the system. There are two types of channels links present in the system namely satellite channel link and terrestrial channel link. The link between source and destination as well as between source and relay is satellite channel link which is assumed to follow Shadowed Rician distribution [15] while the terrestrial channel link is present between relay and destination, assuming to follow Nakagami- m distribution [24],[31].

The Fig-1 shows block diagram of the system model for downlink transmission. The dotted line shows the satellite channel link and the darker line shows terrestrial channel link. For better efficiency, NOMA is applied for two consecutive symbols s1 and s2 to be transmitted simultaneously with an assumption that symbols s2 contains more information than symbols s1. NOMA uses the concept of using superposition coding principle at transmitter and Successive Interference Cancellation (SIC) at the relay. It is assumed that the SIC implemented at relay is perfect to avoid interference. This strict assumption should be relaxed considering the nature of wireless communications. Then, using the superposition coded symbol conveyed to destination and relay, the received signal in the first phase of transmission can be given from [1] as

$$y=\sqrt {{P_s}} (\sqrt \alpha {s_1}+\sqrt {(1 - \alpha )} {s_2}){h_z}+{W_z}$$

1

Where \({h_z}\)is flat fading channel coefficient with z = sd,sr,rd and are source to destination, source to relay and relay to destination respectively and \(\alpha\) is the power allocation coefficient and assumed to be less than 0.5. \({P_s}\) is the transmit power of source. \({W_z}\) denotes additive white Gaussian noise with variance \({N_0}\). \({s_1}\) and \({s_2}\) are the two symbols that are transmitted simultaneously. In the first phase of transmission, symbol \({s_2}\) is detected by relay and destination while symbol \({s_1}\) is treated as noise. To detect the \({s_1}\) symbol, relay uses SIC to estimate \({\hat {s}_1}\) and forward it to the destination in second phase and is given from [1] as

$${y_{rd}}=\sqrt {{P_r}} {\hat {s}_1}{h_{srd}}+{W_{rd}}$$

2

where \({P_r}\) denotes the transmit power of relay. The Eq. (2) shows that symbol \({s_1}\) is detected by the destination based on \({y_{rd}}\). We want to obtain the total Bit Error Probability (BEP) of the NOMA-CRS, therefore, according to the law of probability, the average Bit error probability (ABEP) of the NOMA CRS with symbols \({s_1}\)and \({s_2}\) can be given as

$${P_{e2e}}(e)=\frac{{{P_{{s_1}}}(e)+{P_{{s_2}}}(e)}}{2}$$

3

Where \({P_{{s_1}}}(e)\) and \({P_{{s_2}}}(e)\) are the BEPs of symbols \({s_1}\) and \({s_2}\) respectively. Since only symbol \({s_2}\)is transmitted to the destination in first phase and symbol \({s_1}\) is detected as noise therefore, the BEP for symbol \({s_2}\) appears same as the BEP of far end user in downlink NOMA transmission which is dependent on constellation pairs chosen. Thus, the conditional BEP of the far end user in NOMA given in [1] is used as.

$${P_{{s_2}}}(e|{\gamma _{sd}})=\sum\limits_{{i=1}}^{M} {{P_i}Q(\sqrt {2{\upsilon _i}{\rho _s}{\gamma _{sd}}} )}$$

4

where \({\rho _s}={P_s}/{N_0}\)and \({\gamma _z}=|{h_z}{|^2}\). The coefficients \({P_i}\), M and \({\nu _i}\) vary according to the chosen constellation pair for symbols \({s_1}\)and \({s_2}\). If the chosen constellation pair is BPSK then as given in [1], \({\upsilon _i}=1 \mp 2(\sqrt {(\alpha ) - {{(\alpha )}^2}} )\), \({P_i}\)= 0.5 and M = 2. Since the satellite channel links are supposed to follow Shadowed Rician fading in first phase therefore, we can express the SR fading distribution for symbol \({s_2}\)as given in [15] and [29] and with the help of Eq. (4) as

$$P_{{{s_2}}}^{{sd}}(e)=\int\limits_{0}^{\infty } {\sum\limits_{{i=1}}^{M} {{P_i}Q} } \sqrt {2{\upsilon _i}{\rho _s}{\gamma _{sd}}} {\alpha _i}{e^{ - {\beta _i}x}}{}_{1}{F_1}({m_i};1;{\delta _i}x)dx$$

5

where \({}_{1}{F_1}\) denotes confluent hypergeometric function given in [35], [36] and \({m_i}\) is the shaping parameter of the Shadowed Rician (SR) fading distribution with i = 0,1 and the other parameters are given in [32] as follows.

\({\alpha _i}=0.5{(\frac{{2{b_i}{m_i}}}{{2{b_i}{m_i}+{\Omega _i}}})^{{m_i}}}/{b_i}\) ; \({\beta _i}=\frac{{0.5}}{{{b_i}}}\) ; \({\delta _i}=\frac{{0.5{b_i}}}{{2{b_i}^{2}{m_i}+{b_i}{\Omega _i}}}\)

The parameter \({\Omega _i}\) denotes average power of the LOS component and \(2{b_i}\) denotes the average power of multi-path component. The range of shaping parameter \({m_i}\) varies from 0 to \(\infty\). Now for calculating BEP, the approximated Q function [39] is used and given as

$$Q(x)=\frac{1}{{12}}{e^{\frac{{ - {x^2}}}{2}}}+\frac{1}{4}{e^{\frac{{ - 2{x^2}}}{3}}}$$

6

Now, the Eq. (5) can be rewritten with the help of Eq. (6) and properties of hypergeometric function [36], [37] to obtain the BEP of symbol \({s_2}\) as follows

$$P_{{{s_2}}}^{{sd}}(e)=\frac{1}{{12}}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}})+\frac{1}{4}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}})$$

7

While, the symbols \({s_1}\) are detected in the initial phase of transmission at relay and forwarded to the destination in next phase. The detection is erroneous in both the phases but are totally independent. So, with the help of law of total probability, the BEP of symbol \({s_1}\) is

$${P_{{s_1}}}(e)=P_{{{s_1}}}^{{sr}}(e)[1 - P_{{{s_1}}}^{{rd}}(e)]+[1 - P_{{{s_1}}}^{{sr}}(e)]P_{{{s_1}}}^{{rd}}(e)$$

8

where\(P_{{{s_1}}}^{{sr}}(e)\) and \(P_{{{s_1}}}^{{rd}}(e)\) are BEPs of symbol \({s_1}\) from source to relay in very first phase and relay to destination in the second phase respectively. The second phase transmission is not encountered with interference so, the transmission from relay to destination becomes easier to obtain the BEP of symbol \({s_1}\). For BPSK, the conditional probability of \({s_1}\) is given with the help of [1] as

$${P_{{s_1}}}^{{rd}}(e|{\gamma _{rd}})=Q\sqrt {2{\rho _r}{\gamma _{rd}}}$$

9

Where \({\rho _r}={P_r}/{N_0}\). The use of SIC resulted in detection of symbols \({s_1}\) where Nakagami-m fading distribution is assumed to have followed. Therefore, the BEP of symbol \({s_1}\) from [1] can be given as follows

$$P_{{{s_1}}}^{{rd}}(e)=\frac{1}{{2\sqrt \pi }}\frac{{\sqrt P }}{{{{(1+P)}^{{m_{rd}}+0.5}}}}\frac{{\Gamma ({m_{rd}}+0.5)}}{{\Gamma ({m_{rd}}+1)}}{}_{2}{F_1}(1;{m_{rd}}+0.5;{m_{rd}}+1;\frac{1}{{1+P}})$$

10

Where \(P=\frac{{{\rho _r}{\Omega _{rd}}}}{{{m_{rd}}}}\). The Eq. (10) is applicable when \({m_{rd}}\) is non integer while for integer values, the equation from [1] is given as under

$$P_{{{s_1}}}^{{rd}}(e)=\frac{1}{2}[1 - {\mu ^2}(P)\sum\limits_{{j=0}}^{{{m_{rd}} - 1}} {\left( {_{j}^{{2j}}} \right)} {(\frac{{1 - {\mu ^2}(P)}}{4})^j}]$$

11

Where \(\mu (P)=\sqrt {\frac{P}{{1+P}}}\). In the first phase of transmission, the relay received the superimposed signal for which SIC is employed therefore, the propagation of error during SIC should be considered in the analysis and closed form derivation of BEP. To detect symbol \({s_1}\) the symbol \({s_2}\) should be detected first and the detected symbol \({\hat {s}_2}\) should be subtracted from the received signal. For the analysis, both the erroneous and correct SIC of symbol \({s_2}\) should be taken into consideration. Since the satellite channel link is supposed to follow the Shadowed Rician fading therefore, the conditional BEP of \({s_1}\) symbols in first phase of downlink transmission from [1] can be given as

$${P_{{s_1}}}(e|{\gamma _{sr}})=\sum\limits_{{i=1}}^{N} {{\eta _i}Q\sqrt {2{V_i}{\rho _s}{\gamma _{sr}}} }$$

12

Where \({V_i}\), N and \({\eta _i}\) varies with modulation pairs and for BPSK from [1], \({\eta _i}\) = 0.5 [2,–1, 1, 1,–1], N = 5 and the roots values are \({V_i}=[\alpha ,1 \pm 2\sqrt {\alpha - {\alpha ^2}} ,4 - 3\alpha \pm 4\sqrt {\alpha - {\alpha ^2}} ]\). Now, with the help of approximated Q function given in Eq. (6) and the conditional BEP of Eq. (12), the BEP for symbol \({s_1}\) is evaluated and given as

$$P_{{{s_1}}}^{{sr}}(e)=\frac{1}{{12}}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{{V_i}{\rho _s}+{\beta _{isr}}}})+\frac{1}{4}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}})$$

13

Where the parameters \({\alpha _{isr}}\) and \({\beta _{isr}}\) are defined in the similar way as \({\alpha _i}\) and \({\beta _i}\). \({m_{sr}}\) is the shaping parameter. Now, end to end bit error probability (BEP) of the system can be evaluated by substituting the Eq. (10) and Eq. (13) in Eq. (8) and the obtained result and Eq. (7) in Eq. (3), we get final Eq. (A) for non-integer values of \({m_{rd}}\) as given below

\({P_{e2e}}(e)=\frac{1}{2}[[\frac{1}{{12}}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{{V_i}{\rho _s}+{\beta _{isr}}}})+\frac{1}{4}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}})] \times\) \(\begin{gathered} [1 - \frac{1}{{2\sqrt \pi }}\frac{{\sqrt P }}{{{{(1+P)}^{{m_{rd}}+0.5}}}}\frac{{\Gamma ({m_{rd}}+0.5)}}{{\Gamma ({m_{rd}}+1)}}{}_{2}{F_1}(1;{m_{rd}}+0.5;{m_{rd}}+1;\frac{1}{{1+P}})]+ \hfill \\ [\frac{1}{{2\sqrt \pi }}\frac{{\sqrt P }}{{{{(1+P)}^{{m_{rd}}+0.5}}}}\frac{{\Gamma ({m_{rd}}+0.5)}}{{\Gamma ({m_{rd}}+1)}}{}_{2}{F_1}(1;{m_{rd}}+0.5;{m_{rd}}+1;\frac{1}{{1+P}})] \times \hfill \\ [1 - \frac{1}{{12}}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}) - \frac{1}{4}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}})]+ \hfill \\ [\frac{1}{{12}}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}})+\frac{1}{4}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}})]] \hfill \\ \end{gathered}\)

And for the integer values of \({m_{rd}}\), the end to end bit error probability (BEP) of the system can be calculated by substituting the Eq. (11) and Eq. (13) in Eq. (8) and the obtained result and Eq. (7) in Eq. (3), we get final Eq. (B) as follows

\(\begin{gathered} {P_{e2e}}(e)=\frac{1}{2}[[\frac{1}{{12}}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{{V_i}{\rho _s}+{\beta _{isr}}}})+\frac{1}{4}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}})] \times \hfill \\ [\frac{1}{2}+\frac{{{\mu ^2}(P)}}{2}\sum\limits_{{j=0}}^{{{m_{rd}} - 1}} {\left( {_{j}^{{2j}}} \right)} {(\frac{{1 - {\mu ^2}(P)}}{4})^j}]+[\frac{1}{2} - \frac{{{\mu ^2}(P)}}{2}\sum\limits_{{j=0}}^{{{m_{rd}} - 1}} {\left( {_{j}^{{2j}}} \right)} {(\frac{{1 - {\mu ^2}(P)}}{4})^j}] \times \hfill \\ [1 - \frac{1}{{12}}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{{V_i}{\rho _s}+{\beta _{isr}}}}) - \frac{1}{4}\sum\limits_{{i=1}}^{N} {\frac{{{\eta _i}{\alpha _{isr}}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}}} {}_{2}{F_1}(1;{m_{sr}};1;\frac{{{\delta _i}}}{{\frac{4}{3}{V_i}{\rho _s}+{\beta _{isr}}}})]+ \hfill \\ [\frac{1}{{12}}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{{\upsilon _i}{\rho _s}+{\beta _i}}})+\frac{1}{4}\sum\limits_{{i=1}}^{M} {\frac{{{P_i}{\alpha _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}}} {}_{2}{F_1}(1;{m_i};1;\frac{{{\delta _i}}}{{\frac{4}{3}{\upsilon _i}{\rho _s}+{\beta _i}}})]] \hfill \\ \end{gathered}\)

Thus, it can be seen that the total BEP obtained is the sum of individual BEPs of symbols \({s_1}\) and \({s_2}\). The overall error performance can be further improved by considering the effect of elevation angle. The elevation angle is the angle under which the satellite is observed for fair transmission and reception of signal. The concept of elevation angle is incorporated in our proposed system to improve the transmission by considering Line of Sight between satellite and ground stations (destination and relay). The effect of elevation angle is observed over total Bit Error Probability (BEP). But not all the angles result in successful transmission, therefore only a suitable range defined in [15] and [38] is useful and defined as 20°≤ θ ≤ 80° and the parameters depending on θ are expressed as

\(\begin{gathered} {b_0}(\theta )= - 4.7943 \times {10^{ - 8}}{\theta ^3}+5.5784 \times {10^{ - 6}}{\theta ^2} - 2.1344 \times {10^{ - 4}}\theta +3.2710 \times {10^{ - 2}} \hfill \\ m(\theta )=6.3739 \times {10^{ - 5}}{\theta ^3}+5.8533 \times {10^{ - 4}}{\theta ^2} - 1.5973 \times {10^{ - 1}}\theta +3.5156 \hfill \\ \Omega (\theta )=1.4428 \times {10^{ - 5}}{\theta ^3} - 2.3798 \times {10^{ - 3}}{\theta ^2}+1.2702 \times {10^{ - 1}}\theta - 1.4864 \hfill \\ \end{gathered}\)

where \({b_0}(\theta )\) is power of multipath component, \(m(\theta )\) is shaping parameter and \(\Omega (\theta )\) is average power of the LOS component of proposed system as a function of θ. Their effect is observed over derived expression of total BEP of proposed system for integer and non-integer cases shown in the respective plots.

Now, let us consider power sharing coefficient as \(\beta\) therefore the total power P’ can be shared as \({P_1}\) = \(\beta\) P’ and \({P_2}\) = (1- \(\beta\)) P'. The function of BEP is given as \(F=f(\alpha ,\beta )\) We employed machine learning (ML) algorithm to obtain an appropriate value of \(\alpha\) and \(\beta\) for better Bit Error Probability (BEP). The training of model is done offline consequently reducing the computational complexity and improving the online implementation. A three tier (input, hidden and output layer), well connected neural network (NN) model is built to obtain effective values of \(\alpha\), \(\beta\) pair. The neural network used 10 hidden layers to train the model. The ML algorithm that has been used is Levenberg-Marquardt algorithm. The NN model is trained offline with different set of values to predict the appropriate values of the pair. For epoch count of 1000, 12 iterations and 7 validation checks, the performance of the model has been analyzed. The model performed well in training with the performance metrics 2.287e-8 MSE, regression 0.98189 and gradient 8.56e-8. Then, the trained network was tested for different inputs exclusive of training dataset and the test performance metrics otained is 2.2441e-8 MSE, regression 0.9801 and gradient 8.369e-8.