The ground states for the non-cooperative autonomous systems involving the fractional Laplacian

The aim of this paper is to study the the following non-cooperative autonomous systems involving the fractional Laplacian


Introduction and main results
In these last years a great deal of work has devoted to the study of the weak solutions for the following fractional Schrödinger systems where s ∈ (0, 1) with N > 2s and a(x), f, g satisfying appropriate conditions in order to use a variational method.
The fractional Schrödinger equations are formulated by Laskin [1], they are functional equations of fractional quantum mechanics. Equations involving the fractional Laplacian have attracted much attention in recent years, they appear in several areas such as optimization, In the case of the standard Laplacian operator (s = 1, local case), the existence of solution for the Schrödinger systems have been studied, and relatively complete methods have been formed. However, for the autonomous fractional Schrödinger systems, there are only some literature on the existence of positive solutions, for example, see [19][20][21].
In [19], X.M. He, Marco Squassina and W.M. Zou concerned with the multiplicity of positive solutions for the following elliptic system involving the fractional Laplacian where Ω ⊂ R N is a smooth bounded domain, λ, µ > 0, 1 < q < 2 and α > 1, β > 1 satisfy α + β = 2 * s = 2N N −2s , s ∈ (0, 1). They proved that the system admits at least two positive solutions when the pair of parameters (λ, µ) belongs to a suitable subset of R 2 , with the help of the Nehari manifold.
Edir Junior Ferreira Leite and Marcos Montenegro [20] studied the following strongly coupled systems (−∆) s u = v p in Ω, (−∆) t v = u q in Ω, (1.2) in non-variational form involving fractional Laplace operators, where Ω is a smooth bounded open subset of R N , n 2, 0 < s, t < 1, p, q > 0. They proved Liouville type theorems and by mean of the blow-up method, established a priori bounds of positive solutions. By using those latter, they then derived the existence of positive solutions through topological methods.
Alexander Quaas, Aliang Xia [21] studied the nonexistence of solutions for fractional elliptic problems (1.2) in the case of s = t via a monotonicity result which obtained by the method of moving planes with an improved Aleksandrow-Bakelman-Pucci type estimate for the fractional Laplacian in unbounded domain.
In line with the above works, it worth mentioning that the nonlinearities are cooperative type in [19], hence the energy functions corresponding to them can be proved to have mountain pass structure and Nehari manifold arguments can be used. However, another question arises: for the following more general non-cooperative type fractional systems: where λ > 0 is any constant, whether the ground states and their properties can be obtained? Answering this question constitutes the goal of this paper.
As for the non-cooperative type system (1.2), the existence of solutions was derived through some topological methods in [20][21], but it is not known whether this solution is a least energy solution.
In this paper, motivated by the references mentioned above, we focus our attention on the non-cooperative type fractional systems (1.3) with more general power-type nonlinearities having super-linear and subcritical growth at infinity. The corresponding energy functional is strongly indefinite, that is, the quadratic part of the energy functional has no longer a positive sign, the problems become rather complicated, mathematically. The main purpose of this paper is to obtain the existence of ground states of the system (1.3) through variational method, which are different from the ones of [20] and [21]. Furthermore, regularity and symmetry of solutions are also discussed. By constructing suitable comparison functions based on the Bessel Kernel, we find out that the ground states of (1.3) have a power type decay at infinity.
Since we are interested in positive solutions, we assume the continuous functions f, g satisfy the following conditions: (H 2 ) there exist real numbers l 1 , l 2 > 0 and p, q > 2 such that 1 for every t ∈ R and similarly for g; (H 4 ) for every µ > 0 there exists C µ > 0 such that Similar assumptions have been introduced in [22]. When s = 1, the system (1.3) gives back the classical Schrödinger system. It has been studied by Djairo G. De Figueiredo and Jianfu Yang in [23].
The main result of this paper is stated as follows: (iii) (Decay estimates) there exist some constants 0 < C 1 C 2 such that for all |x| R, where R > 0 is an appropriate constant; (iv) (Symmetry) If we further assume that f, g satisfy (H 5 ), then all positive solutions of (1.3) are radially symmetric with respect to the origin, and u ′ (r), v ′ (r) < 0 for r = |x|.
A typical example of functions verifying the assumption (H 1 ) − (H 5 ) is given by f (t) = l 1 |t| p−2 t, g(t) = l 2 |t| q−2 t with l 1 , l 2 > 0 and p, q > 2 such that 1 Remark 1.2. (i) Although we have a variational problem, the functional associated to it is strongly indefinite, that is, compared to the single equation case and cooperative type systems, the quadratic part of the energy functional has no longer a positive sign, and so we have to recourse to the "Indefinite Functional Theorem" introduced by Benci and Rabinowitiz in [24] which is an extension of both the mountain-pass theorem and the saddle point theorem.
(ii) Theorem 1.1 is a counterpart, precisely a generalization of the main results obtained in [23,25], where the authors considered the case when s = 1. Compared with the operator −∆, which is local, main difficulty of studying of the system (1.3) in the non-local character of the involved operator (−∆) s . To overcome this difficulty we use the Caffarevli-Silestre extension method. This allows us to apply variational techniques to these kinds of problems. However, we would like to point out the ideas as in [23], Liouville-type theorems, classical blow-up arguments in [25] and the moving planes method in [26] are not suitable absolutely for our situation because of emergence of nonlocal operators, some estimates and analysis are more delicate.
(iii) The assumptions on p and q: p, q > 2 and 1 p + 1 q > N −2s N are natural to the system (1.3), which are more general than the ones: 2 < p, q < 2 * s := 2N N −2s . But, the associated functional may not to be well defined in the space H s (R N )×H s (R N ) under the the assumptions, because it may happen that say p < 2 * s < q, here 2 * s denotes the fractional critical Sobolev exponent. However, as explained in Sect.4, we only have to prove Theorem 1.1 in the case 2 < p = q < 2 * s . In fact, given n ∈ N, we can define the truncated functions, where the coefficients are chosen in such a way that g n is C 1 . Thus, in view of (H 2 ), we see that A n = ( l2 p−1 + o(1)) · n q−p , B n = ( l2(p−q) (p−1)(q−1) + o(1)) · n q−1 . In Sect.4, we consider the truncated problem and obtain the existence of the solutions (u n , v n ) to the corresponding system. Then we show that u n ∞ , v n ∞ C for some C > 0 independent of n, therefore they solve the original problem (1.3) if n is taken sufficiently large. Thus, in Sect.2-3, we assume that 2 < p = q < 2 * s .
This paper is organized as follows. In Sect.2, we review certain notations related to the fractional Laplacian and describe the appropriate functional setting for the system (1.3). Sect.3 is devoted to studying the autonomous system (1.3) and giving the proof of main results. In Sect.4, we will show that the solutions to the truncated problem are bounded in L ∞ (R N ), for this, some Liouville-type theorems need be established.
Notatations Here we list some notations which will be used throughout the paper.
• The letters C, C i , i = 0, 1, 2, · · · , will be repeatedly used to denote various positive constants whose exact values are irrelevant.

Preliminaries
In this section, we collect some preliminary results for the fractional Laplacian. Recall that for s ∈ (0, 1), the fractional space H s (R N ) is defined by The fractional Laplacian (−∆) s of a smooth function u : where F denotes the Fourier transform, that is for function ω in the Schwartz class. Also, (−∆) s u can be equivalently represented as Also, we have from [4] that for all u ∈ H s (R N ). For N > 2s, we also know that, for any p ∈ [2, 2 * s ], there exists C p > 0 such that To deal with the nonlocal system (1.3), we will use a method due to Caffarilli and Silvestre in [5] to study a corresponding extension problem, which allows us to investigate the system (1.3) by studying a local problem via classical variational methods. Recall that for In [12] it is proved that where k s is a normalization constant.
Remarking (2.1), we introduce the function space X s (R N +1 + ) that is defined as the It is a Hilbert space endowed with the inner product With the constant k s , we have the extension operator to be an isometry between H s (R N ) and On the other hand, for a function U ∈ X s (R N +1 + ), we will denote its trace on R N × {0} as T r(U ). This trace operator is also well defined and it satisfies For convenience, we will use the following notations: With the above extension (2.1), we can reformulate our problem We are looking for a positive solution Consider the Euler-Lagrange function associated to (2.2) given by dξ, which is C 2 well defined over the Hilbert space E when 2 < p = q < 2 * s , and moreover, the critical points of J λ correspond to the weak solutions of (2.2). If (U, V ) is a solution of (2.2), then the trace (u, v) = (T r(U ), T r(V )) = (U (x, 0), V (x, 0)) is a solution of (1.3). The converse is also true. Therefore, both formulations are equivalent. Moreover, we have that X s (R N +1 + ) is local compactly embedding in L 2 (R N +1 + , y 1−2s ), the weight Lebesgue space endowed with the norm For the proof of the regularity, we utilize the Sobolev inequality on weighted spaces which appeared in [27].
holds for any function U whose support is contained in Ω whenever the right-hand side is well-defined.
What as follows, we recall that the definitions of the relative Morse index and solutions having finite index.
Let E be a real Hilbert space, for a closed subspace of V ⊂ E, we denote by P V the orthogonal projection onto V and by V ⊥ the orthogonal complement of V . Following [28] and [29], we say that the closed subspaces V, W of E are commensurable if P V ⊥ P W and P W ⊥ P V are compact operators.
If V and W are commensurable, the relative dimension of W with respect to V is defined as Commensurability guarantees that both terms in the above formula are finite.
We will also borrow the definition of solutions having finite index as defined in [30].
The proof of Theorem 1.1

The existence of weak solutions
In this subsection, we prove Theorem 1.1 on the existence of weak solutions of the system (1.3), in view of hypothesis 2 < p = q < 2 * s , we work with the space E := X s (R N +1 for any Ψ, Φ ∈ X s (R N +1 + ). So, the critical points of J λ satisfy the equations
It can be observed that the following orthogonal splitting holds ). So that, denoting by Q λ the quadratic term of the energy functional J λ , namely We have that Q λ is positive definite(resp, negative definite) in E + (resp, in E − ). Therefore, J λ is a indefinite functional, we have to refer the "Indefinite functional theorem" introduced by Benci and Rabinowitiz in [22] to obtain a nontrivial critical point of J λ .
The following Lemma will play a significant role in the sequel whose proof is similar with [31] Lemma 2.1, so we omit it.
Lemma 3.1. Let (U n , V n ) be a (P S) c sequence for the functional J λ , namely The proof of Theorem 1.1(i). By the assumptions (H 1 ) − (H 3 ), it is easy to check that the energy function J λ possesses the linking structure, that is, , for some small r > 0, ρ > 0; moreover, if r > 0 is sufficiently large and e = (e 1 , e 2 ) ∈ E, e 1 > 0, e 2 > 0, then Then, according to "Indefinite functional theorem" [22], J λ has a (P S) c sequence {(U n , V n )} ⊂ E, where 0 < ρ c sup E − ⊕R + e J λ , using Lemma 3.1, (U n , V n ) are bounded in E and may assume (U n , V n ) ⇀ (U, V ) as n → ∞, then clearly J ′ λ (U, V ) = 0. Next we need to show that there exists a non-trivial critical point. For this purpose, by concentration compact principle [26], it is possible to find a sequence {x n } ⊂ R N and some constants R > 0 and β > 0 such that Indeed, assuming the contrary, we have But then, for large n and some constants a > 0 and C 1 , C 2 > 0, we have proving a contradiction, since c > 0. Now we define U n (x, y) = U n (x + x n , y), V n (x, y) = V n (x + x n , y), then ( U n , V n ) ⇀ (U 0 , V 0 ) = (0, 0) is a non-trivial critical point of J λ .

The regularity and decay estimates of weak solutions
In this subsection, we prove Theorem 1.1(ii)(iii) stated in the Introduction.
The proof of Theorem 1.1(ii). Let (U, V ) is the positive solution obtained in Theorem 1.1(i). Choose a smooth function η ∈ C ∞ 0 (R N +1 satisfying η = 1 on B + N +1 (0, R) and |∇η| 2 |η|. Multiplying the both side of the first equality of (2.2) by η 2 U β , here β = 1 + 2 N , we discover that For easy reference, let us denote the three above integrals by I 1 , I 2 and I 3 , in the order they appear. We now estimate I 1 , employing the Young's inequality: ab a 2 2δ + δb 2 2 for δ = β 2 to get

(3.4)
On the other hand, applying the identity This gives Combing this with (3.3) (3.4), I 1 I 3 and using the Sobolev trace inequality, we deduce that
Next, we study the decay behavior of positive solutions of (1.3). Before starting to give the proof, let us consider for h ∈ L 2 (R N ) the equation Then in terms of Fourier transform, this problem, for φ ∈ L 2 , reads and has a unique solution φ ∈ H s (R N ) given by the convolution where K is the fundamental solution of (−∆) s + 1 or called the Bessel kernel, Let us recall the main properties of the kernel K that are stated for instance in [33], which are useful in what follows.
We have that (i) K is positive, radially symmetric and smooth in R N \{0}. Moreover, it is non-increasing as a function of r = |x|; (ii) For appropriate positive constants C 1 and C 2 , (iii) There is a constant C > 0 such that The proof of Theorem 1.1(iii). Firstly, we recall the following Claims in [33] that: There are continuous functions w 1 , w 2 in R N satisfying, resp.
By u(x), v(x) → 0 as |x| → ∞ and the condition (H 1 ), we conclude that there is a large R 1 > 0 such that Moreover, by the continuity of solution (u, v) and w 2 , there exists C > 0 such that Therefore, Using comparison arguments, we get that Since (u, v) is a positive solution, then On the other hand, by the continuity of (u, v) and w 1 , there exist constants C 2 , which imply that By the similar comparison arguments, we conclude the second inequality.

The radial symmetry of positive solutions
In this subsection, we show that the positive solutions (u(x), v(x)) of the system (1.3) are radial symmetry . Here we give the proof based on the moving planes method as developed recently in [33], where the radial symmetry and monotonicity properties of the kernel K play a key role. The approach is different from the usual moving planes technique originated in [26] for the case s = 1.
We consider, initially, the planes parallel to the plane x 1 = 0. For each real α, we define x α = (2α − x 1 , x 2 , · · · , x n ) is the reflection of x on the plane T α .

Lemma 3.2. We have
Proof. It follows from the variable substitution and the radially symmetry of K that, Therefore, The similar to the computation of V α (x).
Indeed, firstly Σ 1 is bounded, since V α decay to zero at infinity. If x ∈ Σ 1 , using the fact that |ξ − x α | |ξ − x| in ξ ∈ Σ α , K is decreasing, f is increasing and Lemma 3.3, we have Thus, by Lemma 5.2 in [33] for q = m and r = 1 2 m with m large, such that m > N s and mτ 2, we obtain . Using the Hölder inequality, we get Then, for sufficiently small ε > 0, there is α large enough (negative) such that v α τ L mτ (Σ1) < ε, which conclude together with u, v ∈ L r (r 2) that (3.14) Consequently Σ 1 = ∅. If not, we have from (3.14) that as α → −∞, ε → 0, However, assume that there is x 0 ∈ Σ 1 such that V α (x 0 ) > 0, which implies from (3.12) that u(x 0 ) > 0. By the continuity of u, there is a small neighborhood B N (x 0 , δ) ⊂ Σ 1 such that In conclusion, there exists α * < +∞ such that for all α α * , U α (x) 0 and V α (x) 0 for all x ∈ Σ α . Now Proposition 3.4 allows us to define α 0 = sup{α : Proof. By the continuity, we see that U α0 (x) 0 and V α0 (x) 0 for all x ∈ Σ α0 . It follows from (3.12) and (3.13) that U α0 = 0 in Σ α0 if and only if V α0 = 0. So, by contradiction, that U α0 = 0 and thus also V α0 = 0. Observe from (3.12) and (3.13) that The sequence {x k } and {y k } are bounded, since U α k and V α k decay to zero at infinity, so we may assume that x k → x with x ∈ Σ α0 or y k → y with y ∈ Σ α0 . By the continuity, we have u(x) u α0 (x) or v(y) v α0 (y), which contradict with U α0 (x) < 0 and V α0 (y) < 0.
The proof of Theorem 1.1(iv). By the translation, we may say that α 0 = 0. Thus, we have that u, v are symmetric about the x 1 -axis, i.e. u( Using the same approach in any direction implies that u, v are radially symmetric. 4 The case p = q In Sect.3, we have proved Theorem 1.1 except that we have worked with a truncated problem, as explained in Remark 1.2. The full statement of Theorem 1.1 will be established once we prove uniform bounds in L ∞ of the solutions constructed so far. So, in this section, let us suppose that p, q > 2 are such that 1 p + 1 q > N −2s N with say, 2 < p < 2 * s and p < q. Given n ∈ N, we can define the truncated functions, g n (t) = g(t), t n, A n t p−1 + B n , t > n, where the coefficients are chosen in such a way that g n is C 1 . Thus, in view of (H 2 ), we see that A n = ( l2 p−1 + o(1)) · n q−p , B n = ( l2(p−q) (p−1)(q−1) + o(1)) · n q−1 . The energy functionals associated to the modified problem of (2.2) are given by where G n is the primitive of g n . They are C 2 functionals defined over the Hilbert space E. The critical points of J n correspond to weak solutions of the modified problem For a fixed n, thanks to the Section 3, there are positive solutions (u n , v n ) of the modified problem (4.1) satisfying the conclusion of Theorem 1.1.  The proof of Theorem 4.2 is based on the following simple fact whose proof is the same as Lemma 1.2 in [24].
Next, we will prove a Liouville-type theorem which is crucial for the proof of Theorem 4.2.
Proof. (i) It is obvious.
Making use of well-known Pohozȃve-Rellich type identity, By the condition (c), we deduce that Once Proposition 4.4 is settled, we may use the classical blow-up argument to give the proof of Theorem 4.2 that is the similar with [25], we omit it.

Availability of data and material
Not applicable.

Competing interests
The authors declare that they have no competing interests.