Auto-correlation functions for unitary groups

We compute the auto-correlations functions of order $m\ge 1$ for the characteristic polynomials of random matrices from certain subgroups of the unitary groups $\U(2)$ and $\U(3)$ by applying branching rules. These subgroups can be understood as analogs of Sato--Tate groups of $\USp(4)$ in our previous paper. This computation yields symmetric polynomial identities with $m$-variables involving irreducible characters of $\U(m)$ for all $m \ge 1$ in an explicit, uniform way.

1. Introduction 1.1.Auto-correlation functions.The distribution of characteristic polynomials of random matrices has been of great interest for their applications in mathematical physics and number theory.Since Keating and Snaith [KS00b,KS00a] computed averages of characteristic polynomials of random matrices in 2002 motivated in part by connections to number theory and in part by the importance of these averages in quantum chaos [AS95], it has become clear that averages of characteristic polynomials are fundamental for random matrix models [BDS03, BS06, BH00, BH01, FS02a, FS02b, MN01].
On the way of these developments, the auto-correlation functions of the distributions of characteristic polynomials in the compact classical groups were computed by Conrey, Farmer, Keating, Rubinstein and Snaith [CFK + 03, CFK + 05] and by Conrey, Farmer and Zirnbauer [CFZ08,CFZ05].Later, Bump and Gamburd [BG06] obtained different derivations of the formulas starting from (analogues of) the dual Cauchy identity and adopting a representation-theoretic method.Their results show that the auto-correlation functions are actually combinations of characters of classical groups.
1.2.Sato-Tate groups.The celebrated Sato-Tate conjecture for elliptic curves (i.e.genus 1 curves) predicts that the distribution of Euler factors of an elliptic curve is the same as the distribution of characteristic polynomials of random matrices from SU(2), U(1) or N (U(1)), where N (U(1)) is the normalizer of U(1) in SU(2).The conjecture is proven (under some conditions) by the works of R. Taylor, jointly with L. Clozel, M. Harris, and N. Shepherd-Barron [CHT08,Tay08,HSBT10].For curves of higher genera, J.-P.Serre, N.Katz and P. Sarnak [Ser94,KS99] proposed a generalized Sato-Tate conjecture.Pursuing this direction, K. S. Kedlaya and A. V. Sutherland [KS09] and later together with F. Fité and V. Rotger [FKRS12] made a list of 52 compact subgroups of USp(4) called Sato-Tate groups that would classify all the distributions of Euler factors for abelian surfaces.Recently, Fité, Kedlaya and Sutherland showed that there are 410 Sato-Tate groups for abelian threefolds [FKS21].
1.3.Our previous work.Inspired by the approach of Bump and Gamburd, in a previous paper [LO20], the authors computed the auto-correlation functions of characteristic polynomials for Sato-Tate groups H ≤ USp(4), which appear in the generalized Sato-Tate conjecture for genus 2 curves.The result of [LO20] can be described as follows.Let H ≤ USp(4) be a Sato-Tate group.Then, for arbitrary m ∈ Z ≥1 , we have where the coefficients m (b+2z,b) are the multiplicities of the trivial representation in the restrictions χ Sp(4) (b+2z,b) H and are explicitly given in the paper [LO20] for all the Sato-Tate groups of abelian surfaces.Exploiting the representation-theoretic meaning of m (b+2z,b) , the authors obtained this result by establishing branching rules for χ Sp(4) (b+2z,b) H .Moreover, since most of the Sato-Tate groups are disconnected, we can decompose the integral in (1.3) according to coset decompositions, and find that the characteristic polynomials over some cosets are independent of the elements of the cosets.Combining this observation with the computations of branching rules, we obtain families of non-trivial identities of irreducible characters of Sp(2m, C) for all m ∈ Z ≥1 .For example, we have, for any m ∈ Z ≥1 , where ψ 4 (z, b) is defined on the congruence classes of z and b modulo 4 by the table Notice that the irreducible characters χ Sp(2m) λ are symmetric functions with the number of terms growing very fast as m increases, but that the coefficients ψ 4 (z, b) are independent of m.In order to produce the left-hand side of the identities, there must be systematic cancelations in the right-hand side.
1.4.Schur functions.The Schur functions S λ form the distinguished self-dual basis of the ring of symmetric functions.They appear naturally in representation theory, algebraic combinatorics, enumerative combinatorics, algebraic geometry and quantum physics.In particular, (i) every Schur function corresponds to an irreducible character of the unitary group, which implies the ring of symmetric functions form the Grothendieck ring for unitary groups, (ii) it has various combinatorial realizations in various aspects of algebraic combinatorics.Hence understanding the properties of Schur functions takes a center stage in these research areas.One of the key features of understanding Schur functions is how other symmetric functions can be expressed in the basis of Schur functions; that is, computing the coefficients of S λ , called Schur coefficients, in the expansions.One of the well-known instances is the (inverse of) Kostka matrix, which can be understood as Schur coefficients for monomial (complete) symmetric functions in algebraic combinatorics, and as composition multiplicities of V (λ) in the permutation representation W (λ) in representation theory.Recall that the (inverse of) Kostka matrix is a uni-upper triangular matrix with integer coefficients and entries in the Kostka matrix has a description in terms of semistandard Young tableaux.However, the closed-form formulas for entries in the (inverse of) Kostka matrix are not available in general.Another well-known instance is Littlewood-Richardson rule, which can be understood as Schur coefficients for a product of two Schur functions in algebraic combinatorics, and as composition multiplicities of V (λ)'s in the tensor product V (µ) ⊗ V (η) in representation theory.

Main Result.
In what follows, we describe the main result of this paper and its application.
(M) We compute explicitly the auto-correlation functions for H = U(1) ≤ U(g) (g = 2, 3) and for the subgroups H ≤ U(g) (g = 2, 3) defined in (1.5) below.Namely, for any m ∈ Z ≥1 , we obtain where the coefficients m λ ′ (H) are completely determined.Here S U(m) λ (x) denotes the character of the irreducible representation V (λ) of the unitary group U(m), which are Schur functions.
(A) As an application of the main result, we give closed-form formulas of Schur coefficients c λ g 1 ,g 2 for special infinite families of symmetric functions t (m) g 1 ,g 2 (x), which also have simple expansions in terms of monomial symmetric functions {m (m) λ (x)}.That is, for any m ∈ Z ≥1 and 1 ≤ g 1 + g 2 ≤ 3, we obtain where the Schur coefficients c λ g 1 ,g 2 are given in closed-form formulas.Thus c λ g 1 ,g 2 can be understood as a simple combination of entries in the inverse of Kostka matrix as t (m) g 1 ,g 2 (x) has an expansion in m (m) λ (x) with coefficients from {1, 0, −1}.Let us explain the main result and its application in more detail.To adopt the same strategy as in USp(m), we first introduce several subgroups H g , H ′ g,4 and H g,4 of U(g) (g = 2, 3) as follows, which play the role of Sato-Tate groups in [LO20]: (1.5) where Considering these subgroups in U(g) (g = 2, 3), we present the main result more precisely.
Main Theorem.Let H ≤ U(g) (g = 2, 3) be U(1) or a group in (1.5).Then, for any m ∈ Z ≥1 , we have where the coefficient m λ ′ (H) are the composition multiplicities of the trivial representation in the restriction χ λ ′ | H and are explicitly given in Theorem 3.1, Theorem 3.4, Theorem 4.6 and Theorem 4.14.This theorem can be interpreted as a result on branching rules from U(g) to H in representation theory.To prove the above theorem, we analyze the representation structure of V (λ ′ ) over U(g) with respect to H and determine a certain list of linearly independent subsets in V (λ ′ ).
For an application of Main Theorem, we observe (i) H's are decomposed into disconnected cosets and (ii) the characteristic polynomials in the underlined cosets of H are independent of elements of the coset.These observations enable us to obtain closed-form identities involving Schur functions.
Application.For arbitrary m ∈ Z ≥1 , we have the following identities : (derived from H 3,4 and H ′ 3,4 , Theorem 4.14), where (i) τ (z, b) ∈ {0, 1} is defined on the congruence of z and b modulo 4 as in the following table ) depends on the congruence of z and b modulo 2, 4 and given in Corollary 4.13.
Combining the identities in Application and replacing x i with −x i , we obtain all the other identities in (1.4) (see (3.6), Corollary 4.17, Corollary 4.18 and Remark 4.19).
Note that the identities obtained involves negative Schur coefficients and the number of terms in Schur functions grows enormously as m increases.However, our result implies that such combinations of Schur functions have miraculous cancellations and yield symmetric functions with positive coefficients.Furthermore, the identities state that the Schur coefficients do not depend on m (see Example 3.3, Example 4.8 and Example 4.16).These identities seem intriguing from the viewpoint of representation theory and algebraic combinatorics.It might have been difficult for us to expect that such identities exist, without regard to the auto-correlation functions and branching rules of the newly introduced groups in (1.5).
1.6.Organization of the paper.In Section 2, we review the necessary backgrounds for autocorrelation functions and the dual Cauchy identity.In Section 3, we compute the auto-correlation functions of H's of U(2) and establish the corresponding identities involving Schur functions.In Section 4, we present the auto-correlation functions of H's of U(3) and consider the corresponding identities by analyzing the representation structure of V (λ) with respect to H's.But we postpone a part of the proof to Section 5, which is devoted to determine the composition multiplicities of trivial representations in χ U(g) λ ′ | H 's. This amounts to the proof for U(3).We convert this problem into counting the pairs of integers encoding certain information from representation theory.By expressing the cardinalities as closed-form formulas, we complete the proof.
Convention 1.1.Throughout this paper, we keep the following conventions.
(i) For a statement P , the notation δ(P ) is equal to 1 or 0 according to whether P is true or not.

Dual Cauchy identity and auto-correlation functions
In this section, we fix notations and review the dual Cauchy identity and establish a general formula for the auto-correlation functions of characteristic polynomials.
, we define a partial order λ µ if k ≤ l and λ i ≤ µ i for all i = 1, 2, . . ., k.A partition λ corresponds to a Young diagram, and the transpose λ ′ is defined to be the partition corresponding to the transpose of the Young diagram of λ.
Let U(g) be the unitary group for g ≥ 1.For a partition λ with at most g parts, let S U(g) λ be the Schur function associated with λ.It is well-known that S U(g) λ is the irreducible character of U(g) with highest weight λ.Denote by V g (λ) the representation space of S U(g) λ .When g is clear from the context, we will simply write V (λ).
Definition 2.1.Let H be a closed subgroup of U(g).Define m λ (H) to be the multiplicity of the trivial representation 1 H in the restriction of V (λ) to H.
Proposition 2.3.Let H be a subgroup of U(g) and dγ be the probability Haar measure on H.Then, for each m ≥ 1, the auto-correlation function for the distribution of characteristic polynomials of H is given by Proof.Let t 1 , . . ., t g be the eigenvalues of γ ∈ H. Since we have From Schur orthogonality (for example, [Bum13]), the integral H S U(g) to H, which is equal to m λ ′ (H) by Definition 2.1.This establishes the first identity.The second identity follows from replacing x i with −x i in the first identity.
Our assertion follows from (2.1) and (2.2) by comparing the right-hand sides, since the Schur functions are linearly independent.
We recall the classical branching rule from U(g) to U(g − 1) for g ≥ 2.

Identities for g = 2
In this section, we consider some disconnected subgroups H of U(2) and compute m λ ′ (H) for λ (2 m ) in (2.1).This computation produces identities involving Schur functions S U(m) λ for all m ∈ Z ≥1 .
Then we have Furthermore, for any m ∈ Z ≥1 , we have where we set j := (a − b)/2.Proof.For any γ = t 0 0 t −1 ∈ U(1), we have det(I + xJγ) = 1 + x 2 .Let du = du(γ) be the probability Haar measure on U(1) ≤ U(2).By Proposition 2.3 and (3.1), we have Let v 1 = (1, 0) and v 2 = (0, 1) be the standard unit vectors of V := C 2 , and consider the standard representation of U(2) on V , and let det be the one-dimensional representation of U(2) defined by the determinant.For Thus the trivial U(1)-module is generated by v j 1 v j 2 only when a − b is even, where we set j := (a − b)/2.In other word, we have Note that, when λ ′ = (b + 2j, b), we have λ = (2 b , 1 2j ).Now it follows from (3.3) that m i=1 Remark 3.2.The left hand side of (3.2) can be written as a simple combination of the monomial symmetric functions.Namely, we have where m (m) λ denote the monomial symmetric functions in m-variables associated with partitions λ with ℓ(λ) ≤ m.
and denote by H 2,4 the subgroup of U(2) generated by U(1), J and ζ ζ ζ 4 .That is, we define Then we have Moreover, for any m ∈ Z ≥1 , we have where we set j := (a − b)/2 as before.
Proof.Let λ ′ = (a, b) (m 2 ).We keep the notations in the proof of Theorem 3.1 for As observed in the proof of Theorem 3.1, the vector v j 1 v j 2 is fixed by J if and only if a − b ≡ 4 0. Therefore we have for all γ ∈ U(1), we have where we write ∆(γ) = m i=1 det(I + x i γ) for convenience.Applying Proposition 2.3 to the integrals, we obtain Remark 3.5.
(1) The identity (3.5) can be derived from (3.2).We will consider the alternate proof in Section 3.3.
(2) As in Remark (3.2), we observe that the left hand side of (3.5) is a simple combination of the monomial symmetric functions in m-variables: (2 2k ) (x).
Using a similar argument, we obtain two more identities.
Remark 3.7.The above use of Pieri's rule may not be applicable, in general, if one can try to obtain closed-form formulas for g ≥ 3.For instance, based on Theorem 3.1 about g = 2, one can check the formula in Theorem 4.6 about g = 3 below using Pieri's rule, for first several small values of m.But, when g ≥ 3, obtaining closed-form formula for the coefficient of S U(m) λ , m ≥ 1, seems not easy in this approach.

Identities for g = 3
In this section, we consider some disconnected subgroups H of U(3) and compute m λ ′ (H) for λ (3 m ) in (2.1).As with the case g = 2, our computation yields identities involving Schur functions S U(m) λ for all m ∈ Z ≥1 .
Then one can easily check that J normalizes U(1), and for all γ ∈ U(1).
We prove a useful lemma.
Lemma 4.1.For any k ∈ Z ≥0 , we have Proof.Let det be the one-dimensional representation of U(3) defined by the determinant.Then we have Since det(A) = 1 for any A ∈ U(1) and det(J) = 1, the assertion follows.
Let V and W = 2 V be the fundamental representations of U(3).Take a basis Write Then {w 12 , w 13 , w 23 } is a basis for W , and we have Jw 12 = w 12 , Jw 13 = −w 23 and Jw 23 = w 13 .
(4.6) Proposition 4.4.For a partition λ ′ = (a, b, 0), the multiplicity m λ ′ (U(1)) of the trivial representation in V (λ ′ )| U(1) is equal to the cardinality of the set and the multiplicity m λ ′ (H 3 ) of the trivial representation in V (λ ′ )| H 3 is equal to the cardinality of the set Proof.From the embeddings of U(1) into U(3), we see that the multiplicity m λ ′ (U(1)) is equal to the number of linearly independent vectors in V (λ ′ ) with weight µ such that µ(h 1 ) = 0.
Similarly, the multiplicity m λ ′ (H 3 ) is equal to the number of linearly independent vectors v in V (λ ′ ) with weight µ such that µ(h 1 ) = 0 and Jv = v.
Thus v is fixed only when k is even.Thus the second assertion follows.
The cardinalities of the sets Φ (2) and Φ (4) are computed in the following proposition.
where τ (z, b) ∈ {0, ±1} is defined on the congruence classes of z and b modulo 4 as follows: Proof.The elements (p, q) in Φ(a, b) and the corresponding dimensions p − q + 1 can be each arranged into an array of size (z + 1) × (b + 1) as follows, where we put (p, q) in the left and its corresponding dimensions in the right: By counting the number of odd integers in the right array, we obtain and by counting the number of integers congruent to 1 modulo 4 in the right array, we get Now we state and prove the main theorem of this subsection.
Theorem 4.6.For a partition where we set z := a − b.Furthermore, for any m ∈ Z ≥1 , we have Proof.From Proposition 4.4, we obtain and the formulas for m λ ′ (U(1)) and m λ ′ (H 3 ) are from Proposition 4.5.
(2) Let us consider the case m = 20 in Theorem 4.6.We have (2 6 ,1 5 ) , there are 315 Schur functions with coefficient −1 in the right hand side of (4.11) and the specialization of the left hand side of (4.11) at 1 is equal to 4 20 = 1099511627776.However, checking whether the right hand side of (4.11) coincides with the left hand side of (4.11) may well go beyond the capacity of a regular personal computer.

Subgroup
Then we have Lemma 4.9.For any c ∈ Z ≥2 , we have Proof.As before, let det be the one-dimensional representation of U(3) defined by the determinant.Then we have Since det(A) = 1 for any A ∈ U(1), det(J) = 1 and det(ζ ζ ζ 4 ) = −1, the assertion follows.
Proposition 4.10.For a partition λ ′ = (a, b, ǫ) (ǫ ∈ {0, 1}), the multiplicity m λ ′ (H ′ 3,4 ) is the same as the cardinality of the set and the multiplicity m λ ′ (H 3,4 ) is the same as the cardinality of the set Proof.Let v be a weight vector fixed by H ′ 3,4 .Then, in particular, v is fixed by U(1), and we may write v = f k 1 v (p,q;a,b) for p − q = 2k as in the proof of Proposition 4.4.Since

3
⊗ w q 12 w b−q 13 + (other terms with the same weight), for some constant C, the number of v 1 factors and the number of v 2 factors are the same, which is equal to k + q = (p + q)/2, and we have Thus we have p + q ≡ 4 0, which implies p − q ≡ 2 0. If v is also fixed by J, the we obtain an additional condition p − q ≡ 4 0 as in (the proof of) Proposition 4.4.
The cardinalities of the sets Φ (2,4) (λ ′ ) and Φ (4,4) (λ ′ ) will be computed in the next section, and we present the resulting formulas in the following two propositions.
Now we state and prove the main result of this subsection.
Furthermore, for any m ∈ Z ≥1 , we have where z, b ′ , ǫ are determined by the transpose λ ′ of λ (3 m ).
(2) Let us see the case m = 20 in Theorem 4.14: (2 9 ,1 2 ) (1) = 12342120700.There are 590 Schur functions with negative coefficients in the right hand side of (4.20), and the specialization of the right hand side of (4.20) at 1 is equal to 2 39 .This shows some systematic, even miraculous, cancelations of monomial terms in Schur functions involved in this example.
Combining Theorems 4.6 and 4.14, we obtain the following identity.(1 (1 5. Proofs for the Cardinalities of Φ (2,4) and Φ (4,4) In this section, we will prove the explicit closed-form formulas of Φ (2,4) and Φ (4,4) , which are presented in Propositions 4.10 and 4.11, respectively.Throughout this section, we will explain the reason why the functions κ ǫ (z, b), η ǫ (z, b) and ξ ǫ (z, b) are defined with respect to congruence classes modulo 2 or 4. We start this section with an example.