Genericity of supercuspidal representations of p-adic Sp_{4}

Corinne Blondel and Shaun Stevens^{∗}

August 20, 2007

Abstract We describe the supercuspidal representations of Sp_{4}(F), for F a non-archimedean local field

of residual characteristic different from 2, and determine which are generic.

Introduction

Let F be a locally compact non-archimedean local field, with ring of integers 𝔬_{F}, maximal ideal 𝔭_{F}, and residue field k_{F}. Whereas every irreducible supercuspidal representation of GL_{N}(F) is generic – i.e., has a Whittaker model – this is no longer true for classical groups over F. The existence of non-generic supercuspidal representations of classical groups is significant and bears many consequences; let us cite a few of them in local representation theory (global consequences are heavy as well, see for instance [12]): First, of course, the fact that the definition of the L-function attached to a representation of a classical group is available only for generic representations, thanks to the work of Shahidi; in particular, a characterization through local factors of the local Langlands correspondence for a classical group is not fully available. Also, reducibility of parabolic induction is completely understood for GL_{N}(F) while in classical groups results are complete only in the case of generic inducing representations. The most celebrated example of a non-generic supercuspidal representation is the representation Θ_{10} of Sp_{4}(F), induced from the inflation to Sp_{4}(𝔬_{F}) of the cuspidal unipotent representation θ_{10} of Sp_{4}(k_{F}) constructed by Srinivasan (se [12] again). To our knowledge, until recently, most known non-generic supercuspidal representations were level zero, in particular level zero representations induced from the inflation of a cuspidal unipotent representation of the reductive quotient of a maximal special parahoric subgroup. Our purpose in this paper is to exhaust the non-generic supercuspidal representations of Sp_{4}(F) in odd residual characteristic. It is certainly no surprise, and part of the folklore in the subject, that level zero supercuspidal representations of Sp_{4}(F) coming from cuspidal representations of Sp_{4}(k_{F}) are generic if and only if the corresponding cuspidal representation is, and that the only non-generic cuspidal representation of Sp_{4}(k_{F}) is Srinivasan’s θ_{10}. For the sake of completeness we include proofs of these results. Actually we prove that generic level zero supercuspidal representations of Sp_{2N}(F) are obtained by inducing the inflation of a generic cuspidal representation of the reductive quotient of a maximal parahoric subgroup of Sp_{2N}(F), on the condition that this parahoric subgroup is special. For positive level supercuspidal representations, the situation is not as simple. Our point of view is to use the exhaustive construction given by the second author in [21]: those representations are induced from a set of types, generalizing Bushnell-Kutzko types for GL_{N}(F). We give necessary and sufficient conditions on those types for the induced representation to be generic. We obtain surprisingly many (at least with respect to our starting point) non-generic representations. Let us be more precise. Positive level supercuspidal representations of Sp_{4}(F) fall into four categories, according to the nature of the skew semi-simple stratum [Λ, n, 0, β] at the bottom level of the construction (§2.1):

∗The second author would like to thank the University of Paris 7 for its hospitality in Spring 2005, during which time most of this research was undertaken.

(I) The first category starts with a skew simple stratum with corresponding field extension F [β] of dimension 4 over F. Here non-generic representations are obtained only when F [β]/F is the biquadratic extension, and when a binary condition involving β and the symplectic form is fulfilled.

(II) The second category starts with a skew simple stratum with corresponding field extension F [β] of dimension 2 over F. Non-generic representations are obtained whenever the F [β]- skew hermitian form attached to the symplectic form is anisotropic (or equivalently, when the quotient group J/J^{1} involved in the construction, which is a reductive group over k_{F}, is anisotropic).

(III) In the third category, the stratum is the orthogonal sum of two two-dimensional skew simple strata [Λ_{i}, n_{i}, 0, β_{i}], i = 1, 2. Non-genericity occurs only when F [β_{1}] is isomorphic to F [β_{2}], and again when a binary condition involving β_{1}β_{2} and the symplectic form is fulfilled.

(IV) All representations in the fourth category – when the stratum is the orthogonal sum of a skew simple stratum and a null stratum, both two-dimensional – are non-generic.

The main character in the proof is indeed a would-be character: To a stratum as above is attached a function ψ_{β} on Sp_{4}(F) and the crucial question is whether there exists a maximal unipotent subgroup U of Sp_{4}(F) on which ψ_{β} actually defines a character (see [5]); this question is easily solved in §3, where Proposition 3.4 lists the exact conditions alluded to above. Whenever there is no such subgroup U, we prove in §5 that the corresponding representations are not generic, using a criterion of non-genericity given in §1.2 (this says essentially that if a cuspidal representation c-Ind_{J}^{Sp_{4}(F)}λ is generic, then there is a long root subgroup on which λ is trivial). Now assume that there is a maximal unipotent subgroup U on which ψ_{β} is a character. A type attached to our stratum is a representation κ ⊗ σ of a compact open subgroup J, where κ is a suitable β-extension attached to the stratum and σ is a cuspidal representation of some finite reductive group J/J^{1} attached to the stratum (see §2.1). The fundamental step is Theorem 4.3, stating that the representation κ contains the character ψ_{β} of J ∩ U. This implies genericity in cases (I) and (III), where σ is just a character of an anisotropic group. But for cases (II) and (IV) (in which, we should add, ψ_{β} defines a degenerate character of U) we have to deal with the component σ, with opposite effects. In case (II), the inflation of σ contains the restriction to J ∩ U of a character of U and genericity follows. On the contrary, in case (IV), the inflation of σ does not contain the restriction to J ∩ U of a character of U and the resulting cuspidal representation is not generic.

We have several remarks to add concerning these results. First, genericity for positive level supercuspidal representations of Sp_{4}(F) only depends on the stratum itself, on the symplectic form, and on the representation σ (the “level 0 part”). It does not depend on the choice of a semi-simple character attached to the stratum.

The second remark is that the proofs are quite technical, often on a case-by-case basis. The present work can of course be regarded as a first step towards understanding non-genericity in classical groups, but even the case of Sp_{2N}(F) might not be just an easy generalisation. In particular, the precise conditions for genericity are surprisingly complicated but it seems to us that they do not admit a simple unified description, as for level zero representations. For example, nongeneric positive level supercuspidal representations can be induced from either special or non-special maximal compact subgroups of Sp_{4}(F), and likewise for generic representations. Finally, we have deliberately stuck to the construction of supercuspidal representations of Sp_{4}(F) via types. Another very fruitful point of view uses Howe’s correspondence. Indeed, in a very recent work ([11]), Gan and Takeda study the Langlands correspondence for GSp_{4}(F) and obtain in the process a classification of non-generic supercuspidal representations of GSp_{4}(F) in terms of Howe’s correspondence; they also announce a sequel dealing with Sp_{4}(F). A dictionary between these two points of view would of course be very interesting, especially if it can provide some insight about the way non-generic supercuspidal representations of Sp_{4}(F) fit into L-packets. For example, in case (I), the genericity of the supercuspidal representation depends on the embedding of β in the symplectic Lie algebra: up to the adjoint action of Sp_{4}(F), there are two such embeddings, precisely one of which gives rise to non-generic representations. This suggests that representations in these two sets might be paired to form L-packets. A closer investigation of such phenomena would certainly deserve some effort.

Notation

Let F be a locally compact non-archimedean local field, with ring of integers 𝔬_{F}, maximal ideal 𝔭_{F}, residue field k_{F} and odd residual characteristic p = char k_{F}. On occasion ϖ_{F} will denote a uniformizing element of F. Similar notations will be used for field extensions of F. We let ν_{F} denote the additive valuation of F, normalised so that ν_{F} (F^{×}) = ℤ. We fix, once and for all, an additive character ψ_{F} of F with conductor 𝔭_{F}. Let V be a 2N-dimensional F -vector space, equipped with a nondegenerate alternating form h. By a symplectic basis for V, we mean an ordered basis {e_{−N}, ..., e_{−1}, e_{1}, ..., e_{N}} such that, for 1 ≤ i, j ≤ N,

h(e_{i}, e_{j}) = h(e_{−i}, e_{−j}) = 0, h(e_{−i}, e_{j}) = δ_{ij},

where δ_{ij} is the Kronecker delta. Let A = End_{F} (V) and let denote the adjoint anti-involution on A associated to h, so

h(av, w) = h(v, \overline{a}w), for a ∈ A, v, w ∈ V.

We will let G be the corresponding symplectic group G = \{g ∈ GL_{F} (V)/\overline{g} = g^{−1}\}, or G = Sp_{2N}(F) whenever a symplectic basis is fixed. For most of the paper N will be 2. Similarly Ḡ will denote either GL_{F} (V) or GL_{2N}(F). Skew semi-simple strata in A are the basic objects in what follows. We recall briefly the essential notations attached to those objects and refer to [20] for definitions. Let [Λ, n, 0, β] be a skew semisimple stratum in A. Then Λ is a self-dual lattice sequence in V and defines a decreasing filtration {𝔞_{i}(Λ), i ∈ ℤ} of A by 𝔬_{F} -lattices 𝔞_{i}(Λ) = {x ∈ A/∀k ∈ ℤ, xΛ(k) ⊆ Λ(k + i)}. We put 𝔞(Λ) = 𝔞_{0}(Λ), a self-dual 𝔬_{F} -order in A. We will also need P (Λ) = 𝔞(Λ)^{×} ∩ G and P_{i}(Λ) = (1 + 𝔞_{i}(Λ)) ∩ G for i ≥ 1. Note that the lattice sequence Λ gives rise to a valuation ν_{Λ} on V (6.1) and the filtration {𝔞_{i}(Λ), i ∈ ℤ} gives rise to a valuation on A, also denoted by ν_{Λ}.

Next, β is a skew element in A: \overline{β} = −β, and n is a positive integer with n = −ν_{Λ}(β). Furthermore the algebra E = F [β] is a sum of fields E = ⊕_{i=1}^{l}E_{i}, corresponding to a decomposition V =⊥_{i=1}^{l} V^{i} of V as an orthogonal direct sum, and accordingly of Λ: Λ = Σ_{i=1}^{l}Λ^{i} with Λ^{i}(k) = Λ(k) ∩ V^{i}, and of β: β = Σ_{i=1}^{l}β_{i}. The centralizer B of β in A is B = ⊕_{i=1}^{l}B_{i} where B_{i} is the centralizer of β_{i} in End_{F} (V^{i}). Last, for β a skew element of A we define ψ_{β} as the following function on G: ψ_{β}(x) = ψ (tr(β(x − 1))), x ∈ G.

1 Generalities on genericity

1.1 Genericity

The results of this subsection are valid in a much more general setting than the remainder of this paper so we temporarily suspend our usual notations. Let G = G(F) be the group of F -rational points of a connected reductive algebraic group G defined over F. Let S be a maximal F -split torus in G with G-centralizer T, let B be an F -parabolic subgroup of G with Levi component T, and let U be the unipotent radical of B. Let χ be a smooth (unitary) character of U = U(F). The torus S = S(F) acts on the set of such characters by conjugation. We say that χ is nondegenerate if its stabilizer for this action is just the centre Z of G.

Example 1.1. Let G = Sp_{4}(F), which we write as a group of matrices with respect to some symplectic basis, let T be the torus of diagonal matrices, and let U be the subgroup of upper triangular unipotent matrices in G. Any character χ of U is given by

χ (\begin{matrix}1 & u & x & y \\ 0 & 1 & v & x \\ 0 & 0 & 1 & −u \\ 0 & 0 & 0 & 1 \end{matrix}) = ψ_{F} (au + bv),

for some a, b ∈ F, and it is easy to see that χ is nondegenerate if and only if a, b are both nonzero. Moreover, there are four orbits of nondegenerate characters of U, given by the class of b in F^{×}/(F^{×})^{2}.

Returning to a general connected reductive group G, let π be a smooth irreducible representation of G. We say that π is generic if there exist U = U(F) as above and a nondegenerate character χ of U such that

Hom_{G}(π, Ind_{U}^{G}χ) ≠ 0.

Note that, since all such subgroups U are conjugate in G we may choose to fix one. Moreover, we need only consider nondegenerate characters χ up to T -conjugacy. A basic result here is:

Theorem 1.2 ([15] Theorem 3). Assume G is split over F. Let π be a smooth irreducible representation of G and let χ be a nondegenerate character of a maximal unipotent subgroup U of G. Then

dim_{ℂ}Hom_{G}(π, Ind_{U}^{G}χ) ≤ 1.

On the other hand, and again in a general G, when dealing with supercuspidal representations we may not bother about the nondegeneracy of the character in the definition of genericity, a fact that will be useful in the sequel. Indeed:

Lemma 1.3. Assume G is split over F. Let π be a smooth irreducible supercuspidal representation of G. Let U be a maximal connected unipotent subgroup of G and let χ be a character of U = U(F) such that

Hom_{G}(π, Ind_{U}^{G}χ) ≠ 0.

Then the character χ is nondegenerate.

Proof. Assume χ is degenerate and use the definition of nondegeneracy in [7], 1.2, as well as the corresponding notation: there is a simple root α such that the restriction of χ to U_{(α)} is trivial. The character χ is then trivial on the subgroup ⟨U_{(α)}, U_{der}⟩ of U. We claim that this subgroup contains the unipotent radical N of a proper parabolic subgroup P of G. The assumption Hom_{G}(π, Ind_{U}^{G}χ) ≠ 0 provides us, by Frobenius reciprocity, with a non-zero linear form λ on the space V of π which satisfies λ ∘ π(u) = χ(u)λ for any u ∈ U. Since λ is in particular N-invariant, we get that the space V_{N} of N-coinvariants is non-zero, which contradicts cuspidality. We now prove the claim. Let Δ be the set of simple roots as in [7]. From [2], 21.11, the unipotent radical of the standard F -parabolic subgroup P_{I} of G attached to the subset I = Δ − {α} is U_{Ψ(I)} where Ψ(I) is the set of positive roots that can be written α + β with β either 0 or a positive root. Hence the elements in Ψ(I) other than α are positive roots γ of length at least 2, and U_{Ψ(I)} is directly spanned by U_{(α)} and the U_{(γ)} for non-divisible such roots γ. From [6], Theorem 4.1, for any such γ we have U_{(γ)}(F) ⊂ U_{der} (recall the characteristic of F is not 2), hence N = U_{Ψ(I)}(F) is contained in ⟨U_{(α)}, U_{der}⟩ as asserted. ■

Now let π be an irreducible supercuspidal representation of G. We suppose, as is the case for all known supercuspidal representations, that π is irreducibly compact-induced from some open compact mod centre subgroup of G. Then (the proof of) [5] 1.6 Proposition immediately gives us the following:

Proposition 1.4. Let K be an open compact mod centre subgroup of G and ρ an irreducible representation of K such that π = c-Ind_{K}^{G}ρ is an irreducible, so supercuspidal, representation of G. Then π is generic if and only if there exist a maximal connected unipotent subgroup U of G and a character χ of U = U(F) such that ρ|K ∩ U contains χ|K ∩ U. Moreover, if this is the case then Hom_{G}(π, Ind_{U}^{G}χ) ≠ 0 so χ is nondegenerate, and the character χ|K ∩ U occurs in ρ|K ∩ U with multiplicity 1.

1.2 A criterion for non-genericity

Now we look more closely at the situation for symplectic groups so we return to our usual notation: G = Sp_{2N}(F). In particular, using Proposition 1.4 and a decomposition of G, we will obtain a criterion to determine when an irreducible representation of G is not generic. Let T denote the standard (diagonal) maximal split torus of G, let U be the subgroup of all upper triangular unipotent matrices in G, and put B = T U, the Borel subgroup of all upper triangular matrices in G. Let Φ = Φ(G, T) be the root system and, for γ ∈ Φ, let U_{γ} denote the corresponding root subgroup. Let W denote the Weyl group N_{G}(T)/T; by abuse of notation, we will also use W for a set of representatives in the compact maximal subgroup K_{0} = Sp_{2N}(𝔬_{F}) of G. We write K_{1} for the pro-unipotent radical of K_{0}, so that K_{0}/K_{1} ≃ Sp_{2N}(k_{F}). We note that B ∩ K_{0}/B ∩ K_{1} is the standard Borel subgroup (of upper triangular matrices) of K_{0}/K_{1}, that T ∩ K_{0}/T ∩ K_{1} is the diagonal torus, and that W is the Weyl group.

Let I_{1} denote the inverse image of the maximal unipotent subgroup U ∩ K_{0}/U ∩ K_{1} of K_{0}/K_{1}, that is, the pro-unipotent radical of the standard Iwahori subgroup I consisting of matrices which are upper triangular modulo 𝔭_{F}. Then the Bruhat decomposition for K_{0}/K_{1} gives

K_{0} / K_{1} = (B ∩ K_{0})W I_{1} / K_{1}.

Since K_{1} ⊂ I_{1}, we obtain K_{0} = (B ∩ K_{0})W I_{1}. Finally, using the Iwasawa decomposition G = BK_{0} (since K_{0} is a good maximal compact subgroup of G), we obtain

G = BW I_{1}.  (1.5)

Now we can use this decomposition to translate Proposition 1.4 into a sufficient condition for non-genericity of compactly-induced supercuspidal representations of G.

Proposition 1.6. Let J be a compact open subgroup of G and λ an irreducible representation of J such that π = c-Ind_{J}^{G}λ is an irreducible supercuspidal representation of G. Let U be the subgroup of all upper triangular unipotent matrices in G and we also use the other notations from above. Then π is generic if and only if there exist w ∈ W, p ∈ I_{1} and a character χ of U such that ^{p}λ contains the character χ^{w} of ^{p}J ∩ U^{w}. In particular,

if π is generic then there are p ∈ I_{1} and a long root γ ∈ Φ such that ^{p}λ contains the trivial character of ^{p}J ∩ U_{γ}.

We remark that, in our symplectic basis, the long roots correspond to the entries on the antidiagonal.

Proof. Since all maximal unipotent subgroups of G are conjugate to U, Proposition 1.4 implies that π is generic if and only if there exist g ∈ G and a character χ of U such that λ contains the character χ^{g} of J ∩ U^{g}. Now we use the decomposition (1.5) to write g = bwp, with b ∈ B, w ∈ W and p ∈ I_{1}. Since U^{b} = U and χ^{b} is another character of U, we can absorb the b and the result follows on conjugating by p. The final assertion follows since the derived subgroup U_{der}^{w} contains U_{γ} for some long root γ. ■

2 The supercuspidal representations of Sp_{4}(F)

What we seek is a complete list of which supercuspidal representations of Sp_{4}(F) are generic, which are not. So we start with a description of positive level supercuspidal representations of Sp_{4}(F)

– level zero supercuspidal representations are obtained by inducing from a maximal parahoric subgroup P the inflation of a cuspidal representation of the (finite) quotient of P by its pro-p-radical. We are then in a position to state our main theorem, identifying non-generic representations from our list. We end the section with a proof of the theorem for level zero representations. The proof for positive level occupies the remaining sections.

2.1 The positive level supercuspidal representations

In this section we describe the construction of the positive level supercuspidal representations of Sp_{4}(F). We refer to [21] for more details and for proofs of the results stated here. The construction begins with a skew semisimple stratum [Λ, n, 0, β] in A such that 𝔞(Λ) ∩ B is a maximal self-dual order normalized by E^{×} in B. There are essentially four cases here. In the first two, the stratum is actually simple:

(I) “maximal case”: [Λ, n, 0, β] is a skew simple stratum and E = F [β] is an extension of F of degree 4.

(II) “2 then 0 case”: [Λ, n, 0, β] is a skew simple stratum, E = F [β] is an extension of F of degree 2, and 𝔞_{0}(Λ) is maximal amongst (self-dual) 𝔬_{F} -orders in A normalized by E^{×}.

Otherwise, we have a splitting V = V^{1} ⊥ V^{2} of [Λ, n, 0, β] into two 2-dimensional F -vector spaces, and we write: Λ^{i} for the lattice sequence in V^{i} given by Λ^{i}(k) = Λ(k) ∩ V^{i}, for k ∈ ℤ; β_{i} = 1^{i}β1^{i}, where 1^{i} is the projection onto V^{i} with kernel V^{3−i}; if β_{i} ≠ 0 then n_{i} = −ν_{Λ^{i}}(β_{i}), otherwise n_{i} = 0. Then n = max {n_{1}, n_{2}}.

(III) “2+2 case”: for i = 1, 2, [Λ^{i}, n_{i}, 0, β_{i}] is a skew simple stratum and E_{i} = F [β_{i}] is an extension of F of degree 2.

(IV) “2+0 case”: [Λ^{1}, n_{1}, 0, β_{1}] is a skew simple stratum and E_{1} = F [β_{1}] is an extension of F of degree 2; β_{2} = 0, so that in V^{2} we have the null stratum [Λ^{2}, 0, 0, 0], and 𝔞_{0}(Λ^{2}) is maximal amongst (self-dual) 𝔬_{F} -orders in A^{2} = End_{F} (V^{2}).

We often think of (IV) as a degenerate case of (III) by thinking of a null stratum as a degenerate simple stratum. In each case, we have the subgroups H̄^{1} = H^{1}(β, Λ), J̄^{1} = J^{1}(β, Λ) and J̄ = J(β, Λ) of Ḡ. We write H^{1} = H̄^{1} ∩ G, and similarly for the other groups. There is a family C(β, Λ) of rather special characters of H^{1}, called semisimple characters; one of their properties is the fact that their restriction to P_{i}(Λ) for suitable i is equal to ψ_{β}. For each θ ∈ C(β, Λ), there is a unique irreducible representation η of J^{1} containing θ. In each case, there is a “suitable” extension κ of η to a representation of J, which we call a β-extension – see below for more details of this step. The extensions E, E_{i} in each case come equipped with a non-trivial galois involution, which we write as usual. We use the same notation for the induced involution on the residue fields k_{E}, k_{i}; note that this involution may be trivial. Then the quotient J/J^{1} has one of the following forms:

(I) k_{E}^{1} = \{x ∈ k_{E} : x\overline{x} = 1\};

(II) U(1, 1)(k_{E}/k_{F}) or k_{E}^{1} × k_{E}^{1} if E/F is unramified; SL_{2}(k_{F}) or O_{2}(k_{F}) if E/F is ramified;

(III) k_{1}^{1} × k_{2}^{1};

(IV) k_{1}^{1} × SL_{2}(k_{F}).

Let σ be the inflation to J of an irreducible cuspidal representation of J/J^{1}. (Note that, in the case of O_{2}(k_{F}) in (II), this just means any irreducible representation of the (anisotropic) dihedral group O_{2}(k_{F}).) Now we put λ = κ ⊗ σ and π = c-Ind_{J}^{G}λ is an irreducible supercuspidal representation of G. All irreducible supercuspidal representations of G can be constructed in this way (though we remark that often we cannot, as yet, tell when two such representations are equivalent).

Finally in this subsection, we recall briefly some properties of the β-extensions which we will need, especially in the cases (II) and (IV) where their construction is somewhat more involved. Indeed, in cases (I) and (III), J/J^{1} has no unipotent elements so there is never any problem here. We define another skew semisimple stratum [Λ_{m}, n_{m}, 0, β] as follows:

• in cases (I) and (III), we have Λ_{m} = Λ, n_{m} = n;

• in case (II), Λ_{m} is a self-dual 𝔬_{E}-lattice sequence in V with 𝔞_{0}(Λ_{m}) ⊂ 𝔞_{0}(Λ) minimal amongst (self-dual) 𝔬_{F} -orders normalized by E^{×}, and n_{m} = −ν_{Λ_{m}}(β);

• in case (IV), we take Λ_{m}^{2} a self-dual 𝔬_{F} -lattice sequence in V with 𝔞_{0}(Λ_{m}^{2}) ⊂ 𝔞_{0}(Λ^{2}) minimal amongst (self-dual) 𝔬_{F} -orders, Λ_{m} = Λ^{1} ⊥ Λ_{m}^{2}, and n_{m} = −ν_{Λ_{m}}(β).

In each case, we have the subgroups H̄_{m}^{1} = H^{1}(β, Λ_{m}), J̄_{m}^{1} = J^{1}(β, Λ_{m}) of Ḡ, and we put H_{m}^{1} = H̄_{m}^{1} ∩ G etc. Let θ_{m} ∈ C(β, Λ_{m}) be the transfer of θ, that is θ_{m} = τ_{Λ,Λ_{m},β}θ in the notation of [20] §3.6, and let η_{m} be the unique irreducible representation of J_{m}^{1} containing θ_{m}. We form the group

J̃^{1} = (P_{1}(Λ_{m}) ∩ B)J^{1}.

Then (see [21]) there is a unique irreducible representation η̃ of J̃^{1} which extends η and such that η̃ and η_{m} induce equivalent irreducible representations of P_{1}(Λ_{m}). Moreover, if I_{g}(η̃) denotes the g-intertwining space of η̃, we have

dim I_{g}(η̃) = \begin{cases}1 & if g ∈ J̃^{1}(B ∩ G)J̃^{1}; \\ 0 & otherwise. \end{cases}

A β-extension of η is an irreducible representation κ of J such that κ|J̃^{1} = η̃.

2.2 The main theorem

Theorem 2.1. The non-generic supercuspidal representations of Sp_{4}(F) are the following.

(i) The positive level supercuspidal representations attached to a skew semisimple stratum [Λ, n, 0, β] as above and such that:

• either [Λ, n, 0, β] is a sum of non-zero simple strata (cases (I), (II) and (III)) and there is no maximal unipotent subgroup of G on which ψ_{β} is a character;

• or [Λ, n, 0, β] is the sum of a non-zero simple stratum and a null stratum in dimension 2 (case (IV)).

(ii) The level zero supercuspidal representations induced from the inflation to a maximal parahoric subgroup P of a cuspidal representation σ of P/P^{1}, where P^{1} is the pro-p-radical of P, satisfying one of the following:

(a) P is attached to a non-connected subset of the extended Dynkin diagram of G, that is, P/P^{1} is isomorphic to Sp(2, k_{F}) × Sp(2, k_{F}).

(b) P is isomorphic to Sp_{4}(𝔬_{F}) and σ is a non-regular cuspidal representation of Sp_{4}(k_{F}).

For positive level representations the theorem will follow from Proposition 3.4, which establishes the conditions on β for there to exist a maximal unipotent subgroup of G on which ψ_{β} is a character, and from Theorem 4.5 and section 5. The proof for level zero representations is given below, with a more detailed list.

2.3 The generic level zero representations of Sp_{2N}(F)

Note that for finite reductive groups, the notion equivalent to genericity is called regularity: a representation of Sp_{2N}(k_{F}) is called regular if it contains a nondegenerate character of a maximal unipotent subgroup. Part (ii) of the above theorem, i.e. the level zero case, actually holds for Sp_{2N}(F), as a consequence of Propositions 1.4 and 1.6.

Proposition 2.2. Let P be a maximal parahoric subgroup of Sp_{2N}(F) with pro-p-radical P^{1} and let σ be a cuspidal representation of P/P^{1}. The representation π of Sp_{2N}(F) induced from the inflation of σ to P is irreducible supercuspidal. It is generic if and only if the quotient P/P^{1} is isomorphic to Sp_{2N}(k_{F}) and σ identifies to a regular cuspidal representation of Sp_{2N}(k_{F}).

Proof. Up to conjugacy, we may assume that P is standard; in particular, using the notation in 1.2, P contains I. Then our standard P is the group of invertible and symplectic elements in the order

𝔄 = (\begin{matrix}M_{i}(𝔬_{F}) & M_{i, 2N−i}(𝔬_{F}) & M_{i}(𝔭_{F}^{−1}) \\ M_{2N− i, i}(𝔭_{F}) & M_{2N−2i}(𝔬_{F}) & M_{2N− i, i}(𝔬_{F}) \\ M_{i}(𝔭_{F}) & M_{i, 2N−i}(𝔭_{F}) & M_{i}(𝔬_{F}) \end{matrix}) for some integer i, 0 ≤ i ≤ N.

Assume first that P/P^{1} is isomorphic to Sp_{2N}(k_{F}), i.e. i = 0 or N, and use Proposition 1.4. If σ is regular, then π is generic. Conversely if π is generic, there is a maximal unipotent subgroup U′ and a character χ′ of U′ such that σ contains χ_{|P∩U′}. The subgroup U′ is conjugate to the subgroup U of all upper triangular unipotent matrices so, using the Iwasawa decomposition G = PB as in 1.2, we may replace U′ by U. Since σ is cuspidal, Lemma 1.3, applied to Sp_{2N}(k_{F}), tells us that χ_{|P∩U} identifies with a nondegenerate character of P ∩ U/P^{1} ∩ U, a maximal unipotent subgroup of Sp_{2N}(k_{F}), hence σ is regular.

Assume now that 1 ≤ i ≤ N − 1: then P/P^{1} is isomorphic to Sp_{2i}(k_{F}) × Sp_{2N−2i}(k_{F}), the relevant

entries being those in (\begin{matrix}∗ & 0 & ∗ \\ 0 & ∗ & 0 \\ ∗ & 0 & ∗ \end{matrix}) in the above description of 𝔄. Assume for a contradiction that

π is generic: from Proposition 1.6, plus the inclusion I_{1} ⊂ P, there exist w ∈ W and a character χ of U such that σ contains the character χ^{w} of P ∩ U^{w}. Let \overline{U^{w}} = P ∩ U^{w}/P^{1} ∩ U^{w} and let \overline{χ^{w}} be the character of \overline{U^{w}} defined by χ^{w}. The group \overline{U^{w}} is a maximal unipotent subgroup of P/P^{1} (a simple combinatoric argument suffices here). We will show that, since χ is trivial on U_{der}, the character \overline{χ^{w}} is degenerate, thus contradicting the cuspidality of σ.

To fix ideas, suppose that w = 1. The intersection of the image of U with Sp_{2i}(k_{F}) is the subgroup Ū_{i} of upper triangular unipotent matrices while the image of U_{der} contains the simple long root: the restriction of χ̄ to Ū_{i} is degenerate hence σ cannot be cuspidal (Lemma 1.3).

In general, observe that w must map the N positive long roots (corresponding to the antidiagonal entries in U) onto a set E of N long roots that correspond to the long root entries in U^{w}. Those N long roots separate into i long roots in Ū^{w} ∩ Sp_{2i}(k_{F}) and N − i long roots in Ū^{w} ∩ Sp_{2N−2i}(k_{F}). The N − 1 positive not simple long roots corresponding to antidiagonal entries in U_{der} are sent onto a subset of N − 1 long roots in E: only one is missing, so either in Ū^{w} ∩ Sp_{2i}(k_{F}) or in Ū^{w} ∩ Sp_{2N−2i}(k_{F}), the unique long root entry that does not belong to the derived group does belong to U_{der}^{w}: on this group, the restriction of χ̄^{w} is degenerate, so σ is not cuspidal. ■

2.4 The cuspidal representations of Sp_{4}(𝔽_{q})

We come back to Sp(4). It has been known for a long time that among the cuspidal representations of Sp_{4}(k_{F}), only one is non-regular, the famous representation θ_{10} of Srinivasan ([17] II.8.3, [18]); it is the unique cuspidal unipotent representation of Sp_{4}(k_{F}). Yet this common knowledge lacks of a reference in the modern setting of Deligne-Lusztig characters, we thus pause here to detail the list of cuspidal representations of Sp(4, 𝔽_{q}) as they arise from the Lusztig classification. The necessary background and notations are taken from the book [10], specially chapter 14.

For this subsection only we let G = Sp(4, 𝔽̄_{q}) and we let F be the standard Frobenius on G, acting as x ↦ x^{q} on each entry, so that G^{F} = Sp(4, 𝔽_{q}). We let G^{∗} be the dual group SO(5, 𝔽̄_{q}) with standard Frobenius F^{∗}. Deligne-Lusztig characters of G^{F} are parameterized by pairs (T^{∗}, s), T^{∗} an F^{∗}-stable maximal torus of G^{∗} and s an element of T^{∗}^{F^{∗}}, up to G^{∗}^{F^{∗}}-conjugacy. A rational series of irreducible characters of G^{F} is made of all irreducible components of Deligne-Lusztig characters R_{T^{∗}}^{G}(s) where the rational conjugacy class of s (i.e. the G^{∗}^{F^{∗}}-conjugacy class of s) is fixed. Rational series of characters are disjoint and exhaust irreducible characters of G^{F}. Cuspidal (irreducible) characters are those characters that appear in some R_{T^{∗}}^{G}(s) for a minisotropic torus T^{∗} and do not appear in any R_{T^{∗}}^{G}(s) where the torus T^{∗} is contained in a proper F^{∗}-stable Levi subgroup of G^{∗}.

Let s be a rational semi-simple element contained in an F^{∗}-stable maximal torus T^{∗} of G^{∗}, let C_{G^{∗}}(s) (resp. C_{G^{∗}}^{o}(s)) be its centralizer in G^{∗} (resp. the connected component of its centralizer) and let W (s) (resp. W^{o}(s)) be the Weyl group of C_{G^{∗}}(s) (resp. C_{G^{∗}}^{o}(s)) relative to T^{∗}, contained in the Weyl group W (T^{∗}) of G^{∗} relative to T^{∗}. For w in W (T^{∗}), there exists an F^{∗}-stable maximal torus T_{w}^{∗} of G^{∗} of type w with respect to T^{∗} and containing s if and only if w belongs to W^{o}(s). Letting x be the type of T^{∗} with respect to some split torus, one defines by the formula

χ(s) = (−1)^{l(x)}|W^{o}(s)|^{−1} ∑_{w∈W^{o}(s)}(−1)^{l(w)}R_{T_{w}^{∗}}^{G}(s)

a proper character χ(s) which is a multiplicity one sum of regular irreducible characters, each appearing with multiplicity ±1 in Deligne-Lusztig characters underlying the series attached to s. We have

⟨R_{T_{w}^{∗}}^{G}(s), R_{T_{w}^{∗}}^{G}(s)⟩_{G^{F}} = Card W (s)^{wF^{∗}} and ⟨χ(s), χ(s)⟩_{G^{F}} = |(W (s)/W^{o}(s))^{F^{∗}}|

and the results in loc. cit., §14, imply the following, for the rational series of characters attached to s:

(i) If only minisotropic rational maximal tori contain s, all characters in the series are cuspidal. The number of regular cuspidal characters in the series is the number of components of χ(s).

(ii) If no minisotropic rational maximal torus contains s, there is no cuspidal character in the series.

(iii) If at least one minisotropic rational maximal torus and at least one non-minisotropic rational maximal torus contain s, no cuspidal character in the series (if any) is regular.

(iv) If exactly one minisotropic rational maximal torus (up to rational conjugacy) and at least one non-minisotropic rational maximal torus contain s, there is no cuspidal character in the series.

The Weyl group of G^{∗} has 8 elements. A rational maximal torus of type w with respect to a split torus is minisotropic if and only if w is either a Coxeter element h (there are two of them, conjugate in the Weyl group) or the element of maximal length w_{0}. Rational points of such a torus are conjugate to T_{0}^{∗}^{wF^{∗}}, isomorphic to

𝕂_{4}^{2} = ker N_{𝔽_{q^{4}}^{×}/𝔽_{q^{2}}^{×}} for w = h,
𝕂_{2}^{1} × 𝕂_{2}^{1} = ker N_{𝔽_{q^{2}}^{×}/𝔽_{q}^{×}} × ker N_{𝔽_{q^{2}}^{×}/𝔽_{q}^{×}} for w = w_{0}.

The table below lists the families of geometric conjugacy classes of rational semi-simple elements of G^{∗} through a representative s_{0} (not necessarily rational) in the diagonal torus

T_{0}^{∗} = \{ t^{∗}(λ, μ) = ( \begin{matrix}λ \\ & μ \\ & & 1 \\ & & & μ^{−1} \\ & & & & λ^{−1} \end{matrix} ) / λ, μ ∈ 𝔽̄_{q}^{×}\},

using the following notation: we fix ζ_{4}, a primitive (q^{4} − 1)-th root of unity in 𝔽̄_{q}^{×}, and we let

ζ_{4,2} = ζ_{4}^{q^{2}−1}, ζ_{2} = ζ_{4}^{q^{2}+1}, ζ_{2,1} = ζ_{2}^{q−1} and ζ = ζ_{2}^{q+1}.

In cases 13 and 14, χ(s) = R_{T^{∗}}^{G}(s) is irreducible, cuspidal and regular (it actually contains any

nondegenerate character of a maximal unipotent subgroup) (i); we get \frac{(q − 1)(q − 3)}{8} + \frac{(q^{2} − 1)}{4} equivalence classes of such representations.

Cases 4, 5, 6, 7, 8 and 9 give no cuspidal representations (ii), neither do cases 10 and 12 (iv). Cases 2 and 3 each determine two rational series, in which again (iv) applies: they contain no cuspidal.

Missing cuspidals ([17] II.8.3, [18]) now must come from cases 1 and 11. Indeed case 11 produces two rational series, one of which satisfying (ii), but the other (i), for a torus of type w_{0}. Here χ(s) = R_{T^{∗}}^{G}(s) is the sum of two irreducible, cuspidal and regular (but for different choices of a

nondegenerate character of a maximal unipotent subgroup) representations and we get 2 \frac{(q − 1)}{2} equivalence classes of such representations.

Last, case 1 gives the so-called unipotent series, which for Sp(4, 𝔽_{q}) contains exactly one cuspidal representation ([13], Theorem 3.22), non-regular by (iii).

Case  s_{0}  condition  number  W^{o}(s_{0}) |W (s_{0})/W^{o}(s_{0})|
1  t^{∗}(1, 1)  1  W  1
2  t^{∗}(−1, −1)  1  ⟨s_{α′}, s_{α′+2β′}⟩  2
3  t^{∗}(−1, 1)  1  ⟨s_{β′}⟩  2
4  t^{∗}(ζ^{i}, 1)  ζ^{i} ≠ ±1  \frac{q − 3}{2}  ⟨s_{β′}⟩  1

5  t^{∗}(ζ^{i}, −1)  ζ^{i} ≠ ±1  \frac{q − 3}{2}  1  2

6  t^{∗}(ζ^{i}, ζ^{i})  ζ^{i} ≠ ±1  \frac{q − 3}{2}  ⟨s_{α′}⟩  1
ζ^{i} ≠ ±1
7  t^{∗}(ζ^{i}, ζ^{j})  ζ^{j} ≠ ±1  \frac{(q − 3)(q − 5)}{8}  1  1
ζ^{i} ≠ ζ^{±j}
ζ_{2}^{i} ∉ 𝕂_{2}^{1}
8  t^{∗}(ζ_{2}^{i}, ζ_{2}^{qi})  ζ_{2}^{i} ∉ 𝔽_{q}^{×}  \frac{(q − 1)^{2}}{4}  1  1
ζ^{i} ≠ ±1
9  t^{∗}(ζ^{i}, ζ_{2,1}^{j})  ζ_{2,1}^{j} ≠ ±1  \frac{(q − 1)(q − 3)}{4}  1  1

10  t^{∗}(1, ζ_{2,1}^{i})  ζ_{2,1}^{i} ≠ ±1  \frac{(q − 1)}{2}  ⟨s_{α′+β′}⟩  1

11  t^{∗}(−1, ζ_{2,1}^{i})  ζ_{2,1}^{i} ≠ ±1  \frac{(q − 1)}{2}  1  2

12  t^{∗}(ζ_{2,1}^{i}, ζ_{2,1}^{−i})  ζ_{2,1}^{i} ≠ ±1  \frac{(q − 1)}{2}  ⟨s_{α′+2β′}⟩  1
ζ_{2,1}^{i} ≠ ±1
13  t^{∗}(ζ_{2,1}^{i}, ζ_{2,1}^{j})  ζ_{2,1}^{j} ≠ ±1  \frac{(q − 1)(q − 3)}{8}  1  1
ζ_{2,1}^{i} ≠ ζ_{2,1}^{±j}

14  t^{∗}(ζ_{4,2}^{qi}, ζ_{4,2}^{i})  ζ_{4,2}^{i} ≠ ±1  \frac{q^{2} − 1}{4}  1  1

3 The function ψ_{β} on maximal unipotent subgroups

A key step in the determination of Whittaker functions in GL_{N}(F) in [5] is the construction of a maximal unipotent subgroup U of GL_{N}(F) on which ψ_{β} defines a character (loc. cit. propositions 2.1 and 2.2). This will be a key step indeed in the determination of generic supercuspidal representations of Sp_{4}(F): the existence of such a subgroup on which ψ_{β} defines a non degenerate character will turn out to be a sufficient condition for genericity (see §4), whereas the non existence will imply non genericity (see §5). In cases where such a U exists but ψ_{β} is degenerate, we will find both generic and non generic representations.

3.1 The quadratic form h(v,βv)

Proposition 3.1. Let β be an element of A such that β̄ = −β and let ψ_{β} be the function on G defined by ψ_{β}(x) = ψ (tr(β(x − 1))), x ∈ G. The following are equivalent:

(i) There exists a maximal unipotent subgroup U of G such that the restriction of ψ_{β} to U is a character of U.

(ii) There exists a maximal unipotent subgroup U of G such that ψ_{β}(x) = 1 for all x ∈ U_{der}.

(iii) There exists a totally isotropic flag of subspaces of V:

{0} ⊂ V_{1} ⊂ V_{2} ⊂ V_{3} ⊂ V

such that βV_{i} ⊂ V_{i+1} for i = 1, 2.

(iv) The quadratic form v ↦ h(v, βv) on V has non trivial isotropic vectors.

Proof. The equivalence of the first three statements is straightforward and a variant of [5] 2.1; note that a maximal flag of subspaces of V determines a maximal unipotent subgroup of G if and only if it is totally isotropic. Certainly (iii) implies (iv): a basis vector v for V_{1} satisfies h(v, βv) = 0 since V_{2} is its own orthogonal. Assuming (iv), let v be a non-zero vector in V such that h(v, βv) = 0 and put V_{1} = Span {v}, V_{3} = V_{1}^{⊥}. If βv and v are colinear, let V_{2} be any totally isotropic 2-dimensional subspace of V containing V_{1}, otherwise put V_{2} = Span {v, βv}: (iii) is satisfied since, for a totally isotropic flag as in (iii), the conditions βV_{1} ⊂ V_{2} and βV_{2} ⊂ V_{3} are equivalent (recall β̄ = −β). ■

Remark 3.2. Assume the conditions in Proposition 3.1 hold and let U be a maximal unipotent subgroup of G attached to a totally isotropic flag {0} ⊂ V_{1} ⊂ V_{2} ⊂ V_{3} ⊂ V such that βV_{i} ⊂ V_{i+1} for i = 1, 2. A simple inspection shows that the character ψ_{β} of U is non degenerate if and only if βV_{1} is not contained in V_{1} and βV_{2} is not contained in V_{2}.

We need to investigate those cases where the element β appears in a skew semi-simple stratum [Λ, n, 0, β] as listed in §2.1. We need an extra piece of notation in cases I or II, where the stratum is simple: the field extension E = F [β] has degree 4 or 2; we let E_{0} be the field of fixed points of the involution x ↦ x̄ on E, so that [E : E_{0}] = 2, and we define a skew-hermitian form δ on V relative to E/E_{0} by

h(av, w) = tr_{E/F} (aδ(v, w)) for all a ∈ E, v, w ∈ V.  (3.3)

(This notation will also be used in case III when E_{1} and E_{2} are isomorphic, with E = E_{1}.) The determinant of δ belongs to F^{×} if [E : F] = 2; it is a skew element in E^{×} if [E : F] = 4.

Proposition 3.4. Let β be an element of A appearing in a skew semi-simple stratum [Λ, n, 0, β] as in §2.1. The only cases in which there does not exist a maximal unipotent subgroup U of G on which ψ_{β} is a character are the following:

(i) The element β generates a biquadratic extension E = F [β] of F (case I) and the coset β det (δ) N_{E/E_{0}}(E^{×}) in E_{0}^{×} is the N_{E/E_{0}}(E^{×})-coset that does not contain the kernel of tr_{E_{0}/F}.

(ii) The element β generates a quadratic extension E = F [β] of F (case II) and the skew-hermitian form δ on V, dim_{E}V = 2, is anisotropic – that is, det (δ) ∉ N_{E/F} (E^{×}).

(iii) The symplectic space V decomposes as V = V^{1}⊥V^{2} and the element β decomposes accordingly as β = β_{1} + β_{2} where for i = 1, 2, β_{i} generates a quadratic extension E_{i} = F [β_{i}] (case III), E_{1} is isomorphic to E_{2} and β_{1}/β_{2} ∉ det (δ) N_{E/F} (E^{×}).

Remark 3.5. Let β be as above. Assume there exists a maximal unipotent subgroup U of G on which ψ_{β} is a character. Then:

• in cases I and III the character ψ_{β} of U is non degenerate;

• in cases II and IV the character ψ_{β} of U is degenerate.

The proof of those statements occupies the next two subsections. We recall that, up to isomorphism, there is exactly one anisotropic quadratic form on V: its determinant is a square and its Hasse-Minkowski symbol is equal to −(−1, −1)_{F} ([14], §63C).

3.2 The biquadratic extension

Let us examine the case of a maximal simple stratum (case I). The determinant of the quadratic form v ↦ h(v, βv) on V is the determinant of β, i.e. N_{E/F} (β).

Lemma 3.6. The norm N_{E/F} (β) of β is a square in F^{×} if and only if E is biquadratic. If this holds we have: N_{E/E_{0}}(E^{×}) = F^{×}E_{0}^{×}^{2}.

Proof. A four-dimensional extension of F is called biquadratic if it is Galois with Galois group ℤ/2ℤ × ℤ/2ℤ. Biquadratic extensions of F are all isomorphic, their norm subgroup is F^{×2}(class field theory). The “if” part is then clear. Now assume that N_{E/F} (β) is a square. Since β is skew and generates a degree four field extension of F, its square β^{2} generates E_{0} over F and is not a square in E_{0}^{×}, while N_{E_{0}/F} (β^{2}) = N_{E/F} (β) must be a square in F^{×}. We proceed according to ramification. If E_{0} is ramified over F: β^{2} must have even valuation, its squareroot generates an unramified extension of E_{0}. So E/E_{0} is unramified and N_{E/E_{0}}(E^{×}) is made of even valuation elements, i.e. is equal to F^{×}E_{0}^{×}^{2} since 𝔬_{E_{0}}^{×} = 𝔬_{F}^{×}(1 + 𝔭_{E_{0}}). It follows that N_{E/F} (E^{×}) = F^{×}^{2}. If E_{0} is unramified over F, we have in the residual field k_{E_{0}}:

u ∈ k_{E_{0}}^{×} is a square in k_{E_{0}}^{×} if and only if N_{k_{E_{0}}/k_{F}} (u) is a square in k_{F}^{×}.

We write β^{2} = ϖ_{F}^{j} u with u ∈ 𝔬_{E_{0}}^{×}. Then N_{E_{0}/F} (β^{2}) = ϖ_{F}^{2j}N_{E_{0}/F} (u). It follows that u must be a square and β^{2} must have odd valuation: its square root generates a ramified extension of E_{0}. Then E is the extension E_{0}[α] where α^{2} is a uniformizing element in F, and N_{E/E_{0}}(E^{×}) = (−α^{2})^{ℤ}𝔬_{E_{0}}^{×2} = F^{×}E_{0}^{×}^{2} because k_{F}^{×} ⊂ k_{E_{0}}^{×2}. It follows that N_{E/F} (E^{×}) = F^{×}^{2}. ■

If N_{E/F} (β) is not a square, we are done. Assume from now on that E is biquadratic and use the form δ defined in 3.3. Since β is skew and δ is skew-hermitian, the element βδ(v, v) belongs to E_{0}. Since V is one-dimensional over E the form δ(v, v) is anisotropic and the subset D(V) = {βδ(v, v) / v ∈ V, v ≠ 0} of E_{0}^{×} is one of the two cosets of N_{E/E_{0}}(E^{×}) in E_{0}^{×}. On the other hand, we have N_{E/E_{0}}(E^{×}) = F^{×}E_{0}^{×}^{2} hence the set of non-zero elements in Ker tr_{E_{0}/F} is fully contained in one of those two cosets. Proposition 3.4 follows in this case (and Remark 3.5 directly follows from 3.2 since β generates a degree 4 extension of F).

Remark. We can be more precise about this condition: if E_{0} is unramified over F and |k_{F} | ≡ 3 [4], then h(βv, v) is anisotropic if and only if βδ(v, v) ∉ F^{×}E_{0}^{×}^{2}; otherwise h(βv, v) is anisotropic if and only if βδ(v, v) ∈ F^{×}E_{0}^{×}^{2}.

3.3 Cases II, III and IV

The case numbered IV in §2.1 is obvious. The set of isotropic vectors for the quadratic form h(v, βv) is the subspace V^{2}. The flags that satisfy 3.1(iii) are the flags that can be written in the form {0} ⊂ F e_{−2} ⊂ F e_{−2} + F e_{−1} ⊂ F e_{−2} + F e_{−1} + F e_{1} ⊂ V where {e_{−i}, e_{i}} is a symplectic basis of V^{i} for i = 1, 2.

Case II is also quite clear: as in case I the element βδ(v, v) belongs to E_{0} = F^{×} hence h(βv, v) = 2βδ(v, v) has isotropic vectors if and only if δ(v, v) does. Furthermore a flag {0} ⊂ V_{1} ⊂ V_{2} ⊂ V_{3} ⊂ V as in 3.1 must have the form V_{1} = F v, where v is non zero and isotropic for δ, and V_{2} =< v, βv >. Since β^{2} belongs to F^{×} we always have βV_{2} = V_{2} so, if ψ_{β} defines a character of the corresponding unipotent subgroup of G, this character is degenerate.

We finish with case III. We have V = V^{1}⊥V^{2} and β = β_{1} + β_{2}. For v ∈ V, writing v = v_{1} + v_{2} on V = V^{1}⊥V^{2}, we get h(v, βv) = h(v_{1}, β_{1}v_{1}) + h(v_{2}, β_{2}v_{2}). The determinant of this form is the product N_{E_{1}/F} (β_{1})N_{E_{2}/F} (β_{2}). For the form to be anisotropic, the determinant must be a square hence

N_{E_{1}/F} (β_{1}) ≡ N_{E_{2}/F} (β_{2}) mod F^{×2}.

Each β_{i} is skew with characteristic polynomial X^{2} − (−N_{E_{i}/F} (β_{i})): the class of −N_{E_{i}/F} (β_{i}) mod the squares determines, up to isomorphism, the extension E_{i}. So if E_{1} and E_{2} are not isomorphic we are done.

We pursue assuming they are and let E = E_{1} ≃ E_{2}. We may see V as a vector space over E and define δ as in 3.3. The decomposition V = V^{1}⊥V^{2} is orthogonal for δ as well, and for i = 1, 2, v_{i} ↦ h(v_{i}, β_{i}v_{i}) is an anisotropic quadratic form on V^{i}: a (non zero) isotropic vector for h(v, βv) must have the form v = v^{1} + v^{2} with v^{i} ∈ V^{i}, v^{i} ≠ 0. We then have:

h(v, βv) = h(v^{1}, β_{1}v^{1}) + h(v^{2}, β_{2}v^{2}) = 2 β_{1} δ(v^{1}, v^{1}) + 2 β_{2} δ(v^{2}, v^{2}),

and 3.4 follows.

The remark on the non degeneracy of ψ_{β} on U whenever it defines a character follows from the fact that V^{1} and V^{2} are anisotropic for h(v, βv): the corresponding totally isotropic flag has the form (with notations as above) {0} ⊂ V_{1} =< v^{1} + v^{2} >⊂ V_{2} =< v^{1} + v^{2}, β_{1}v^{1} + β_{2}v^{2} >⊂ V_{1}^{⊥} ⊂ V. We never have βV_{2} ⊆ V_{2} unless β_{1}^{2} = β_{2}^{2} – but this belongs to case II, not to case III.

4 Generic representations

In section 3, we have found necessary and sufficient conditions for there to exist a maximal unipotent subgroup U of G on which ψ_{β} defines a character. When there is such a U, this gives us a candidate for trying to build a Whittaker model, as is the case in GL_{N}(F) (see [5]). In this section we consider the case where there is such a U.

4.1 Characters and β-extensions

Proposition 4.1 (cf. [5, Lemma 2.10]). Let [Λ, n, 0, β] be a skew semisimple stratum in A. Let θ ∈ C(Λ, 0, β) be a skew semisimple character and let U be a maximal unipotent subgroup of G such that ψ_{β}|_{U_{der}} = 1. Then

θ|_{H^{1}∩U} = ψ_{β}|_{H^{1}∩U}.

Unfortunately, we have been unable to find a unified proof of this Proposition; the proof is therefore rather ugly, on a case-by-case basis, and we postpone it to the appendix. We continue with the notation of Proposition 4.1. Then we can define a character Θ_{β} of H̃^{1} = (J ∩ U)H^{1} by

Θ_{β}(uh) = ψ_{β}(u)θ(h), for u ∈ J ∩ U, h ∈ H^{1}.

Notice that this is a character, since J normalizes H^{1} and intertwines θ with itself.

Corollary 4.2 (cf. [5, Lemma 2.11]). Let η be the unique irreducible representation of J^{1} which contains θ. Then the restriction of η to J^{1} ∩ U contains the character ψ_{β}|_{J^{1}∩U}.

Proof. The proof is essentially identical to that of [5, Lemma 2.11]. We recall that k_{θ}(x, y) = θ[x, y] defines a nondegenerate alternating form on the finite group J^{1}/H^{1} ([20, Proposition 3.28]). Notice also that the image of H̃^{1} ∩ J^{1} in J^{1}/H^{1} is a totally isotropic subspace for the form k_{θ}, since θ extends to a character Θ_{β} of H̃^{1}. Now we can construct η by first extending Θ_{β} to (the inverse image in J^{1} of) a maximal totally isotropic subspace of J^{1}/H^{1} and then inducing to J^{1}. In particular, η contains Θ_{β} and hence ψ_{β}|_{J^{1}∩U}. ■

Now put J̃^{1} = (J ∩U)J^{1} = (P (Λ)∩B^{×}∩U)J^{1}. Note that, if J/J^{1} is anisotropic then J ∩U = J^{1}∩U so J̃^{1} = J^{1}.

Theorem 4.3 (cf. [16, Theorem 2.6]). Let [Λ, n, 0, β] be a skew semisimple stratum in A as listed in §2.1 and let κ be a β-extension of η to J as described there. Assume there is a maximal unipotent subgroup U of G such that ψ_{β}|_{U_{der}} = 1 and use the notation above. Then:

κ|J̃^{1} ≃ IndJ̃^{1} ; H̃^{1}Θ_{β}.

In particular, we deduce that the restriction of κ to J ∩ U contains the character ψ_{β}|_{J∩U} (cf. [5, Lemma 2.12]). Moreover, except in case IV, and case II when J/J^{1} is isotropic, this means that the restriction of the simple type λ to J ∩ U contains ψ_{β}|_{J∩U}, since λ = κ.

Proof. The proof is in essence the same as that of [16, Theorem 2.6], which it may be useful to read first: because of the similarities, we do not give all the details here. We begin by proving

Ind_{(J^{1}∩U)H^{1}}^{J^{1}}Θ_{β} ≃ η.  (4.4)

We prove this first in cases I and III. Here E^{1} = {x ∈ E/xx̄ = 1} is a maximal torus of G. Then J = E^{1}J^{1} and π = c-Ind_{J}^{G}κ. Since J ∩ U = J^{1} ∩ U, Corollary 4.2 implies that κ contains ψ_{β}|_{J∩U}. Hence π contains ψ_{β} and, since ψ_{β} is then a nondegenerate character, κ contains ψ_{β}|_{J∩U} with multiplicity one (Proposition 1.4). Hence Θ_{β}|_{(J^{1}∩U)H^{1}} occurs in η with multiplicity precisely one and (4.4) follows (see [16, Lemma 2.5]). Note that this already gives the Theorem cases I and III, since E^{1} is maximal and J̃^{1} = J^{1} in these cases. Now we consider the other cases II and IV. Recall that U is given by a flag {0} ⊂ V_{1} ⊂ V_{2} ⊂ V_{3} ⊂ V (see Proposition 3.1), described in §3.3. What we need here is to define a parabolic subgroup P_{0} of G, with unipotent radical U_{0} contained in U and with a specific Levi factor M_{0} conforming to Λ in the sense of [4, §10]. We achieve this according to the case as follows.

(II) From §3.3, the unipotent subgroup U is attached to a flag

{0} ⊂ F w_{−1} ⊂ F w_{−1} + F βw_{−1} ⊂ F w_{−1} + F βw_{−1} + F βw_{1} ⊂ V

where {w_{−1}, w_{1}} is a Witt basis for V over E. We then let M_{0} be the stabilizer of the decomposition V = Ew_{−1} ⊕ Ew_{1} and P_{0} be the stabilizer of the flag {0} ⊂ Ew_{−1} ⊂ V.

(IV) Here U is attached to a flag {0} ⊂ F e_{−2} ⊂ F e_{−2} + F e_{−1} ⊂ F e_{−2} + V^{1} ⊂ V. We can pick e_{2} so that {e_{−2}, e_{2}} is a symplectic basis of V^{2} adapted to Λ^{2}. We then let M_{0} be the stabilizer of the decomposition V = F e_{−2} ⊕ V^{1} ⊕ F e_{2} and P_{0} be the stabilizer of the flag {0} ⊂ F e_{−2} ⊂ F e_{−2} ⊕ V^{1} ⊂ V.

Let k_{θ} be the form defined in the proof of Corollary 4.2. In each case, we have the following properties (see [4, §10] and [21, Lemma 5.6, Corollary 5.10]):

(i) U, H^{1} and J^{1} have Iwahori decompositions with respect to (M_{0}, P_{0});

(ii) J^{1} ∩ U_{0}/H^{1} ∩ U_{0} is a totally isotropic subspace of J^{1}/H^{1} with respect to the form k_{θ};

(iii) J^{1} ∩ M_{0} ∩ U/H^{1} ∩ M_{0} ∩ U is a maximal totally isotropic subspace of J^{1} ∩ M_{0}/H^{1} ∩ M_{0};

(iv) there is an orthogonal sum decomposition

\frac{J^{1}}{H^{1}} = \frac{J^{1} ∩ M_{0}}{H^{1} ∩ M_{0}} ⊥ (\frac{J^{1} ∩ U_{0}}{H^{1} ∩ U_{0}} × \frac{J^{1} ∩ U_{0}^{−}}{H^{1} ∩ U_{0}^{−}} ) ,

where U_{0}^{−} is the unipotent subgroup opposite to U_{0} relative to M_{0}.

Then (J^{1} ∩ U)H^{1} has an Iwahori decomposition with respect to (M_{0}, P_{0}) and (J^{1} ∩ U)H^{1}/H^{1} ≃ J^{1} ∩ M_{0} ∩ U/H^{1} ∩ M_{0} ∩ U ⊥ J^{1} ∩ U_{0}/H^{1} ∩ U_{0} is a maximal totally isotropic subspace of J^{1}/H^{1}. In particular, from the construction of Heisenberg extensions, equation (4.4) follows. For the final stage, as in the construction of β-extensions, there is an 𝔬_{E}-lattice sequence Λ_{m} such that J̃^{1} = (P^{1}(Λ_{m})∩B)J^{1}, and P^{1}(Λ^{m}) still has an Iwahori decomposition with respect to (M_{0}, P_{0}). Defining η̃ to be IndJ̃^{1} ; H̃^{1}Θ_{β}, we see that η̃|J^{1} = η, and one checks that

IndP^{1}(Λ_{m}) ; J̃^{1}η̃ ≃ Ind_{J_{m}^{1}}^{P^{1}(Λ_{m})}η_{m}.

Since this uniquely determines η̃, from the definition of β-extension we get that κ|J̃^{1} = η̃, and the result follows. ■

4.2 The positive level generic supercuspidal representations of Sp_{4}(F)

Here we show that all the positive level supercuspidal representations which are not in the list of Theorem 2.1 are indeed generic.

Theorem 4.5. Let π = c-Ind_{J}^{G}λ be a positive level irreducible supercuspidal representation of G = Sp_{4}(F) with underlying skew semisimple stratum [Λ, n, 0, β]. Suppose that this stratum is not in case IV and that there exists a maximal unipotent subgroup U of G such that ψ_{β}|_{U_{der}} = 1. Then π is generic.

Proof. Except in case II when J/J^{1} is isotropic, this is immediate from Theorem 4.3, since λ|J̃^{1} = κ|J̃^{1} in these cases so λ|_{J∩U} contains the character ψ_{β}|_{J∩U}; hence, by Proposition 1.4, π is generic. Suppose now we are in case II and choose a Witt basis {w_{−1}, w_{1}} for V over E attached to the unipotent subgroup U as in the proof of Theorem 4.3. The quotient J/J^{1} ≃ P (Λ)∩B^{×}/P_{1}(Λ)∩B^{×} is then isomorphic to SL(2, k_{F}) if E/F is ramified, to U(1, 1)(k_{E}/k_{F}) if E/F is unramified.

We have U ∩ B^{×} = \{(\begin{matrix}1 & x \\ 0 & 1 \end{matrix}) /x ∈ F \} with respect to the E-basis \{w_{−1}, w_{1}\} and

U ∩ B^{×} = \{u(x) = (\begin{matrix}1 & 0 & 0 & N(β)x \\ 0 & 1 & x & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix}) /x ∈ F \}

in the symplectic basis \{w_{−1}, βw_{−1}, \frac{β}{N(β)} \frac{w_{1}}{2} , \frac{w_{1}}{2} \} of V over F.

Now recall that λ = κ ⊗ σ, for σ some irreducible cuspidal representation of J/J^{1}. Note that all cuspidal representations σ of U(1, 1)(k_{E}/k_{F}) or SL(2, k_{F}) are generic, since the maximal unipotent subgroup is abelian and σ cannot contain the trivial character, by Lemma 1.3. Here we only need the fact that (the inflation of) σ restricted to J ∩ U ∩ B^{×} contains some character of J ∩ U ∩ B^{×},

which must have the form u(x) ↦ ψ_{F} (αx) for some α in F. Since χ : (\begin{matrix}1 & a & b & c \\ 0 & 1 & x & y \\ 0 & 0 & 1 & z \\ 0 & 0 & 0 & 1 \end{matrix}) ↦ ψ_{F} (αx) is

a character of U, we see that σ|J ∩ U contains χ|J ∩ U. Since κ|_{J∩U} contains ψ_{β} (Theorem 4.3), we deduce that λ|_{J∩U} contains the restriction to J∩U of the character χψ_{β} of U and, by Proposition 1.4, π = c-Ind_{J}^{G}λ is generic (and the character χψ_{β} is non-degenerate). ■

4.3 Case IV

In case IV, any maximal unipotent subgroup U of G on which ψ_{β} defines a character can be written as upper triangular unipotent matrices in a symplectic basis {e_{−2}, e_{−1}, e_{1}, e_{2}} as in the beginning of §3.3. Here {e_{−2}, e_{2}} is a symplectic basis of V^{2} hence U ∩ B^{×} is made of matrices of

the form ( \begin{matrix}1 & 0 & 0 & x \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ). Even though σ, as a cuspidal hence generic representation of SL(2, k_{F}), does

contain a non trivial character of J ∩ U ∩ B^{×}, this character is by no means the restriction of a character of U. This is an heuristic explanation of the fact that none of the case IV supercuspidal representations are generic, unfortunately not a proof. In this section, we prove a crucial result towards non-genericity. The last step will be taken in section 5.4.

Proposition 4.6. Let π = c-Ind_{J}^{G}λ be a positive level supercuspidal representation from case IV and let U be a maximal unipotent subgroup of G such that ψ_{β}|_{U_{der}} = 1. Then the restriction of λ to J ∩ U ∩ B^{×} is a sum of non-trivial characters.

Proof. We retain all the notation of section 4.1 and write U as in the beginning of this section. We also write U^{2} = U∩B^{×} = U∩ Sp(V^{2}). Since J = (J∩P (Λ)∩B^{×})J^{1} we have J∩U = (J∩U^{2})(J^{1}∩U) hence: J̃^{1} = (J ∩ U^{2})J^{1} and H̃^{1} = (J ∩ U^{2})(J^{1} ∩ U)H^{1}. We claim that

(i) J^{1} ∩ U^{2} = H^{1} ∩ U^{2};

(ii) IndJ̃^{1} ; H̃^{1}Θ_{β} ≃ 1_{J∩U^{2}} ⊗ η.

The first claim comes from the definitions in [19] on p.131. Indeed J^{1} ∩ Sp(V^{2}) = H^{1} ∩ Sp(V^{2}) = P_{1}(Λ^{2}). For the second, we note first that 1_{J∩U^{2}} ⊗ η does define a representation of J̃^{1} since J ∩ U^{2} normalizes (J^{1}, η) and η|J^{1} ∩ U^{2} = η|H^{1} ∩ U^{2} is a multiple of ψ_{β}, trivial on U^{2} (β_{2} = 0). Frobenius reciprocity gives a non-zero intertwining operator between those representations, which are irreducible (recall from §4.1 that IndJ̃^{1} ; H̃^{1}Θ_{β}|J^{1} = η).

We have λ = κ⊗σ, with σ an irreducible cuspidal representation of J/J^{1} ≅ SL_{2}(k_{F}). In particular, σ is generic and its restriction to J̃^{1}/J^{1} is a sum of nondegenerate characters χ. On the other hand, the second claim above, plus Theorem 4.3, tell us that κ is trivial on J ∩ U^{2}, q.e.d. ■

5 Non-generic representations

Our goal in this section is to show that the positive level supercuspidal representations which are in the list of Theorem 2.1 are indeed non generic. For cases (I), (II) and (III) we will prove that whenever there is no maximal unipotent subgroup on which the function ψ_{β} is a character (see Proposition 3.4), an irreducible supercuspidal representation of G = Sp_{4}(F) with underlying skew semisimple stratum [Λ, n, 0, β] is not generic. The main tool here is the criterion given in Proposition 1.6, hence we will work out in some detail the restriction of ψ_{β} to one-parameter subgroups. This technique will also provide us with a last piece of argument to settle non-genericity in case (IV).

5.1 The function ψ_{β} on some one-parameter subgroups

In a given symplectic basis (e_{−2}, e_{−1}, e_{1}, e_{2}) of V, we will denote by U_{k}, for k ∈ {−2, −1, 1, 2}, the following root subgroup :

U_{k} = \{1 + x t_{k} ; x ∈ F \} with t_{k}(e_{j}) = \begin{cases}e_{−k} & if j = k; \\ 0 & otherwise. \end{cases}

It is attached to a long root and has a filtration indexed by s ∈ ℤ: U_{k}(s) = {1 + x t_{k} : x ∈ 𝔭_{F}^{s} }. Let β be an element of A and let ψ_{β} be the function on G defined by ψ_{β}(x) = ψ(trβ(x − 1)), x ∈ G. For k in {−2, −1, 1, 2}, let ϵ(k) be the sign of k.

Lemma 5.1. Fix g ∈ G and k in {−2, −1, 1, 2}. Let s ∈ ℤ.

(i) For any x in F we have: ψ_{β} (1 + x^{g}t_{k}) = ψ (ϵ(k) x h(ge_{−k}, βge_{−k})).

(ii) The character ψ_{β} of ^{g}U_{k} is non trivial on ^{g}U_{k}(s) if and only if s ≤ −ν_{F} h(ge_{−k}, βge_{−k}).

Proof. We keep the usual notation ^{g}u = gug^{−1}, u^{g} = g^{−1}ug. We have for x ∈ F:

ψ_{β} (1 + x^{g}t_{k}) = ψ (tr(βx^{g}t_{k})) = ψ (tr(xβ^{g} t_{k})) = ψ(x(β^{g})_{k,−k})

while the (k, −k) entry of β^{g} is (β^{g})_{k,−k} = ϵ(k)h(e_{−k}, β^{g}e_{−k}) = ϵ(k)h(ge_{−k}, βge_{−k}). ■

5.2 Proof for a maximal simple stratum

We work in this paragraph with a maximal simple stratum [Λ, n, 0, β] in End_{F} V, satisfying the assumption in Theorem 2.1 namely, in terms of the form δ (see (3.3) and §3.2):

(i) E = F [β] is a biquadratic extension of F with fixed points E_{0} under x ↦ x̄; in particular N_{E/E_{0}}(E^{×}) = F^{×}E_{0}^{×2}.

(ii) The subset D(V) = {βδ(v, v) : v ∈ V, v ≠ 0} of E_{0}^{×} is the N_{E/E_{0}}(E^{×})-coset in E_{0}^{×} that does not contain the kernel of tr_{E_{0}/F}.

Note that lattice duality with respect to the form δ, defined by L^{#} = {v ∈ V : δ(v, L) ⊆ 𝔭_{E}} for L an 𝔬_{E}-lattice in V, coincides with lattice duality with respect to h. We may and do assume that the self-dual lattice chain Λ satisfies Λ(i)^{#} = Λ(d − i), i ∈ ℤ, with d = 0 or 1. We also may and do assume that Λ is strict, i.e. has period 2.

Lemma 5.2. Pick v_{0} in V such that Λ(i) = 𝔭_{E}^{i}v_{0} for i ∈ ℤ. We have ν_{E}δ(v_{0}, v_{0}) = 1 − d and, for v ∈ V and s in ℤ:

\begin{matrix}ν_{F} h(v, βv) = \frac{1}{e(E_{0}/F)}ν_{E_{0}}(βδ(v, v)); \\ ν_{F} h(v, βv) = s ⟺ v ∈ Λ(s + \frac{1}{2}(n + d − 1)) − Λ(s + \frac{1}{2}(n + d + 1)). \end{matrix}

Proof. We have h(v, βv) = −2 tr_{E_{0}/F} (βδ(v, v)) (§3.2); the first statement is thus a consequence of the following property:

Let x be an element of the F^{×}E_{0}^{×2}-coset in E_{0}^{×} that does not contain the kernel of tr_{E_{0}/F}. Then ν_{F} tr_{E_{0}/F} x = \frac{1}{e(E_{0}/F)}ν_{E_{0}}x.

Indeed we have ν_{F} tr_{E_{0}/F} x = \frac{1}{e(E_{0}/F)} [ν_{E_{0}}x + ν_{E_{0}} (1 + \frac{x̃}{x})] where x ↦ x̃ is the Galois conjugation

of E_{0} over F. Let x and y in E_{0}^{×} such that \frac{x̃}{x} and \frac{ỹ}{y} belong to −1 + 𝔭_{E_{0}} and let u = x/y. Then

ũ/u belongs to 1 + 𝔭_{E_{0}}, which implies that u belongs to F^{×}E_{0}^{×2} (easy checking according to the ramification of E_{0} over F). Hence x and y are in the same F^{×}E_{0}^{×2}-coset, that must be the coset containing the kernel of tr_{E_{0}/F}, q.e.d.

The second statement is now immediate. We can write v ∈ V as v = u v_{0} with u ∈ E. Then:

ν_{F} h(v, βv) = \frac{1}{e(E_{0}/F)} \frac{1}{e(E/E_{0})} [ν_{E}(βδ(v_{0}, v_{0})) + ν_{E}(uū)]

hence ν_{F} h(v, βv) = \frac{1}{2}(−n + 1 − d) + ν_{E}u, q.e.d. ■

We now fix a symplectic basis (e_{−2}, e_{−1}, e_{1}, e_{2}) adapted to Λ: there are non-decreasing functions α_{s} : ℤ → ℤ such that

Λ(j) = ⨁_{s∈{−2,−1,1,2}} 𝔭_{F}^{α_{s}(j)}e_{s} (j ∈ ℤ).  (5.3)

Proposition 5.4. For any g ∈ P (Λ), for any k ∈ {−2, −1, 1, 2}, the character ψ_{β} of ^{g}U_{k} is non trivial on ^{g}U_{k} ∩ P_{[_{\frac{n}{2}}]+1}(Λ).

Proof. Note first that P (Λ) normalizes P_{[_{\frac{n}{2}}]+1}(Λ), so ^{g}U_{k} ∩ P_{[_{\frac{n}{2}}]+1}(Λ) is equal to ^{g}U_{k}(s), where s ∈ ℤ is defined by U_{k} ∩ P_{[_{\frac{n}{2}}]+1}(Λ) = U_{k}(s). Using Lemma 6.2 for our lattice sequence Λ of period 2 we get, for x ∈ F:

1 + x t_{k} ∈ P_{[_{\frac{n}{2}}]+1}(Λ) ⟺ ∀j, x t_{k}Λ(j) ⊂ Λ(j + [\frac{n}{2}] + 1)
⟺ ∀j, e_{−k} ∈ Λ(j − 2α_{k}(j) + [\frac{n}{2}] + 1 − 2ν_{F} x)
⟺ ν_{Λ} (e_{−k}) ≥ max \{j − 2α_{k}(j); j ∈ ℤ\} + [\frac{n}{2}] + 1 − 2ν_{F} x
⟺ ν_{Λ} (e_{−k}) ≥ ν_{Λ} (e_{k}) + [\frac{n}{2}] + 1 − 2ν_{F} x
⟺ 2ν_{F} x ≥ 2ν_{Λ} (e_{k}) + [\frac{n}{2}] + 2 − d.

Hence s = ν_{Λ} (e_{k}) + 1 + [\frac{[\frac{n}{2}] − d + 1}{2}]; we have to prove s ≤ −ν_{F} h(ge_{−k}, βge_{−k}) (Lemma 5.1).

Since g belongs to P (Λ) we have by Lemmas 5.2 and 6.2:

ν_{F} h(ge_{−k}, βge_{−k}) = ν_{F} h(e_{−k}, βe_{−k}) = −ν_{Λ} (e_{k}) + d − 1 − \frac{1}{2}(n + d − 1)).

The condition we need is thus [\frac{[\frac{n}{2}] − d + 1}{2}] ≤ \frac{1}{2}(n − d − 1) which holds for n ≥ 1 (the right hand

side is an integer by Lemma 5.2). ■

We now derive Theorem 2.1 in this case: the representation π is not generic. Indeed if it was, we could find, by Proposition 1.6, some g ∈ P (Λ) and some k ∈ {−2, −1, 1, 2} such that λ contains the trivial character of ^{g}U_{k} ∩ J. In particular the restriction of λ to P_{[_{\frac{n}{2}}]+1}(Λ) would contain the trivial character of ^{g}U_{k} ∩ P_{[_{\frac{n}{2}}]+1}(Λ). Since this restriction is a multiple of ψ_{β} this is impossible.

5.3 Proof for a maximal semi-simple or non-maximal simple stratum

We will in this section treat simultaneously cases (II) and (III) in Theorem 2.1, granted that:

Lemma 5.5. Under the assumption of case (II) in Theorem 2.1, any splitting V = V^{1}⊥V^{2} of V into two one-dimensional E-vector spaces splits the lattice chain Λ, that is, for any t ∈ ℤ:

Λ(t) = Λ^{1}(t)⊥Λ^{2}(t) with Λ^{i}(t) = Λ(t) ∩ V^{i}, i = 1, 2.

Proof. We know from section 3 that the given assumption amounts to the fact that the quadratic form v ↦ h(v, βv) on V has no non trivial isotropic vectors, which implies that the anti-hermitian form δ on V defined by h(av, w) = tr_{E/F} (aδ(v, w)) for all a ∈ E, v, w ∈ V, is anisotropic. From [1]A.2, the lattice chain underlying Λ is thus the unique self-dual 𝔬_{E}-lattice chain in V. On the other hand self-dual 𝔬_{E}-lattice chains Λ^{i} in V^{i}, i = 1, 2, can be summed into a self-dual 𝔬_{E}-lattice chain in V. Unicity implies that Λ is obtained in this way. ■

We can now let [Λ, n, 0, β], with Λ = Λ^{1}⊥Λ^{2}, β = β_{1} + β_{2} and n = max {n_{1}, n_{2}}, be a maximal skew semi-simple stratum or a non-maximal skew simple stratum in End_{F} V: we place ourselves in the situation of case (II) or case (III) in Proposition 3.4, case (II) being obtained by letting E_{1} = E_{2} = E and β_{1} = β_{2} = β. In particular E_{1} = F [β_{1}] is always isomorphic to E_{2} = F [β_{2}] and one checks easily that the conditions (ii) or (iii) in Proposition 3.4 are equivalent to saying that the symplectic form satisfies:

∀v_{1} ∈ V^{1}, ∀v_{2} ∈ V^{2}, − h(v_{1}, βv_{1}) h(v_{2}, βv_{2}) ∉ N_{E_{i}/F} (E_{i}^{×}).

For an homogeneous treatment regardless of the ramification over F of the quadratic extensions involved, we use the conventions explained in section 6.2, namely the lattices sequences Λ^{1}, Λ^{2} and Λ are all normalized in such a way that they have period 4 over F and duality given by d = 1.

Lemma 5.6. Let v_{1} ∈ V^{1}, v_{2} ∈ V^{2} and v = v_{1} + v_{2} ∈ V. Let e = e(E_{i}/F).

(1) ν_{F} h(v, βv) = min {ν_{F} h(v_{1}, β_{1}v_{1}), ν_{F} h(v_{2}, β_{2}v_{2})};

(2) ν_{Λ}(v) ≥ 2ν_{F} h(v, βv) − max \{\frac{2ν_{E_{1}}β_{1}}{e} , \frac{2ν_{E_{2}}β_{2}}{e} \}.

Proof. (1) We have h(v, βv) = h(v_{1}, β_{1}v_{1}) + h(v_{2}, β_{2}v_{2}). The assertion follows if the valuations of h(v_{i}, β_{i}v_{i}), i = 1, 2, are distinct. If they are equal and finite, write h(v, βv) = h(v_{1}, β_{1}v_{1})(1 + \frac{h(v_{2},β_{2}v_{2})}{h(v_{1},β_{1}v_{1})}). Since \frac{h(v_{2},β_{2}v_{2})}{h(v_{1},β_{1}v_{1})} cannot be congruent to −1 mod 𝔭_{F} (its opposite would be a square hence a norm) the result follows.

(2) From Lemma 6.4 and (1) we get ν_{Λ}(v) = min_{i=1,2}ν_{Λ_{i}}(v_{i}) = min_{i=1,2}(2ν_{F} h(v_{i}, β_{i}v_{i}) − \frac{2ν_{E_{i}}β_{i}}{e} ) whence the result. ■

We fix symplectic bases {e_{i}, e_{−i}} of V^{i}, i = 1, 2, adapted to Λ_{i} and use notation 5.3. We denote by 1^{1} and 1^{2} the orthogonal projections of V onto V^{1} and V^{2} respectively. We need information on the intersection of one-parameter subgroups ^{g}U_{k} with subgroups of G of the following form (with a_{1}, a_{2} positive integers and a = max {a_{1}, a_{2}}):

L = (1 + 𝔞_{a_{1}}(Λ^{1}) + 𝔞_{a_{2}}(Λ^{2}) + 𝔞_{a}(Λ)) ∩ G.

Lemma 5.7. Let g ∈ P (Λ); let g_{i} = ν_{Λ} (1^{i} (ge_{−k})) ∈ ℤ ∪ +∞. Then

gU_{k} ∩ L = ^{g}U_{k}(s) with s = [\frac{ν_{Λ} (e_{k}) + max_{i=1,2}\{a_{i} − g_{i}\} + 3}{4}] .

Proof. We have for x ∈ F and g ∈ G (see [9] 2.9):

1 + x^{g}t_{k} ∈ L ⟺ x^{g}t_{k}Λ(j) ⊆ Λ^{1}(j + a_{1}) + Λ^{2}(j + a_{2}) for any j ∈ ℤ.

Since t_{k} Λ(j) = 𝔭_{F}^{α_{k}(j)}e_{−k} we have, if g belongs to P (Λ):

\begin{aligned}1 + x^{g}t_{k} ∈ L & ⟺ ∀j, xge_{−k} ∈ Λ^{1}(j + a_{1}− 4α_{k}(j)) + Λ^{2}(j + a_{2}− 4α_{k}(j)) \\ & ⟺ for i = 1, 2, ∀j, 4ν_{F} x + g_{i} ≥ j − 4α_{k}(j) + a_{i}. \end{aligned}

We conclude with Lemma 6.2. ■

We now apply this lemma to the subgroup L obtained with a_{i} = [\frac{n_{i}}{2} ]+1, i = 1, 2. Define integers l_{1} and l_{2} by l_{i} = ν_{F} h(1^{i} (ge_{−k}) , β_{i}1^{i} (ge_{−k})). From Lemmas 5.1 and 5.7, the character ψ_{β} of gU_{k}g^{−1} is non trivial on gU_{k}g^{−1} ∩ L if and only if

[\frac{ν_{Λ} (e_{k}) + max_{i=1,2}\{[\frac{n_{i}}{2} ] + 1 − g_{i}\} + 3}{4}] ≤ −min \{l_{1}, l_{2}\}.

On the other hand we have g_{i} = 2l_{i} − \frac{2ν_{E_{i}}β_{i}}{e} by Lemma 6.4, and Lemma 6.3 relates n_{i} = −ν_{Λ_{i}}(β_{i}) and ν_{E_{i}}β_{i}; in any case, one checks:

max_{i=1,2}\{[\frac{n_{i}}{2} ] + 1 − g_{i}\} = ε − 2min \{l_{1}, l_{2}\} with ε = 0 or 1.

It follows that ψ_{β} is non trivial on gU_{k}g^{−1} ∩ L if and only if ν_{Λ} (e_{k}) ≤ −2min {l_{1}, l_{2}} − ε, that is, ν_{Λ} (e_{k}) ≤ −2ν_{F} h(ge_{−k}, βge_{−k}) − ε. Now Lemma 5.6 gives us:

ν_{Λ} (ge_{−k}) ≥ 2ν_{F} h(ge_{−k}, βge_{−k}) − max \{\frac{2ν_{E_{1}}β_{1}}{e} , \frac{2ν_{E_{2}}β_{2}}{e} \}.

Since g belongs to P (Λ), we have ν_{Λ} (ge_{−k}) = ν_{Λ} (e_{−k}) = −ν_{Λ} (e_{k}) (Lemma 6.2), which implies the desired inequality. We have just proven:

Proposition 5.8. For any g ∈ P (Λ), for any k ∈ {−2, −1, 1, 2}, the character ψ_{β} of gU_{k}g^{−1} is non trivial on gU_{k}g^{−1} ∩ L, where

L = (I + 𝔞_{[_{\frac{n_{1}}{2}} ]+1}(Λ^{1}) + 𝔞_{[_{\frac{n_{2}}{2}} ]+1}(Λ^{2}) + 𝔞_{[_{\frac{n}{2}}]+1}(Λ)) ∩ G.

At this point we can derive Theorem 2.1 in cases (II) and (III) as in the previous subsection: the representation π is not generic. Indeed if it was, we could find, by Proposition 1.6, some g ∈ P (Λ) and some k ∈ {−2, −1, 1, 2} such that λ contains the trivial character of ^{g}U_{k} ∩ J. In particular the restriction of λ to the above subgroup L would contain the trivial character of ^{g}U_{k} ∩ L. Since this restriction is a multiple of ψ_{β} this is impossible.

5.4 Non-genericity in the degenerate case (IV)

We let again [Λ, n, 0, β], with Λ = Λ^{1} ⊕ Λ^{2}, β = β_{1} + β_{2} and n = max {n_{1}, n_{2}}, be a skew semisimple stratum in End_{F} V, but we assume that one of the two simple strata involved is null (case IV). Although there does exist a maximal unipotent subgroup on which ψ_{β} is a character, this character is then degenerate (Proposition 3.4 and Remark 3.5). We will show that the corresponding supercuspidal representation is non-generic, using Proposition 4.6 and the criterion 1.6. Criterion 1.6 involves conjugacy by elements in an Iwahori subgroup. We will find convenient to use the standard Iwahori subgroup, and to use an Iwahori subgroup normalizing the lattice chain Λ. These conditions can both be fulfilled at the possible cost of exchanging Λ^{1} and Λ^{2}, that is, we have to complicate notations and let {1, 2} = {r, s} with [Λ^{r}, n_{r}, 0, β_{r}] not null and [Λ^{s}, n_{s}, 0, β_{s}] = [Λ^{s}, 0, 0, 0]; in particular n = n_{r}. Since the proof below is rather technical, we first sketch it, assuming β_{2} = 0. In a symplectic basis as in §3.3, ψ_{β} does define a character of the upper and lower triangular unipotent subgroups, trivial on the long root subgroups U_{±2} corresponding to the null stratum but non trivial on the other long root subgroups U_{±1}. As in the previous case, we have a subgroup L of J on which λ restricts to a multiple of ψ_{β}. We will show that for any Iwahori conjugate ^{g}U_{±1}, ψ_{β} is non trivial on ^{g}U_{±1} ∩ L. Next we will identify subsets X_{2} and X_{−2} of I such that, for g ∈ X_{±2}, ψ_{β} is again non trivial on gU_{±2} ∩ L. The last step will be to show that for g ∈ I − X_{±2}, if the representation λ contains the trivial character of ^{g}U_{±2} ∩ J, then it contains the trivial character of U_{±2} ∩ J – this last possibility being excluded by Proposition 4.6. Hence, by Proposition 1.6, the representation induced from λ cannot be generic. We pick a symplectic basis (e_{−2}, e_{−1}, e_{1}, e_{2}) of V adapted to Λ and such that P (Λ) contains the standard Iwahori subgroup I consisting of matrices with entries in 𝔬_{F} which are upper triangular modulo 𝔭_{F}. We normalize lattice sequences Λ_{1} and Λ_{2} such that they have period 4 over F and duality invariant 1 and define ϵ = 0 if Λ_{r} contains a self-dual lattice, ϵ = 1 otherwise (§6.2). Let:

L = (I + 𝔞_{1}(Λ^{s}) + 𝔞_{[_{\frac{n}{2}}]+1}(Λ^{r}) + 𝔞_{[_{\frac{n}{2}}]+1}(Λ)) ∩ G.

This is a subgroup of H^{1} (from the definition [19] p.131) on which θ restricts to ψ_{β} ([20] Lemma 3.15).

Lemma 5.9. Let k in {−2, −1, 1, 2}. For y ∈ P (Λ), define y_{r} = ν_{Λ}(1^{r}(ye_{−k})), y_{s} = ν_{Λ}(1^{s}(ye_{−k})) and l_{r} = ν_{F} h(1^{r} (ye_{−k}) , β_{r}1^{r} (ye_{−k})). The character ψ_{β} of yU_{k}y^{−1} is non trivial on yU_{k}y^{−1} ∩ L if and only if either 1^{r}(ye_{−k}) = 0 or 1^{r}(ye_{−k}) ≠ 0 and

\{ \begin{matrix}y_{r} ≤ ν_{Λ}(e_{−k}) + [\frac{n}{2}] + 1 − 2ϵ \\ y_{s} ≥ −ν_{Λ}(e_{−k}) + 4 l_{r} + 1 \end{matrix}

Proof. We have h(v, βv) = h(v_{r}, β_{r}v_{r}) (for v = v_{1} + v_{2}, v_{i} ∈ V^{i}), so with Lemmas 6.4 and 6.3:

ν_{F} h(v, βv) = t ⟺ 1^{r}(v) ∈ Λ^{r}(2t + [\frac{n}{2}] + 1 − ϵ) − Λ^{r}(2t + [\frac{n}{2}] + 1 − ϵ + 1).

In particular, if 1^{r}(ye_{−k}) = 0, the character ψ_{β} is trivial on yU_{k}y^{−1} (Lemma 5.1). We now assume 1^{r}(ye_{−k}) ≠ 0 and get from Lemma 6.4:

y_{r} = 2 l_{r} + [\frac{n}{2}] + 1 − ϵ.  (5.10)

We apply Lemma 5.7 to L:

yU_{k}y^{−1} ∩ L = yU_{k}(t)y^{−1} with t = [\frac{ν_{Λ}(e_{k}) + max \{1 − y_{s}, ϵ − 2 l_{r}\} + 3}{4}] .

Using Lemma 5.1 we conclude that the character ψ_{β} of yU_{k}y^{−1} is non trivial on yU_{k}y^{−1} ∩ L if and only if t ≤ −l_{r} whence the result (note that ν_{Λ}(e_{k}) = −ν_{Λ}(e_{−k}) by Lemma 6.2). ■

Lemma 5.11. Let y ∈ I. If |k| = r, or if |k| = s and 1^{r}(ye_{−k}) ∉ Λ^{r}(ν_{Λ}(e_{−k}) + [\frac{n+1}{2}]), the character ψ_{β} of yU_{k}y^{−1} is non trivial on yU_{k}y^{−1} ∩ L.

Proof. Since y belongs to P (Λ) we certainly have 1^{r}(ye_{−k}) ∈ Λ^{r}(ν_{Λ}(e_{−k})), 1^{s}(ye_{−k}) ∈ Λ^{s}(ν_{Λ}(e_{−k})), and:

either (a) 1^{r}(ye_{−k}) ∉ Λ^{r}(ν_{Λ}(e_{−k}) + 1)
or (b) 1^{r}(ye_{−k}) ∈ Λ^{r}(ν_{Λ}(e_{−k}) + 1) and 1^{s}(ye_{−k}) ∉ Λ^{s}(ν_{Λ}(e_{−k}) + 1).

(5.12)

Assume first (a). Then the first condition in Lemma 5.9 is satisfied and the second will hold if −ν_{Λ}(e_{−k}) + 4 l_{r} + 1 ≤ ν_{Λ}(e_{−k}). But we have y_{r} = 2 l_{r} + [\frac{n}{2}] + 1 − ϵ (5.10) whence

4l_{r} = 2(ν_{Λ}(e_{−k}) − [\frac{n}{2}] − 1 + ϵ) ≤ 2ν_{Λ}(e_{−k}) − 1.

Hence:

if y ∈ P (Λ) and 1^{r}(ye_{−k}) ∉ Λ^{r}(ν_{Λ}(e_{−k}) + 1), ψ_{β} is non trivial on yU_{k}y^{−1} ∩ L.  (5.13)

The discussion now will rely on |k|. If |k| = r and y ∈ I, then (a) holds and (5.13) gives the result. If now y ∈ I and |k| = s, we are left with case (b) in (5.12); in particular y_{s} = ν_{Λ}(e_{−k}). We only have to check that the assumption 1^{r}(ye_{−k}) ∉ Λ^{r}(ν_{Λ}(e_{−k}) + [\frac{n+1}{2}]) implies the two inequalities in 5.9, which is straightforward granted that, when ϵ = 1, n is even (6.3). ■

Lemma 5.14. Let |k| = s and y ∈ I such that 1^{r}(ye_{−k}) ∈ Λ^{r}(ν_{Λ}(e_{−k})+[\frac{n+1}{2}]). Define z = I+Z−Z̄ with Z(e_{−k}) = −y_{−k,−k}^{−1}1^{r}(ye_{−k}) and Z(e_{t}) = 0 for t ≠ −k. Then z belongs to P[_{\frac{n+1}{2}}](Λ), contained in J ∩ I, and 1^{r}(zye_{−k}) = 0.

Proof. One checks easily that z belongs to I and 1^{r}(zye_{−k}) = 0. It remains to show that Z belongs to 𝔞[_{\frac{n+1}{2}}](Λ). We have, using notation 6.1 and Lemma 6.2:

Z ∈ 𝔞[_{\frac{n+1}{2}}](Λ) ⟺ Z 𝔭_{F}^{α_{−k}(t)}e_{−k} ⊆ Λ(t + [\frac{n+1}{2}]) for any t ∈ ℤ
⟺ Ze_{−k} ∈ Λ^{r}(t − 4α_{−k}(t) + [\frac{n+1}{2}]) for any t ∈ ℤ
⟺ 1^{r}(ye_{−k}) ∈ Λ^{r}(max_{t∈ℤ}\{t − 4α_{−k}(t)\} + [\frac{n+1}{2}])
⟺ 1^{r}(ye_{−k}) ∈ Λ^{r}(ν_{Λ}(e_{−k}) + [\frac{n+1}{2}]).
■

With this in hand we are ready to conclude:

Proposition 5.15. If the representation Ind_{J}^{G}λ has a Whittaker model, there exists k ∈ {−s, s} such that λ contains the trivial character of U_{k} ∩ J = U_{k} ∩ P (Λ^{s}).

Proof. Recall Proposition 1.6: if the representation Ind_{J}^{G}λ has a Whittaker model, there exists k ∈ {−2, −1, 1, 2} and y ∈ I such that λ contains the trivial character of yU_{k}y^{−1} ∩ J. Note that if (k, y) is such a pair, so is (k, zyx) for any z ∈ J ∩ I and x ∈ I ∩ N_{G}(U_{k}). Assume we are in this situation and pick such a pair (k, y). Since the restriction of λ to L is a multiple of ψ_{β}, Lemma 5.11 tells us that we must have |k| = s and 1^{r}(ye_{−k}) ∈ Λ^{r}(ν_{Λ}(e_{−k})+[\frac{n+1}{2}]). The Proposition is then an immediate consequence of the following fact:

(∗) The double class (J ∩ I) y (I ∩ N_{G}(U_{k})) contains an element of J ∩ I, indeed of I ∩ P (Λ^{s}).

Let us now prove this fact. Assume first that k = s = 2 (then r = 1). Using the standard Iwahori decomposition of y ∈ I and the fact that upper triangular matrices normalize U_{2} we may assume that y is a lower triangular unipotent matrix. Now since 1^{1}(ye_{−k}) belongs to Λ^{r}(ν_{Λ}(e_{−k}) + [\frac{n+1}{2}]) we may change y into zy where z is defined in Lemma 5.14. Note that z is also lower triangular unipotent, hence so is zy. Since 1^{1}(zye_{−k}) = 0, zy has the following shape:

zy = (\begin{matrix}1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & b & 1 & 0 \\ c & 0 & 0 & 1 \end{matrix}) .

The middle block (\begin{matrix}1 & 0 \\ b & 1 \end{matrix}) centralizes U_{2} so the double class contains (\begin{matrix}1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ c & 0 & 0 & 1 \end{matrix}) which belongs to

I ∩ Sp(V^{2}) hence to P (Λ) ∩ Sp(V^{2}) = P (Λ^{2}), q.e.d. The case k = −s = −2 is identical, replacing lower triangular by upper triangular. The cases k = s = 1 and k = −s = −1 are obtained similarly,

using conjugation by w =(\begin{matrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ ϖ_{F} & 0 & 0 & 0 \\ 0 & ϖ_{F} & 0 & 0 \end{matrix}) or w^{−1}, elements of GSp_{4}(F) normalizing I, to get a

convenient Iwahori decomposition for the initial element y. ■

We are at last ready to prove non-genericity of supercuspidal representations coming from case (IV), which will finish the proof of Theorem 2.1. The group U_{k} ∩ J above is equal to J ∩ U ∩ B^{×} for some unipotent subgroup U chosen as in Proposition 4.6 – indeed we have k ∈ {−s, s} so U_{k} is a long root subgroup attached to the two-dimensional space in which we have a null stratum. We know from Proposition 4.6 that the restriction of λ to U_{k} ∩ J, a sum of non trivial characters, cannot contain the trivial character. Hence Ind_{J}^{G}λ doesn’t have a Whittaker model.

6 Appendix: normalization of lattice sequences

We gather in the first two subsections the technical information about direct sums of lattice sequences that we need in several parts of the paper. Specifically, in most cases we deal with direct sums of self-dual lattice sequences in two-dimensional symplectic spaces and it will be convenient to homogeneize their F -periods and duality invariants. Then we proceed with preliminary results in view of the proof, in §6.4, of Proposition 4.1

6.1 Self-dual lattice sequences

Let Λ be an 𝔬_{F} -lattice sequence in a finite dimensional F -vector space V and let 𝔞_{k}(Λ) be the corresponding filtration of A. We define:

ν_{Λ}(v) = max {i ∈ ℤ; v ∈ Λ(i)} (v ∈ V); ν_{Λ}(g) = max {i ∈ ℤ; g ∈ 𝔞_{i}(Λ)} (g ∈ A).

When V is equipped with a symplectic form h, lattice duality with respect to h is defined by L^{#} = {v ∈ V : h(v, L) ⊆ 𝔭_{F} } for L an 𝔬_{F} -lattice in V. An 𝔬_{F} -lattice sequence Λ in V is self-dual if there is an integer d(Λ), the duality invariant of Λ, such that Λ(t)^{#} = Λ(d(Λ) − t). We now let Λ be a self-dual lattice sequence of period e in V and we pick a symplectic basis (e_{−2}, e_{−1}, e_{1}, e_{2}) adapted to Λ: there are non-decreasing functions α_{s} : ℤ → ℤ such that

Λ(j) = ⨁_{s∈{−2,−1,1,2}} 𝔭_{F}^{α_{s}(j)}e_{s} (j ∈ ℤ).  (6.1)

We will need the following straightforward property:

Lemma 6.2. For k in {−2, −1, 1, 2}, we have

ν_{Λ} (e_{k}) = −ν_{Λ} (e_{−k}) + d(Λ) − 1 = max {j − eα_{k}(j); j ∈ ℤ}.

Proof. Since the period of Λ is e, the two sets {j −eα_{k}(j); j ∈ ℤ} and {i ∈ ℤ; α_{k}(i) = 0} coincide: the valuation of e_{k} is the maximum of either one. Next we use duality to check that

e_{−k} ∉ Λ(x + d) ⟺ e_{−k} ∉ Λ(−x)^{#} ⟺ h(e_{−k}, 𝔭_{F}^{α_{k}(−x)}e_{k}) ∉ 𝔭_{F} ⟺ α_{k}(−x) ≤ 0.
■

6.2 Normalization of our lattice sequences

We start as in §2 case (III) with an orthogonal decomposition V = V^{1} ⊥ V^{2} and not null skew simple strata [Λ^{i}, n_{i}, 0, β_{i}] in End_{F} (V^{i}). We let E_{i} = F [β_{i}]. We will find convenient to normalize the lattice sequences Λ^{i} in such a way that their sum Λ is given by Λ(t) = Λ^{1}(t)⊥Λ^{2}(t) for any t ∈ ℤ, and is self-dual. This will be the case provided that Λ^{1} and Λ^{2} have the same period and d(Λ^{1}) = d(Λ^{2}) = 1 (see [20] for instance). The quadratic form v ↦ h(v, β_{i}v) on V^{i} has no non trivial isotropic vectors, since the anti-hermitian form δ_{i} on V^{i} defined by h(av, w) = tr_{E_{i}/F} (aδ_{i}(v, w)) for all a ∈ E_{i}, v, w ∈ V^{i}, is anisotropic. Let v_{i} be a basis of V^{i} over E_{i}. There is some u_{i} in E_{i}, satisfying ū_{i} = −u_{i}, such that this form reads δ_{i}(xv_{i}, yv_{i}) = uxȳ for all x, y ∈ E_{i}. Since E_{i} is tame over F, lattice duality is the same for h and δ_{i}, namely Λ^{i}(t)^{#} = {v ∈ V^{i}; δ_{i}(v, Λ^{i}(t)) ⊂ 𝔭_{E_{i}}}. We have here (𝔭_{E_{i}}^{t}v_{i})^{#} = 𝔭_{E_{i}}^{d_{i}−t}v_{i} with d_{i} = 1 − ν_{E}u_{i}. Hence, up to a translation in indices, the unique self-dual 𝔬_{E_{i}}-lattice chain (L_{i})_{i∈ℤ} in V^{i} satisfies one of three possibilities:

(i) E_{i} is ramified over F and d_{i} = 0;

(ii) E_{i} is unramified over F and d_{i} = 0;

(iii) E_{i} is unramified over F and d_{i} = 1.

In the two first cases we put L′_{i}(t) = L_{i}([\frac{t}{2}]) and get a lattice sequence with duality invariant d′_{i} = 1; in the third case we keep L′_{i} = L_{i}. In the ramified case we put Λ_{i} = L′_{i}: it has period 4. We now need to put Λ_{i} = 2L′_{i} in the second case, Λ_{i} = 4L′_{i} in the third case, and we get a normalization of our 𝔬_{E_{i}}- lattice sequences in V_{i} such that their period is 4 and duality invariant 1. We will use the following straightforward properties of Λ_{i}, in each of the three cases above:

Lemma 6.3. Normalize the lattice sequence Λ^{i} such that its period over F is 4 and d(Λ^{i}) = 1.

(i) If E_{i} is ramified over F then ν_{Λ_{i}}(V^{i} − {0}) = 2ℤ + 1 and ν_{Λ_{i}}(β_{i}) = 2ν_{E_{i}}(β_{i}) + 1;

(ii) If E_{i} is unramified over F and Λ_{i} contains a self-dual lattice then ν_{Λ_{i}}(V^{i} − {0}) = 4ℤ + 2 and ν_{Λ_{i}}(β_{i}) = 2(2ν_{E_{i}}(β_{i}) + 1);

(iii) If E_{i} is unramified over F and Λ_{i} does not contain a self-dual lattice then ν_{Λ_{i}}(V^{i} − {0}) = 4ℤ and ν_{Λ_{i}}(β_{i}) = 4ν_{E_{i}}(β_{i}).

We need to relate, under these conventions, the valuations relative to the lattice chains Λ_{i} with the valuations over F of the quadratic forms h(v, β_{i}v), v ∈ V^{i}.

Lemma 6.4. Let e_{i} = e(E_{i}/F) and normalize the lattice sequence Λ^{i} such that its period over F is 4 and d(Λ^{i}) = 1. For any v ∈ V^{i} we have:

ν_{Λ_{i}}(v) = 2ν_{F} h(v, β_{i}v) − \frac{2ν_{E_{i}}β_{i}}{e_{i}} .

Proof. For v in V^{i} and a in E_{i} we have h(av, β_{i}av) = N_{E_{i}/F} (a)h(v, β_{i}v). The map v ↦ h(v, β_{i}v) on V_{i} is thus constant on the sets Λ_{i}(t) − Λ_{i}(t + 1) hence factors through the valuation ν_{Λ_{i}}: there is a map ϕ, defined on the image of ν_{Λ_{i}} and with values in ℤ, such that ν_{F} h(v, β_{i}v) = ϕ(ν_{Λ_{i}}(v)) for any non zero v in V^{i}. Periodicity over E_{i} implies ϕ(t + 2e_{i}) = ϕ(t) + e_{i}, t ∈ ℤ: computing one

value of ϕ is enough. We certainly have ν_{F} h(v, β_{i}v) = \frac{1}{e_{i}} (ν_{E_{i}}β_{i} + ν_{E_{i}}δ_{i}(v, v)) whence:

(i) if E_{i} is ramified over F, then Λ_{i}(1) = Λ_{i}(0) = Λ_{i}(1)^{#} so ν_{Λ_{i}}(v) = 1 implies ν_{E_{i}}δ_{i}(v, v)) = 1

and ν_{F} h(v, β_{i}v) = \frac{1}{2}(ν_{E_{i}}β_{i} + 1);

(ii) if E_{i} is unramified over F and Λ_{i} contains a self-dual lattice then Λ_{i}(2)^{#} = Λ_{i}(−1) = Λ_{i}(2) so ν_{Λ_{i}}(v) = 2 implies ν_{E_{i}}δ_{i}(v, v)) = 1 and ν_{F} h(v, β_{i}v) = ν_{E_{i}}β_{i} + 1;

(iii) If E_{i} is unramified over F and Λ_{i} does not contain a self-dual lattice then Λ_{i}(0)^{#} = Λ_{i}(1) = ϖ_{F} Λ_{i}(0) so ν_{Λ_{i}}(v) = 0 implies ν_{E_{i}}δ_{i}(v, v)) = 0 and ν_{F} h(v, β_{i}v) = ν_{E_{i}}β_{i}.

■

6.3 Some intersections

The following Lemma and Corollary were suggested by Vytautas Paskunas. In the proofs, we may (and often will) ignore the condition that the stratum and character be skew, since the results in the skew case follow immediately by restriction to G – that is, we actually prove the statements for σ-stable semisimple strata in GL_{4}. To ease notation, this will be implicit so that, in the proofs below, U should be a σ-stable maximal unipotent subgroup of GL_{4}, etc.

First we need some notation in the semisimple case. For [Λ, n, 0, β] a semisimple stratum in V with splitting V = ⨁_{i=1}^{2} V^{i}, we write A = ⊕_{i,j=1}^{2}A^{ij} in block notation, where A^{ij} = Hom_{F} (V^{j}, V^{i}) = 1^{j}A1^{i} and 1^{i} is the projection onto V^{i} with kernel V^{3−i}. For k = (k_{1}, k_{2}) ∈ ℤ^{2}, with k_{1} ≥ k_{2} ≥ 1, define

𝔞_{k} = 𝔞_{k}(Λ) = (\begin{matrix}𝔞_{k_{1}}^{11} & 𝔞_{k_{1}}^{12} \\ 𝔞_{k_{1}}^{21} & 𝔞_{k_{2}}^{22} \end{matrix})

and U^{k}(Λ) = 1 + 𝔞_{k}.

Lemma 6.5. Let [Λ, n, 0, β] be a semisimple stratum in A, with splitting V = ⨁_{i=1}^{l} V^{i}, 1 ≤ l ≤ 2. Suppose also that F [β] is of maximal degree over F, and let U be a maximal unipotent subgroup of G. Write B for the centralizer in A of β. For k ≥ m ≥ 1, we have

((U^{m}(Λ) ∩ B)U^{k}(Λ)) ∩ U = (U^{k}(Λ) ∩ U).

We remark that the proof below is just for the case which interests us here (so that there are at most 2 pieces in the splitting) but it is straightforward to generalize the Lemma to the case where there are an arbitrary number of pieces.

Proof. We will prove the corresponding additive statement. Writing U = 1+ℕ, 𝔟_{m} = 𝔞_{m}(Λ)∩B, it is:

(𝔟_{m} + 𝔞_{k}) ∩ ℕ = 𝔞_{k} ∩ ℕ.

Notice that, since B = F [β], the lattice 𝔟_{m} contains no non-trivial nilpotent elements. We will show that, for 1 ≤ m < k,

(𝔟_{m} + 𝔞_{k}) ∩ ℕ ⊂ (𝔟_{m+1} + 𝔞_{k}) ∩ ℕ

and the result follows at once by an easy induction. So suppose ϵ ∈ 𝔟_{m} is such that (ϵ+𝔞_{k})∩ℕ ≠ ∅. In particular, (ϵ + 𝔞_{m+1}) ∩ ℕ ≠ ∅ so there exists s > 0 such that ϵ^{s} ∈ 𝔞_{sm+1}. But then ϵ^{s} ∈ 𝔟_{sm+1} so (by [3], working block-by-block), the coset ϵ + 𝔟_{m+1} contains a nilpotent element, which must be 0. We deduce that ϵ ∈ 𝔟_{m+1}, as required. ■

For [Λ, n, 0, β] a semisimple stratum in V with splitting V = ⨁_{i=1}^{2} V^{i}, we write M_{sp} = Aut_{F} (V^{1})× Aut_{F} (V^{2}), U_{sp} = 1 + A^{12}, P_{sp} = M_{sp}U_{sp} and Ū_{sp} = 1 + A^{21}.

Corollary 6.6. Let [Λ, n, 0, β] be a semisimple stratum in A, with splitting V = ⨁_{i=1}^{l} V^{i}, 1 ≤ l ≤ 2, and let U be a maximal unipotent subgroup of G. Write B for the centralizer in A of β. We suppose that

(i) U has an Iwahori decomposition with respect to (M_{sp}, P_{sp});

(ii) U ∩ B^{×} is a maximal unipotent subgroup of B^{×}.

Then, for k_{1} ≥ ⋯ ≥ k_{l} ≥ m ≥ 1 and k = (k_{1}, …, k_{l}), we have

((U^{m}(Λ) ∩ B)U^{k}(Λ)) ∩ U = (U^{m}(Λ) ∩ B ∩ U)(U^{k}(Λ) ∩ U).  (6.7)

Again the statement is easily generalized to the case when the splitting has more than two pieces. Note also that, in the simple case, condition (i) is empty while condition (ii) is implied, for example, by the condition ψ_{β}|_{U_{der}} = 1 ([5] Proposition 2.2). This is not true for semisimple strata.

Proof. First we reduce to the simple case. We notice that U^{k}(Λ) and B^{×} ⊂ M_{sp} have Iwahori decompositions with respect to (M_{sp}, P_{sp}). Since U also has such a decomposition, we reduce to proving that we have equality in (6.7) when we intersect both sides with U_{sp}, with M_{sp} and with Ū_{sp} respectively. Since B ⊂ M_{sp}, this is immediate for the unipotent radicals Ū_{sp}, U_{sp}. Hence we are reduced to the intersection with M_{sp}, which, block-by-block, is just the simple case. So now suppose [Λ, n, 0, β] is a simple stratum. As in Lemma 6.5, we will prove the corresponding additive statement: writing U = 1 + ℕ, it is:

(𝔟_{m} + 𝔞_{k}) ∩ ℕ = (𝔟_{m} ∩ ℕ) + (𝔞_{k} ∩ ℕ),  (6.8)

where k = k_{1}. We will reduce to the case where E = F [β] is maximal and invoke Lemma 6.5. Write d = [E : F]; then, in the flag corresponding to U,

{0} = V_{0} ⊂ V_{1} ⊂ ⋯ ⊂ V_{N} = V,

the subspace V_{di} is an E-subspace, for 0 ≤ i ≤ N/d. Let U_{0} = 1 + ℕ_{0} be the unipotent subgroup corresponding to the maximal E-flag

{0} = V_{0} ⊂ V_{d} ⊂ ⋯ ⊂ V_{di} ⊂ ⋯ ⊂ V_{N} = V

and let P_{0} = 1 + ℙ_{0} the corresponding parabolic subgroup. There exists an E-decomposition V = ⊕_{i=1}^{N/d} W_{i} of V such that for 0 ≤ i ≤ N/d, V_{di} = ⊕_{j=1}^{i}W_{j} and such that for every t ∈ ℤ, Λ(t) = ⊕_{i=1}^{N/d} Λ(t) ∩ W_{i} (as a suitable variant of, e.g., [22] §II.1). Let L_{0} be the corresponding Levi component of P_{0} and let Ū_{0} = 1 + ℕ̄_{0} be the unipotent subgroup opposite U_{0} with respect to L_{0}. The lattices 𝔟_{m} and 𝔞_{k} have (additive) Iwahori decompositions with respect to ℕ̄_{0}, ℙ_{0} ([4] §10) so

(𝔟_{m} + 𝔞_{k}) ∩ ℙ_{0} = 𝔟_{m} ∩ 𝕃_{0} + 𝔟_{m} ∩ ℕ_{0} + 𝔞_{k} ∩ 𝕃_{0} + 𝔞_{k} ∩ ℕ_{0}

and we have

(𝔟_{m} + 𝔞_{k}) ∩ ℕ = (𝔟_{m} + 𝔞_{k}) ∩ ℙ_{0} ∩ ℕ = (𝔟_{m} ∩ 𝕃_{0} + 𝔞_{k} ∩ 𝕃_{0}) ∩ ℕ + 𝔟_{m} ∩ ℕ_{0} + 𝔞_{k} ∩ ℕ_{0}

Hence we are reduced to showing

(𝔟_{m} ∩ 𝕃_{0} + 𝔞_{k} ∩ 𝕃_{0}) ∩ ℕ = 𝔟_{m} ∩ 𝕃_{0} ∩ ℕ + 𝔞_{k} ∩ 𝕃_{0} ∩ ℕ,

which is the same as (6.8) in 𝕃_{0}, where we can work block-by-block. In each block, the field extension E is maximal, so we have indeed reduced to the maximal case, which is Lemma 6.5. ■

Finally, we will need one more similar result, in the case of a minimal semisimple stratum – in this case the extra conditions of the previous Corollary are not satisfied.

Lemma 6.9. Let [Λ, n, 0, β] be a skew semisimple stratum, with splitting V = V^{1} ⨁ V^{2}, 1 ≤ l ≤ 2. Suppose that β_{i} = 1^{1}β1^{i} is minimal, for i = 1, 2 and that [Λ, n, n − 1, β] is not equivalent to a simple stratum. Assume also (without loss of generality) that n_{1} ≥ n_{2} ≥ 1. Put k_{i} = [\frac{n_{i}}{2} ] + 1, for i = 1, 2, and k = (k_{1}, k_{2}). Then H^{1} = (U^{1}(Λ) ∩ B)U^{k}(Λ) and, if U is a maximal unipotent subgroup of G on which ψ_{β} defines a character:

H^{1} ∩ U = U^{k}(Λ) ∩ U.

Proof. We work under the conventions made in the previous subsection. The subgroup U is necessarily given (see Proposition 3.4) by a flag of the following form:

{0} ⊂ ⟨v_{1} + v_{2}⟩ ⊂ ⟨v_{1} + v_{2}, v_{−1} − v_{−2}⟩ ⊂ ⟨v_{1}, v_{2}, v_{−1} − v_{−2}⟩ ⊂ V,

where v_{i} ∈ V^{i} are non zero vectors such that h(v_{1}, β_{1}v_{1}) = −h(v_{2}, β_{2}v_{2}) = μ and v_{−1} = μ^{−1}β_{1}v_{1}, v_{−2} = −μ^{−1}β_{2}v_{2}. In particular {v_{i}, v_{−i}} is a symplectic basis for V^{i}, adapted to Λ^{i}.

That H^{1} = (U^{1}(Λ) ∩ B)U^{k}(Λ) is clear from the definition. For the intersection property, we will need:

Lemma 6.10. Put ν = ν_{Λ_{1}}(v_{1}) − ν_{Λ_{2}}(v_{2}) and let E_{i,j} be the linear map sending v_{j} to v_{i} and all other basis elements to 0. We have

ν_{Λ}(E_{1,2}) = ν,  ν_{Λ}(E_{−2,−1}) = n_{1} − n_{2} − ν.  (6.11)

Furthermore, if n_{1} ≥ n_{2}, then 0 ≤ ν ≤ n_{1} − n_{2}.

Proof. Computing the valuations of E_{1,2} and E_{−2,−1} is simple checking. Lemma 6.4 combined

with our asssumption on v_{1}, v_{2} implies ν = −\frac{2ν_{E_{1}}β_{1}}{e_{1}} + \frac{2ν_{E_{2}}β_{2}}{e_{2}} . The inequality 0 ≤ ν ≤ n_{1} − n_{2}

then follows from Lemma 6.3. ■

Write U = 1 + ℕ. Then the elements of ℕ can be written (as matrices with respect to the basis {v_{1}, v_{−1}, v_{2}, v_{−2}})

x = (\begin{matrix}a & e & −a & c \\ −d & −b & d & −b \\ a & e + f & −a & c + f \\ d & b & −d & b \end{matrix}) ,

for a, b, c, d, e, f ∈ F. Now suppose that x as above also lies in the lattice (in block matrix form, each block 2 × 2)

(\begin{matrix}𝔞_{m} & 𝔞_{k_{1}} \\ 𝔞_{k_{1}} & 𝔞_{k_{2}} \end{matrix}) ,

for some m < k_{1}. Then, using (6.11), we get:

• aE_{2,1} ∈ 𝔞_{k_{1}} so aE_{1,1} = aE_{1,2}E_{2,1} ∈ 𝔞_{k_{1}+ν};

• bE_{−1,−2} ∈ 𝔞_{k_{1}} so bE_{−1,−1} = bE_{−1,−2}E_{−2,−1} ∈ 𝔞_{k_{1}+n_{1}−n_{2}−ν};

• (c + f)E_{2,−2} ∈ 𝔞_{k_{2}} so (c + f)E_{1,−2} = (c + f)E_{1,2}E_{2,−2} ∈ 𝔞_{k_{2}+ν};

cE_{1,−2} ∈ 𝔞_{k_{1}} so fE_{1,−2} ∈ 𝔞_{min{k_{1},k_{2}+ν}}

and fE_{1,−1} = fE_{1,−2}E_{−2,−1} ∈ 𝔞_{min{k_{1}+n_{1}−n_{2}−ν,k_{2}+n_{1}−n_{2}}}.

(e + f)E_{2,−1} ∈ 𝔞_{k_{1}} so (e + f)E_{1,−1} = (e + f)E_{1,2}E_{2,−1} ∈ 𝔞_{k_{1}+ν}.

Writing t = min {ν, n_{1}−n_{2}−ν, k_{2}+n_{1}−n_{2}−k_{1}} ≥ 0, we have aE_{1,1}, bE_{−1,−1}, eE_{1,−1} ∈ 𝔞_{k_{1}+t} ⊂ 𝔞_{m+1}. In particular, looking at the “top-left” block of x, we have

(\begin{matrix}a & e \\ −d & −b \end{matrix}) ∈ (\begin{matrix}0 & 0 \\ −d & 0 \end{matrix}) + 𝔞_{m+1}.

Now we prove that, for m ≤ k_{2} and k = (k_{1}, k_{2}),

((U^{m}(Λ) ∩ B)U^{k}(Λ)) ∩ U = U^{k}(Λ) ∩ U.

By Lemma 6.5, we have

((U^{m}(Λ) ∩ B)U^{k}(Λ)) ∩ U ⊂ ((U^{m}(Λ) ∩ B)U^{k_{2}}(Λ)) ∩ U = U^{k_{2}}(Λ) ∩ U.

Then we need only prove, for k_{2} ≤ m < k_{1} (the additive statement)

(\begin{matrix}𝔟_{m} + 𝔞_{k_{1}} & 𝔞_{k_{1}} \\ 𝔞_{k_{1}} & 𝔞_{k_{2}} \end{matrix}) ∩ ℕ ⊂ (\begin{matrix}𝔟_{m+1} + 𝔞_{k_{1}} & 𝔞_{k_{1}} \\ 𝔞_{k_{1}} & 𝔞_{k_{2}} \end{matrix}) .

But we have seen above that, if ϵ ∈ (\begin{matrix}𝔟_{m} & 0 \\ 0 & 0 \end{matrix}) is such that ϵ + 𝔞_{k} contains an element of ℕ,

then ϵ + (\begin{matrix}𝔞_{m+1} & 0 \\ 0 & 0 \end{matrix}) also contains a nilpotent element. But then (as in the proof of Lemma 6.5)

ϵ + (\begin{matrix}𝔟_{m+1} & 0 \\ 0 & 0 \end{matrix}) also contains a nilpotent element, which must be 0. Hence ϵ ∈ 𝔟_{m+1}, as required.

■

6.4 Proof of Proposition 4.1

It remains to prove Proposition 4.1: If θ ∈ C(Λ, 0, β) is a skew semisimple character and U is a maximal unipotent subgroup of G such that ψ_{β}|_{U_{der}} = 1, then

θ|_{H^{1}∩U} = ψ_{β}|_{H^{1}∩U}.

As in the previous section, we actually prove the same statement for σ-stable semisimple strata in GL_{4}, and the result follows by restriction to G. We remark that a skew element β generating a field such that [F [β] : F] = 2 is necessarily minimal. We proceed on a case-by-case basis: Simple case (Cases (I) and (II)) We proceed by induction on r = −k_{0}(β, Λ). We make the following additional hypothesis:

(∗) there exists a simple stratum [Λ, n, r, γ] equivalent to [Λ, n, r, β] such that ψ_{γ}|_{U_{der}} = 1.

In particular, we can then use the inductive hypothesis for the stratum [Λ, n, 0, γ] with the same unipotent subgroup U. Note that this hypothesis is certainly satisfied when β is minimal, since we can take γ = 0. We will show later that it is also satisfied in the other cases that are relevant to us here. We have H^{1}(β, Λ) = (U^{1}(Λ)∩B)H^{[ \frac{r}{2} ]+1}(β, Λ) so, by Corollary 6.6, H^{1} ∩U = H^{[ \frac{r}{2} ]+1} ∩U. Moreover, H^{[ \frac{r}{2} ]+1}(β, Λ) = H^{[ \frac{r}{2} ]+1}(γ, Λ) and θ|_{H^{[ \frac{r}{2} ]+1}} = θ_{0}|_{H^{[ \frac{r}{2} ]+1}}ψ_{β−γ}, for some θ_{0} ∈ C(Λ, 0, γ). But, by the inductive hypothesis, θ_{0} agrees with ψ_{γ} on H^{[ \frac{r}{2} ]+1} ∩ U and the result follows. To finish, we must show that (∗) is satisfied. The only case to consider is when [Λ, n, 0, β] is a skew simple stratum with F [β] maximal (of degree 4) and r = −k_{0}(β, Λ) < n. Then [Λ, n, r, β] is equivalent to some skew simple stratum [Λ, n, r, γ_{0}], with γ_{0} minimal and F [γ_{0}] of degree 2 over F. Note that, since p ≠ 2, all extensions are tame here. Note also that the flag corresponding to the unipotent subgroup U is given by V_{i} = ⟨v, βv, ⋯, β^{i−1}v⟩, for some v ∈ V with h(v, βv) = 0. Also put ζ = h(βv, β^{2}v) ≠ 0, and d = −v_{F} (ζ). Let P (X) = X^{2} + λ ∈ F [X] be the minimal polynomial of γ_{0} and put e_{−2} = v, e_{−1} = βv, e_{1} = ζ^{−1}P (β)v, e_{2} = −βζ^{−1}P (β)v − kβv, where k = ζ^{−2}h(βv, β^{2}v) − 2λζ^{−1} (this is a symplectic basis). With respect to this basis, β has matrix

(\begin{matrix}0 & −λ & 0 & μ \\ 1 & 0 & k & 0 \\ 0 & ζ & 0 & λ \\ 0 & 0 & −1 & 0 \end{matrix})

for some μ ∈ F. For i, j ∈ {±1, ±2}, write E_{i,j} for the linear map sending e_{j} to e_{i} and all other basis vectors to 0. We then have

E_{−2,−1} ∈ 𝔞_{n}, E_{1,−1} ∈ 𝔞_{de−r}.

Write β = γ_{0} + c_{0}, with c_{0} ∈ 𝔞_{−r}. Then β^{2} + λ = c_{0}γ_{0} + γ_{0}c_{0} + c_{0}^{2} ∈ 𝔞_{−n−r}. From the matrix description of β, we get that

β^{2} + λ = (\begin{matrix}0 & 0 & −(μ + kλ) & 0 \\ 0 & kζ & 0 & (μ + kλ) \\ ζ & 0 & kζ & 0 \\ 0 & −ζ & 0 & 0 \end{matrix})

Then:

• (μ + kλ)E_{−1,2} ∈ 𝔞_{−n−r} and (μ + kλ)E_{−2,2} = (μ + kλ)E_{−2,−1}E_{−1,2} ∈ 𝔞_{−r};

• ζE_{1,−1} ∈ 𝔞_{−r}.

Hence

c = (\begin{matrix}0 & 0 & 0 & μ + kλ \\ 0 & 0 & 0 & 0 \\ 0 & ζ & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix}) ∈ 𝔞_{−r}.

Put γ = β − c. Then [Λ, n, r, γ] is a stratum as required by hypothesis (∗).

Minimal semisimple case (Case (III)) Now suppose [Λ, n, 0, β] is a skew semisimple stratum, with splitting V = V^{1} ⨁ V^{2}, 1 ≤ l ≤ 2, that β_{i} = 1^{1}β1^{i} is minimal, for i = 1, 2 and that [Λ, n, n − 1, β] is not equivalent to a simple stratum. Assume also (without loss of generality) that n_{1} ≥ n_{2} ≥ 1. Put k_{i} = [\frac{n_{i}}{2} ] + 1, for i = 1, 2, and k = (k_{1}, k_{2}). Then, by Lemma 6.9, H^{1} = (U^{1}(Λ) ∩ B)U^{k}(Λ) and H^{1} ∩ U = U^{k}(Λ) ∩ U. But, by definition of θ, it agrees with ψ_{β} on U^{k}(Λ) so the result follows.

Degenerate semisimple case (case (IV)) Now suppose [Λ, n, 0, β] is again a skew semisimple stratum, but with β_{2} = 0. In this case it is straightforward to see that the flag defining U must be given by

{0} ⊂ V_{1} = ⟨v_{2}⟩ ⊂ V_{2} = ⟨v_{2}, v_{1}⟩ ⊂ V_{3} = V_{1} ⊕ V^{1} ⊂ V_{4} = V,

where v_{2} ∈ V^{2} and v_{1} ∈ V^{1}. In particular, U satisfies the extra conditions in Corollary 6.6. Putting k_{1} = [\frac{n_{1}}{2} ] + 1 and k_{2} = 1, we have H^{1} = (U^{1}(Λ) ∩ B)U^{k}(Λ) so, by Corollary 6.6,

H^{1} ∩ U = (U^{1}(Λ) ∩ B ∩ U)(U^{k}(Λ) ∩ U) = U^{k}(Λ) ∩ U.

As in semisimple case (i), the result now follows by the definition of semisimple characters.

Non-minimal semisimple case (case (III)) Finally, suppose [Λ, n, 0, β] is a skew semisimple stratum, with splitting V = V^{1} ⨁ V^{2}, 1 ≤ l ≤ 2, that β_{i} = 1^{1}β1^{i} is minimal, for i = 1, 2 but that [Λ, n, n − 1, β] is equivalent to a simple stratum. Let r = −k_{0}(β, Λ). As in the simple case, we will show that

(∗) there exists a simple stratum [Λ, n, r, γ] equivalent to [Λ, n, r, β] such that ψ_{γ}|_{U_{der}} = 1.

(Indeed, γ will be minimal.) The proof is then the same as that in the simple case, since we can use the simple case for γ. In this case we invoke Lemma 6.5 to show that H^{1} ∩ U = H^{[ \frac{n}{2} ]+1} ∩ U. As in the simple case, the flag corresponding to the unipotent subgroup U is given by V_{i} = ⟨v, βv, ⋯, β^{i−1}v⟩, for some v ∈ V with h(v, βv) = 0. So v = v_{1} + v_{2}, with v_{i} ∈ V^{i} such that h(v_{1}, β_{1}v_{1}) = −h(v_{2}, β_{2}v_{2}). Note that, for i = 1, 2, {v_{i}, β_{i}v_{i}} is a basis for V^{i}, with respect to which β_{i} has matrix

(\begin{matrix}0 & λ_{i} \\ 1 & 0 \end{matrix}) ,

for some λ_{i} ∈ F. Also, let E_{i} denote the linear map in A^{ii} which send β_{i}v_{i} to v_{i} and v_{i} to 0; then E_{i} ∈ 𝔞_{n}^{ii}. Let [Λ, n, r, γ_{0}] be a skew simple stratum equivalent to [Λ, n, r, β], with γ_{0} ∈ M_{sp}, and let X^{2} − λ be the minimal polynomial of γ_{0} over F. For i = 1, 2, let γ_{i} ∈ A^{ii} have matrix

(\begin{matrix}0 & λ \\ 1 & 0 \end{matrix}) ,

with respect to the basis {v_{i}, β_{i}v_{i}} of V^{i}. Since β − γ_{0} ∈ 𝔞_{−r}, we get that λ − λ_{i} ∈ 𝔞_{−n−r}^{ii}, for i = 1, 2, so (λ − λ_{i})E_{i} ∈ 𝔞_{−r}^{ii}. Hence β_{i} − γ_{i} ∈ 𝔞_{−r}^{ii} and γ = γ_{1} + γ_{2} is as required, since γv = βv.

This completes the proof of Proposition 4.1. ■

Remarks It surely will not have escaped the reader’s notice that the methods in each case are rather similar. It may well be possible to unify the cases into a single proof but we have not been able to do this. We also note that we could not have used [5] Lemma 2.10 here, since the proof given there unfortunately does not work. It seems likely that the result there is true (at least in the tame case, as here) but we have not (yet) been able to find a proof.

References

[1] L.Blasco, Description du dual admissible de U(2, 1)(F) par la théorie des types de C. Bushnell et P. Kutzko, Manuscripta Math. 107, 2002, 151–186.

[2] A.Borel, Linear algebraic groups, Graduate Texts in Mathematics 126, Springer-Verlag, New York, 1991.

[3] C.J.Bushnell, Hereditary orders, Gauss sums and supercuspidal representations of GL_{N}, J. Reine Angew. Math. 375/376, 1987, 184–210.

[4] C.J.Bushnell and G.Henniart, Local tame lifting for GL(N) I: simple characters, Pub. Math. IHES 83,1996, 105233.

[5] C.J.Bushnell and G.Henniart, Supercuspidal representations of GL_{n}: explicit Whittaker functions, J. Algebra 209(1), 1998, 270–287.

[6] C.J.Bushnell and G.Henniart, On the derived subgroups of certain unipotent subgroups of reductive groups over infinite fields, Transform. Groups 7(3), 2002, 211–230.

[7] C.J.Bushnell and G.Henniart, Generalized Whittaker models and the Bernstein center, Amer. J. Math. 125(3), 2003, 513–547.

[8] C.J.Bushnell and P.C.Kutzko, The admissible dual of GL(N) via compact open subgroups, Annals of Mathematics Studies 129, Princeton, 1993.

[9] C.J.Bushnell and P.C.Kutzko, Semisimple types in GL_{n}, Compositio Math. 119, 1999, 53–97.

[10] F.Digne and J.Michel, Representations of Finite Groups of Lie Type, London Mathematical Society Student Texts 21, Cambridge University Press, 1991.

[11] W.T.Gan and S.Takeda, The local Langlands conjecture for GSp(4), Preprint, June 2007, arXiv:math.NT/0706.0952v1.

[12] R.Howe and I.I.Piatetski-Shapiro, A counterexample to the “generalized Ramanujan conjecture” for (quasi)-split groups, Automorphic Forms, Representations and L-functions (Proc. Sympos. Pure Math., Vol. XXXIII, part 1, Corvallis 1977), 315–322, Amer. Math. Soc., Providence, R.I., 1979.

[13] G.Lusztig, Representations of Finite Chevalley Groups, C.B.M.S. 39 , A.M.S. 1978.

[14] O.T.O’Meara, Introduction to quadratic forms, Springer-Verlag, 1963.

[15] F.Rodier, Whittaker models for admissible representations of reductive p-adic split groups, Harmonic analysis on homogeneous spaces (Proc. Sympos. Pure Math., Vol. XXVI, Williams Coll., Williamstown, Mass., 1972), 425–430, Amer. Math. Soc., Providence, R.I., 1973.

[16] V.Paskunas, S.Stevens, On the realisation of maximal simple types and epsilon factors of pairs, Amer. J. Math. to appear, available from arXiv.org/abs/math.RT/0603051

[17] T.A.Springer, Characters of special groups, in Seminar on Algebraic Groups and Related Finite Groups, Lecture Notes in Math. 131, Springer-Verlag 1970.

[18] B.Srinivasan, The characters of the finite symplectic group Sp(4, q), Trans. A.M.S. 131 (1968), 488–525.

[19] S.Stevens, Intertwining and supercuspidal types for p-adic classical groups, Proc. London Math. Soc. 83, 2001, 120–140.

[20] S.Stevens, Semisimple characters for p-adic classical groups, Duke Math. J. 127(1), 2005, 123–173.

[21] S.Stevens, The supercuspidal representations of p-adic classical groups, Preprint, July 2006, arXiv:math.RT/0607622.

[22] A.Weil, Basic Number Theory, Springer Verlag, 1974.

Corinne Blondel Shaun Stevens C.N.R.S. - Théorie des Groupes - Case 7012 School of Mathematics Institut de Mathématiques de Jussieu University of East Anglia Université Paris 7 Norwich NR4 7TJ F-75251 Paris Cedex 05 United Kingdom France blondel@math.jussieu.fr ginnyshaun@bigfoot.com