It comes from chapter 7, Steinberg_2012_Representation theory of finite groups.pdf.

permutation representation

Let be a group action. Define a representation by setting

One calls the permutation representation associated to .

fixed subspace

Let be a representation. Then

is the fixed subspace of .

We can prove that (see Proposition 7.2.7), is the direct sum of all the copies of the trivial representation in . In the other word, for a representation and the trivial character of , there is

Proposition

Let be a group action. Let be the orbits of on and define . Then is a basis for , the fixed subspace of , and hence is the number of orbits of on .

By ^a5ygsc, we know that for every permutation representation , it has at least trivial representation as a constituent, where is the number of orbits of . Thus is not irreducible if its degree is not . As a corollary of this proposition, we prove the Burnside’s lemma.

Burnside's lemma

Let be a group action and let be the number of orbits of on . Then $$ m=\frac{1}{|G|}\sum_{g\in G}|\mathrm{Fix}(g)|.

Consider the group action $\tau:G\to S_{X\times X}$, then we get the following corollary. [!corollary] Let $\sigma:G\to S_X$ be a transitive group action. Then the equalities

\mathrm{rank}(\sigma)=\frac{1}{|G|}\sum_{g\in G}|\mathrm{Fix}(g)|^2=\langle \chi_{\tilde\sigma},\chi_{\tilde\sigma}\rangle.

^8a0152 [!definition] For a transitive group action $\sigma:G\to S_X$, by [[#^a5ygsc|^a5ygsc]], it has only one trivial sub-representation, denoted by $\mathbb{C} v_0$. Since $\sigma(g)$ is a permutation matrix for all $g$, the representation of $\sigma$ is unitary and so it is decomposable by [[Maschke's Theorem#^0m7u7u|^0m7u7u]]. Then $\mathbb{C} X$ can be decomposed as $\mathbb{C}X=\mathbb{C}v_0\oplus V_0$ where $V_0=\mathbb{C}v_0^\bot$. Let $\tilde\sigma'$ be the restriction of $\tilde\sigma$ to $V_0$, then it is called the *augmentation representation* associated to $\sigma$. By [[#^8a0152|^8a0152]], we can use the *augmentation representation* $\tilde \sigma'$ to describe the rank of $\sigma$, which deduces the following theorem. [!theorem] Let $\sigma:G\to S_X$ be a transitive group action. Then the argumentation representation $\tilde\sigma'$ is irreducible iff $G$ is $2$-transitive on $X$. `\begin{proof}` Since $\sigma$ is transitive, by [[#^a5ygsc|^a5ygsc]], $\tilde\sigma$ has exactly one trivial sub-representation and so $\left\langle \chi_1,\chi_{\tilde\sigma'}\right\rangle=0$. Then $\tilde\sigma'$ is irreducible iff $\langle\chi_{\tilde\sigma'},\chi_{\tilde\sigma'}\rangle=1$ iff $\mathrm{rank}(\sigma)=2$, because

\left\langle \chi_\sigma,\chi_\sigma\right\rangle =\left\langle \chi_1,\chi_1\right\rangle +2\mathrm{Re}\left\langle \chi_1,\chi_{\tilde\sigma’}\right\rangle +\left\langle \chi_{\tilde\sigma’},\chi_{\tilde\sigma’}\right\rangle=1+0+1=2.

Now we finish the proof. `\end{proof}` **Remark.** Similarly, one can prove that if $\tilde\sigma'$ can be expressed as sum of two irreducible sub-representations, it is of rank $3$. It is used in [[Kantor_Liebler_1982_The Rank 3 Permutation Representations of the Finite Classical Groups.pdf]].