In this post, we will prove the following monstrous looking identity from Gradshteyn and Ryzhik (3.255): $$ \int _0^1 \frac{x^{\mu+\frac{1}{2}} (1-x)^{\mu-\frac{1}{2}}}{(c+2bx-ax^2)^{\mu+1}}dx = \frac{\sqrt{\pi}}{\left\{a + \left(\sqrt{c+2b-a} + \sqrt{c}\right)^2\right\}^{\mu+\frac{1}{2}}\sqrt{c+2b-a}} \frac{\Gamma \left(\mu + \frac{1}{2}\right)}{\Gamma\left(\mu+1\right)}$$ where $c+2b-a>0$, $a + \left(\sqrt{c+2b-a} + \sqrt{c}\right)^2 > 0$ and $\text{Re }\mu > -\frac{1}{2}$
Let us denote the integral by $I$. We will first use a Möbius transformation to get rid of the quadratic in the denominator. Pick $\rho > 0$ such that $$ c\rho^2 + 2b\rho – a = 0 \; \implies \; \rho = \frac{-b+\sqrt{b^2+ac}}{c} > 0 $$ Now, set $x=\frac{u}{1+\rho u}$ and $dx = \frac{du}{(1+\rho u)^2}$ to get: $$I = \frac{1}{c^{\mu+1}}\int_0^U \frac{u^{\mu+\frac{1}{2}} (1+(\rho-1)u)^{\mu-\frac{1}{2}}}{(1+\kappa u)^{\mu+1}}du$$ where $U=\frac{1}{1-\rho}$ and $\kappa = 2\left(\rho+\frac{b}{c}\right)=\frac{2\sqrt{b^2+ac}}{c}$.
We can now scale back this integral to $[0, 1]$ using the substitution $u = U v$ and $du = U dv$: $$I = \frac{U^{\mu+\frac{3}{2}}}{c^{\mu+1}} \int_0^1 \frac{v^{\mu+\frac{1}{2}} (1-v)^{\mu-\frac{1}{2}}}{(1+\lambda v)^{\mu+1}}dv$$ where $\lambda = U \kappa$.
We can immediately recognize the above expression as Euler’s integral for the Gauss hypergeometric function: $$I = \frac{U^{\mu+\frac{3}{2}}}{c^{\mu+1}}B\left(\mu+\frac{3}{2}, \mu+\frac{1}{2}\right) \;_{2}F_{1}\left(\mu+\frac{3}{2},\mu+1;2(\mu+1);-\lambda\right)$$
Apply Pfaff Transformation: $${\;}_{2}F_{1}\left(a,b;c;z\right) = (1-z)^{-a}{\;}_{2}F_{1}\left(a,c-b;c;\frac{z}{z-1}\right)$$ with $a=\mu+\frac{3}{2}$, $b=\mu+1$, $c=2(\mu+1)$, $z=-\lambda$ to get: $${\;}_{2}F_{1}\left(\mu+\frac{3}{2},\mu+1;2(\mu+1);-\lambda\right) = (1+\lambda)^{-\mu-\frac{3}{2}} {\;}_{2}F_{1}\left(\mu+\frac{3}{2},\mu+1;2(\mu+1);\frac{\lambda}{1-\lambda}\right)$$ For this precise configuration there is a classical quadratic reduction: $${\;}_{2}F_{1}\left(b+\frac{1}{2},b;2b;t\right)=\frac{1}{\sqrt{1-t}}\left(\frac{1+\sqrt{1-t}}{2}\right)^{1-2b}, \quad \text{Re }b>0$$
Put everything together: $$I = \frac{U^{\mu+\frac{3}{2}}}{c^{\mu+1}} \frac{\Gamma\left(\mu+\frac{3}{2}\right)\Gamma\left(\mu+\frac{1}{2}\right)}{\Gamma\left(2(\mu+1)\right)}\frac{1}{\sqrt{1+\lambda}}\left(\frac{1+\frac{1}{\sqrt{1+\lambda}}}{2}\right)^{-2\mu-1}$$
All that remains is to simplify the above expression. Firstly, we can use $\Gamma\left(\mu+\frac{3}{2}\right)=\frac{2\mu+1}{2}\Gamma\left(\mu+\frac{1}{2}\right)$ and $\Gamma(2(\mu+1))=2^{2\mu+1}\sqrt{\pi}\Gamma(\mu+1)\Gamma\left(\mu+\frac{1}{2}\right)$ to get the $\sqrt{\pi} \frac{\Gamma\left(\mu+\frac{1}{2}\right)}{\Gamma\left(\mu+1\right)}$ factor. The remaining purely algebraic constants simplify, with $$\begin{align*} U &= \frac{c}{b+c-\sqrt{b^2+ac}} \\ \lambda &= \frac{2\sqrt{b^2+ac}}{b+c-\sqrt{b^2+ac}} \end{align*}$$ to $$\frac{1}{\sqrt{c+2b-a}} \frac{1}{\left(a+\left(\sqrt{c+2b-a}+\sqrt{c}\right)^2\right)^{\mu+\frac12}}$$ One convenient algebraic identity used here is the following: $(c+b)^2 – (b^2+ac) = c(c+2b-a)$.