Don't build gmp and mpfr when using external toolchains (based on patch from #6035)