It is possible to install custom software (another than cgminer) on Antminer hardware? For example on antminer s5?
Should be possible. There was custom firmware available back in circa 2015/2016. Check out Bitcointalk forum for older posts on Antminer S5: https://bitcointalk.org/index.php?topic=1452009.0. But be aware that you could brick your miner.