I am a post graduate electrical engineering student, specialising in digital electronics. I would like to foray into ASICs for bitcoin mining. But first I need to understand the mathematics of Bitcoin mining. What subjects do I need to study to get a good idea of what my ASIC has to tackle?
Your ASIC basically only does one thing: SHA256 hashing. It gets a block template, hashes it, compares the result as if it was a number to some target to see if it is less than the target, if not it modifies the block slightly, hashes again, repeat...
So all you need to understand is the SHA-256 hash algorithm. It is not overly complicated, you will need to understand some logical bitwise operations like AND and XOR, hopefully understand how bits, bytes, hexadecimal, etc. are all related, it involves some bit-rotations which should be mostly self-explanatory, how modulo works, and maybe a couple of other similar-level concepts. As a postgrad digital electronics engineer I expect most of these concepts are bread and butter to you, so you should be ok :)
Of course, that's just to read through how the algorithm works, understanding why certain things are done in the algorithm is much more complicated and would require a lot more reading about hash functions and cryptography. But understanding why something is done is not a requirement to use it.
Wikipedia page for SHA-256 which contains some pseudocode is found here: https://en.wikipedia.org/wiki/SHA-2