I often come across papers written in the Bitcoin space involving some math that I am not formally versed in.
Could anyone point out general topics in math that I should have a formal grasp on in order to really understand cryptography?
The signatures in Bitcoin (and Bitcoin Cash, Ethereum, and many other cryptocurrencies) are generated using elliptic curve curve cryptography. I found these videos very helpful in that regard: https://www.youtube.com/watch?v=jQLmaBjYrk4
Other than that, you probably want to look into hash functions and asymmetric cryptography in general.
As a general introduction to cryptography, I took the cryptography course on Coursera which I can really recommend. This goes into the role of the adversary and what he can achieve. The teacher of this course, Dan Boneh, is, I believe, among the ten most renowned and best cryptographers in the world. On top of that, he is a very good teacher. The course does go into both symmetric and asymmetric cryptography but in the asymmetric field, it only teaches you RSA and Diffie-Hellman, not elliptic curves. Use the above videos for that.