Are there any hard math problems or math contests that have rewards paid in Bitcoin?
There is a site where one can submit bounties for proofs of theorems in Coq formal proof management system, and get paid in bitcoin:
Additionally there are other sites that have done competitions in the past:
Another project in development (but at the moment not specifically tied to bitcoin) is ProofPeer
Additional information about these proof markets can be found in this article: http://blogchain.fr/programmarket-en/