If you are talking about finding a better algorithm than SHA(SHA(block header)) - it might be possible to optimise the algorithm for some features, like being GPU resistant and so forth, but generally that would be a bit pointless. Changing that algorithm now would undermine the stability of Bitcoin and wouldn't benefit it much. Generally, the current algorithm does it job and does it well - it provides scalable difficulty, the mechanics are still unbroken, and so forth.
As for using AI to try solving the SHA(SHA(block header)) algorithm - it probably would be hard if not computationally impossible at the moment. The algorithm doesn't appear to be broken, meaning that the output of the calculation appears to be quite random, which is not faring good for trying to use AI to solve it.