Bitcoin doesn't use RSA and there is no encryption or cryptography that hides transaction details. Bitcoin is pseudonymous, not anonymous. There are unique identifers for every participant in Bitcoin, the somewhat hard part is tying those identifiers to actual people. So there is no RSA, number theory, or ECC involved in what the IRS is doing because none of that is involved in the details they are concerned about.
What observers like the IRS can do is follow transactions and see how money changes hands from some identifers to others. Some identifiers may be known, such as exchanges and other large services. Most are not and likely to be individuals. Using various software tools, you can try to work out how money is changing hands, but it is error prone, can have a lot of false positives, and is actively being counteracted with things like CoinJoins.
But the IRS probably isn't following those transactions. They are doing something much easier: subpoena the exchanges for customer information. By law, exchanges need to gather identifying information about their customers. This includes things like Social Security Number, passport numbers, driver's license numbers, etc. They have to store all of these personally identifying information and they are all tied to user accounts. Additionally, all of these user accounts have trade and transaction history tied to them as well.
So the IRS is basically just asking (or compelling) exchanges to hand over all of this information about their users and comparing the numbers that were filed in their tax returns to the numbers that they calculate based upon the trade and transaction data that they have gotten from the exchanges. No cryptography, machine learning, or even programming involved whatsoever.
