This is an interesting problem, one that has been studied and discuss within the bitcoin community quite a bit.
The basic way to do this is to keep track of a normalized TXID alongside of the actual TXID used in the protocol. Then to calculate the normalized ID of a transaction, you serialize it:
- With the inputs' txids replaced by their normalized transaction
- Removing the scriptSigs (as those are the most malleable parts of a transaction)
The only caveat is that that this doesn't need to be done for coinbase transactions, as the scriptSig is not malleable (since it's already in a block) and there are no real txids in the inputs. You also don't want to remove the scriptSig from the coinbase transaction because it contains the height of the block (per BIP34), which guarantees the TXID (and normalized TXID) will be unique. Hence, for coinbase transactions, the normalized transaction id is the same as the transaction id (nTXID == TXID).
Note: this is a lot of work to keep track of normalized IDs for every transaction. It has not been implemented in Bitcoin Core yet, and implementing your own version of this may be time consuming.
If you can guarantee the set of outputs in all transactions within your system will be unique (i.e. you are using new receiving addresses and new change addresses for every transaction), then you might be able to use a hash of the set of outputs as a unique identifier. This would be more specific to your use case, as a temporary fix while Bitcoin Core doesn't yet have support for normalized transaction ids.
Some further resources: