Unfortunately, I'm not sure there are great answers to these questions right now, but I'll try to explain how things work today.
If doing a 2nd CPFP should you boost the original tx, or boost the last boost?
I think it depends on what you're trying to accomplish. If you're ok with the first CPFP attempt taking a while to confirm and you just want the very first transaction to confirm soon, then you might want to just CPFP the original transaction again (assuming you have another output to use?), so that you only are paying for two transactions to quickly confirm, rather than 3.
If instead you need the first CPFP attempt to confirm quickly as well, then you could spend one of its outputs and try to fee bump both the first and second transactions when you create this third one. Transactions are selected for mining based on their "ancestor feerate" score, which is the total feerate of a transaction along with all the unconfirmed parents that must be confirmed as well in order for the transaction to be valid to be mined. So this would do exactly what you want, I think.
Alternatively, you could also try to RBF the first two transactions at the same time, by issuing a transaction that conflicts with the first one and includes the outputs you want to include from both those transactions. Whether this is more economical will depend on what other transactions are in the mempool that spend outputs of these two transactions (see below).
What happens to CPFP txs that are already in the mempool and the original tx gets RBF'd?
Any transactions that depend on the original will be removed from the mempool when the original transaction is RBF'd. Generally, RBF is more efficient (lower required fees) than CPFP if the goal is to fee-bump. However, for anti-DoS reasons, if there are child transactions in the mempool, then RBF's of the parent are required to not only pay for the parent to be evicted (a higher fee and higher feerate than the parent), but they're also required to pay for all the children being evicted, too (so the new transaction must have a higher fee than that paid by all transactions that are removed from the mempool). This is described in BIP 125.
So, if you have a transaction that someone else has already chained a CPFP onto, then it might be more expensive for you to RBF your original transaction than it is for you to chain your own CPFP transaction, because the RBF would not just pay for replacing the parent but also the child that someone else created.
Also, a limitation of the current mining algorithm is that in a situation where a parent transaction has two different children trying to CPFP it (ie "sibling transactions" that each only depend on the parent, and not each other), the priority with which the parent will be mined will be at the higher of the two ancestor feerates of those children -- even though it's possible that the best feerate would be the one that includes all children taken together.
Would it be more cost-effective to perform several CPFP txs, or should you consider replacing your original CPFP tx with RBF?
In the absence of other transactions that would spend outputs of your original transaction -- which can lead to RBF "pinning", where a transaction is difficult to replace due to the rules requiring replacements to pay more total fee than all transactions that would have to be removed -- it should always be cheaper to RBF than CPFP, because you have to pay for more total bytes of transaction data to be mined when you generate new transactions than when you replace a transaction.
However, RBF can be tricky as well, because if you RBF a transaction repeatedly you have to be careful about how you manage your inputs,
so that if you are generating multiple replacements for a transaction, then each one should conflict with everyone that you signed before, or else you might end up issuing a payment twice. So some thought is needed in designing how a given application or wallet is designed to make sure that this is used properly.
In the future, the pinning issues related to RBF may be mitigated by new policy rules, such as the "v3" transaction relay policy that has been proposed, which would limit RBF pinning by limiting the size and number of child transactions that transactions opting-in to that policy are permitted. However, that's not something that has been deployed yet.
