The transaction fee is specified by the signed transaction itself and is consumed when the transaction is first processed, regardless of the transaction type or amount or its success or failure. For example, an offer than is later cancelled takes one transaction fee to place the offer and another to later cancel it and a properly specified payment transaction that fails (lack of funds, lack of destination trust lines, paths dry, etc, etc) still charges the transaction fee. Only transactions that are outright dropped by the server (e.g. badly formatted, invalid in some way, fee too low, etc) return a local server error and do not charge the fee.
The nominal transaction fee is 0.000010 XRP (10 drops) but this is dynamically increased when the server(s) are under load.
It's up to the client to set a sane amount based on the amount the server(s) are currently requesting.
For example, a client could set the fee to the current amount and re-try with a higher fee if it fails, it could set a slightly higher fee than requested to prevent such retries if the fees are changing. It could have a configurable or hard coded maximum beyond which it stops and asks the user if they want to proceed with the high fee or try again later.
Completely unrelated are transit fees that an issuer may have set.