Transactions compete with each other in regard to the fee they offer to the miner divided by the amount of blockspace they require to be included in a block.
Where does the fee come from?
When you craft a transaction, you first determine the amount that you want to send. After creating the sending outputs, you collect inputs that have sufficient value to pay for the sending outputs as well as that. Your software then crafts a payment order (transaction) that lists the inputs to be consumed authenticated by the corresponding signatures, as well as the outputs to be created. If the inputs total value exceeds what you wanted to send, you will often create an additional change output to send back the remainder to an address that you control.
Obviously, you may not create outputs worth more than the inputs that you spend, but the opposite is required: The difference between the sum of the inputs and the sum of the outputs, i.e. the value that you have not reassigned to a recipient, is free-floating and may be collected by the miners in the coinbase transaction of the confirming block.
How big is a transaction?
The amount of blockspace required, is dependent on the type of script that you use in your input. e.g.:
- a P2PKH input has 147 or 148 bytes
- a P2SH 2 of 3 multisignature input has 295 bytes
- variable for custom P2SH scripts
As far as I know, all outputs right now are 34 bytes.
To calculate the whole transaction size, you add up the transaction header (10 bytes), the inputs with their corresponding size (n*148 bytes for P2PKH), and the outputs (m*34 bytes).
Fee Rate
The fee rate is now the amount of freefloating fee in satoshis per the size of the transaction in bytes:
feeRate [sat/byte] = fee [sat] / txsize [byte]
Miners will generally select transactions by order of highest fee rate first.