There's 2 parameters that affect the value of a transaction fee: the transaction's size and how long you're willing to wait for it to confirm.
Smaller transactions pay a smaller fee than larger transactions since miners can include more of them. At a higher level, you pay for each byte of data you want to include in a block.
The price per byte (denominated in satoshis per byte) that will confirm you in n blocks is estimated by looking at the past data (and maybe at the current transactions waiting to be confirmed).
Once you have that price, you can multiply the size of your transaction by it to get the fee you have to pay.
As for formulas, if you use standard addresses (not P2SH), the formula is:
fee = (n_inputs * 148 + n_outputs * 34 + 10) * price_per_byte
SegWit slightly changes this, where instead of paying per byte, you pay per unit of weight: 1 byte of non witness data = 4 weight, 1 byte of witness data = 1 weight.