I read in several paper about "chaff coins" but I could not find a clear definition for this phenomenon. Does it refer to a specific coin? Is there any "formal" definition for that ?
The term is used specifically in Monero's ring signature based spending scheme.
Monero transactions do not exactly specify which previous outputs are being spent. Instead, a list of alternative inputs is provided (consisting of the real one, and a number of chaff coins). A ring signature is then used to prove authority to spend from the real one, without revealing which one that is.
This has the advantage of obscuring the transaction graph significantly. It also comes at a very high price - as it means validation cannot use a UTXO set based approach, and must instead maintain an ever-growing database of spent coins to prevent double spending.