There is no size limit to the mempool from what I can find. It might be possible to flood it but rather difficult and expensive / slow. I see two approaches:
A) minimum relay fee
- connect to 1000 nodes
- send plenty of txs that meet the minimum relay requirements
- how many?: 32 000 000, 32 000 on each node
- nLocktime in the far future so they don't get into blocks
- tx fee: minimum relay fee: 0.0001
- amount: very low
- size per tx: 250bytes
- mempools are filled with 8gb of dust
It would take a while to generate that many addresses and tx and get them broadcasted. Also it would cost at least 3200BTC.
B) 0 fee txs with minimum output of 0.01 so they still get relayed
Mempool is gated by a rate limiter for free tx:
// Continuously rate-limit free transactions
// This mitigates 'penny-flooding' -- sending thousands of free transactions just to
// be annoying or make others' transactions take longer to confirm.
// Use an exponentially decaying ~10-minute window:
// -limitfreerelay unit is thousand-bytes-per-minute
// At default rate it would take over a month to fill 1GB
So this will be at least very slow. Also you need to have 320 000 BTC for 8GB. :)
There might be ways to artificially make txs larger. Alternate clients might do things differently.