We need to distinguish the limitations for validity and standardness. The first is determined by the consensus rules (which cannot change without a hard fork). The second ones are determined by local relay and mining policies implemented in the reference client, and can change from version to version.
In addition, we need to distinguish raw multisig (a simple OP_CHECKMULTISIG based script), and P2SH multisig (where the OP_CHECKMULTISIG script is only revealed upon spending).
Finally, because scripts include public keys, their sizes depend on their lengths. Both compressed (33 bytes) and uncompressed (65 bytes) keys exist, depending on which version of a wallet was used to generate them.
Raw multisig
Every m-of-n combination is valid (up to n=20), but standardness rules only allow up to n=3.
This limitation is enforced when sending (i.e., sending to a 2-of-4 multisig will not be considered standard).
P2SH multisig
validity rules require that the P2SH redeem script is at most 520 bytes. As the redeem script is [m pubkey1 pubkey2 ... n OP_CHECKMULTISIG], it follows that the length of all public keys together plus the number of public keys must not be over 517.
- For compressed public keys, this means up to n=15
- For uncompressed ones, up to n=7.
For standardness, it depends on the version. Up to v0.9.*, the reference client required the total spending script to be at most 500 bytes.
- For compressed keys, this means m*73 + n*34 <= 496 (up to 1-of-12, 2-of-10, 3-of-8 or 4-of-6).
- For uncompressed keys, this means m*73 + n*66 <= 496 (up to 1-of-6, 2-of-5, 3-of-4).
In 0.10, these standardness rules will likely be relaxed, allowing to go up to the validity limits (15-of-15 for compressed keys, 7-of-7 for uncompressed keys).
Also, for P2SH, standardness and validity restrictions are only applied when spending (as the spending script is only revealed at that time). This may mean that sending your coins to a P2SH script may make them irrecoverably lost (especially when violating a validity rule).