The other answers describe the rules currently implemented in the client, but I expect those rules to be replaced once block size limits become relevant. Since those rules aren't enforced when clients verify the blockchain, they are only guidelines, and will be violated if they're against the interest of miners.
Miners will simply choose the subset of candidate transactions that respects the limits (blocksize and signature count) and maximizes the fee.
Disregarding a few minor complications (knapsack and dependent transactions) this means that miners sort the transactions by
fee/transactionSize if the maximal size is the limiting factor, and
fee/signatureCount if the signature count is the limiting factor.