In the very early days of Bitcoin blocks were typically mined by one individual person, often referred to as "solo mining"). This made sense because with only CPU mining each person pretty much had the same probability of solving a block as the next person (unless they had several CPUs). Eventually as mining expanded, became more distributed, and hardware such as FPGAs, GPUs, and ASICs came along, the idea of pooled mining was developed. This allowed a group of people to put their computational power together and share the rewards across each "pooled" miner in accordance with their contributions once the pool had solved the block. Each pool is different, and will use different reward algorithms such as PPLNS, PPS, etc.
why bother mining, if there is no prize for second place?
Mining has very profitable rewards, especially if you are lucky enough to solve the block while solo mining. If you do not have to share the reward among others, 12.5 BTC will likely be more than enough to pay for the electricity used to mine the block depending on location, etc. For smaller miners, pooled mining will give them a portion of the rewards depending on how much hash rate is directed to the pool, regardless of whether they have personally solved the block themselves.
does the randomness still ensure the smaller pools get lucky just enough to keep them interested in mining BTC?
Exactly. Imagine you and a friend were the only people mining in a pool you have created. Lets say you both have one 14Th miner. If the pool solves the block, you will split the reward because you have both agreed to work together and share the rewards.
