The Bitcoin Protocol (consensus rules) has two relevant rules for the timestamps in block headers:
- A node will not accept a block whose timestamp is more than two hours in the future.
- A node will not accept a block unless it has a timestamp greater than the median of the previous 11 blocks. In Bitcoin, we call this Median-Time-Past (MTP).
As you mention in your question, difficulty changes are calculated based on the times of the first and last blocks in a 2,016-block difficulty period. (Technically there's an off-by-one error there, but that's not important here.)
Given the above rules, if all miners agreed, they could simply increment the clock the minimum amount of one second MTP for the first 2,015 blocks and then set the time to two hours in the future. That would basically give them just a small decrease in the difficulty, but think about what happens to the MTP when they add that last slightly-future datapoint: the actual median doesn't change much at all. Actual time stamps are in seconds, but here's a set of 11 timestamps in days delta from present time:
[-13, -13, -13, -13, -13, -13, -13, -13, -13, -13, 0]
The median of the above is -13, meaning that after miners create the slightly-future-time block at the end of the difficulty period, they don't need to move their timestamps forward more than the minimum one second---so the next difficulty period starts out -13 days.
At the end of the next difficulty period, miners again move the timestamp forward as far as possible, so the protocol thinks it took 28 days to mine the blocks---half the expected speed---and so decreases the difficulty by about half. Now the values used for MTP look like:
[-27, -27, -27, -27, -27, -27, -27, -27, -27, -27, 0]
So miners can continue keeping timestamps far in the past and repeat the attack, lowering difficulty every period until the point where it takes them less than 2,016 seconds to produce 2,016 blocks, at which point they can't lower difficulty any further because the MTP function requires time increase by a minimum of one second over the median every block.
Now, your main question was how can this attack work without collusion by a majority of miners. Now that you've seen how the attack works with all miners participating, it should be clear than selecting the median time can allow an attacking miner who's lucky enough to find blocks reliably to prevent median time from jumping forward to an honest value. For example, imagine these are the times of the previous 11 blocks, in block chain sequence:
[-27, 0, -27, 0, -27, 0, -27, 0, -27, 0, -27]
If you sort those numbers to find the median, it's -27 even though 5/11ths (45%) of the hashrate is mining accurately. But wait, doesn't that mean the attacking miner has 55% of the hashrate? Maybe not, for a large miner with about 30% or more of the hashrate could obtain an advantage over other miners using a selfish mining attack, or the miner could simply threaten to attempt to make stale ("orphan") other miners blocks who have accurate time stamps, causing those honest miners to earn less income.
I myself don't consider the attack particularly likely on Bitcoin because it's slow to execute and publicly obvious, but it's something protocol designers do need to keep in mind for when they change parameters, as those changes could make the attack easier to execute.