It looks like the difficulty is deterministic, but only if the timestamps are trusted.
What's to prevent a large miner from starting back much earlier in the chain (when difficultly was miniscule) and quickly making their own chain with forged timestamps? Wouldn't it be possible to quickly "catch up" and surpass the block height of the real chain, and thus convince clients to switch to a chain built with a non-competitive difficulty?