It is a thing that can be done, but it requires trust. Any user may, at any time, confirm the chain they have is a legitimate chain by starting from the genesis block. They merely need to crank through a few gigabytes of data and hash it all. Okay, maybe more than a few.
There are algorithms to create a "truncated" chain. In this, we observe that most transactions that have been created (coins sent to it) have also been used to create other transactions (coins sent out of it). Once this happens, a transaction can no longer be used, so could be culled. Much of the block chain can be culled in this way.
This sort of approach can be very useful for a pragmatic bitcoiner, who just wants to make sure the people who are paying them are paying with legit coins. The blockchain they need is quite a lot shorter. However, someone had to shorten it. Do you trust them? The only way to verify a truncated chain is to go verify it against the bitcoin blockchain all the way from the genesis block, which is the step you were trying to avoid.
As for everyone agreeing, this is trickier. Remember that a newly minted block doesn't magically become the tip of the block chain. There's a consensus algorithm which makes sure it most likely eventually becomes part of the chain. Orchestrating this could be quite a feat.
If you got people to agree to do this, it would be handled as a "soft fork," in that your new shortened chain would be compatible with bitcoin. You would merely add a layer of trust in the person that built the chain. I'd argue this would be a minimal layer of trust. It would be easy for the miners to publish a substantial consensus saying "we have verified chain XYZ is consistent with Bitcoin."
Verifying off of the genesis block is a clever way around this trust issue. When you look at bitcoin from the genesis block on, it's not that "my bitcoins only have value as long as I can trust that the genesis block was indeed the start of the bitcoin chain," its "my bitcoins value is intrinsically tied into the genesis block."
Would it be bitcoin if you forked it? Amaclin, in their comments argues no, but I'd say that topic gets really murky. Read up on the Ship of Theseus and then make your own decision on that.
As for shortening block chains, there is a great interest these days in "off chain" transactions to shorten the chains. However, as far as I am aware, Bitcoin does not currently have support for these.