If it's a website - what if it goes bust or goes down?
There are private key storage companies that specialise in backup procedures and use multi-signature keys. If they go bust, you still have full control of your funds and just choose another provider.
Read more about how Bitgo employs these services on behalf of its users:
https://blog.bitgo.com/bitgo-release-open-source-key-recovery-service/
If it's on my phone - then what if I lose my phone?...
Most Bitcoin wallets are now Hierarchical Deterministic (HD) style wallets, which means all private keys can be restored by entering a set of random English words. As long as you don't lose those words, you can easily restore your bitcoins if something happens to your device.
Correct me if I'm wrong but presumably a bitcoin is essentially a long string of numbers, so presumably I could physically write this down on paper, make a few copies as backup and bury them in the garden (or whatever), then I wouldn't have to worry about the website going down.
You're not wrong, but most people would find this process tedious and difficult. And if you die, there's a good chance your family would have nobody to guide them through the restore process.
I guess that my ultimate question is, how can I store some bitcoins for many years without having to worry about the security of my wallet?
The best solution currently is a hardware wallet called Trezor. It is a usb device that is a mini computer capable of generating and keeping your private keys offline, even when plugged in.