First thing to understand is that a balance of some address is not really a balance in the same sense as a bank account balance. It is actually closer to how you think of a physical wallet balance: it will have multiple cash bills and the balance of the wallet will be the collection of the cash bills you happen to have in your wallet, it's not just 1 number - it's a set of distinct bills. Each cash bill will have some value and an unique serial number printed on it. Let's try and apply this model of thinking to Bitcoin.
Digital Bitcoin wallet will have some magic not found in cash wallets. With Bitcoin, first you must program your wallet to subscribe to some set of addresses. Then, when you connect to the network, the wallet will scan the blockchain and "load" all the Bitcoin bills that belong to any of the addresses. How does it know which bills belong to it? Every Bitcoin bill has 3 things printed on it: unique serial number (TXID/index), the address (technically "locking script") and the value (technically "satoshi amount") Multiple bills may have the same address printed but they'll each have different serial numbers and may have different values, too so each address may have multiple bills of various denominations associated with it. In Bitcoin, each such "bill" is called an unspent-transaction-output (UTXO).
When you spend Bitcoin, you take some set of these UTXO-bills from your wallet, and create a new set of UTXO-bills of equal or smaller amount (the difference goes to miners as fee). Old bills get destroyed, new ones get created - and you can freely set the value of each.
To actually spend an UTXO-bill, your wallet programming must know how to unlock it, and each UTXO-bill is really a smart bill, it has a little program/firmware, where the bill itself decides the conditions which must be satisfied for it to be spent. The address really encodes a specific firmware, so you will only look for addresses where you know how to make the program unlock the bill. Most bills require you to key-in a signature and a public key, and the bill's firmware checks the key and signature against the value hard-coded in its firmware and if it passes it then releases the Bitcoins and lets them go to some new bill(s) with firmware set by the spender.
Normally a wallet will store a collection of secret keys, and then it will compute matching addresses and then it will scan the blockchain to "load" any UTXOs that belong to it, and when it loads all of them it will be ready to spend by submitting transactions to the blockchain.
With that I hope you will get a better understanding of how it works. Let's now try answer the questions.
How does a Ledger Nano S work? Is it simply a wallet?
The hardware unit is not a full wallet. Some software on your computer, connected to the Internet, must scan the blockchain to find all those UTXO-bills that belong to your address(es) for which your hardware unit holds the keys. The same software prepares a transaction that will spend them, and then talks to your hardware unit just to fill in the missing piece - the signature. Then, it is again your online software that publishes the signed transaction to the network, and updates the view of what cash UTXO-bills you own (it will remove the ones just spent from your wallet's balance).
Is a wallet simply a folder/directory containing a number (1 or more) of private keys, each able to generate a public address?
That's the minimal information required to reconstruct a wallet. But a wallet ready to spend contains more - a list of all UTXOs that belong to it which it "loads" into itself by scanning the blockchain for addresses for which it knows the secret keys.
What ensures that a private key, generated in a wallet, is compatible with, for instance, the Bitcoin blockchain?
Almost any string of 64 hexadecimal characters (32 bytes) is a valid secret key.
Nearly every 256-bit number is a valid ECDSA private key. Specifically, any 256-bit number from 0x1 to 0xFFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFE BAAE DCE6 AF48 A03B BFD2 5E8C D036 4140 is a valid private key.
If a wallet happens to generate a 256-bit number outside the range, it can just try again, or make it wrap around by applying the modulo operation.
Can I choose my own private key? If I have a favourite sequence of numbers, is it possible to encrypt these numbers and use the encrypted value as my private key, enabling me to have a private key without writing it down anywhere (I know I sound paranoid :-D)
Yes. You just have to have some way of mapping your secret sauce to a 256-bit number. You can just hash your secret information using sha256 and get a 256-bit value. This is NOT A GOOD IDEA because humans are terrible at producing enough entropy, and it is more likely that not that a computer could guess your secret. It is better to let your wallet generate a random mnemonic and then spend some time learning it, and have it backed up on paper somewhere safe.
If I, theoretically, don't intend to spend my Bitcoins, wouldn't it be more efficient to simply write down my private key on a piece of paper and never use a wallet until I actually intend to spend the coins?
That's ok, but be sure you use some hierarchical-deterministic wallet (ideally BIP32/BIP39/BIP44 compatible) so writing down just 1 secret (mnemonic phrase / master key) will be enough to deterministically generate a whole bunch of secret keys/addresses.
For my hardware wallet, I am asked to fill out a recovery sheet. How can I recover my funds using my recovery sheet in case I lose my hardware wallet? I'm uncertain how the recovery sheet is linked to the private keys stored in my hardware wallet?
If I can actually create multiple private keys in the hardware wallet, I can't image how the recovery sheet can be linked to the private keys stored on the wallet.
All the keys will be generated using a formula, from the one secret written down on the recovery sheet. The "formula" is cryptographically-safe meaning nobody can recover the "master key" from any 1 key, and that nobody can tell that some 2 public keys were created from the same master key. If you lose your wallet, any software that knows the formula (defined in BIPs 32/39/44) can reconstruct all your addresses and then scan the blockchain to load your UTXOs and by doing so fully reconstruct your wallet.
How can I see the balance of a wallet? Is it just the number of coins sent to the public address subtracted the number of coins sent from the public address (do you actually say that coins are sent FROM a public address or does public addresses only exist for the purpose of receiving)?
If that's the case, could I simply dig down my hardware wallet in the ground in my garden, and still keep track of the balance using the public addresses associated with the private keys stored on the hardware wallet?
Yes, this is called a watch-only wallet. It will keep a list of just the public keys/addresses and not keep the secret keys. So it can look but not touch. With BIP39, using some crypto magic, it's possible to have a piece of information that lets you reregenerate all the addresses but without generating the associated secret keys, which is rather convenient - all you need is 1 piece of information to be able to watch all your addresses.
Is there any differences between the terms public key and public address? Is the blockchain technology almost similar to PGP encryption software?
A public address doesn't have to have a public key at all! Remember that address encodes a specific firmware for the UTXO-bill. The most common type of firmware is pay-to-public-key-hash (P2PKH) but it doesn't have to be! Also, the actual public keys of an address will remain secret until you spend from the address the 1st time, because the address typically encodes the hash of the key, which is why address re-use is discouraged.
