So I'm in the process of starting a new service, but want to make sure I build the foundation correctly from the start. When customers deposit to their user account on my service, I need to understand whether I should be using one giant wallet with a new address for every customer and assign that address in the database against their user, or if I should generate a new wallet (with, presumably, new seeds) on signup for every customer, so each customer has their own unique wallet. What is the best course of action here? One thing to be aware of is how the backup process will go, and any potential race conditions (say someone generates themselves an address and sends coins, but the server gets wiped out and the backup doesn't have that last 10 seconds of activity, so the address that was generated can't be tied back to the appropriate user).
If you are looking to accept deposits from your customers then you should create one deterministic wallet using software like electrum and the install the extended public key aka xpub aka master public key (MPK) on the server. Then from that you can generate all the addresses you need.
The above creates a watch-only setup where you can generate addresses to receive bitcoin but not spend. So in case someone compromises your server they learn your entire transaction history but they can't steal from you. In order to spend the money you need to use the seeded wallet on your PC.
In order to associate addresses with users you would make a note of it in your app's database.
If you want to associate addresses with users in some deterministic fashion then you can assign each user an ID number and a counter which will serve as an address index. Then you can devise an appropriate derivation path that incorporates those two things. So for example Alice could be user ID 5430. Alice's deposit addresses are m/5430/i where i is the counter in the database. First Alice address would be m/5430/0, second would be m/5430/1 and so on. The user ID would not change once a user registers so even with slightly older backups as long as you have the user ID you can increment the counter until you get to the correct one.
The best thing to do is make sure that each customer has their own wallet, with a recoverable phrase.
You mentioned you are using a server, which is unwise as hackers will be looking at that as a big bull's eye.
It sounds like you are making an exchange.
So what they do, is they have receive addresses which are real and on the blockchain. Each user is assigned an address where they can receieve X amount of BTC.
The send addresses however, are in the database, and the database keeps track of how many BTC you have internally.
When you ask to send BTC to someone, the database checks whether that someone is also in their database based on the toAdress, if so, then just minus amount from you and add amount to other person. This happens off the chain.
If not, then they minus the amount from you and send the BTC from one of their own addresses.
You should get someone to build your system for you, as you are dealing with people's finances, and one wrong move could leave you like Mt. Gox