Are there any resources/best practises on running a highly available/high throughput lightning node? If my lightning service where to take off, should I be running multiple lightning nodes or a single lightning node? Would it be possible to scale horizontally? Maybe each channel could be managed on a different machine?
So there are a couple things to consider here:
- The database, it needs to be remotely available if you are going to failover your Lightning node. We talk about adding support to eclair for a remote database (postgres) in our blog post here
- Safely accessing the database. If you have two nodes writing to the database at the same time, it is possible that they write conflicting versions which could result in a penalty transaction on the blockchain. This means you need some sort of locking mechanism to access the database. We have implemented this on eclair as well, you can find the blog post about it here and the actual PR on eclair here.
- The final thing that needs to be considered is how to make the key material available in a highly available, yet secure way. This is still something we are exploring at Suredbits. The "dumb" thing to do is just replicate the keys across every node you have.
Now you should be able to spin up an arbitrary amount of nodes. There should be 1 master node that is actively updating the database, and
n shadow nodes that are waiting for the master node to fail. For implementing this, you can use something like Amazon's ELB and health check's to determine when the current master node fails, and you should promote one of the "shadow" nodes.
You can see a demo of this here on youtube
Are there any resources/best practises on running a highly available/high throughput lightning node?
In Lightning Network, any disturbances (electricity outage, server crash) while running the application can result in permanent loss of funds. It is thus recommended that you run your server in a managed facility like cloud or colo if you are not able to provide high reliability of the servers yourself. This should especially be true in your case since you will be running multiple nodes that will route payments. Any server malfunction can have direct monetary losses.
For attaining high throughput (higher lightning routing) you can use autopilot feature to determine which nodes to connect to and what amount of liquidity you must have in each channel. The things that you might want to consider include increasing the centrality of your node, increase probability to have well funded routes, and improve your presence within your own geography.
If my lightning service where to take off, should I be running multiple lightning nodes or a single lightning node?
Depends on the service that you want to provide. If you want to provide a custodial solution, then running one lightning node may be considered. However, increasing the number of nodes will increase the probability of your routing capabilities, prevent bottlenecks and hence provide a better customer service.
If you want the users to be in charge of their funds, you have no option but to use multiple lightning nodes, one for each user.
Maybe each channel could be managed on a different machine?
No need to run different nodes on a different machine. You can create separate directories within the same machine and run the lightning node.
re: "It is thus recommended that you run your server in a managed facility like cloud or colo if you are not able to provide high reliability of the servers yourself." - I'm not sure I like the idea of hosting a high-capacity hot wallet on someone else's servers, as there are other security methods worth mentioning, eg watchtowers. Also, a lightning node does not require perfect uptime, it just needs to be online enough to ensure no channel partners have attempted to claim an outdated channel state (how much uptime this is will vary according to channel settings).– chytrikDec 10, 2019 at 22:25
@chytrik I agree with your reasoning, but one of the issue that needs to be tackled is preventing force closes of channels. If OP is running a custodial solution, then their node will have high routing. If the server crashes at a time when multiple HTLCs are being routed, then it can lead to the peers force closing the channels if those peers are not able to fulfill those HTLCs. Watchtowers can play a role in ensuring that the peer does not cheat me, but honest force closing of channels is a legitimate concern that can cost high fees if this situation happens often. Dec 11, 2019 at 5:06
that is a fair point; there is cost to manage channels. Thanks for elaborating– chytrikDec 11, 2019 at 6:14