Strong proponent here! Having a dedicated list of bootstrap nodes in different geographies is key for a robust tBTCv2 launch.
I think it would be great if someone could give a more technical summary of the job bootstrap nodes will perform.
What happens if one or multiple of these nodes:
a) vanishes completely or ends up having significantly less than 99% uptime
b) acts dishonest/malicious
Why 4 nodes and not 2? 5? 10? 40?
What impacts does this group of node have on the decentralization of tBTC - assuming they (4) nodes start a conspiracy against tBTC, what impact can they have?
$18 750 per month is a decent chunk. Companies able to facilitate a deal of this nature will already have 24/7 technical staff on standby so the impact of adding another “product” shouldn’t result in major cost increases. What really is the cost here then is maintenance / servers and for that it seems excessive. I’d like to know more about the breakdown for this number.
What was the selection process for these initial nodes? Has everyone been given a chance to submit a proposal to operate as a node (I just heard of this today, but might have missed it). Should we consider future removal / addition or nodes? If not, why not?
How will these bootnodes be monitored to ensure requirements are met? Will they monitor one another and/or will the DAO arrange monitoring
What service(s) will be provided? Bootnodes generally supply the set of active peers via a reliable/hardcoded IP.
I agree that 18750 / month / bootstrapper is a considerable amount, requiring 5 9’s uptime does not seem unreasonable
Re: bootnode stakes - is there a minimum requirement? I agree that if a stake is required to run such a node, the operators must be compensated. The min amount should be known.
Bootstrap nodes are hardcoded in the client code and used by other nodes connecting to the network to discover the rest of the nodes. If one or some part of bootstrap nodes vanishes completely, other bootstrap nodes will still be available and will let the clients discover other nodes. Bootstrap nodes are not supposed to participate in random beacon or tBTC protocol but if they get modified and start doing so, the risk of malicious behavior and consequences (slashing stake) are the same as for any other node.
The proposal talks about “Two Random Beacon and ECDSA nodes” and it means two nodes - Random Beacon and ECDSA (tBTC) are the same client software in v2. The engineering team considers 4 independent parties running 2 geographically distributed nodes each safe enough for a robust tBTC v2 launch. There is no advantage in having a higher number of bootstrap nodes assuming the 8 instances proposed work reliably. Every bootstrap node enables network discovery in the same way.
They can’t steal BTC but may render the protocol unable to perform day-to-day tasks if off-chain clients can’t discover the network. If all the parties start a conspiracy against tBTC, the community can work together to self-point the nodes to each other. The client firewall rules ensure that the client can only connect to other node if 1) that other node is a bootstrap node; OR 2) that other node has a minimum authorization and is participating in random beacon and/or tBTC.
Would that be paid in T? Calculated on 30 day TWAP or price at execution of payment? Is there a SLA agreement that the providers would have with the DAO? Whats the min stake for a bootstrap node?
Thanks Doug. Based on that, seems each seednode operator requires two sites with the following:
- a reserved IP or reliable dynDNS resolution
- some sufficient stake, in T
- appropriate OS and the tBTCv2 client software
- extremely reliable, well positioned connectivity
- highly reliable, sufficient CPU/RAM to comfortably maintain peer list, as neither storage nor greater processing will be required - a small system
- expertise running the above, 24/7/365 staff
So each site will cost the DAO approx 9375 USD / month.
Is that accurate and/or have I left anything important out?
I would recommend lowering the SLA requirement from 99.99% to 99.9%.
99.9% is still sufficient and only one client instance with the given operator address should be running which makes the higher SLA really hard to achieve in practice. With 4 independent parties running 2 geographically distributed nodes, 99.9% SLA is enough.
Every party is solely responsible for monitoring their nodes, making sure they are up, and reacting to alerts. The development team will monitor if the requirements are met and if we see that some party does not fulfil their duties, we will notify that party and the DAO.
What service(s) will be provided? Bootnodes generally supply the set of active peers via a reliable/hardcoded IP.
Just like @dougvk explained, the addresses will be hardcoded in the client. Hostname with a reserved IP, no dynamic DNS. We must be sure the bootstrap node is reachable with no delays.
Whats the min stake for a bootstrap node?
The goal is to have the bootstrap nodes not needing a minimum stake. This is different from v1. We are working on updating the client code.
Note on the history here: for the launch of Keep & tBTC v1 we also needed bootstrap nodes, which we achieved with locked KEEP grants. We chose those nodes based on testnet performance + reputation of the service provider.
I believe today’s list has been a product of…
- Which reputable staking providers are involved at this stage, and planning to support tBTC v2 at launch?
- Which have been involved in client development, and have shown good uptime with v1?
There hasn’t been a public bid process or anything like that, to my knowledge. In some circumstances I’d push for one, but I don’t think we’ll have many reputable, proven staking providers come out of the woodwork that haven’t already.
Just my two cents
Should we consider future removal / addition or nodes? If not, why not?
We’ve had issues with providers on the last iteration of the network, so this isn’t just hypothetical. Removing them means removing an IP address from the default client config, with stakers welcome to upgrade to the new list, or not. In practice, this is a “developers propose, stakers ratify” governance… I’m not sure we need more process than that right now, and there’s no way to enforce it on-chain.
Bootstrap Nodes are very effective, essentially working like a light house for other nodes; I find the operation to work like a type of self-healing mesh.
I am a huge fan of this proposal because I believe Bootstrap Nodes offer an essential advantage to our network, and advance stability.
There are a growing number of reasons why any node in any network can go offline, for example individual economics and local environmental hazards like recent heat waves, forest fires, and rolling electrical blackouts.
Putting a grant in place to ensure there are at minimum 4 long term, enterprise grade, and geographically unique Bootstrap Nodes will certainly set our network apart, and create a truly robust environment.
This is an excellent proposal, thank you Victor.
Thanks pdryaga - updated seednode requirements:
- a reserved IP
- no T stake
- appropriate OS + tBTCv2 client software set only for p2p gossip
- reliable, well positioned connectivity
- reliable, sufficient CPU/RAM to comfortably maintain peer list, as neither storage nor greater processing will be required - a small system
- reasonable expertise running the above, 24/7/365 staff
- tBTC dev team will monitor these 8 nodes on the DAO’s behalf.
I’ll point out that for 900k / year we can easily come up with other ways to achieve same for far less cost.
Example: DAO hires a person for 100k USD / year to tend 8 cloud situations for which the DAO pays the bill.
I would like to point out that a single person is a single point of failure. We also have no room for experimentation during the launch, we must be sure bootstrap nodes are in a good hands of proven, reputable staking providers. I imagine that 4 years from now, the bootstrap mechanism and network recovery may look completely different and may not require such a high SLA.
Yes, that example was meant more as illustration than a complete solution.
No one is debating the requirement for predictable seed-nodes via hard-coded IP’s.
I am pointing out that 900k / year is a high amount for the required services and wanted those better quantified. Along with expectations/monitoring of these pre-selected operators spelled out.
I suspect the issues leading to this proposal included some subtleties, including:
- time is short, we need this solved.
- DAO mechanisms to purchase appropriate IP’s/+VPS plus admin overhead could be administratively tough, and DAO outright ownership might be sub-optimal, long-term, and who can head that up anyway?
- Locating competent people to config and keep the nodes secure/up/patched AND be responsive to project can be tough.
9375 USD per month per node still appears to be quite a lot.
Also - thank you all for your time, I know the issue but one of many
I’m pretty new to this and am not very technical when it comes to node operationality. However, the bootstrapped node idea seems like a solid plan to ensure quality uptime for T once TBtc2 is live. I agree it’s definitely a hefty cost to pay as well. I’m currently using P2P for staking my T with a Pre Node, and P2P has been great work with. They also charge a high fee to stakers for using their nodes, which can deter many from using them. If this passes, and we pay them to run these bootstrap nodes, it may be beneficial to the network (increase in nodes), if they are required to provide a discount for T holders to stake with them?
Thanks @Vict0r for setting this up ! Thanks @pdyraga for your feedback.
I support this proposal, this is needed and the staking providers are solid and reliable afaik.
I have two concerns though :
$900k per year is a big number for our DAO, more so since we’re not generating revenue.
How much can the monthly grant be decreased with an 99.9% uptime SLA ?
Could we negotiate a number that starts lower and increases over time in sync with DAO’s revenue ?
We should have a reliable process in place where dev team and DAO members are communicating/collaborating on ongoing monitoring this. I think the IG would be a fit for this.
That’s definitely a question that should be discussed. Thank you @Agoristen! I totally agree that the high price should have some reasons, and let me share thoughts about it
Actually, the major piece of costs is salaries. We don’t have one 24/7 support department due to security reasons and huge differences between the blockchain projects. The most robust approach is allocating a dedicated team responsible for 24/7 monitoring and reacting. The team consists of a Product Manager and 3 DevOps engineers (1 main and 2 back-ups). Of course, there are no tasks for full-time work, but even part-time is expensive.
The second thing we should consider is developing secure & HA infrastructure. That’s one-time work, but it will take 1-2 months during testnet and early mainnet to set up monitoring, alerting, automation scripts, etc.
And finally, we shouldn’t forget about boring, but still important point - inflation which is not 0 for a 4-year commitment
a few more comments from my side
@blitmore1 it proposed to set up: 2 random beacon v2 bootstrap nodes, 2 tBTC v2 bootstrap nodes. And It is not just the cost of a VM for a node. It’s also a cost of Ethereum node, at least two because there has to be a redundancy, and bootstraps will receive a lot of traffic and will execute a lot of calls to Ethereum.
Thank you for the warm feedback @C-Nu! That will definitely influence our T-staking offer since the big part of work like setting up monitoring, alerting, etc. will be already done. I haven’t got exact numbers yet, but we will share them with the staking guide closer to the mainnet goes live
I (along with Kuba) am a Keep Network Contributor who also has a financial interest in Boar, a group which will potentially benefit from the outcome of this proposal. Because of the perception of a potential conflict of interest with this proposal process and in the interest of full transparency, I am disclosing my relationship with Boar to Threshold DAO members. If you have questions or concerns about my relationship to Boar in regards to this proposal please leave comments on this forum thread directly.
For clarity, Keep does not have any financial interest in Boar.
Thank you for the disclosure, @pdyraga, and your note, @Will.
My view is that we need node and staking providers, clearly the economics for those companies are very different from 2 years and even a year ago - and so I find it both natural and beneficial that we have strong relationships with firms providing these services. Further, I’ve seen nothing but highly ethical behavior from the Keep and NuCypher teams and dedication to our shared mission/vision, so I trust their processes for contributor disclosure.
This proposal has been superseded by TIP-045 Bootstrap Node Proposal v2