TIP-035 T grants for tBTC v2 bootstrap node providers

Proposals
1

TL;DR
Proposal for establishing Threshold grants for tBTC v2 bootstrap node providers. Bootstrap nodes are essentially gateways to the tBTC network. They are NOT the same as other Threshold nodes, including nodes acting as signers for tBTC. Bootstrap nodes must be deployed on enterprise-grade hardware and require considerable resources and high technical competence to ensure 99.99% availability. Further, their bonded stakes are not inflation protected as bootstrap nodes do not participate in rewards because they cannot be exposed to slashing risk. Four providers have been identified that are willing to provide long-term support to Threshold (4 years). This proposal establishes a grant of $18,750/month for each provider.

Background
Bootstrap nodes are critical building blocks of infrastructure for the Threshold Networkā€™s tBTC application and are very different from regular nodes. Their operator addresses are hard-coded in off-chain client code.

Bootstrap Node Requirements
Bootstrap nodes require a long-term commitment and need to be deployed on enterprise-grade infrastructureā€¦ Additionally, Bootstrap node providers need to run 2 Random Beacon and ECDSA nodes as well as a minimum of 2 Ethereum nodes to ensure failover protection and guarantee availability of at least 99.99%.

Stakes bonded to a Bootstrap node do not participate in rewards, which means the T locked in their stake is not inflation protected. The DAO will stake the necessary T from its treasury for each bootstrap node

Bootstrap operators also need to be in separate geographic regions to protect against outages, ensure availability and provide stability to the network.

Launch Providers
Four professional node operators have been identified that are willing to make this commitment to Threshold:

Boar - https://boar.network/

InfStones - https://infstones.com/ - InfStones is dedicated to building the next evolution of a better world through limitless Web3 innovation by supporting tens of thousands of nodes on more than 60 major blockchains. The company is dedicated to bringing down the barriers to connecting with the blockchain. InfStonesā€™ platform is battle-tested and supports over 15,000 nodes on over 60 blockchains.

P2P - https://p2p.org/ - P2P Validator provides high uptime, secure staking with advanced monitoring & support. P2P was established in early 2018 and proved the best quality of staking services as one of the top operators in Threshold, Solana, Polkadot/Kusama, Tezos, Graph, Moonbeam, Regen, Chainlink, and Connext Networks. Currently, P2P secures more than 1,5 billion USD value across 30+ networks, provides highly-available RPC services, and contributes to web3 as a core developer for Lido on Solana and Neutron - Cosmos Hub secured smart-contract chain.

Staked - https://staked.us/ - Staked operates the most secure, performant, and cost-effective block production nodes for decentralized PoS protocols on behalf of institutional investors. Their multi-tier signing and listening node architecture delivers stakeholders the ideal combination of security, scalability and decentralization.

Proposal
This proposal would create a grant for each Bootstrap operator of $18,750 per month in liquid T tokens, calculated at the TWAP for the period This grant is designed to cover the operational and staffing costs for bootstrap providers and ensure proper alignment and commitment to the network.

10 Likes
2

Thank you @Vict0r! P2P supports Threshold in the long term since the very beginning of Nucypher testnet, where we invested our own funds in the project and collaborated with NuCypher team to provide a powerful suite of node monitoring and alerting. And we are happy to continue our partnership as a valuable part of tBTC v2 bridge.

We are ready to ensure 99,99% uptime for bootstrap nodes. Our infrastructure is geographically distributed and always under advanced monitoring. Our team provides 24/7 technical support and utilizes world-best DevOps practices. We use multi-step alerting to ensure high-availability and fast reaction speed in case of urgenсies or critical updates. Our DevOps maintain regular snapshot and backup procedures to ensure data availability. To achieve better focus we allocate a small dedicated team responsible for operations that consist of a Product Manager, 1-2 DevOps engineers, and Data Analyst.

About P2P

P2P Validator was established in early 2018 and proved the best quality of staking services provided. Our team has valuable experience in operating highly-available infrastructure utilizing world-best DevOps practices. We took part in multiple testnets and provided non-custodial staking services from the first block of mainnet in 15+ blockchains. Currently, P2P secures more than 1,5 billion USD value across 30+ networks, including Cosmos Hub, Ethereum, Polkadot/Kusama, Solana, TheGraph, Flow and many more. We have 150+ professionals across 10+ countries and our expertise goes far beyond secure validation:

  • P2P is a core developer for Lido on Solana
  • Agoric API solution to power insights and analytics tools
  • Development of Neutron - Cosmos Hub secured smart-contract chain
  • Active governance participation
  • Highly-available RPC service
  • Multiblockchain ETL solution with an API for the collected data
  • Bridges & relayers operation (Wormhole, IBC relayers, Gravity bridge, Ad-Astra etc.)
  • Oracle operation (Chainlink, Pyth, internal oracles in Kava, Agoric etc.)
  • Custom node monitoring suits (NuCypher node monitoring suite; Marlin network node monitoring)
  • Validator performance dashboards for foundations and community
  • P2P Economy blog to promote the most groundbreaking projects in the field and educate the community

Regarding our achievements, Iā€™m proud to highlight the following

  • P2P is one of the top operators in Solana, Polkadot, Kusama, Tezos, Graph, Moonbeam, Regen, Chainlink, Agoric
  • Cosmos Game-of-Zones two first place awards
  • Cosmos Game-of-Stakes ā€œNever Jailedā€ award winners
  • Top ten reliable baker in Tezos
  • Participation in mainnet bootstrapping and seed peer program in Mina
  • Multiple Best Availability Category award in Oasis Quest
  • Top 5 verified staking provider

Letā€™s create a bright decentralized future together!

6 Likes
3

Code name: TIP-035.

1 Like
4

Strong proponent here! Having a dedicated list of bootstrap nodes in different geographies is key for a robust tBTCv2 launch.

1 Like
5

I think it would be great if someone could give a more technical summary of the job bootstrap nodes will perform.

Questions

  1. What happens if one or multiple of these nodes:
    a) vanishes completely or ends up having significantly less than 99% uptime
    b) acts dishonest/malicious

  2. Why 4 nodes and not 2? 5? 10? 40?

  3. What impacts does this group of node have on the decentralization of tBTC - assuming they (4) nodes start a conspiracy against tBTC, what impact can they have?

  4. $18 750 per month is a decent chunk. Companies able to facilitate a deal of this nature will already have 24/7 technical staff on standby so the impact of adding another ā€œproductā€ shouldnā€™t result in major cost increases. What really is the cost here then is maintenance / servers and for that it seems excessive. Iā€™d like to know more about the breakdown for this number.

  5. What was the selection process for these initial nodes? Has everyone been given a chance to submit a proposal to operate as a node (I just heard of this today, but might have missed it). Should we consider future removal / addition or nodes? If not, why not?

3 Likes
6

Furthermore -
How will these bootnodes be monitored to ensure requirements are met? Will they monitor one another and/or will the DAO arrange monitoring

What service(s) will be provided? Bootnodes generally supply the set of active peers via a reliable/hardcoded IP.
I agree that 18750 / month / bootstrapper is a considerable amount, requiring 5 9ā€™s uptime does not seem unreasonable

Re: bootnode stakes - is there a minimum requirement? I agree that if a stake is required to run such a node, the operators must be compensated. The min amount should be known.

3 Likes
7

  1. Bootstrap nodes are hardcoded in the client code and used by other nodes connecting to the network to discover the rest of the nodes. If one or some part of bootstrap nodes vanishes completely, other bootstrap nodes will still be available and will let the clients discover other nodes. Bootstrap nodes are not supposed to participate in random beacon or tBTC protocol but if they get modified and start doing so, the risk of malicious behavior and consequences (slashing stake) are the same as for any other node.

  2. The proposal talks about ā€œTwo Random Beacon and ECDSA nodesā€ and it means two nodes - Random Beacon and ECDSA (tBTC) are the same client software in v2. The engineering team considers 4 independent parties running 2 geographically distributed nodes each safe enough for a robust tBTC v2 launch. There is no advantage in having a higher number of bootstrap nodes assuming the 8 instances proposed work reliably. Every bootstrap node enables network discovery in the same way.

  3. They canā€™t steal BTC but may render the protocol unable to perform day-to-day tasks if off-chain clients canā€™t discover the network. If all the parties start a conspiracy against tBTC, the community can work together to self-point the nodes to each other. The client firewall rules ensure that the client can only connect to other node if 1) that other node is a bootstrap node; OR 2) that other node has a minimum authorization and is participating in random beacon and/or tBTC.

4 Likes
8

Would that be paid in T? Calculated on 30 day TWAP or price at execution of payment? Is there a SLA agreement that the providers would have with the DAO? Whats the min stake for a bootstrap node?

1 Like
9

Thanks Doug. Based on that, seems each seednode operator requires two sites with the following:

  1. a reserved IP or reliable dynDNS resolution
  2. some sufficient stake, in T
  3. appropriate OS and the tBTCv2 client software
  4. extremely reliable, well positioned connectivity
  5. highly reliable, sufficient CPU/RAM to comfortably maintain peer list, as neither storage nor greater processing will be required - a small system
  6. expertise running the above, 24/7/365 staff

So each site will cost the DAO approx 9375 USD / month.
Is that accurate and/or have I left anything important out?

1 Like
10

I would recommend lowering the SLA requirement from 99.99% to 99.9%.

99.9% is still sufficient and only one client instance with the given operator address should be running which makes the higher SLA really hard to achieve in practice. With 4 independent parties running 2 geographically distributed nodes, 99.9% SLA is enough.

Every party is solely responsible for monitoring their nodes, making sure they are up, and reacting to alerts. The development team will monitor if the requirements are met and if we see that some party does not fulfil their duties, we will notify that party and the DAO.

What service(s) will be provided? Bootnodes generally supply the set of active peers via a reliable/hardcoded IP.

Just like @dougvk explained, the addresses will be hardcoded in the client. Hostname with a reserved IP, no dynamic DNS. We must be sure the bootstrap node is reachable with no delays.

Whats the min stake for a bootstrap node?

The goal is to have the bootstrap nodes not needing a minimum stake. This is different from v1. We are working on updating the client code.

2 Likes
11

Note on the history here: for the launch of Keep & tBTC v1 we also needed bootstrap nodes, which we achieved with locked KEEP grants. We chose those nodes based on testnet performance + reputation of the service provider.

I believe todayā€™s list has been a product ofā€¦

  • Which reputable staking providers are involved at this stage, and planning to support tBTC v2 at launch?
  • Which have been involved in client development, and have shown good uptime with v1?

There hasnā€™t been a public bid process or anything like that, to my knowledge. In some circumstances Iā€™d push for one, but I donā€™t think weā€™ll have many reputable, proven staking providers come out of the woodwork that havenā€™t already.

Just my two cents :person_shrugging:

Should we consider future removal / addition or nodes? If not, why not?

Weā€™ve had issues with providers on the last iteration of the network, so this isnā€™t just hypothetical. Removing them means removing an IP address from the default client config, with stakers welcome to upgrade to the new list, or not. In practice, this is a ā€œdevelopers propose, stakers ratifyā€ governanceā€¦ Iā€™m not sure we need more process than that right now, and thereā€™s no way to enforce it on-chain.

2 Likes
12

Bootstrap Nodes are very effective, essentially working like a light house for other nodes; I find the operation to work like a type of self-healing mesh.

I am a huge fan of this proposal because I believe Bootstrap Nodes offer an essential advantage to our network, and advance stability.

There are a growing number of reasons why any node in any network can go offline, for example individual economics and local environmental hazards like recent heat waves, forest fires, and rolling electrical blackouts.

Putting a grant in place to ensure there are at minimum 4 long term, enterprise grade, and geographically unique Bootstrap Nodes will certainly set our network apart, and create a truly robust environment.

This is an excellent proposal, thank you Victor.

2 Likes
13

Thanks pdryaga - updated seednode requirements:

  1. a reserved IP
  2. no T stake
  3. appropriate OS + tBTCv2 client software set only for p2p gossip
  4. reliable, well positioned connectivity
  5. reliable, sufficient CPU/RAM to comfortably maintain peer list, as neither storage nor greater processing will be required - a small system
  6. reasonable expertise running the above, 24/7/365 staff
  7. tBTC dev team will monitor these 8 nodes on the DAOā€™s behalf.

Iā€™ll point out that for 900k / year we can easily come up with other ways to achieve same for far less cost.

Example: DAO hires a person for 100k USD / year to tend 8 cloud situations for which the DAO pays the bill.

14

I would like to point out that a single person is a single point of failure. We also have no room for experimentation during the launch, we must be sure bootstrap nodes are in a good hands of proven, reputable staking providers. I imagine that 4 years from now, the bootstrap mechanism and network recovery may look completely different and may not require such a high SLA.

3 Likes
15

Yes, that example was meant more as illustration than a complete solution.
No one is debating the requirement for predictable seed-nodes via hard-coded IPā€™s.

I am pointing out that 900k / year is a high amount for the required services and wanted those better quantified. Along with expectations/monitoring of these pre-selected operators spelled out.

I suspect the issues leading to this proposal included some subtleties, including:

  1. time is short, we need this solved.
  2. DAO mechanisms to purchase appropriate IPā€™s/+VPS plus admin overhead could be administratively tough, and DAO outright ownership might be sub-optimal, long-term, and who can head that up anyway?
  3. Locating competent people to config and keep the nodes secure/up/patched AND be responsive to project can be tough.

9375 USD per month per node still appears to be quite a lot.
Also - thank you all for your time, I know the issue but one of many

16

Iā€™m pretty new to this and am not very technical when it comes to node operationality. However, the bootstrapped node idea seems like a solid plan to ensure quality uptime for T once TBtc2 is live. I agree itā€™s definitely a hefty cost to pay as well. Iā€™m currently using P2P for staking my T with a Pre Node, and P2P has been great work with. They also charge a high fee to stakers for using their nodes, which can deter many from using them. If this passes, and we pay them to run these bootstrap nodes, it may be beneficial to the network (increase in nodes), if they are required to provide a discount for T holders to stake with them?

17

Thanks @Vict0r for setting this up ! Thanks @pdyraga for your feedback.
I support this proposal, this is needed and the staking providers are solid and reliable afaik.

I have two concerns though :

  1. $900k per year is a big number for our DAO, more so since weā€™re not generating revenue.
    How much can the monthly grant be decreased with an 99.9% uptime SLA ?
    Could we negotiate a number that starts lower and increases over time in sync with DAOā€™s revenue ?

  2. We should have a reliable process in place where dev team and DAO members are communicating/collaborating on ongoing monitoring this. I think the IG would be a fit for this.

2 Likes
18

Thatā€™s definitely a question that should be discussed. Thank you @Agoristen! I totally agree that the high price should have some reasons, and let me share thoughts about it

Actually, the major piece of costs is salaries. We donā€™t have one 24/7 support department due to security reasons and huge differences between the blockchain projects. The most robust approach is allocating a dedicated team responsible for 24/7 monitoring and reacting. The team consists of a Product Manager and 3 DevOps engineers (1 main and 2 back-ups). Of course, there are no tasks for full-time work, but even part-time is expensive.

The second thing we should consider is developing secure & HA infrastructure. Thatā€™s one-time work, but it will take 1-2 months during testnet and early mainnet to set up monitoring, alerting, automation scripts, etc.

And finally, we shouldnā€™t forget about boring, but still important point - inflation which is not 0 for a 4-year commitment

19

a few more comments from my side

  1. @blitmore1 it proposed to set up: 2 random beacon v2 bootstrap nodes, 2 tBTC v2 bootstrap nodes. And It is not just the cost of a VM for a node. Itā€™s also a cost of Ethereum node, at least two because there has to be a redundancy, and bootstraps will receive a lot of traffic and will execute a lot of calls to Ethereum.

  2. Thank you for the warm feedback @C-Nu! That will definitely influence our T-staking offer since the big part of work like setting up monitoring, alerting, etc. will be already done. I havenā€™t got exact numbers yet, but we will share them with the staking guide closer to the mainnet goes live

20

I (along with Kuba) am a Keep Network Contributor who also has a financial interest in Boar, a group which will potentially benefit from the outcome of this proposal. Because of the perception of a potential conflict of interest with this proposal process and in the interest of full transparency, I am disclosing my relationship with Boar to Threshold DAO members. If you have questions or concerns about my relationship to Boar in regards to this proposal please leave comments on this forum thread directly.

1 Like