Updated to v0.6.3 of the specification, progress with networking & interoperability, and more human resources.
We've made steady progress over the last few weeks with our networking stack and upgrades to the latest specification:
- Up-to-date with spec
- RESTful HTTP API merged into eth2.0-spec.
- Significantly improved metrics and monitoring.
- Lots of progress on
- Fuzzing Lighthouse.
- BLS standardisation and optimisation.
- More dedicated Lighthouse resources.
- Merge feature branches.
- Meeting Eth2 implementers in Toronto and Ithaca.
- Collaborate on networking interoperability.
- Replace gRPC RESTful HTTP.
We've made steady progress this month with a focus on improving developer experience.
Up-to-date and passing tests
Since our last update, we have updated
Lighthouse to specification
As described in our previous blog post, the
specification has introduced a more comprehensive test suite. We are happy to
announce that Lighthouse now passes all of these new test vectors (except a
few which cannot be passed yet, but will be fixed in
The test vectors are provided as large YAML files which are not appropriate for storage in a regular git repository. As such, those who wish to build and test Lighthouse are now required to install the git extension: git-lfs.
Defined RESTful HTTP API
Luke has finalised the REST API which will allow the validator client binary to communicate with a running beacon node. The proposal has been merged into the Eth2.0 specification repository, and is also maintained on SwaggerHub for easy viewing.
Metrics and Monitoring
Paul has implemented a range of new metrics across the Lighthouse code base, which helps to monitor the health of and interactions between nodes. These metrics are intended to be a utility for developers to measure how code changes affect operation, and are implemented using the Prometheus monitoring system.
The metrics are provided for developer use only, and as such there is no guarantee that any metric will remain the same between different software versions. In future, a more stable API for fetching operational data is likely to be provided via the REST API.
In our previous update Adrian described Discovery v5
(discv5) and it's role in Eth2.0. We are currently testing an initial implementation of
discv5 within a libp2p context. This implementation is focused around the
discovery aspect of discv5 (
FINDNODE RPC requests) with the
topic advertisement and registration development being delayed until the discovery aspect
is complete and tested. We will be focusing more time on client interoperability
and less on building out all the features on the network protocols over the
next stretch and imagine the completion of extended features of discv5 to be delayed.
Fuzzing for bugs
As Mehdi wrote about in a
blog post last year, we are implementing fuzzing
techniques to detect crashes (a.k.a panics in Rust) and memory allocation errors in Lighthouse. Work had already started on this and has intensified this week, targeting primarily our networking stack and leading to the identification of a few bugs. Some of these bugs impact external crates (dependencies) included in Lighthouse. For example,
libFuzzer identified a memory allocation bug in the ASN.1-DER encoder/decoder used by
rust-libp2p - see the related GitHub issue for further details.
Expect to see more details and information about fuzzing Lighthouse in the next update.
BLS standardisation and optimisation
The standard for BLS is fast approaching a stable state. We're happy to announce that we now have a fully functional implementation of the standard, using Apache Milagro as the backing library, which can be found here. Included in this library is a small list of JSON test vectors which have been verified against the reference implementation.
Furthermore, Vitalik recently published an optimisation for verifying multiple signatures from a variety of public keys and messages, by reducing the number pairings that need to be done. This optimisation was implemented and benchmarked, where we recorded a reduction of approximately 40% in verification times when verifying 300 messages.
@michaelsproul was employed by Sigma Prime on a part-time basis in March to work on Lighthouse and has been a tremendous help since then. We are very excited that Michael has recently decided to join us full-time and therefore increase his excellent contributions to the Lighthouse project.
The next few weeks will see some further code clean-up, testing, and collaboration.
Merging feature branches
While we have been working on separate tasks such as networking, spec. updates, and various other fixes, there is currently a lot of unmerged code. Dubbed by some as "the great merge of 2019", consolidating all of our changes into the master branch is a major task that will happen over the next couple of weeks.
Collaboration for interoperability
In our push towards a multi-client testnet We have been collaborating with other Eth2.0 implementers to achieve interoperability. As such, Paul has been catching up with other implementers in North America at the Scaling Ethereum research workshop and the IC3 Blockchain Boot Camp. We are also in communication with other Eth2.0 teams in an effort to standardise our network communication.
Swapping gRPC for REST
With the publication of the RESTful HTTP API, we will soon be replacing the gRPC interface used for communication between the validator client and beacon node. The new REST interface should make it easier for validator clients, written in any language, to easily interact with Lighthouse. As an added bonus, this change removes Lighthouse's dependency on both gRPC and protobuf.
v0.7.0 of the specification
was released the same day as this update, and so the next few weeks will also
see a push to get Lighthouse up-to-spec with this latest version and passing all the tests.