Blockchain Engineering - class of 2024 - Team Democracy-1: blockchain networking (a)

[synctext]

Project assignment for Delft University of Technology master course called "blockchain engineering".
Democracy-1: Blockchain networking

• Replay historical voting rounds on the live network. Emulate 10..50 identities from a single smartphone.
• Understand and re-use existing ready-to-go datasets with DAO votes.
• Create transaction blocks within Trustchain in a custom IPv8 community using Kotlin superapp
• Share votes using a ledger-based gossip protocol
• Binary transfer of bulk votes using QUIC or uTP.
• The outcome of your entire project is a single number. Remember, running code is required for a passing grade 💥. Your focus is the amounts of blockchain-data bits you can transfer between two phones.
• BONUS, add carrier-grade NAT puncturing

Background reading:

• The SIM-card experiment with carrier-grade NAT puncturing. Real details: msc placeholder: 5G overlay infrastructure for decentralised learning ??? #7258
• Read about EVA protocol!

Upcoming sprint assignment

• fetch major code update for superapp modules and IPv8 sub-module
• understand existing code
• NAT puncturing testing, using the app feature. Try to get a receiving UDP packet!
• get a cmdline Java transfer going between 2 peers, with uTP.

[Eragoneq]

For the task to get the uTP transfer going, should we just make a completely fresh project and somehow already integrate it into the current project?

uTP Thesis - link

[synctext]

First task, get uTP working, minimal changes. No superapp integration yet.

[synctext]

• not as much progress as needed for a successful mid-term
• Found null pointer bug in uTP lib, typos, and changed internals
• not as much time spend on course as required (12-13 hours/week)
• EVA, IPv8, NAT puncturing initial investigations
• did small trails with different laptops 🚀

Sprint goal this week: structural measurement table of uTP. read example table
Quality of networking protocol determines the performance. Read background thesis with details
investigate tc utility, or netem wrapper

Measurement environment with packet loss and latency with reproducible performance test. One script to produce the example table with various conditions: 2 laptop and 1% packet loss rate

[synctext]

• organised into 2 teams:
  • 1. sub-app team and debug (@Eragoneq and @juul-j)
  • 2. benchmark team ()
• test mode inside Superapp
  • benchmark utility for connecting and profiling 2 phones
  • Enhancement of https://canyouseeme.org/ (without central server)
  • add to superapp, wireshark debugging, emulator port forwarding
  • proxy setup
  • no success yet 2 phones connecting with uTP
  • hint: no fancy overengineering, generic, universal connectivity API, minimal viable code.
• Benchmarking team
  • got the 11 years old testplan.csv to work MICROS_WAIT_BURST is only parameter being changed
  • 91 test case scenarios, 90 lines
  • burst size seems to be important says @CallumH2410
  • no idea yet why 3 KBit/sec or 9 MByte/sec
  • Structural testing of default parameter setting and changing 1 setting and the time plus determine outcome
  • Even Docket to Docker operational and profiled 🚀

FROM python:3.8-alpine
RUN apk add iproute2 vim iputils-ping python3 py3-pip openjdk11 bash 

ToDo:

• sprint and documentation how to get a connectable port from Android Studio with forwarding.
• exact line of code to uTP with null-pointer bug?
• rounding errors in uTP test code
• We can keep assignment as is; or make it more interesting:
  • focus exclusively on test transfer between 2 phones
  • just 1 networking debug screen, no new sub-app, no new dependancy, just add existing code to running code
  • debug screen can test transfer 2 hard-coded files: votes.csv (3.3. MBytes) and votes.csv (13.7 MBytes)
  • change the parameters for the test, such as,
    • packet size/mode,
    • packet drop 100% 🏎️
    • packet re-ordering
    • fast retransmit
    • congestion control.
    • How about 1-way receive mode, send mode?
    • 2-way send&receive mix traffic mode?
  • Decide between full tuning and both phones or 1 phone is the leader and automagically sets up the remote receiving parameters.
  • Make the uTP library mode "self-testing".
  • Inspiration from more (too generic) generic approach
  • no docker needed, extend the testplan.csv with minimal changes needed for 1 new column: packet drop rate {0..100%}. So all in same Java lib.

[PieterCarton]

Varying Package size/mode

Skipped packages needed before fast retransmit

Target Delay of Congestion Control

Test performed with 1% package loss under 250 ms latency, transfer speed taken as average of 10 runs

Benchmark Program Location: https://github.com/iiljkic/utp4j/blob/master/src/main/java/net/utp4j/examples/configtest/ConfigTestWrite.java

[Eragoneq]

We've actually contacted the author of the utp4j library and he personally suggested to rewrite it from scratch.
Main reasons being code quality issues and being in quite experimental state.

Still in the contact about some other details, but this was his main reply about the maintenance/taking over the project.
-- EDIT: After further conversation, the author agreed to transfer ownership (to me for now)

Current versions:

• utp4j (fork) https://github.com/PieterCarton/utp4j
• utp4j (original) https://github.com/Eragoneq/utp4j
• trustapp (fork) https://github.com/Eragoneq/trustchain-superapp

Current week progress update:

• Unifed the workflow on github
• Pushed changes to our forks
• Managed to exchange data between 2 phones (with uTP on local network) (253 KB/s)
• Fixed existing issue in utp4j library and updated dependencies

Current view in the app:

[PieterCarton]

Week Update:

• Forked UTP4J library and pushed previous modifications (see comment of @Eragoneq for more details surrounding situation)
• Implemented basic benchmarking "hand-shake": sending party in benchmark determines testplan to follow, receiving party follows plan
• Migrated benchmarking from docker to work on local machine
• Partially redid current benchmarks
  • Drastic improvement in performance compared to previous weeks when no latency is present (speeds of up to 50mB/s at its peak, avg. speed around 25mB/s).
  • Packet loss has little to no effect on these tranfer rates
  • However, even small amount of latency makes the transfer rate collapse (50x decrease in speed at 50ms latency)
• investigation of slow speed through wireshark
  • transmitter only actively sending packets fraction of the time, then idle for multiple ms
  • likely congestion window too small -> congestion control not optimally configured for medium to large latencies

Example: fast restransmission Benchmark without latency

Example: same fast restransmission Benchmark with 50ms latency

[synctext]

• "the other group is using the library wrong", inverse logic of client/server. Server == receiving in common case, resulting sequence number ==1 or random. ???? 😂
• 7390 line of code added. What do other master courses say about mixing fixes and global textual changes?
• Problems with utp4j lib:
  • overengineering, too abstract, boilerplate
  • bad inconsistent naming
  • CPU performance heavy, not optimised and not profiled. (5% cpu usage on laptop)
  • it collapses with latency, so is incapable of keeping the packet request queue filled (or window size wrong).
• please focus on the 500 ms case, which is bad 4G/5G networking. But still realistic.
• For mid-term, focus on an Android-14 compatible APK on this issue for transfer testing 👓
• Create an example IPv8 community with utp4j and use endpoint-send-ipv8 to send a single UDP via endpoint override
• Clear 500ms-case library/performance fixes could be cardinal outcome of your Master course project!

[Eragoneq]

Starting off this week updates:

• Transferred the existing utp4j library to the organisation

  • Further pull requests should be reviewed by other team members
  • Post issues of library directly on the repo

• Managed to fetch peer data

• Sending over wireless network (local)

• Attempted to implement sending over IPv8 socket

  Current app version: apk

[CallumH2410]

Callum: Was sick for most of last week so will need to do some catching up the upcoming week(s).
Some issues getting our forked repositories working based on space issues with my laptop (Michals contributions), got it working for physical device so managed to get some front-end changes in.
Will be looking to pick up the lost hours of last week in the upcoming week(s).

[PieterCarton]

Weekly Update Benchmarking Team:

• upon investigation of UTP with 500ms latency, congestion window was tiny as expected
• 2 issues found:
  • when connection is very stable, UTP estimates the variance of the latency to be 0ms. Since UTP uses the variance to determine when a packet times out, this leads to phantom packet loss and limits window size
  • when latency is very high (~500ms), the congestion window only increases very slowly. Upon investigation, congestion window growth is heavily dependent on Round Trip Time. Increasing the congestion window gain parameter greatly increases performance in the benchmarks
  • max congestion window gain does not accurately respresent window gain in practice, there is factor 10 decrease in window gain https://github.com/PieterCarton/utp4j/blob/7bd348b8ca849d6c4ab2327fdd4a22ed23082bcf/src/main/java/net/utp4j/channels/impl/alg/UtpAlgorithm.java#L223

Benchmark results (performed on 4mB file):
(Default parameters coloured blue)
(Benchmark script: https://github.com/PieterCarton/UTPBenchmark/blob/186d5a8a10a0e6afb857330ed2d96cdbaac05663/benchmark.bash)

At 500ms delay, with 10ms jitter and 1% pkt loss

At 50ms delay, with 2ms jitter and 1% pkt loss

At 0ms delay with 1% pkt loss

[synctext]

• Dream goal of MBytes/sec transfers in a phone-2-phone network
• uTP library has no test code for packet timings and formulas like RTO (retransmission timeout) {thesis})
  
• untested code of this complexity will have bugs 🤔
• parameter setting like max congestion window gain seem to be optimised in simulations, not 4G and 5G networks.
• Create various example unit tests for private void updateWindow() function which contains lots of complexity ?
  • loss of 1 packet
  • packet arriving late
  • focus on 3 packets and 10 packets?
• uTP owns the UDP socket()! deep integration with IPv8 requires big stripping action
• Read about IPv8 peer discovery, see basics
  • active demo community walking every 5 seconds
  • use 3 colors please plus incoming and outgoing connections.
  • orange, just heard recently about a peer, no connection.
  • red, heard about a peer over 60 seconds ago, never connected.
  • green, got an incoming packet in last 60 seconds
  • yellow, got an incoming packet in more than 60 seconds ago
  • grey, got an incoming packet in more than 120 seconds ago
  • remove dead peers after 300 seconds
• expand your puncture, send, and receive into full network debugger
  • see design and running code from 8 years ago 😲
  • app-to-app communicator on Android #2131 (comment)
  • 
• Robustness: packet loss can happen to the initial send_binary(). Make sure to re-try the initial request for a uTP transmission, if nobody is answering you.
• awesome bonus, birthday attack details

[Eragoneq]

This week:

• work on NAT puncturing
  • direct open-port connection works
  • less restrictive NAT works
  • still has errors
• making the send/receive more seamless
• integrating data size and status messages into requests
• updating the UI

[PieterCarton]

Week 6 Report Benchmarking team:
Last week there were Prof. Pouwelse addressed some concerns about the complex piece of code that controls the congestion window

This week:

• We compared the UTP4J code to the UTP specification and LEDBAT specification, and it seemed to be up to specifications
• Unit tests were created to verify correctness of UTP4j congestion control, where we test the effect of latency caused by congestion
• Unit test were also made to test the packet time out behaviour
• Some benchmarks were rerun with larger file sizes, effectiveness of boosting congestion window gain is decreased
  Example:
  Benchmark of transfer of 50mB file at 50ms latency
  

[PieterCarton]

Current UI Design Proof of Concept:

[grimadas]

For the next week:

• The UI looks a bit rough. Need to keep working on it

• The receive button is irrelevant, needs to removed, or hidden. The server for receiving is running all times

• Separate port for UTP is used but there are still issues. Own public address/port not know, and there is reliable way to know if other peer is running UTP. One simplest option might be use third party to help bootstrap and discover UTP port. Any ideas, @qstokkink, what is the simplest and most elegant way to integrate UTP and IPv8?

• Focus on the demo with an app for the next week.

• The colours show the Pv8 status to see if peer is online.

• Think how to combine two status of the peer: UTP and IPv8

• Start working on the documentation.

[qstokkink]

@qstokkink, what is the simplest and most elegant way to integrate UTP and IPv8?

These are two opposites. The simple way is not elegant and the elegant way is not simple.

First, the difficult/elegant way would be to have channel separation for control logic and data streaming. You would have to puncture the NAT in a separate community with a separate socket and "hand off" this socket to your uTP streaming protocol. In a fantasy world, this would be on a separate physical channel as well (but this is obviously not realistic for consumer connections). You would need to use some form of DHT to couple the peer identity in the streaming community to the identity in your control logic community.

Second, the simple way: create a new IPv8 message to wrap your uTP packets. You will not get optimal performance, you will hinder other communities, and you will not have nice logical separation.. but it works 🤷 If your goal is to "just get something running", pick this option.

[Eragoneq]

@qstokkink

I think that still the "difficult" solution would be actually easier if we're supposed to reuse the library code. As right now all the socket stuff, threads and listeners are quite coupled with the logic of uTP making it hard to just reuse the existing IPv8 socket. I've already attempted it and made quite a bit of progress, but still there would need to be quite a lot of refactoring required in the library to actually get it working well, which then makes it even less sense to use the library and probably just better to rewrite it from scratch with the separation (due to other known issues).

For now I was able to get the separate socket and everything working on local networks only reusing the IPv8 to exchange status messages about file transfer request and other metadata, then the separate socket is used to actually send binary data. I'm now considering what would be the way to actually properly get the NAT puncture to work. Do you have any suggestion on what to use for it to work in a simple way? I don't think it's possible to somehow open another port with the existing p2p connection, we'd still have to use 3rd party to establish the external ports and that's what I'm currently considering.

[PieterCarton]

Weekly update:

• added logging of certain utp connection data in UTP debug fragment inside app
• started documentation of UTP benchmarking repository (link)

[synctext]

• {repeating} Dream goal of MBytes/sec transfers in a phone-2-phone network
• {repeating} please focus on the 500 ms case, which is bad 4G/5G networking. But still realistic.
  • 1 person 1 day
  • debugging, ensure it works, flawless as possible binary transfer
  • test setup: 3 hardware Android phone, 4 emulators, and the "Week 10" 2 test phones.
• After 8 weeks the effective magic setting for the congestion window increase is not in the code
• Please eliminate the server!!!
  • hard-coded IPv4 address
  • helpful to get something working
  • internal machine, will never scale
  • testing only
• IPv8 details: https://py-ipv8.readthedocs.io/en/latest/reference/peer_discovery.html
  • you talk to somebody
  • they already talk to somebody else
  • that somebody else also send you a pucture message 😲
  • this stranger is ready to receive your next "walk" in the network
  • 1 introduction-request message trigger 3 packet transfers (DoS 👎).
  • 
• Disagrees with @qstokkink unfortunately.
  • the design requirement is not efficiency, but fault tolerance. Hence use a single socket.
  • puncturing is a stochastic process that not always works. Thus avoid at all cost
  • we do not want "double puncture" logic in the networking stack. Puncture for signaling port and streaming port.
  • requires the 3 packets punctures for the stream socket 🤔
  • use IPv8 UDP transport port to transfer the uTP stream. 20 Bytes of overhead on 1500 Bytes is OK.
• Single repo for uTP. Include unit test, include benchmark

[CallumH2410]

UTP_demo.webm

Here is the demo of how it looks at the moment, as soon as we receive a package from one of these peers, the status indicator should turn to green.
Red: status idle
Orange: sent
Green: received.

Post-meeting notes:
These will be changed to the following:
Sent without ack: yellow
Sent with ack: Green
More than a minute without communication: yellow
More than 3 minutes without communication: red

[Eragoneq]

This week progress:

• Managed to use IPv8 socket to transfer data
  • Using the low-level prefix in UDPEndpoint layer
  • Overriden socket to use new channel to be fillled with data
  • Connection works in every case that peer is visible in the community
• Created separate UtpCommunity to show clients capabilities
• For now receiver always accepts any incoming data
• Deleted old connection code
• Still working on custom payloads for heartbeat and transfer request

[grimadas]

Great progress. The apk is working with speed up to 1 MB/s

For next week:

• Make sure you have running demo
• Finish the documentation, compile the benchmarks into one place and write the Gotchas into one file (big file transfer ..), what decisions you made
• Apk ready to be installed and repo with the code.

[Eragoneq]

Project content

Download APK here.

For documented details regarding the app usage, click here.

For documented details regarding benchmarking done over the course time, click here.

All relevant work for the project on respective repositories is linked below:

uTP4j

For documented details on relevant changes and benchmarking, click here.

trustchain-superapp

kotlin-ipv8

For more documented details about the uTP integration, click here

Overall project overview

Library

• Contacted the original author and got a hold of the repository
• Found and addressed the basic issues of dependencies, formatting and testing
• Updated the version and migrated to use gradle
• TODO: Update license and readme

Benchmarking

• Created a lot more test files to test parameters
• Analyzed the algorithm and its parameters
• Created a small framework for further testing in simulated network environment
• Wrote additional unit tests for real life cases

App

• Created a debug screen for uTP connections
• Used dynamically updating peer status
• Created a debug log screen

IPv8

• Created proof of concept for opening another port with NAT puncturing
• Extensive discussion about the IPv8 socket vs own socket
• Finished with the extended socket multiplexing approach
• Using separate UtpCommunity to inform users about the capabilities