Initial Conway Support and Plutus Machine Optimizations

Objectives

• Provide initial support for the Turbo sync to work in the Conway era.
• Optimizations of the C++ Plutus Machine to enable full transaction witness validation.

Next steps

• Prepare benchmarks of the C++ Plutus Machine and full transaction witness validation.
• Refine support for Conway governance actions.
• Re-enable the binary state export for the latest version of the Cardano Node.

Latest builds of DT Explorer

DT Explorer is a demo application that allows non-technical users to test the Turbo sync and Turbo wallet-history-reconstruction algorithms.

To keep your data secure, DT Explorer is designed to work without private keys. For those who want an added layer of protection, the app can also be run in a virtual machine, with only a moderate impact on performance.

Status

Development build. Tested only in a minimal set of hardware and software environments.
Provided for the general public to persuade themselves of the benefits of the technology and collect feedback.

Compatibility

This release changes the binary format of the local chain state. Thus, a deinstallation of all previous versions including the respective chain data is necessary.

Windows Installer

• dt-explorer-0.8.0.exe can be downloaded from the Assets section below.
• SHA256 hash of the installer: 52184146e42260304dfdba755af03b6de770f26f5c40e956f797b3770cf1a971

Mac (Arm64) Disk Image

• dt-explorer-0.8.0.dmg can be downloaded from the Assets section below.
• SHA256 hash of the image: b72e3b05a6361567e08420b03abd79486499ca570cc57f5dce4dca553b062928

Installation instructions

For installation instructions consult "Test it yourself" section of this repository's README file.

C++ Plutus Machine

Objectives

This is a code-only release that adds a major technical component: a C++ implementation of the Plutus machine. The intended use case is the evaluation of on-chain script witnesses during chain synchronization. The source code is located in the lib/dt/plutus subdirectory of this repository and includes:

• an implementation of the Plutus machine;
• an implementation of the UPLC (Untyped Plutus) parser;
• an implementation of a decoder of the Flat format used for storing Plutus scripts on-chain;
• an implementation of Plutus builtins up to batch 4 (Plutus V3);
• an implementation of Plutus cost models and all necessary costing functions.

Quality

The quality of the code has been evaluated using a copy of the official Plutus conformance tests excluding tests of builtins that were not part of the Plutus core spec as of September, 2024. All 487 test cases have been successfully passed. The copied test cases are located in data/plutus/conformance subdirectory of this repository and the code of the test is located in lib/dt/plutus/machine.test.cpp.

Test it

Clone this repository and build and start a Docker container with the test environment:

git clone https://github.com/sierkov/daedalus-turbo.git dt
cd dt
docker build -t dt -f Dockerfile.test .
docker run -it --rm dt

Within the terminal window opened by the previous command, run the following command to evaluate an example Plutus script:

./dt plutus-eval ../data/plutus/conformance/example/factorial/factorial.uplc

../data/plutus/conformance/ directory comes with 487 example Plutus scripts that were used to test the implementation.

To test your own scripts, you can use the following command to start the test container; Replace <script-dir> with a directory with your scripts on the host machine:

docker run -it --rm -v <script-dir>:/script dt

To evaluate a Plutus script in the binary on-chain format, use the following command:
./dt plutus-eval --format=flat /script/my-script.bin

Next steps

• Test and optimize the Plutus machine on all scripts stored on the Cardano mainnet.
• Finish implementing the Chang fork features and use the Plutus machine to validate on-chain script witnesses.
• Prepare user-testable builds of DT Explorer for Win and Mac.

Hybrid Turbo+P2P Sync and Daedalus Export

Objectives

This release introduces two major features which are described below. Hardware requirements, testing instructions, and technical documentation are provided in this repository's README file.

Hybrid Turbo+P2P Sync

Hybrid Sync is a new synchronization procedure that is a combination of the synchronization using the Turbo protocol and the Cardano Network protocol executed in that sequence:

• The Turbo protocol offers a much faster delivery of blockchain data thanks to its data compression.
• The successive synchronization using the Cardano Network protocol allows to verify and update the local chain against the best chain known on the Cardano Network using the Ouroboros Genesis chain selection rule (highest local chain density).

This synchronization procedure provides both the performance benefits of the Turbo protocol and the robustness of verifying/updating the blockchain data using the standard Cardano Network protocol. For additional safety, the app will use the bootstrap peer provided by the IOG relays-new.cardano-mainnet.iohk.io. This feature is available as sync-hybrid CLI command and is used by the DT Explorer app.

Daedalus Export

Release 0.6 introduces support for exporting the blockchain data and the ledger state in the binary format used by Cardano Node 8.9.2 (used in the latest version of Daedalus 5.4.0). The exported data are binary compatible and allow for quick bootstrapping of new Daedalus instances. The export procedure generates the Cardano Node's data directory consisting of:

• Raw blockchain data files (both immutable and volatile).
• Primary and secondary indices for the immutable blockchain data files.
• A binary snapshot of the Cardano Node's state

This feature is available as node-export CLI command and is integrated into the DT Explorer app. To test it, one needs to specify the directory with the Cardano Node's database and to have at least 200 GB of free storage space. Normally, that directory is:

• On Windows: %APPDATA%\Daedalus Mainnet\chain
• On Mac OS: ~/Library/Application Support/Daedalus Mainnet/chain

Latest builds of DT Explorer

Status

Development builds. Tested only in a minimal set of hardware and software environments. It's recommended to test them with an installation of Daedalus dedicated for testing (not your main installation). For additional safety, the builds can be tested on a virtual machine. However, in that case, a degradation of the performance is possible (normally, stays within 30% of the bare metal's performance.

Compatibility

This release changes the binary format of the local chain state, so a full deinstallation of any previous software versions including the respective chain data is necessary.

Windows Installer

• dt-explorer-0.6.0.exe can be downloaded from the Assets section below.
• SHA256 hash of the installer: 1be389fc7211a6085b38539ab42048ee5f03e2b3388f0911c77bc6ab058d41e6

Mac (Arm64) Disk Image

• dt-explorer-0.6.0.dmg can be downloaded from the Assets section below.
• SHA256 hash of the image: 265f816aeddace96d0e516b1f2d182dcd25c22d2c3a7e6f43fa3631d530e67db

Installation instructions

For installation instructions consult "Test it yourself" section of this repository's README file.

Cardano Network Sync and Ouroboros Genesis

Objective

This release adds initial support for blockchain synchronization over the Cardano Network protocol and the Ouroboros Genesis chain selection rule. This synchronization method is aimed at scenarios when the quicker download times of the main method, thanks to the data compression, are not relevant, such as synchronizing the tail of the chain or using it as a fallback mechanism for extra robustness. This implementation performs the consensus data validation as explained in the On the Security of Wallet Nodes in the Cardano Blockchain note.

P2P synchronization over the Cardano network protocol can be launched using the new sync-p2p
command. It takes two arguments: a local chain directory and a peer address to synchronize with.
This command can be used interchangeably with the main sync-turbo, with each command synchronizing only differences. The option for P2P synchronization is available only in the command-line client at the moment.

Testing instructions are provided in this repository's README file.

Latest builds of DT Explorer

Status

Development build. Tested only in a minimal set of hardware and software environments. Testing and refinement are ongoing.

Compatibility

This release changes the binary format of the local chain state, so a full deinstallation of any previous software versions including the respective chain data is necessary.

Windows Installer

• dt-explorer-0.5.0.exe can be downloaded from the Assets section below.
• SHA256 hash of the installer: 15ec34fad245b95680366f82a136b70d6860e35cfc9d61309eba805d2ca41cc2

Mac (Arm64) Disk Image

• dt-explorer-0.5.0.dmg can be downloaded from the Assets section below.
• SHA256 hash of the image: 94cf928cc47f1e326f985957d082984ade8786d53c278eaaf5eab9c919b724cf

Installation instructions

For installation instructions consult "Test it yourself" section of this repository's README file.

DT Explorer (Win+Mac): Performance and Stability Refinements for Fast Internet Connections

Objective

The objective of the DT Explorer application is to show the new synchronization and history-reconstruction algorithms in a safe and easy-to-test way. The application does not need private keys and can reconstruct the history of any stake or payment address to prove its performance claims. Once the algorithms are proven, milestone 4 of the project will begin, integrating them into Daedalus builds. M4 will include additional testing and code review of the new components to ensure their security and robustness. At the same time, the goal of the current release and milestone 3 is to prove the performance claims by doing all operations that are non-trivial performance-wise so that the feasibility of providing both performance and security for Daedalus users is beyond reasonable doubt.

The technical details are explained in:

1. On the Security of Wallet Nodes in the Cardano Blockchain explains the security approach.
2. Parallelized Ouroboros Praos explains the approach to parallelized data validation.
3. Highly Parallel Reconstruction of Wallet History in the Cardano Blockchain explains the approach to wallet history reconstruction.
4. Scalability of Bulk Synchronization in the Cardano Blockchain explains the network infrastructure requirements, the need for compressing proxies, and how the approach can be scaled to support up to a billion Cardano wallet nodes.

Status

Development build. Tested only in a minimal set of hardware and software environments. Testing and refinement are ongoing.

Supporting network infrastructure

The high-speed delivery of blockchain data depends on a network of compressing proxies (Cardano nodes that can share their local chain with other nodes in a compressed format and which are needed until the support for compressed data transfers becomes part of the regular Cardano network protocol). Even though the current configuration is minimal and consists of three servers, it is sufficient to support up to a thousand high-speed (with a download speed of 200+ Mbps) full blockchain synchronizations per week.

Requirements

• a modern (2022+) CPU with 8 or more physical cores
• 16 GB of RAM
• a modern NVME SSD drive with 80+ GB of available space
• a fast Internet connection (normally will be the bottleneck): 250+ Mbps is recommended to reach the full sync times of one hour or less
• Mac OS Sonoma for Mac builds (earlier versions may work but have yet to be be tested).
• Windows 11 for Windows builds (earlier versions may work but have yet to be be tested).

The app will check the hardware and refuse to work if it’s weaker than that of an Orange Pi 5 Plus.
Some examples of the synchronization performance (1 Gbps Internet; full Cardano mainnet up to epoch 475):

• Intel 13900HX Laptop (24 cores, 64 GB RAM): ~10 minutes.
• Macbook Air M2 Laptop (8 cores, 16 GB RAM): ~30 minutes.
• Orange Pi 5 Plus SBC (8 cores, 16 GB RAM): ~70 minutes.

How to install on Mac OS (Arm64)

• Download the Mac image from the Assets section under the post.
• Open the image. This is a development (unsigned) image, so Mac OS will ask you if you trust the developer: See Apple's explanation and instructions.
• Copy dt-ui app to your Applications folder.
• Both program and blockchain data will be stored in that folder, so when deleted all used space will be recovered.
• Launch the app from the Applications folder. If Mac OS says that the app is damaged, follow these instructions from Stackoverflow.

How to install on Windows 11

• Download and launch the installer from the Assets section under the post.
• Choose locations for the program files and blockchain data (each will be asked individually). For optimal performance, it's important to select your fastest SSD drive if you have multiple storage devices.

How to use (any OS)

• Synchronization (full or partial) always happens at the app's launch; to catch up, simply restart the app. If you restart before the synchronization is finished, the app will reuse already downloaded data but may reprocess and revalidate some of them.
• History reconstruction happens through a simple search for a transaction, stake, or payment address, either entered explicitly or when clicked as part of blockchain exploration. The easiest starting point for most users would be to search for their own stake address (e.g.: stake1uxw70wgydj63u4faymujuunnu9w2976pfeh89lnqcw03pksulgcrg), as its history will be the most representative of their wallet's history.
• When searching for Cardano addresses starting with "addr1" prefix, the app may ask you if want to explore the payment or stake history. The reason for that is that many cardano addresses contain two keys. If in doubt, select stake history as it will normally discover more transactions. This is necessary so that the app can work without private keys. However, when integrated into Daedalus, the same aglorithms can reconstruct the full wallet history by finding all payment and stake keys generated from a wallet private key.
• Once synchronized, users can turn off their Internet connection and test history reconstruction with new transaction or stake addresses to prove that the app reconstructs all histories interactively and uses only the downloaded blockchain data.

Share your results

Every time the sync takes longer than 1 minute, the app will show an additional screen with an overview of the synchronization performance. Users are invited to post screenshots of the results they especially like (or not) on Twitter, mentioning @TurboDaedalus.

Reporting bugs

Please create a GitHub issue in this project to report an issue.

Download the Mac (Arm64) Build

• The Mac application image, dt-explorer-0.4.16.dmg, can be downloaded from the Assets section under this post.
• SHA256 hash of the image: dac903d7b331e2f8c44b7e59e65ba977ea5e31fdcfc98d55be94ea7902ce6a5b
• For your convenience, it has been pre-scanned for viruses by VirusTotal: the confirmation link

Download the Windows Build

• The Windows installer, dt-explorer-0.4.16.exe, can be downloaded from the Assets section under this post.
• SHA256 hash of the installer: f6537280df89bf9ce0c35e33ff3cbf2e5831a45f2dd3afee1430c8a156dca65d
• For your convenience, it has been pre-scanned for viruses by VirusTotal: the confirmation link

DT Explorer For Mac (Arm64) and Windows (x64): Preview Builds

Objective

The objective of the DT Explorer application is to show the new synchronization and history-reconstruction algorithms in a safe and easy-to-test way. The application does not need private keys and can reconstruct the history of any stake or payment address to prove its performance claims. Once the algorithms are proven, milestone 4 of the project will begin, integrating them into Daedalus builds. M4 will include additional testing and code review of the new components to ensure their security and robustness. At the same time, the goal of the current release and milestone 3 is to prove the performance claims by doing all operations that are non-trivial performance-wise so that the feasibility of providing both performance and security for Daedalus users is beyond reasonable doubt.

The technical details are explained in:

1. On the Security of Wallet Nodes in the Cardano Blockchain explains the security approach.
2. Parallelized Ouroboros Praos explains the approach to parallelized data validation.
3. Highly Parallel Reconstruction of Wallet History in the Cardano Blockchain explains the approach to wallet history reconstruction.
4. Scalability of Bulk Synchronization in the Cardano Blockchain explains the network infrastructure requirements, the need for compressing proxies, and how the approach can be scaled to support up to a billion Cardano wallet nodes.

Status

Development build. This is the first public build for Mac, tested only in a minimal set of hardware and software environments. Testing and refinement are ongoing, and updated builds are upcoming.

Supporting network infrastructure

The high-speed delivery of blockchain data depends on a network of compressing proxies (Cardano nodes that can share their local chain with other nodes in a compressed format and which are needed until the support for compressed data transfers becomes part of the regular Cardano network protocol). Even though the current configuration is minimal and consists of three servers, it is sufficient to support up to a thousand high-speed (with a download speed of 200+ Mbps) full blockchain synchronizations per week.

Requirements

• a modern (2022+) CPU with 8 or more physical cores
• 16 GB of RAM
• a modern NVME SSD drive with 80+ GB of available space
• a fast Internet connection (normally will be the bottleneck): 250+ Mbps is recommended to reach the full sync times of one hour or less
• Mac OS Sonoma for Mac builds (earlier versions may work but have yet to be be tested).
• Windows 11 for Windows builds (earlier versions may work but have yet to be be tested).

The app will check the hardware and refuse to work if it’s weaker than that of an Orange Pi 5 Plus.
Some examples of the synchronization performance (1 Gbps Internet; full Cardano mainnet up to epoch 475):

• Intel 13900HX Laptop (24 cores, 64 GB RAM): ~10 minutes.
• Macbook Air M2 Laptop (8 cores, 16 GB RAM): ~30 minutes.
• Orange Pi 5 Plus SBC (8 cores, 16 GB RAM): ~70 minutes.

How to install (Mac)

• Download the Mac image from the Assets section under the post.
• Open the image. This is a development (unsigned) image, so Mac OS will ask you if you trust the developer: See Apple's explanation and instructions.
• Copy dt-ui app to your Applications folder.
• Both program and blockchain data will be stored in that folder, so when deleted all used space will be recovered.
• Launch the app from the Applications folder. If Mac OS says that the app is damaged, follow these instructions from Stackoverflow.

How to install (Windows)

• Download and launch the installer from the Assets section under the post.
• Choose locations for the program files and blockchain data (each will be asked individually). For optimal performance, it's important to select your fastest SSD drive if you have multiple storage devices.

How to use (any OS)

• Synchronization (full or partial) always happens at the app's launch; to catch up, simply restart the app. If you restart before the synchronization is finished, the app will reuse already downloaded data but may reprocess and revalidate some of them.
• History reconstruction happens through a simple search for a transaction, stake, or payment address, either entered explicitly or when clicked as part of blockchain exploration. The easiest starting point for most users would be to search for their own stake address (e.g.: stake1uxw70wgydj63u4faymujuunnu9w2976pfeh89lnqcw03pksulgcrg), as its history will be the most representative of their wallet's history.
• When searching for Cardano addresses starting with "addr1" prefix, the app may ask you if want to explore the payment or stake history. The reason for that is that many cardano addresses contain two keys. If in doubt, select stake history as it will normally discover more transactions. This is necessary so that the app can work without private keys. However, when integrated into Daedalus, the same aglorithms can reconstruct the full wallet history by finding all payment and stake keys generated from a wallet private key.
• Once synchronized, users can turn off their Internet connection and test history reconstruction with new transaction or stake addresses to prove that the app reconstructs all histories interactively and uses only the downloaded blockchain data.

Share your results

Every time the sync takes longer than 1 minute, the app will show an additional screen with an overview of the synchronization performance. Users are invited to post screenshots of the results they especially like (or not) on Twitter, mentioning @TurboDaedalus.

The image above shows the best-case time with pre-downloaded data on a laptop with a 24-core Intel 13900HX CPU.

Reporting bugs

Please create a GitHub issue in this project to report an issue.

Download the Mac (Arm64) Build

• The Mac application image, dt-explorer-0.4.15.dmg, can be downloaded from the Assets section under this post.
• SHA256 hash of the image: e9d5adcb26f1a5e1048f963fdd52aec8538ca1be8b664d9d4440b819a4387281
• For your convenience, it has been pre-scanned for viruses by VirusTotal: the confirmation link

Download the Windows Build

• The Windows installer, dt-explorer-0.4.15.exe, can be downloaded from the Assets section under this post.
• SHA256 hash of the installer: 96ad1b89c12c2f0baea24f7f7699d483f2b74e643d2ad3f8306c7a28fdb71b58
• For your convenience, it has been pre-scanned for viruses by VirusTotal: the confirmation link

DT Explorer For Windows: A Preview Build

Objective

The objective of the DT Explorer application is to show the new synchronization and history-reconstruction algorithms in a safe and easy-to-test way. The application does not need private keys and can reconstruct the history of any stake or payment address to prove its performance claims. Once the algorithms are proven, milestone 4 of the project will begin, integrating them into Daedalus builds. M4 will include additional testing and code review of the new components to ensure their security and robustness. At the same time, the goal of the current release and milestone 3 is to prove the performance claims by doing all operations that are non-trivial performance-wise so that the feasibility of providing both performance and security for Daedalus users is beyond reasonable doubt.

The technical details are explained in:

1. On the Security of Wallet Nodes in the Cardano Blockchain explains the security approach.
2. Parallelized Ouroboros Praos explains the approach to parallelized data validation.
3. Highly Parallel Reconstruction of Wallet History in the Cardano Blockchain explains the approach to wallet history reconstruction.
4. Scalability of Bulk Synchronization in the Cardano Blockchain explains the network infrastructure requirements, the need for compressing proxies, and how the approach can be scaled to support up to a billion Cardano wallet nodes.

Status

Development build. This is the first public build for Windows, tested only in a minimal set of hardware and software environments. Testing and refinement are ongoing, and updated builds are upcoming.

Supporting network infrastructure

The high-speed delivery of blockchain data depends on a network of compressing proxies (Cardano nodes that can share their local chain with other nodes in a compressed format and which are needed until the support for compressed data transfers becomes part of the regular Cardano network protocol). Even though the current configuration is minimal and consists of three servers, it is sufficient to support up to a thousand high-speed (with a download speed of 200+ Mbps) full blockchain synchronizations per week.

Requirements

• a modern (2022+) CPU with 8 or more physical cores
• 16 GB of RAM
• a modern NVME SSD drive with 80+ GB of available space
• a fast Internet connection (normally will be the bottleneck): 250+ Mbps is recommended to reach the full sync times of one hour or less
• Windows 11 (earlier Windows versions may work but have yet to be be tested).

The app will check the hardware and refuse to work if it’s weaker than that of an Orange Pi 5 Plus.
Some examples of the synchronization performance (1 Gbps Internet; full Cardano mainnet up to epoch 475):

• Intel 13900HX Laptop (24 cores, 64 GB RAM): ~10 minutes.
• Macbook Air M2 Laptop (8 cores, 16 GB RAM): ~30 minutes. (a Mac build will be released later).
• Orange Pi 5 Plus SBC (8 cores, 16 GB RAM): ~70 minutes.

How to use

• Download and launch the installer.
• Choose locations for the program files and blockchain data (each will be asked individually). For optimal performance, it's important to select your fastest SSD drive if you have multiple storage devices.
• Synchronization (full or partial) always happens at the app's launch; to catch up, simply restart the app. If you restart before the synchronization is finished, the app will reuse already downloaded data but may reprocess and revalidate some of them.
• History reconstruction happens through a simple search for a transaction, stake, or payment address, either entered explicitly or when clicked as part of blockchain exploration. The easiest starting point for most users would be to search for their own stake address (e.g.: stake1uxw70wgydj63u4faymujuunnu9w2976pfeh89lnqcw03pksulgcrg), as its history will be the most representative of their wallet's history.
• When searching for Cardano addresses starting with "addr1" prefix, the app may ask you if want to explore the payment or stake history. The reason for that is that many cardano addresses contain two keys. If in doubt, select stake history as it will normally discover more transactions. This is necessary so that the app can work without private keys. However, when integrated into Daedalus, the same aglorithms can reconstruct the full wallet history by finding all payment and stake keys generated from a wallet private key.
• Once synchronized, users can turn off their Internet connection and test history reconstruction with new transaction or stake addresses to prove that the app reconstructs all histories interactively and uses only the downloaded blockchain data.

Share your results

Every time the sync takes longer than 1 minute, the app will show an additional screen with an overview of the synchronization performance. Users are invited to post screenshots of the results they especially like (or not) on Twitter, mentioning @TurboDaedalus.

The image above shows the best-case time with pre-downloaded data on a laptop with a 24-core Intel 13900HX CPU.

Reporting bugs

Please create a GitHub issue in this project to report an issue.

Download

• The Windows installer, dt-explorer-0.4.14.exe, can be downloaded from the Assets section under this post.
• SHA256 hash of the installer: 88b4cb569fbb3e87b8060a4640a01ed1991dbff018845566a7f5b588b0c7caa9
• For your convenience, it has been pre-scanned for viruses by VirusTotal: the confirmation link

Parallelized Ouroboros Praos

• Parallelized Ouroboros Praos paper presenting a new data-validation method for Cardano.
• Source code implementing Parallelized Ouroboros Praos. Consult README.md for testing instructions.
• Source code of the experiments supporting the paper. Consult experiment/README.md for more details.

Desktop Blockchain Explorer User Interface

• The initial version of the desktop blockchain explorer user interface. The repository’s README provides instructions on how to test it locally.
• An HTTP API server that provides access to the library’s features to the user interface and other clients.
• Build scripts use CMake to make multi-platform builds easier.
• Search indices are merged on the go during downloads to reduce the end-to-end synchronization time.

Incremental Synchronization over the Internet

• Support for incremental synchronization of compressed blockchain data over the network.
• Two-step merge of large index updates to minimize chances of big index truncate operations.
• An asynchronous http downloader maximizing network bandwidth utilization during synchronization.
• Leverage the new sync-http command in the Test it yourself section of the README.
• Introduce DT_WORKERS environment variable to override the number of worker threads.
• Introduce DT_LOG environment variable to override the location of the log file.
• Make sync-http and sync-local commands warn the user about non-fulfilled hardware requirements.
• Automatic clean-up of unmerged index chunks on sync start
• Make tx-info command show human-readable output instead of JSON.
• Various optimizations and bug fixes.