Development (#20)

* Documentation (#7)

* documentation v1

* documentation v2

Added a paragraph about the longest chain rule.
Added the image about the Scalachain node and the actor hierarchy.
Small grammar corrections

* Cluster (#19)

* Initial cluster integration

- added cluster configuration
- added cluster manager and listener
- added /status/members endpoint
- added docker-compose file
- improved application configuration

* Transaction broadcast first implementation

* Transaction broadcast refactoring

- transaction broadcast has been refactored.

* Docker improvement

Improved docker image creation. Sbt is downloaded during image creation
to reduce start time of containers.
Source code can be mounted in a folderinside the container and then
copied in the /development one. This in order to prevent problems due to
concurrent r/w in the same volume by multiple containers.

* Fixed tests

Tests are fixed to support the new cluster architecture

* New block broadcast

The broadcast of new blocks has been implemented.
Fixed the persistence of snapshots in containers.

* Updated documentation for the new cluster feature

* Improved style

.gitignore

@@ -3,4 +3,5 @@
 .idea
 target/
 persistence/journal/*
-persistence/snapshots/*
+persistence/snapshots/*
+.DS_Store

docker/Dockerfile → Dockerfile

@@ -4,7 +4,7 @@ RUN apk add --no-cache git openssh && \
     mkdir /development
 
 # GIT
-ADD repo-key /
+COPY docker/repo-key /
 RUN chmod 600 /repo-key && \  
     echo "IdentityFile /repo-key" >> /etc/ssh/ssh_config && \  
     echo -e "StrictHostKeyChecking no" >> /etc/ssh/ssh_config && \
@@ -19,7 +19,10 @@ RUN apk add --no-cache bash && \
     chmod 0755 /usr/local/bin/sbt && \
     apk del build-dependencies
 
-ADD init.sh /etc/init.sh
+RUN cd /development && \
+    sbt sbtVersion
+
+COPY docker/init.sh /etc/init.sh
 
 RUN chmod +x /etc/init.sh
 

README.md

@@ -2,20 +2,22 @@
 
 # Scalachain
 Scalachain is a blockchain implemented using the Scala programming language and the actor model (Akka Framework). The main purpose of the project is:
-
-* Understand how a simple blockchain works 
-* Learn the Scala programming language and Akka
-* Share what I learned
-
+*   Understand how a simple blockchain works 
+*   Learn the Scala programming language and Akka
+*   Share what I learned
 For this reason, any contribution is welcome! :-)
 
-I published a story on freeCodeCamp where I explain in depth the development of Scalachain. Read it [here](https://medium.freecodecamp.org/how-to-build-a-simple-actor-based-blockchain-aac1e996c177)! :-)
+I published a story on freeCodeCamp where I explain in depth the development of Scalachain [v1.0](https://github.com/elleFlorio/scalachain/releases/tag/v.1.0). Read it [here](https://medium.freecodecamp.org/how-to-build-a-simple-actor-based-blockchain-aac1e996c177)! :-)
 
-### TL;DR
-Run the Scalachain node using ```sbt``` (```sbt run```) or the Docker container (see ```/docker``` folder).
-The API to interact with Scalachain are documented [here](https://documenter.getpostman.com/view/4636741/RWaHw8yx).
+## TL;DR
+Run Scalachain following these steps:
+1.  build the Docker image as described in the ```/docker``` folder
+2.  Run the cluster of 3 nodes using the ```docker-compose up``` command in the root folder of the project. 
+3.  Use The API documented [here](https://documenter.getpostman.com/view/4636741/RWaHw8yx) to interact with Scalachain nodes.
 
-### Quick introduction to the blockchain
+Enjoy! :grin:
+
+## Quick introduction to the blockchain
 There a lot of good articles that explain how a blockchain works, so I will do a high level introduction just to provide some context to this project.
 
 The blockchain is a distributed ledger: it registers some transaction of values (e.g., coins) between a sender and a receiver. What makes a blockchain different from a traditional database is the decentralized nature of the blockchain: it is distributed among several communicating nodes that guarantee the validity of the transactions registered.
@@ -27,45 +29,53 @@ The solution of the cryptographic puzzle depends on the values stored in the las
 
 It may happen that different nodes mine a block at the same time, creating different "branches" from the same blockchain - this is called a fork in the blockchain. when this happens, sets of nodes will continue mining starting from the different blocks of the new branches, that will grow in different ways. This situation is solved automatically when a branch becomes longer than the others: the longest chain always wins, so the winning branch becomes the new blockchain.
 
-### The Blockchain Model
+## The Blockchain Model
 Scalachain is based on a blockchain model that is a simplification of the Bitcoin one.
 The main components of our blockchain model are the **transaction**, the **chain**, the **proof of work** (PoW) algorithm, and the **Node**.
 The transactions are stored inside the blocks of the chain, that are mined using the PoW. The node is the server that runs the blockchain.
 
-##### Transaction
+### Transaction
 Transactions register the movement of coins between two entities. Every transaction is composed by a sender, a recipient and an amount of coin. Transactions will be registered inside the blocks of our blockchain.
 
-##### Chain
+### Chain
 The chain is a linked list of blocks containing a list of transactions. Every block of the chain has an index, the proof that validates it (more on this later), the list of transactions, the hash of the previous block, the list of previous blocks, and a timestamp. Every block is chained to the previous one by its hash, that is computed converting the block to a JSON string and then hashing it through a ```SHA-256``` hashing function.
 
-##### PoW
+### PoW
 The PoW algorithm is required to mine the blocks composing the blockchain. The idea is to solve a cryptographic puzzle that is hard to solve, but easy to verify having a proof. The PoW algorithm that is implemented in Scalachain is similar to the Bitcoin one (based on [Hashcash](https://en.wikipedia.org/wiki/Hashcash)), and consists in finding a hash with N leading zeros that is computed starting from the hash of the previous block and a number, that represents the proof of our algorithm. 
 We can formalize it as:
 
 ``` NzerosHash = SHA-256(previousNodeHash + proof) ```
 
 The higher is N, the harder is to find the proof. In Scalachain N=4 (It will be configurable eventually).
 
-##### Node
+### Node
 The Node is the server running our blockchain. It provides some REST API to interact with it and perform basic operations such as send a new transaction, get the list of pending transactions, mine a block, and get the current status of the blockchain.
 
-### The Actor Model
+## The Actor Model
 Scalachain has been implemented using the actor model through the use of the [Akka Framework](https://akka.io/).
 The hierarchy of actors is depicted in the following diagram of the Scalachain node.
 
 ![Scalachain node](img/scalachain_node.png)
 
-* **Broker Actor** is responsible of managing the transactions. It can add a new transaction, read the pending transactions (the ones that are not yet included in a block), and clear the list of pending transactions.
-
-* **Miner Actor** is responsible of the mining process. It validates the proof of added nodes and executes the PoW algorithm to mine a new block. It is implemented as a state machine composed of two states (ready and busy) in order to handle mining concurrency - If the node is mining a block, it cannot start to mine a new one.
-
-* **Blockchain Actor** is responsible of managing the chain. It manages the addition of new blocks, and provides information about the chain: the last index, the last hash, and the status of the chain itself. This is a *Persistent Actor*, so it keeps a journal of events and saves a snapshot of the state of the chain when a new block is added. If the node goes down, when it is up again the Blockchain Actor can restore the status of the chain loading the last snapshot. The journal of events and the snapshots are stored inside the ```persistence``` folder.
-
-* **Node Actor** is on the top of the hierarchy of actors: it manages all the operations executed by the other actors, and it is the junction point between the API and the business logic implemented inside the actors.
-
-### Scalachain API
-Scalachain node runs a server at default address ```http://localhost:8080/``` using Akka HTTP.
+*   **Broker Actor** is responsible of managing the transactions. It can add a new transaction, read the pending transactions (the ones that are not yet included in a block), and clear the list of pending transactions.
+*   **Miner Actor** is responsible of the mining process. It validates the proof of added nodes and executes the PoW algorithm to mine a new block. It is implemented as a state machine composed of two states (ready and busy) in order to handle mining concurrency - If the node is mining a block, it cannot start to mine a new one.
+*   **Blockchain Actor** is responsible of managing the chain. It manages the addition of new blocks, and provides information about the chain: the last index, the last hash, and the status of the chain itself. This is a *Persistent Actor*, so it keeps a journal of events and saves a snapshot of the state of the chain when a new block is added. If the node goes down, when it is up again the Blockchain Actor can restore the status of the chain loading the last snapshot. The journal of events and the snapshots are stored inside the ```persistence``` folder.
+*   **Node Actor** is on the top of the hierarchy of actors: it manages all the operations executed by the other actors, and it is the junction point between the API and the business logic implemented inside the actors.
+*   **ClusterManager Actor** is the manager of the cluster. Its in charge of everything cluster related, such as return the current active members of the cluster.
+*   **ClusterListener Actor** listens to the events that happen in the cluster - a member is up/down, etc. - and logs it for debugging purpose.
+
+## Scalachain cluster
+Since scalachain v2.0 is possible to run a cluster of Scalachain nodes. The easiest way to create a simple cluster is to run the docker-compose.yml file with the command ```docker-compose up```. This will spawn a 3 nodes cluster with 1 seed and 2 nodes. You can send a transaction to one node that will be propagated to the other ones in the cluster. Once a node mines a block, it will send it to the other nodes of the cluster that will add it to their local blockchain.
+
+## Scalachain API
+The cluster is composed of 3 nodes, mapped to the following addresses:
+*   ```http://localhost:8000``` - ```seed``` 
+*   ```http://localhost:8001``` - ```node1``` 
+*   ```http://localhost:8002``` - ```node2```
+
 The API to interact with the node is documented [here](https://documenter.getpostman.com/view/4636741/RWaHw8yx)
 
-### How to run Scalachain
-Scalachain node can be run using ```sbt``` (```sbt run```) or a Docker container. The instructions on how to run it using Docker can be found inside the ```/docker``` folder.
+## How to run Scalachain
+Run Scalachain following these steps:
+1.  build the Docker image as described in the ```/docker``` folder
+2.  Run the cluster of 3 nodes using the ```docker-compose up``` command in the root folder of the project.

build.sbt

@@ -6,23 +6,28 @@ scalaVersion := "2.12.4"
 
 resolvers ++= Seq(
   "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/",
+  "dnvriend" at "http://dl.bintray.com/dnvriend/maven",
   Resolver.jcenterRepo
 )
 
-lazy val akkaVersion = "2.5.3"
-lazy val akkaHttpVersion = "10.1.4"
+lazy val akkaVersion = "2.5.21"
+lazy val akkaHttpVersion = "10.1.7"
+lazy val akkaPersistenceInmemoryVersion = "2.5.15.1"
+lazy val scalaTestVersion = "3.0.5"
 
 libraryDependencies ++= Seq(
-  "com.typesafe.akka" %% "akka-testkit" % "2.5.14" % Test,
-  "org.scalatest" %% "scalatest" % "3.0.4" % "test",
+  "com.typesafe.akka" %% "akka-testkit" % akkaVersion % Test,
+  "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "com.typesafe.akka" %% "akka-persistence" % akkaVersion,
   "org.iq80.leveldb" % "leveldb" % "0.10",
   "org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8",
   "com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion,
   "com.typesafe.akka" %% "akka-actor" % akkaVersion,
   "com.typesafe.akka" %% "akka-http" % akkaHttpVersion,
   "com.typesafe.akka" %% "akka-stream" % akkaVersion,
-  "com.github.dnvriend" %% "akka-persistence-inmemory" % "2.5.1.1"
+  "com.typesafe.akka" %% "akka-cluster" % akkaVersion,
+  "com.typesafe.akka" %% "akka-cluster-tools" % akkaVersion,
+  "com.github.dnvriend" %% "akka-persistence-inmemory" % akkaPersistenceInmemoryVersion
 )
 
-
+fork in Test := true

docker-compose.yml

@@ -0,0 +1,60 @@
+version: '3.7'
+
+networks:
+  scalachain-network:
+
+services:
+  seed:
+    networks:
+      - scalachain-network
+    image: elleflorio/scalachain
+    volumes:
+      - ~/Development/scala/scalachain/:/tmp/scalachain
+    ports:
+      - '2552:2552'
+      - '8000:8000'
+    environment:
+      SERVER_IP: 0.0.0.0
+      CLUSTER_IP: seed
+      CLUSTER_SEED_IP: seed
+      SCALACHAIN_PERSISTENCE_DIR: '/persistence/seed/journal'
+      SCALACHAIN_SNAPSHOTS_DIR: '/persistence/seed/snapshots'
+    command: ["local", "/tmp/scalachain"]
+
+  node1:
+    networks:
+      - scalachain-network
+    image: elleflorio/scalachain
+    volumes:
+      - ~/Development/scala/scalachain/:/tmp/scalachain
+    ports:
+      - '8001:8000'
+    environment:
+      SERVER_IP: 0.0.0.0
+      CLUSTER_IP: node1
+      CLUSTER_PORT: 1600
+      CLUSTER_SEED_IP: seed
+      CLUSTER_SEED_PORT: 2552
+      SCALACHAIN_NODE_ID: node1
+      SCALACHAIN_PERSISTENCE_DIR: '/persistence/node1/journal'
+      SCALACHAIN_SNAPSHOTS_DIR: '/persistence/node1/snapshots'
+    command: ["local", "/tmp/scalachain"]
+
+  node2:
+    networks:
+      - scalachain-network
+    image: elleflorio/scalachain
+    volumes:
+      - ~/Development/scala/scalachain/:/tmp/scalachain
+    ports:
+      - '8002:8000'
+    environment:
+      SERVER_IP: 0.0.0.0
+      CLUSTER_IP: node2
+      CLUSTER_PORT: 1600
+      CLUSTER_SEED_IP: seed
+      CLUSTER_SEED_PORT: 2552
+      SCALACHAIN_NODE_ID: node2
+      SCALACHAIN_PERSISTENCE_DIR: '/persistence/node2/journal'
+      SCALACHAIN_SNAPSHOTS_DIR: '/persistence/node2/snapshots'
+    command: ["local", "/tmp/scalachain"]

docker/README.md

@@ -1,15 +1,15 @@
 # Scalachain Docker image
 The Dockerfile lets you build the Docker Image of a scalachain node. In details, the Dockerfile:
-* copy the ssh private key to pull from the repo - **N.B. the key is only for read operations**
-* download the [sbt](https://www.scala-sbt.org/) version specified through the image ```build-arg``` parameter
-* install git and openssh
-* checkout the project
-* copy the script file init.sh
-* ```EXPOSE``` port 8080 
+*   copy the ssh private key to pull from the repo - **N.B. the key is only for read operations**
+*   download the [sbt](https://www.scala-sbt.org/) version specified through the image ```build-arg``` parameter
+*   install git and openssh
+*   checkout the project
+*   copy the script file init.sh
+*   ```EXPOSE``` port 8080 
 
 The image can be built using the command:
 
-```docker build --build-arg SBT_VERSION="1.1.6" -t elleflorio/scalachain .```
+```docker build --build-arg SBT_VERSION="1.2.7" -t elleflorio/scalachain .```
 
 The docker image is already available in the [Dockerhub](https://hub.docker.com/r/elleflorio/scalachain/)
 
@@ -18,13 +18,13 @@ The docker container compiles the source code and run the scalachain node. For t
 
 To run a branch of the remote repo simply pass the name of the branch as the argument. e.g.:
 
-```docker run --name scalachain-dev -p 8080:8080 elleflorio/scalachain development```
+```docker run --name scalachain-dev -p 8000:8000 elleflorio/scalachain development```
 
 This will run the code of the branch ```development```.
 
-To run the local source code mounting the local folder to the container /development one, use the ```-v /your/source/folder:/development``` and pass ```local``` as the argument to the script. e.g.:
+To run the local source code mounting the local folder to the container ```/development``` one, use the ```-v /your/source/folder:/development``` and pass ```local``` as the argument to the script. You can optionally specify a folder inside the container where the source in the volume should be mounted, e.g. ```/tmp/scalachain```. The source will be automatically copied inside the ```/development``` folder. This is useful when you are running multiple containers in the same machine. This is a complete example:
 
-```docker run --name scalachain -p 8080:8080 -v ~/Development/scala/scalachain/:/development elleflorio/scalachain local```
+```docker run --name scalachain -p 8080:8080 -v ~/Development/scala/scalachain/:/tmp/scalachain elleflorio/scalachain local /tmp/scalachain```
 
 This will run the code contained in the folder ```~/Development/scala/scalachain/```.
 

docker/init.sh

@@ -1,9 +1,15 @@
 #! /bin/sh
 SOURCE=$1
-if [ $SOURCE == "local" ]
+SOURCE_FOLDER=$2
+if [[ ${SOURCE} == "local" ]]
 then
     echo "Running local source code"
-    cd /development
+    if ! [[ -z "$SOURCE_FOLDER" ]]
+    then
+        echo "copying local sources from $SOURCE_FOLDER to /development"
+        cp -r ${SOURCE_FOLDER}/. /development
+    fi
+        cd /development
 else
     echo "Runnig repo souce code"
     cd scalachain

img/scalachain_node.drawio

@@ -0,0 +1 @@
+<mxfile modified="2019-03-26T21:22:48.602Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/8.8.0 Chrome/61.0.3163.100 Electron/2.0.2 Safari/537.36" etag="CtiQ_G8Z8pSDfTESHG_I" version="10.5.8" type="device"><diagram id="F2S_QHZglgRF0Ka7Bkp2" name="Page-1">3Vpdc+I2FP01flzGlixjHgNJt53ZtJlmO90+KpbAygqLkUWA/vrKWMYWMqxJAbPhgdG9lmTpnHuPrz88OJmvP0u8SB8FodwDPll78N4DYIhi/V84NqUDxaB0zCQjpSuoHc/sX2qcvvEuGaG51VEJwRVb2M5EZBlNlOXDUoqV3W0quH3WBZ5Rx/GcYO56/2ZEpaU3BsPa/ytls7Q6cxCNyiNzXHU2O8lTTMSq4YIPHpxIIVTZmq8nlBfYVbiU4345cHS3MEkz1WXAK3z9dP9HToIxeUpJ8tfXMAw/mW28Yb40GzaLVZsKASmWGaHFJL4Hx6uUKfq8wElxdKUp175Uzbm2At2cMs4nggup7UxkutPYnIFKRdcHlx7sANGBRMWcKrnRXcwAMDIYmiACQ2OvakqQb3xpgw4QGyc2YTDbzV0jpRsGrBOAq+KzARQlOnKMKaRKxUxkmD/U3rENZd3nixALA+ArVWpj0gAvlbDhpWumvjXa/xRTDQAy5v3aTL01NpWR6Q1/axrNYYVdj9ta1UCC83S72mBHY7HH4yRqSMRSJvQYdiaPsZxRdaQfag8KSTlW7M1ex/kJdjPhigT7A9SgOOjI7wBZDP+A3h4Yhb0y6mjdhC9zReUjzrSMSw9EXC9+/FK0ZkXrLlFayQ4LYnCaIHoAEkzjaaL9uZLiO20ciZKYvkzPo5fQR5ZewtjVywC06GV0KbkEDvaPLCsg9y+AMaIxCdswjsELjKLzYBwhG+Owd4zhofj+wvR/diTA990/IR37Id8/HaFDx7jY/0eKedA7yEHgwHjNq7RVhg07lmGBXYYN+yrDUMeL9oGYuNJFG95Onf2+MvtH9PbAaNQro+HtMPrOO6cbpDTslVJ0O5R2VeHbpxT0SSlyqpffBaF17XLZkvEqd0m7EvFm7pIit2TkIvmepJhlH6hs7B/oUa9V4xWe7VyqZow7alevj+5iJ43+fHj+WiTQ028nKdeJD7x12kynU5C0KheJXiJ0poTaV6625+GtCQUvdh/mSlfxEgVX0uXCXl5O9qDXkCgbXwdHf/vzWl81YM5mmTYTjaO+0YbjAmWm13FnDswZIdtkbmPV5v0MNHV5bTFEAzAKg2AYhUMfxTFsIQ1dijQ3UVxdzMhd8RqtQJXjPGeJzc+7hMZFzHqR40JQ+TrrhznDk2D6xDtCgj1CINjLh1L/zKgaXWcigNAAHZ+qlEhnqi1Ru43/D+5GDncTMZ8vMx3uiuZbSDWipyaYSaTbz61gXwLj0Mmt0TUVsMqdpgIuF1S+sbytXhbTDleiD8sWRK4SXpct92H5ydJ3+2oH9kWqSpLT9S4+PtG71U6b9ScQZff6OxL48B8=</diagram></mxfile>

project/build.properties

@@ -1 +1 @@
-sbt.version = 1.1.6
+sbt.version = 1.2.7

src/main/resources/application.conf

@@ -1,9 +1,63 @@
-akka.persistence.journal.plugin = "akka.persistence.journal.leveldb"
-akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot-store.local"
+akka {
+  actor {
+    provider = "cluster"
+  }
+  persistence {
+    journal {
+      plugin = "akka.persistence.journal.leveldb"
+      leveldb {
+        dir = ${scalachain.persistence.dir}
+        # DO NOT USE THIS IN PRODUCTION !!!
+        # See also https://github.com/typesafehub/activator/issues/287
+        native = false
+      }
+    }
+    snapshot-store {
+      plugin = "akka.persistence.snapshot-store.local"
+      local.dir = ${scalachain.snapshots.dir}
+    }
+  }
+  remote {
+    log-remote-lifecycle-events = on
+    netty.tcp {
+      hostname = ${clustering.ip}
+      port = ${clustering.port}
+    }
+  }
+  cluster {
+    seed-nodes = [
+      "akka.tcp://"${clustering.cluster.name}"@"${clustering.seed-ip}":"${clustering.seed-port}
+    ]
+    auto-down-unreachable-after = 10s
+  }
+}
+http {
+  ip = "127.0.0.1"
+  ip = ${?SERVER_IP}
 
-akka.persistence.journal.leveldb.dir = "persistence/journal"
-akka.persistence.snapshot-store.local.dir = "persistence/snapshots"
+  port = 8000
+  port = ${?SERVER_PORT}
+}
+clustering {
+  ip = "127.0.0.1"
+  ip = ${?CLUSTER_IP}
 
-# DO NOT USE THIS IN PRODUCTION !!!
-# See also https://github.com/typesafehub/activator/issues/287
-akka.persistence.journal.leveldb.native = false
+  port = 2552
+  port = ${?CLUSTER_PORT}
+
+  seed-ip = "127.0.0.1"
+  seed-ip = ${?CLUSTER_SEED_IP}
+
+  seed-port = 2552
+  seed-port = ${?CLUSTER_SEED_PORT}
+
+  cluster.name = "scalachain"
+}
+scalachain {
+  node.id = "node0"
+  node.id = ${?SCALACHAIN_NODE_ID}
+  persistence.dir = "persistence/journal"
+  persistence.dir = ${?SCALACHAIN_PERSISTENCE_DIR}
+  snapshots.dir = "persistence/snapshots"
+  snapshots.dir = ${?SCALACHAIN_SNAPSHOTS_DIR}
+}