Writing Distributed and Replicated State Machine in Golang Using Raft | by Sanad Haj | May, 2022

Well, you may hear a lot about “distributed system” or “raft” itself. But, you may wonder how to use it.

Photo by Joshua Earle on Unsplash

In this tutorial, we’ll be discussing how to handle kv database running in cluster mode using Golang and Raft library. This tutorial is more focused on the code and using raft clusters rather than discussing the raft algorithms.

Raft is a protocol with which a cluster of nodes can maintain a replicated state machine.

The state machine is kept in sync through the use of a replicated log. However, the details of the Raft protocol are outside the scope of this tutorial, For more details on Raft, see In Search of an Understandable Consensus Algorithm

Golang implementation of the raft

Raft algorithm comes in search of an understandable consensus algorithm. Unfortunately, most of the go libraries out there required a deep knowledge of their implementation and APIs.

The raft library we’ll be using in this tutorial was born to align with the understandability raft principle and its sole purpose is to provide consensus with the minimalistic, simple, clean, and idiomatic API.

Etcd Raft is the most widely used Raft library in production. But, it follows a minimalistic design philosophy by only implementing the core raft algorithm which leaves gaps and ambiguities.

So, instead of reinventing the wheel, shaj13/raft library uses etcd raft as its core.

That’s how you can benefit from the power and stability of etcd raft, with an understandable API. Indeed, it keeps your focus on building awesome software.

We are going to start by creating our project.

mkdir raft && cd raft && go mod init raft && touch raft.go

This will create a new directory called raft and initialize the project with go.mod.

Before we write any code we need to write some mandatory code to make the program run.

package main
import (
func main() {
log.Println("Raft !!")

We are going to remove the line that prints out Raft!!, add the flag package and initialize it.

package main 
import "flag"
func main() {
addr := flag.String("raft", "", "raft server address")
join := flag.String("join", "", "join cluster address")
api := flag.String("api", "", "api server address")
state := flag.String("state_dir", "", "raft state directory (WAL, Snapshots)")

We are going to implement a struct named stateMachine define kv database that reads and applies a key value alongside taking database snapshot and restoring it.

We are going to add the gorilla mux package and initialize the router inside the main function.

router := mux.NewRouter()

Now we are going to establish the endpoints of our API, the way we will set this up is to create all of our endpoints in the main function, every endpoint needs a function to handle the request and we will define those below the main function.

router.HandleFunc("/", http.HandlerFunc(save)).Methods("PUT", "POST")
router.HandleFunc("/{key}", http.HandlerFunc(get)).Methods("GET")
router.HandleFunc("/mgmt/nodes", http.HandlerFunc(nodes)).Methods("GET")
router.HandleFunc("/mgmt/nodes/{id}", http.HandlerFunc(removeNode)).Methods("DELETE")

Now we just need to define the functions that will handle the requests.
Before we start we need to declare two variables to allow routes access data.

  • Node represents raft process
  • FSM represents raft process state machine

In the main function and below the router we need to declare our raft node and gRPC server to allow current raft node to communicate with other raft nodes.

Your file should now look something like this:

Building raft cluster

go mod tidy && go build raft.go 

Running single node raft

First, start a single-member cluster of raft:

./raft -state_dir=$TMPDIR/1 -raft :8080 -api :9090

Each raft process maintains a single raft instance and a key-value server.

raft server address (-raft), state dir (-state_dir), and http key-value server address (-api) are passed through the command line.

Next, store a value (“medium”) to a key (“hello”):

curl -L -X PUT -d '{"Key":"hello", "Value":"medium"}'

Finally, retrieve the stored key:

curl -L

Running a local cluster

Let’s bring two additional raft instances.

./raft -state_dir $TMPDIR/2 -raft :8081 -api :9091 -join :8080
./raft -state_dir $TMPDIR/3 -raft :8082 -api :9092 -join :8080

Now it’s possible to write a key-value pair to any member of the cluster and likewise retrieve it from any member.

Fault Tolerance

To test cluster recovery, write a value “foo” to key “foo”:

curl -L -X PUT -d '{"Key":"foo", "Value":"foo"}'

Next, stop a node (9092) and replace the value with “bar” to check cluster availability:

curl -L -X PUT -d '{"Key":"foo", "Value":"bar"}'
curl -L

Finally, bring the node back up and verify it recovers with the updated value “bar”:

curl -L

Cluster Reconfiguration

Nodes can be added, removed, or updated from a running cluster. Let’s remove node using requests to the REST API.

First, list all raft cluster nodes, and get node id.

curl -L

Then remove a node using a DELETE request:

curl -L<ID> -X DELETE

The node will shut itself down once the cluster has processed this request.

News Credit

%d bloggers like this: