Go has had the biggest impact in distributed systems. The developers of projects like Docker, Kubernetes, Etcd, and Prometheus all decided to use Go for good reason. Google developed Go and its standard library as an answer to software problems at Google: multi-core processors, networked systems, massive computation clusters—in other words, distributed systems. Because you likely use systems like these and want to know how they work, how to debug them, and how to contribute to them, or you want to build similar projects of your own.
So how do you start ? Building a distributed service isn’t the easiest or smallest project in the world. If you try to build all the pieces at once, all you’ll end up with is a big, stinking mess of a code base and a fried brain. You build the project piece by piece. A good place to start is a commit log JSON over HTTP service. Even if you’ve never written an HTTP server in Go before, follow to know how to make an accessible application programming interface (API) that clients can call over the network. You’ll learn about commit log APIs.
How JSON over HTTP Services Fits into Distributed Systems
JSON over HTTP APIs are the most common APIs on the web, and for good reason. They’re simple to build since most languages have JSON support built in. And they’re simple and accessible to use since JSON is human readable and you can call HTTP APIs via the terminal with curl.
JSON/HTTP isn’t limited to small web services. Most tech companies that provide a web service have at least one JSON/HTTP API acting as the public API of their service
But For their internal web APIs, the company may take advantage of technologies like protobuf/gRPC for features that JSON/HTTP doesn’t provide—like type checking , versioning and improve efficiency—but their public one will still be JSON/HTTP for accessibility.
JSON/HTTP is a great choice for the APIs of infrastructure projects like Elasticsearch (a popular open source, distributed search engine) and Etcd (a popular distributed key-value store used by many projects, including Kubernetes) also use JSON/HTTP for their client-facing APIs, while employing their own binary protocols (protobuf) for communication between nodes to improve performance.
JSON/HTTP is no toy—you can build all kinds of services with it.
Building an HTTP server
it can indeed be a straightforward process especially in GoLang, thanks to the robust features provided by the language's standard library. However, before getting into the technical details, it's essential to clarify the purpose behind what is our purpose of building a HTTP server. Today, our aim is to construct a HTTP server specifically tailored for logging commits in JSON format over HTTP. Thankfully, Go offers a comprehensive solution for this task through its net/http package, enabling us to effortlessly craft a custom HTTP server capable of serving as a reliable commit log service. In this article, we'll explore the steps involved in creating such a server, leveraging the power and simplicity of Go's HTTP capabilities. By the end, you'll have a solid understanding of how to build a robust HTTP server tailored to your specific requirements. Let's dive in!
that's a basic example of how to write an http server in golang using net/http standard package
package main
import (
"fmt"
"net/http"
)
func main() {
addr := ":8080" // Specify the port to listen on
// Create a new multiplexer (router)
mux := http.NewServeMux()
// Define routes
mux.HandleFunc("/", handleRoot)
mux.HandleFunc("/hello", handleHello)
// Create a new HTTP server
server := &http.Server{
Addr: addr,
Handler: mux,
}
// Start the server
fmt.Printf("Server listening on %s\n", addr)
if err := server.ListenAndServe(); err != nil {
fmt.Printf("Error starting server: %s\n", err)
}
}
func handleRoot(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to the homepage!")
}
func handleHello(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, World!")
}
in the above code
define two handler functions:
handleRootfor the root path"/"andhandleHellofor the"/hello"path. These functions simply write a response back to the client.create a new ServeMux (router) and register the handler functions for specific routes.
configure an http.Server instance with the desired address to listen on (addr) and the multiplexer (mux) as the handler.
Finally, we start the server by calling ListenAndServe() on the server instance.
Multiplexer
In Go, a multiplexer, often referred to as a router, is a mechanism used to handle incoming HTTP requests and direct them to the appropriate handler functions based on the request path.
The http.NewServeMux() function creates a new ServeMux instance. ServeMux is a lightweight HTTP request router (or multiplexer) provided by the net/http package in Go. It acts as a request multiplexer, matching the URL of incoming requests against a list of registered patterns and calls the handler for the pattern that most closely matches the URL.
Let me Introduce Gorilla Mux: A Powerful URL Router for Go
Gorilla Mux is a popular third-party package for Go that provides a powerful URL router and dispatcher. The name mux stands for "HTTP request multiplexer". It enhances the capabilities of Go's built-in net/http package by offering advanced routing features and more flexibility in handling HTTP requests.
package main
import (
"fmt"
"net/http"
"github.com/gorilla/mux"
)
func main() {
addr := ":8080" // Specify the port to listen on
// Create a new Gorilla Mux router
router := mux.NewRouter()
// Define routes with specified HTTP methods
router.HandleFunc("/", handleRoot).Methods("GET")
router.HandleFunc("/hello", handleHello).Methods("GET")
// Create a new HTTP server
server := &http.Server{
Addr: addr,
Handler: router, // Use Gorilla Mux router as the handler
}
// Start the server
fmt.Printf("Server listening on %s\n", addr)
if err := server.ListenAndServe(); err != nil {
fmt.Printf("Error starting server: %s\n", err)
}
}
func handleRoot(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to the homepage!")
}
func handleHello(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, World!")
}
Why Use Gorilla Mux:
Enhanced Routing: Gorilla Mux offers more powerful and flexible URL routing capabilities compared to the standard
net/httppackage.Middleware Support: It provides built-in middleware support, allowing for easier integration of cross-cutting concerns like logging and authentication.
Parameter Extraction: Gorilla Mux simplifies the extraction of URL parameters, facilitating cleaner and more concise route handling code.
HTTP Method-Based Routing: It enables intuitive and structured handling of different HTTP methods (GET, POST, etc.) for each route.
Subrouter Functionality: Gorilla Mux supports subrouters, promoting modularization and better organization of route structures in large applications.
net/http.Server
net/http.Server is a struct type provided by Go's standard library in the net/http package. It represents an HTTP server that listens for incoming connections on a specific network address and dispatches those connections to registered handlers for processing. Here's an overview of its fields:
Addr: A string representing the network address the server should listen on, in the format "host:port". For example, ":8080" means the server will listen on port 8080 of all available network interfaces.
Handler: An interface value that specifies the handler to invoke to process incoming HTTP requests. The handler must satisfy the http.Handler interface, which requires implementing the ServeHTTP method.
there's more fields you can check the docs but those who matter to us for now.
also net/http provides methods like ListenAndServe which used to start the HTTP server and make it listen for incoming HTTP requests on the network address specified by the Addr field of the http.Server instance.
Here's a breakdown of what the server.ListenAndServe() method does:
Starts Listening: When invoked, ListenAndServe starts listening for incoming connections on the network address specified by the Addr field
Accepts Connections: As soon as a new connection is established, the method accepts it. It waits for incoming connections and handles them as they arrive.
Dispatches Requests: Upon accepting a connection, ListenAndServe dispatches incoming HTTP requests to the appropriate handler based on the URL path specified in the request.
Handles Requests: Once a request is dispatched, the server invokes the ServeHTTP method of the registered handler to handle the incoming HTTP request. If you've registered handlers using
http.HandleFuncorhttp.Handle, the server automatically invokes the appropriate handler function or type's ServeHTTP method, respectively.Responds to Requests: After the handler processes the request, it writes the response back to the client through the connection established earlier.
Continues Listening: The ListenAndServe method continues to listen for more incoming connections and repeats the process for each new request, effectively creating an event loop that keeps the server running indefinitely.
Blocking Behavior: ListenAndServe is a blocking method, meaning it will prevent the program from exiting as long as the server is running. It will continuously handle incoming requests until an error occurs or the server is explicitly shut down.
that's enough about go http servers let's get to know more about commit log json.
Why Logs ?
at first let's know why we will talk about logs? the main concept about logs or let's answer first what is a log ? most of the time your thoughts about logs is that you throw some exceptions and put them into a table or file, i won't return to that table unless something bad happen, issue raised, but log is more than this, log (write ahead log, transactional log in databases, ACID transactions how they happen, commit logs in databases, but by the time you had logs and transactional logs that become change logs it has different uses in internal databases and distributed system.
so in the past days what developers used to have is an application log that is mostly meant for humans to read kind of series of loosely structured requests, errors in rotating text files.
nowadays when you have many services and servers involved,That old approach quickly becomes unmanageable. The purpose of logs quickly becomes an input to queries and graphs in order to understand behavior across many machines, something that English text in files is not nearly as appropriate for as the kind of structured log that's more general and closer to what in the database, it's a commit log or journal ( append-only sequence of records by time)
Each rectangle represents a record that was appended to the log. Records are stored in the order they were appended. Reads proceed from left to right. Each entry appended to the log is assigned a unique, sequential log entry number that acts as its unique key.
The ordering of records defines a notion of “time” since entries to the left are defined to be older then entries to the right. The log entry number can be thought of as the “timestamp” of the entry also it called the "logical time".
Describing this ordering as a notion of time seems a bit odd at first, but it has the convenient property of being decoupled from any particular physical clock. This property will turn out to be essential as we get to distributed systems.
In fact, the use of logs in much of the variations in database internals:
The log is used as a publish/subscribe mechanism to transmit data to other replicas
The log is used as a consistency mechanism ordering the updates that are applied to multiple replicas Somehow,
perhaps because of this origin in database internals, the concept of a machine readable log is not widely known, although, this abstraction is ideal for supporting all kinds of messaging, data flow, and real-time data processing.
" A Log is perhaps the simplest possible storage abstraction. it is an append-only, totally-ordered sequence of records ordered by time (logical time). "
Set Up the Project
The first thing we need to do is create a directory for our project’s code. We’ll call our project proglog, so open your terminal to wherever you like to put your code and run the following commands to set up your module:
$ mkdir proglog
$ cd proglog
$ go mod init github.com/user/proglog
Build a Commit Log Prototype
commit logs is a data structure for an append-only sequence of records, ordered by time, and you can build a simple commit log with a slice.
so let's define some structs that we can depend on in building a log library.
Log : log struct is a group of records which will be slice
Record : record struct is a
value which is a group of bites (slice) representing the the log message
offset which will be integer refers to the unique index that we can start to read the record value from
Create an internal/server directory tree in the root of your project and put the following code under the server directory in a file called log.go:
# /internal/server/log.go
package server
import (
"fmt"
"sync"
)
type Record struct {
Value []byte `json:"value"`
Offset uint64 `json:"offset"`
}
type Log struct {
mu sync.Mutex
records []Record
}
func NewLog() *Log {
return &Log{}
}
var ErrOffsetNotFound = fmt.Errorf("offset not found")
func (c *Log) Append(record Record) (uint64, error) {
c.mu.Lock()
defer c.mu.Unlock()
record.Offset = uint64(len(c.records))
c.records = append(c.records, record)
return record.Offset, nil
}
func (c *Log) Read(offset uint64) (Record, error) {
c.mu.Lock()
defer c.mu.Unlock()
if offset >= uint64(len(c.records)) {
return Record{}, ErrOffsetNotFound
}
return c.records[offset], nil
}
To append a record to the log, you just append to the slice. Each time we read a record given an index, we use that index to look up the record in the slice. If the offset given by the client doesn’t exist, we return an error saying that the offset doesn’t exist. All really simple stuff
Build a JSON over HTTP Server
Go web server comprises one function — a net/http HandlerFunc(ResponseWriter, *Request)—for each of your API’s endpoints.
Our API has two endpoints:
Produce for writing to the log
Consume for reading from the log.
When building a JSON/HTTP Go server, each handler consists of three steps:
Unmarshal the request’s JSON body into a struct.
Run that endpoint’s logic with the request to obtain a result.
Marshal and write that result to the response.
create our HTTP server. Inside your internal/server directory, create a file called http.go that contains the following code:
# internal/server/http.go
package server
import (
"encoding/json"
"net/http"
"github.com/gorilla/mux"
)
type httpServer struct {
Log *Log
}
func newHTTPServer() *httpServer {
return &httpServer{
Log: NewLog(),
}
}
func NewHTTPServer(addr string) *http.Server {
httpsrv := newHTTPServer()
r := mux.NewRouter()
r.HandleFunc("/", httpsrv.handleProduce).Methods("POST")
r.HandleFunc("/", httpsrv.handleConsume).Methods("GET")
return &http.Server{
Addr: addr,
Handler: r,
}
}
We now have a server referencing a log for the server to defer to in its handlers. using NewHTTPServer that start with capital N which refers that it's exposed to be used outside the package, We create our server and use the popular gorilla/mux library to write RESTful routes that match incoming requests to their respective handlers.
An HTTP
POSTrequest to/matches the produce handler and appends the record to the logan HTTP
GETrequest to/matches the consume handler and reads the record from the log.
then wrap our server with a net/http.Server so the user just needs to call ListenAndServe() func to listen for and handle incoming requests.
Next, we’ll define the request and response structs by adding this snippet below
# internal/server/http.go
type ProduceRequest struct {
Record Record `json:"record"`
}
type ProduceResponse struct {
Offset uint64 `json:"offset"`
}
type ConsumeRequest struct {
Offset uint64 `json:"offset"`
}
type ConsumeResponse struct {
Record Record `json:"record"`
}
A produce request contains the record that the caller of our API wants appended to the log, and a produce response tells the caller what offset the log stored the records under.
A consume request use the offset to specifies which records the caller of our API wants to read and the consume response to send back those records to the caller.
implement the server’s handlers
# /internal/server/http.go
func (s *httpServer) handleProduce(w http.ResponseWriter, r *http.Request) {
var req ProduceRequest
err := json.NewDecoder(r.Body).Decode(&req)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
off, err := s.Log.Append(req.Record)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
res := ProduceResponse{Offset: off}
err = json.NewEncoder(w).Encode(res)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
The produce handler implements the three steps we discussed before:
unmarshaling the request into a struct
using that struct to produce to the log and getting the offset that the log stored the record under
marshaling and writing the result to the response.
Our consume handler looks almost identical.
func (s *httpServer) handleConsume(w http.ResponseWriter, r *http.Request) {
var req ConsumeRequest
err := json.NewDecoder(r.Body).Decode(&req)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
record, err := s.Log.Read(req.Offset)
if err == ErrOffsetNotFound {
http.Error(w, err.Error(), http.StatusNotFound)
return
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
res := ConsumeResponse{Record: record}
err = json.NewEncoder(w).Encode(res)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
The consume handler calls Read(offset uint64) to get the record stored in the log.
Run Your Server
write is a main package with a main() function to start your server. In the root directory of your project, create a cmd/server directory tree, and in the server directory create a file named main.go with this code:
# /cmd/server/main.go
package main
import (
"log"
"github.com/user/proglog/internal/server"
)
func main() {
srv := server.NewHTTPServer(":8080")
log.Fatal(srv.ListenAndServe())
}
Our main() function just needs to create and start the server, passing in the address to listen on (localhost:8080) and telling the server to listen for and handle requests by calling ListenAndServe().
Wrapping our server with the *net/http.Server in NewHTTPServer(). we’d create an HTTP server. It’s time to test our slick new service.
Test Your API
run and test by hitting the endpoints with curl. Run the following command in your terminal to start the server:
$ go run main.go
Open another tab in your terminal and run the following commands to add some records to your log:
$ curl -X POST localhost:8080 -d '{"record": {"value": "TGV0J3MgR28gIzEK"}}'
$ curl -X POST localhost:8080 -d '{"record": {"value": "TGV0J3MgR28gIzIK"}}'
$ curl -X POST localhost:8080 -d '{"record": {"value": "TGV0J3MgR28gIzMK"}}'
Go’s encoding/json package encodes []byte as a base64-encoding string. The record’s value is a []byte, so that’s why our requests have the base64 encoded forms. You can read the records back by running the following commands and verifying that you get the associated records back from the server:
$ curl -X GET localhost:8080 -d '{"offset": 0}'
$ curl -X GET localhost:8080 -d '{"offset": 1}'
$ curl -X GET localhost:8080 -d '{"offset": 2}'
Congratulations 🥳🥳🥳 —you have built a Commit log JSON/HTTP service and confirmed it works!
Refs :-
Distributed Services with Go by Travis Jeffery : Your Guide to Reliable, Scalable, and Maintainable Systems