Go Microservices with Go-kit using Mongodb and Mysql

Hello folks,In this post I will explain about developing the Go microservices with Go kit which is a toolkit for microservices in Go. As I have faced a lot of problems while learning this concept because of not having proper references I just wanted to write this post. The source code of the examples is provided here.

Before Getting into the topic let us discuss about the Go kit Architecture which will be helpful while implementing the Go microservices. This architecture is also be called as clean Architecture.

Go-Kit Architecture:

source: gokit.io

Basically there are three layers or components in the Go kit based application.

1. Transport layer
2. Endpoint layer
3. Service layer

Transport Layer:

Transport layer is useful when you are writing microservices in any language to facilitate the communication between the service to service. The transport domain is bound to concrete transports like HTTP or gRPC. Go kit supports various transports using HTTP, gRPC, NATS, AMQP and Thrift. In a world where microservices may support one or more transports, this is very powerful; you can support a legacy HTTP API and a newer RPC service, all in a single microservice.

Endpoint Layer:

An endpoint is like an action/handler on a controller; it’s where safety and antifragile logic lives. If you implement two transports (HTTP and gRPC), you might have two methods of sending requests to the same endpoint. Each and every service method in Go kit needs to be converted into an endpoint to make RPC style communication between clients and servers (like Request and Response).

Service Layer:

Services are where all of the business logic is implemented. A service usually glues together multiple endpoints. In Go kit, services are typically modeled as interfaces, and implementations of those interfaces contain the business logic. Each service method then interacts with databases with the help of Repository interface created and the methods implemented for each repository abstract method.

Example Services :

Basically, Go kit provides different features such as tracing with zipkin, logging middleware, metrics with prometheus monitoring tools,rate-limiting and circuit breaking (hystrix). These are essential for any microservice. Here I am discussing about logging middleware, metrics with prometheus, database interaction with Mongodb and Mysql.

We’ll explore the example service in a manner that is how the Go kit microservices follows. Please make habit of writing all the layers inside the same package to reduce the risk overthinking and making mistakes while accessing the different functions. In the below example Iam explaining a POST request “/create”, remaining CRUD operations will be available in the provided Github repo.

1. Define the model , make sure that each field in a struct should start with a capital letter otherwise the field cannot considered by the struct.

// model.go
package main
type User struct {
Id         string `json:"id" omitempty`
Userid     int    `json:"userid" binding:"required"`
Email      string ` json:"email" binding:"required"`
Password   string ` json:"password" binding:"required"`
Phone      string ` json:"phone" binding:"required"`
}

2. Declare the service interface and all the abstract methods inside it which you are going to implement in the service layer. Also write a service struct and NewService which are useful when you are interacting with the database through repository. Then implement all the service methods.

Declaration and Implementation of service methods.

// service.go
package main
import (
"context"
"github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/gofrs/uuid"
)
// service implements the User Service
// logger for logging middleware
type userservicestruct struct {
repository Repository
logger     log.Logger
}
// Service describes the User service.
type UserService interface {
CreateUser(ctx context.Context, user User) (string, error)
}
// NewService creates and returns a new User service instance
func NewService(rep Repository, logger log.Logger) UserService {
return &userservicestruct{
repository: rep,
logger:     logger,
}
}
func (s userservicestruct) CreateUser(ctx context.Context, user User) (string, error) {
logger := log.With(s.logger, "method", "Create")
uuid, _ := uuid.NewV4()
var id = uuid.String()
user.Id = id
userDetails := User{
Id:         user.Id,
Userid:   user.Userid,
Email:      user.Email,
Password:   user.Password,
Phone:      user.Phone,
}
if err := s.repository.CreateUser(ctx, userDetails); err != nil {
level.Error(logger).Log("err", err)
}
return id, nil
}

3. Declare the Handlers and routes inside main.go which is the root file for a microservice. The handlers will call Endpoints, DecodeRequests & encodeResponse. Create an instance of your service with Repository and NewService. NewService returns the address of repository instance.

// main.go
package main
import (
"fmt""net/http"
"os"
"github.com/gorilla/mux"
"github.com/go-kit/kit/log"
httptransport "github.com/go-kit/kit/transport/http"
"github.com/go-kit/kit/log/level"
)
func main() {
r := mux.NewRouter()
dbmname := "mysql" // change to mongo for mongodb
var svc UserService
//For the service implementation with mongodb
{
if dbmname == "mongo" {
db := GetMongoDB()
repository, err := NewRepo(db, logger)
if err != nil {
level.Error(logger).Log("exit", err)
os.Exit(-1)
}
svc = NewService(repository, logger)
} else {
// for the service implementaion with Mysql
sqldb := GetSqlDB()
repository, err := NewSqlRepo(sqldb, logger)
if err != nil {
level.Error(logger).Log("exit", err)
os.Exit(-1)
}
svc = NewService(repository, logger)
}
}
CreateUserHandler := httptransport.NewServer(
makeCreateUserEndpoint(svc),
decodeCreateUserRequest,
encodeResponse,
)

http.Handle("/", r)
http.Handle("/create", CreateUserHandler)
logger.Log("msg", "HTTP", "addr", ":8000")
logger.Log("err", http.ListenAndServe(":8000", nil))
}

4. Write the Adapters which will convert the service methods into endpoints to make RPC style communication with HTTP transport.

// transport.go starts here
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strconv"
"github.com/go-kit/kit/endpoint"
"github.com/gorilla/mux"
)
// Endpoint for the User service.
func makeCreateUserEndpoint(s UserService) endpoint.Endpoint {
fmt.Println("make createUser endpoint called")
return func(ctx context.Context, request interface{}) (interface{}, error) {
req := request.(CreateUserRequest)
id, err := s.CreateUser(ctx, req.user)
return CreateUserResponse{Id: id, Err: err}, nil
}
}

5. Write DecodeRequests to decode the incoming request into the json format and encodeResponse to send the response in json format. Sometimes the incoming request data may be in the form XML, HTML or JSON etc… To understand the incoming data Go-kit architecture uses DecodeRequests and to send the response in the same format, it uses encodeResponse.

// transport.go continues here 
func decodeCreateUserRequest(_ context.Context, r *http.Request) (interface{}, error) {
fmt.Println("decodeCreateUserRequest called")
var req CreateUserRequest
if err := json.NewDecoder(r.Body).Decode(&req.user); err != nil {
return nil, err
}
return req, nil
}
//  encodes the output
func encodeResponse(_ context.Context, w http.ResponseWriter, response interface{}) error {
fmt.Println("Encode Response called")
w.Header().Set("Content-Type", "application/json; charset=utf-8")
return json.NewEncoder(w).Encode(response)
}
type (
CreateUserRequest struct {
user User
}
CreateUserResponse struct {
Id  string `json:"id"`
Err error
}
)

• Structs are necessary for each service method to receive the request and pass it on to the method implementation through end points and to send the response.

6. Now you are ready with basic setup of Go kit microservice. Let us now add database to this service.

//dbconnection.go
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/go-sql-driver/mysql"
mgo "gopkg.in/mgo.v2"
)
// for mongodb
var db *mgo.Database
func GetMongoDB() *mgo.Database {
host := "MONGO_HOST"
dbName := "Go_mic_service"
fmt.Println("connection info:", host, dbName)
session, err := mgo.Dial("mongodb://localhost:27017")
if err != nil {
fmt.Println("session err:", err)
os.Exit(2)
}
db = session.DB(dbName)
return db
}
// for mysql
var sqldb *sql.DB
func GetSqlDB() *sql.DB {
var err error
dbUser := "root" // mysql username
dbPass := "root"  // mysql password
dbName := "godb" // mysql dbname
sqldb, err := sql.Open("mysql", dbUser+":"+dbPass+"@/"+dbName)
if err != nil {
panic(err.Error())
}
return sqldb
}

7. Create a Repository interface which consists of all the abstract methods which will interact with the database to get the things done. Also create a repo structure to return the database name .

// repo.go stars here
package main
import (
"context"
"database/sql"
"errors"
"fmt"
"github.com/go-kit/kit/log"
_ "github.com/go-sql-driver/mysql"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var RepoErr = errors.New("Unable to handle Repo Request")
const UserCollection = "gotestuser"
type Repository interface {
CreateUser(ctx context.Context, user User) error
}
// Repo structure for mongodb
type repo struct {
db     *mgo.Database
logger log.Logger
}
// Repo for mongodb
func NewRepo(db *mgo.Database, logger log.Logger) (Repository, error) {
return &repo{
db:     db,
logger: log.With(logger, "repo", "mongodb"),
}, nil
}

Using Mysql

// Repo structure for mysql
type sqlrepo struct {
sqldb  *sql.DB
logger log.Logger
}
// Repo for mysql
func NewSqlRepo(sqldb *sql.DB, logger log.Logger) (Repository, error) {
return &sqlrepo{
sqldb:  sqldb,
logger: log.With(logger, "sqlrepo", "sqldb"),
}, nil
}

8. Implement all the abstract repository methods declared inside the Repository interface.

Using Mongodb..

// repo.go continues here

func (repo *repo) CreateUser(ctx context.Context, user User) error {
fmt.Println("create user mongo repo", db)
err := db.C(UserCollection).Insert(user)
if err != nil {
fmt.Println("Error occured inside CreateUser in repo")
return err
} else {
fmt.Println("User Created:", user.Email)
}
return nil
}

Using Mysql..

func (sqlrepo *sqlrepo) CreateUser(ctx context.Context, user User) error {fmt.Println("create user sql repo", sqldb)
dbt := sqlrepo.sqldb
sqlresp, err := dbt.Prepare("INSERT INTO gotestuser (id,userid,email,password,phone) VALUES (?,?,?,?,?)")
sqlresp.Exec(user.Id, user.Userid, user.Email, user.Password, user.Phone)
fmt.Println("User Created:", sqlresp, err)
return nil
}

Write go.mod to build and execute this service in an easy way.

// go.mod
module UserService
go 1.14
require (
github.com/go-kit/kit v0.10.0
github.com/gofrs/uuid v3.3.0+incompatible
github.com/prometheus/client_golang v1.3.0
gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22
)

Now our service is ready to test using postman, but let’s wait to add more features to this service.

Note : Don’t get confused by seeing the two databases. There are notable differences for them.

1. main.go → service creation
2. dbconnection.go → GetSqlDB() and GetMongoDB()
3. repo.go →sqlrepo structure (for mysql), repo structure (for mongo),NewRepo()(for mongo) , NewSqlRepo (for mysql)
4. repo.go → repository methods implementations.
5. Before going to use mysql service methods create a database with name “godb” and a table inside it with the name “gotestuser” with the model fields.

Apart from these changes all the remaining is same.

9. Logging Middleware:

Logging middleware is used to get the logs of service methods throughout the service starting from receiving request from client to traversing through transport , endpoint and service layers and sending the response to the client.

To have logging to your microservice add the following to the respective files.

// Add this lines to main.go
logger := log.NewLogfmtLogger(os.Stderr) // creating logger
svc = loggingMiddleware{logger, svc} // passing the service to the loggingMiddleware

Write logging.go file to enable and configure the logging for all the serrvice methods.

package main
import (
"context"
"time"
"github.com/go-kit/kit/log"
)
type loggingMiddleware struct {
logger log.Logger
next   UserService
}
func (mw loggingMiddleware) CreateUser(ctx context.Context, user User) (Email string, err error) {
defer func(begin time.Time) {
_ = mw.logger.Log(
"method", "creatuser",
"Email", user.Email,
"err", err,
"took", time.Since(begin),
)
}(time.Now())
Email, err = mw.next.CreateUser(ctx, user)
return
}

10. Instrumentation with prometheus :

Instrumentation is necessary to monitor the application in every aspect with the help of metrics generated.prometheus is a monitoring tool for the metrics. We can export and view this metrics graphically in grafana by just mentioning the address where in the metrics are generated.

Add the Following imports and methods to main.go.

"github.com/prometheus/client_golang/prometheus/promhttp"
stdprometheus "github.com/prometheus/client_golang/prometheus"
kitprometheus "github.com/go-kit/kit/metrics/prometheus"
// Adding requestcount,latency,countResult to metrics
fieldKeys := []string{"method", "error"}
requestCount := kitprometheus.NewCounterFrom(stdprometheus.CounterOpts{
Namespace: "my_group",
Subsystem: "string_service",
Name:      "request_count",
Help:      "Number of requests received.",
}, fieldKeys)
requestLatency := kitprometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
Namespace: "my_group",
Subsystem: "string_service",
Name:      "request_latency_microseconds",
Help:      "Total duration of requests in microseconds.",
}, fieldKeys)
countResult := kitprometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
Namespace: "my_group",
Subsystem: "string_service",
Name:      "count_result",
Help:      "The result of each count method.",
}, []string{})
svc = instrumentingMiddleware{requestCount, requestLatency, countResult, svc}
// Route to export the metrics
http.Handle("/metrics", promhttp.Handler())

Add a instrumenting.go file to generate the metrics for all the service methods.

// instrumenting.go
package main
import (
"context"
"fmt"
"time"
"github.com/go-kit/kit/metrics"
)
type instrumentingMiddleware struct {
requestCount   metrics.Counter
requestLatency metrics.Histogram
countResult    metrics.Histogram
next           UserService
}
func (mw instrumentingMiddleware) CreateUser(ctx context.Context, user User) (output string, err error) {
defer func(begin time.Time) {
lvs := []string{"method", "createuser", "error", fmt.Sprint(err != nil)}
mw.requestCount.With(lvs...).Add(1)
mw.requestLatency.With(lvs...).Observe(time.Since(begin).Seconds())
}(time.Now())
output, err = mw.next.CreateUser(ctx, user)
return
}

Hurray!!, you are all set to test the example microservice now.

1. Build the Go service with : go build .
2. Run it using the exe file generated ,in this case it is “UserService”.
3. Go to the postman and test the following routes.

i. localhost:8000/create

ii. localhost:8000/user/{userid}

iii. localhost:8000/deleteuser/{userid}

iv. localhost:8000/updateuser/

v. localhost:8000/getall

4. Go to the Browser and type in the address : localhost:8000/metrics to view the metrics.

You can write as many as microservices and make communication between them with the help of transports like Http and gRpc.

That’s it guys, I hope you’ll understood the things well. Still if anyone have any suggestions or doubts do comment here.I am always ready to help you out at any point of time.

For any queries feel free to reach me out at LinkedIn B Narsimha and narsimhulu.464@gmail.com.

Thank You !!!