quapona technologies

Today we live in a containerized world. Applications are getting larger and larger but still need to be as modular as possible. This is a major challenge for companies, especially for big corporations. Modularity will boost companies by allowing smaller development teams and giving the possibility to reuse application parts for different kinds of products. This will work out of the box if every piece of software is continuously and automatically tested, whereas the interface between the different application parts needs to be well-defined.

At quapona we live this approach and call these solutions Self-Contained Systems (SCS). A SCS should work as a whole software solution for a certain problem by splitting it into single parts.


One Self-Contained System could consist of the storage solution (Data), the storage driver (Logic) and the application which generates the data to be stored (Interfaces). You can think of the certain parts as Microservices, which will be connected together to a single working unit. The main challenge is to decide which functionality belongs to the Microservice and which does not. This automatically leads into iterative development since Microservices are getting larger and can thus be split into multiple ones. A good company communication culture is mandatory in order to make this work as developers always need to rethink their work.

General Microservice conditions

Every Microservice can basically do whatever is required to get its job done, but there are still a few rules to be considered:

  • The interface technology for Microservices within a SCS has to be the same
  • Every Microservice needs to be fully documented
  • Every Microservice must be tested with unit, module and integration tests
  • The API of a Microservice needs to follow Semantic Versioning
  • Manual testing and Code Review is a must-have

At quapona we call the combination of these rules Continuous Quality Assurance (CQA). The excessive testing of the application is one of our key drivers to ensure the portability of the Microservice and keep Self-Contained Systems stable and the data reliable. One of the main mistakes companies are making is to safe money in testing which leads into cruel software solutions everyone might have examples for: They are monolithic, they mostly solve problems of only a few use cases and they eventually cost way too much money when they have to be extended or maintained.

Besides the general CQA approach, it is also vital to abstract the implementation of the Microservice to be as general as possible. This enables making the Microservice (or the whole solution) Open Source, which allows feedback from all over the world and endowes other developers with the option to reuse the solution. The sensitivity for providing fulfilling documentation and testing is much higher when developers consider making the software Open Source than to just develop for a small team where they know each other too well. The feedback you will get from the certain community is to be worth a mint. There are lots of experts out there which can be reached instantly by today.

A Microservice template

How to achieve this? Which interface is to be used and how to deploy the application? Which language should I choose? Should I write the communication interface between two Microservices on my own? No. Another big flaw by companies is to write everything from scratch. It should also be avoided to rely too hard on existing solutions to maintain the necessary flexibility of your business. The best approach is to always keep the eyes open for new technologies by picking the right solutions for your needs without creating a second world around them where you might get stuck. Our example of a Microservice template uses the following, well-known technologies:

  • Go: As main programming language
  • GRPC: As an interface to another Microservice
  • GoMock: As mocking framework for unit tests
  • Docker: As containerization solution
  • Kubernetes: As module and integration testing, container orchestration and deployment solution

What is outstanding here is that a lot of named technologies are made by a company called Google. Yeah, that is true, Google provides lots of great open Source Solutions which might help smaller companies to be successful. But keep in mind: Every part of the Microservice needs to be exchangeable if necessary. For example our template provides the ability to switch from Docker to rkt, another container engine developed by CoreOS.

Using the template

Here we go, our Microservice template can easily be retrieved via GitHub:

> git clone https://github.com/quaponatech/microservice-template

Some software requirements have to be installed on your development system if you want to use the templates' full capabilities. To test the functionality of the Microservice template, simply run go run main.go in the root directory of the repository. Then you should see the following output:

2017/05/26 08:43:16 GRPC server: Listening on port 42302
2017/05/26 08:43:16 GRPC server: Prepare server options (without TLS)
2017/05/26 08:43:16 GRPC server: Creating new RPC server
2017/05/26 08:43:16 Microservice - [SERVICE] Setting up Service
2017/05/26 08:43:16 Microservice - [LOGGER]  Starting Server Logger
2017/05/26 08:43:16 Microservice - [LOGGER]  Started Server Logger
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to debug channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to warning channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to log channel
2017/05/26 08:43:16 Microservice - [INFO]    Initialized Server
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to error channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to status channel
2017/05/26 08:43:16 Microservice - [STATUS]  StateInitialized
2017/05/26 08:43:16 Microservice - [INFO]    Register microservice
2017/05/26 08:43:16 Microservice - [STATUS]  StateStarting
2017/05/26 08:43:16 Microservice - [STATUS]  StateStarted
2017/05/26 08:43:16 Microservice - [STATUS]  StateRunning

We now have a fully working GRPC enabled server running and waiting for incoming connections and messages. This can be tested easily via Telnet:

> telnet localhost 42302
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Now we are able to type anything which comes to our mind:


After a few words the server will respond with:

Connection closed by foreign host.

The server does logging per default and tells me what happened here:

2017/05/26 08:46:18 transport: http2Server.HandleStreams received bogus greeting from client: "Hello\r\nWorld\r\nTest\r\nTest"

This is correct, since GRPC is communicating via HTTP/2 my messages do not make any sense at all. The template also has a well-formatted command line help, which can be seen when running go run main.go -h:

> go run main.go -h
microservice 0.1.0
A microservice template
    main [global options] command [command options] [arguments...]
    This microservice provides...
    Prename Surname 
    help, h  Shows a list of commands or help for one command
    --port value, -p value         server port of the service (default: 42302)
    --usetls, -t                   use TLS if true, else plain TCP communication
    --certfile value, -c value     TLS certificate file
    --privkeyfile value, -k value  TLS private key file
    --logdir value, -d value       log directory
    --loglevel value, -l value     defines the log output level from Debug (0) to Quiet (5) (default: 0)
    --dry-run, -n                  do a dry run without starting the server
    --client-ip value              client IP to connect (default: "localhost")
    --client-port value            client port to connect (default: "42303")
    --help, -h                     show help
    --version, -v                  print the version
© 2017 quapona technologies GmbH

You want to use the template for your own projects? Great, all code parts which need your interaction are flagged with TODO comments, like the adaption of the contact details on the help command output:

> rg TODO src/cmd/service/main.go
20:     authorName     = "Prename Surname"               // TODO
21:     authorMail     = "p.surname@quapona.com"         // TODO
22:     appUsage       = "A microservice template"       // TODO
23:     appName        = "microservice"                  // TODO
24:     appDescription = "This microservice provides..." // TODO

The template structure

The general file structure of the Microservice looks like this:

├── ci/                                 // Continuous Integration related scripts and data
│   └── prepare                         // - Prepares the test environment
├── deploy/                             // Continuous Deployment related scripts and data:
│   ├── Dockerfile                      // - The input for the Docker image build process
│   ├── k8s/                            // - Preparation test deployments for Kubernetes
│   │   └── mongo.yml                   //   - Just an example mongodb deployment
│   └── Rkt.acb                         // - The input for the Rkt image build process
├── main.go -> src/cmd/service/main.go  // Handy symlink to the main executable command
├── Makefile                            // Wraps the whole build and test targets
├── protobuf/                           // Protobuf message and service definitions
│   └── microservice.proto              // - A hello world example for the microservice
├── README.md                           // The repository README
├── LICENSE.md                          // The license
└── src/                                // Every source code goes in here
    ├── bench_test.go                   // - Example benchmarks for performance testing
    ├── client.go                       // - Client definition to easily connect to this microservice
    ├── client_test.go                  //   - The unit tests for the client
    ├── cmd/                            // - The main directory for the different executables
    │   ├── demo/demo.go                //   - A demo client implementation
    │   └── service/main.go             //   - The microservice executable
    ├── integration_test.go             // - A sample integration test implementation
    ├── lib.go                          // - The main library logic
    ├── lib_test.go                     //   - The unit tests for the main library
    ├── mock_client/                    // - Generated mocking sources from mockgen
    │   └── mock_client.go              //   - Example client connection mocking
    ├── protobuf/                       // - Generated protocol buffer and GRPC output
    │   └── microservice.pb.go          //   - The output from `../../protobuf/microservice.proto`
    ├── rpc.go                          // - The actual RPC logic
    ├── rpc_test.go                     //   - The unit tests for the RPC logic
    └── testmain_test.go                // - Unit and integration test entry point

Source Code, Continuous Integration and Continuous Deployment scripts are separated to achieve maximum modularity here as well.

Connecting two Microservices

It is no problem to connect multiple Microservices by starting a server, listening on port 42303:

> make && ./deploy/main -p 42303
2017/05/26 14:25:06 Microservice - [STATUS]  StateRunning

The StateRunning indicates that the server is ready and listens for incoming connections or messages. To connect to this instance we can simply uncomment the block in lib.go:57-77 that should look like this:

if client.AccessInfo != nil { // The real connection path
    service.LogChan - "Connect to microservice"
    if err := client.Connect(client.AccessInfo); err != nil {
        service.ErrorChan - err
        service.StatusChan - server.StateError
        return nil
    service.LogChan - "Successfully connected to microservice!"
} else if client.MicroServiceClient == nil { // The failure path
    service.ErrorChan - fmt.Errorf("Microservice client equals nil")
    service.StatusChan - server.StateError
    return nil
} else { // The mocking path
    service.LogChan  make && ./deploy/main
2017/05/26 14:28:56 GRPC server: Listening on port 42302
2017/05/26 14:28:56 GRPC server: Prepare server options (without TLS)
2017/05/26 14:28:56 GRPC server: Creating new RPC server
2017/05/26 14:28:56 Microservice - [SERVICE] Setting up Service
2017/05/26 14:28:56 Microservice - [LOGGER]  Starting Server Logger
2017/05/26 14:28:56 Microservice - [LOGGER]  Started Server Logger
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to debug channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to error channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to status channel
2017/05/26 14:28:56 Microservice - [STATUS]  StateInitialized
2017/05/26 14:28:56 GRPC client: Initialize connection to grpc server
2017/05/26 14:28:56 GRPC client: Setup connection options
2017/05/26 14:28:56 GRPC client: Connect to server
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to warning channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to log channel
2017/05/26 14:28:56 Microservice - [INFO]    Initialized Server
2017/05/26 14:28:56 Microservice - [INFO]    Register microservice
2017/05/26 14:28:56 Microservice - [INFO]    Connect to microservice
2017/05/26 14:28:56 Microservice - [INFO]    Successfully connected to microservice!
2017/05/26 14:28:56 Microservice - [STATUS]  StateStarting
2017/05/26 14:28:56 Microservice - [STATUS]  StateStarted
2017/05/26 14:28:56 Microservice - [STATUS]  StateRunning

The log message Successfully connected to microservice! indicates that we now have a working interconnection of both Microservices.

A demo client

A demo client in src/cmd/demo/demo.go can be used to test an example GRPC message:

> go run main.go &
> go run ./src/cmd/demo/demo.go
2017/05/26 14:55:44 GRPC client: Initialize connection to grpc server
2017/05/26 14:55:44 GRPC client: Setup connection options
2017/05/26 14:55:44 GRPC client: Connect to server
Sending message:"Hello, world!"
Got response message:"Hello, world!"
Closed connection.
> fg

Great, our Microservice does nothing more than sending the request as a response back to the client. This is exactly the behavior the RPC implementation in src/rpc.go hints at:

// Hello returns a Response to the given Response
func (service *MicroService) Hello(ctx context.Context, request *protobuf.Request) (*protobuf.Response, error) {
    return &protobuf.Response{Message: request.GetMessage()}, nil

Building a container with Docker

If you have docker installed, simply run make all docker to build the statically linked executable, pack it in a small Alpine Linux image in deploy/microservice.tar. This tarball can be loaded with docker load or simply make dockerload. Then the image should be available within your local docker installation, which can be verified with docker images:

> docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
microservice-template   b9c4bbd             8ae4d2eed2ca        18 seconds ago      11.8MB
alpine                  latest              a41a7446062d        7 hours ago         3.97MB

The template is only approximately 12 megabyte large and contains the statically linked executable. This makes it pretty portable and easy to deploy. Running the image can simply be done via the docker run command:

> docker run -it microservice-template:b9c4bbd
2017/05/26 06:52:37 GRPC server: Listening on port 42302

Testing the Microservice

The unit tests should be able to fully run locally and can be executed with make utest. GoMock is used to simulate connections to other services by specifying input and output data. make lint uses gometalinter to apply a lot of code analysis tools to the local source code. The target make mtest just simulates the setup of the Microservice. You can think of it as a dry-run, which does not listen to incoming messages.

make itest runs the integration tests. This needs a running Kubernetes cluster and a working kubectl connection. Kubernetes can be installed locally for testing by using Minikube or installed remotely via kubeadm. Explaining Kubernetes is a little bit out of scope of this post, but there is an extensive documentation what you can do with this software beast. Furthermore a valid docker registry is needed to make the images available for Kubernetes. The integration tests will run isolated on a separate Kubernetes DNS namespace on the cluster to allow parallel testing. Integration tests should create all necessary components in Kubernetes, and then test it with a set of data as input and validate the output. These tests usually are the hardest to create, but are still needed to do real software quality assurance. The last quality assurance step of the Microservice will be done in a SCS test, which is not part of the template and will be done in a different repository.

GRPC and Protocol buffers

The connection to other services will be done using a GRPC Client and Server implementation, which uses protocol buffers (see the *.proto files in the proto folder) and TLS encrypted HTTP/2 to ensure a secure communication. This functionality is also included within our template. Just a reminder: every component needs to be exchangeable inside the Microservice as well to ensure the full power of modularity. So we could replace the GRPC stuff with other technologies like Apache Thrift if we want, which will not influence the core logic of the Microservice.

Finally the full testing toolchain looks like this:

CI Pipeline

Here, we execute some build tests in the first step to check if the source code is consistent and complete. Then comes the unit testing stage which should report the test coverage. The third step is the containerization step, which bundles the Microservice together with all needed data. Information like database credentials should stay outside of the container image so that they can be passed inside only when it is really necessary. The module tests can be skipped in some cases and mainly validate the binary within the container image. In the integration test all needed components of the Microservice will be put together and tested with data. This ensures the working interconnection to other dependencies.

Versioning Microservices

Following Semantic Versioning is an important key aspect when developing this way, to ensure stable connection interfaces between the different Microservices. This means in general that it should be safe to update the MINOR version of a dependency, when following the format MAJOR.MINOR.PATCH. A microservice should start at version 0.1.0, whereas everything lower than version 1.0.0 is intended to be unstable. The target is then to develop a stable 1.0.0 version which ensures the overall reliability of the SCS.

The whole Self-Contained System will later be tested in a separate repository and then deployed with the tested combination of Microservices. It should then be possible to reuse the Microservices in other SCS if you succeed in modularity and abstraction within your application logic.

Performance testing

It is important to keep track of the overall performance of a Self-Contained System, especially when the application grows and the number of Microservices increases over time. Testing the single Microservice template can be done via make bench, which runs two example tests: A single-threaded and a multi-threaded client server communication. On our local test machines a client-to-server RPC communication took about 33000 nanoseconds, which results in approximately 30000 RPCs per second. The overall networking overhead is virtually nonexistent in this test since we run the client and the server locally. Our measurement with real network connection and a single-server instance showed up around 700 RPCs per second, which sounds more realistic. Kubernetes has a number of features like replication and horizontal autoscaling to target performance bottlenecks in production environments. The knowledge about the theoretically achievable performance of the Microservice is important to adjust such settings in a meaningful way. This means the performance of the single Microservice should be measured continuously to detect performance regressions on a daily basis as early as possible. To put it all in a nutshell the overall SCS performance should be measured like this as well. This helps to identify the hot paths within the SCS and get to know where to adjust the right settings.

Kommentare powered by CComment

Beitrag teilen