Practical Guide to Self-Contained Systems
Today we live in a containerized world. Applications are getting larger and larger but still need to be as modular as possible. This is a major challenge for companies, especially for big corporations. Modularity will boost companies by allowing smaller development teams and giving the possibility to reuse application parts for different kinds of products. This will work out of the box if every piece of software is continuously and automatically tested, whereas the interface between the different application parts needs to be well-defined.
At quapona we live this approach and call these solutions Self-Contained Systems (SCS). A SCS should work as a whole software solution for a certain problem by splitting it into single parts.
One Self-Contained System could consist of the storage solution (Data), the storage driver (Logic) and the application which generates the data to be stored (Interfaces). You can think of the certain parts as Microservices, which will be connected together to a single working unit. The main challenge is to decide which functionality belongs to the Microservice and which does not. This automatically leads into iterative development since Microservices are getting larger and can thus be split into multiple ones. A good company communication culture is mandatory in order to make this work as developers always need to rethink their work.
General Microservice conditions
Every Microservice can basically do whatever is required to get its job done, but there are still a few rules to be considered:
- The interface technology for Microservices within a SCS has to be the same
- Every Microservice needs to be fully documented
- Every Microservice must be tested with unit, module and integration tests
- The API of a Microservice needs to follow Semantic Versioning
- Manual testing and Code Review is a must-have
At quapona we call the combination of these rules Continuous Quality Assurance (CQA). The excessive testing of the application is one of our key drivers to ensure the portability of the Microservice and keep Self-Contained Systems stable and the data reliable. One of the main mistakes companies are making is to safe money in testing which leads into cruel software solutions everyone might have examples for: They are monolithic, they mostly solve problems of only a few use cases and they eventually cost way too much money when they have to be extended or maintained.
Besides the general CQA approach, it is also vital to abstract the implementation of the Microservice to be as general as possible. This enables making the Microservice (or the whole solution) Open Source, which allows feedback from all over the world and endowes other developers with the option to reuse the solution. The sensitivity for providing fulfilling documentation and testing is much higher when developers consider making the software Open Source than to just develop for a small team where they know each other too well. The feedback you will get from the certain community is to be worth a mint. There are lots of experts out there which can be reached instantly by today.
A Microservice template
How to achieve this? Which interface is to be used and how to deploy the application? Which language should I choose? Should I write the communication interface between two Microservices on my own? No. Another big flaw by companies is to write everything from scratch. It should also be avoided to rely too hard on existing solutions to maintain the necessary flexibility of your business. The best approach is to always keep the eyes open for new technologies by picking the right solutions for your needs without creating a second world around them where you might get stuck. Our example of a Microservice template uses the following, well-known technologies:
- Go: As main programming language
- GRPC: As an interface to another Microservice
- GoMock: As mocking framework for unit tests
- Docker: As containerization solution
- Kubernetes: As module and integration testing, container orchestration and deployment solution
What is outstanding here is that a lot of named technologies are made by a company called Google. Yeah, that is true, Google provides lots of great open Source Solutions which might help smaller companies to be successful. But keep in mind: Every part of the Microservice needs to be exchangeable if necessary. For example our template provides the ability to switch from Docker to rkt, another container engine developed by CoreOS.
Using the template
Here we go, our Microservice template can easily be retrieved via GitHub:
> git clone https://github.com/quaponatech/microservice-template
Some software requirements have
to be installed on your development system if you want to use the templates' full capabilities. To test the
functionality of the Microservice template, simply run go run main.go in the root directory of the repository. Then
you should see the following output:
2017/05/26 08:43:16 GRPC server: Listening on port 42302
2017/05/26 08:43:16 GRPC server: Prepare server options (without TLS)
2017/05/26 08:43:16 GRPC server: Creating new RPC server
2017/05/26 08:43:16 Microservice - [SERVICE] Setting up Service
2017/05/26 08:43:16 Microservice - [LOGGER] Starting Server Logger
2017/05/26 08:43:16 Microservice - [LOGGER] Started Server Logger
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to debug channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to warning channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to log channel
2017/05/26 08:43:16 Microservice - [INFO] Initialized Server
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to error channel
2017/05/26 08:43:16 Microservice - [CHANNEL] Listening to status channel
2017/05/26 08:43:16 Microservice - [STATUS] StateInitialized
2017/05/26 08:43:16 Microservice - [INFO] Register microservice
2017/05/26 08:43:16 Microservice - [STATUS] StateStarting
2017/05/26 08:43:16 Microservice - [STATUS] StateStarted
2017/05/26 08:43:16 Microservice - [STATUS] StateRunning
We now have a fully working GRPC enabled server running and waiting for incoming connections and messages. This can be tested easily via Telnet:
> telnet localhost 42302
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Now we are able to type anything which comes to our mind:
Hello
World
Test
Test
After a few words the server will respond with:
Connection closed by foreign host.
The server does logging per default and tells me what happened here:
2017/05/26 08:46:18 transport: http2Server.HandleStreams received bogus greeting from client: "Hello\r\nWorld\r\nTest\r\nTest"
This is correct, since GRPC is communicating via HTTP/2 my messages do not make
any sense at all. The template also has a well-formatted command line help, which can be seen when running
go run main.go -h:
> go run main.go -h
microservice 0.1.0
A microservice template
USAGE:
main [global options] command [command options] [arguments...]
DESCRIPTION:
This microservice provides...
AUTHOR:
Prename Surname
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--port value, -p value server port of the service (default: 42302)
--usetls, -t use TLS if true, else plain TCP communication
--certfile value, -c value TLS certificate file
--privkeyfile value, -k value TLS private key file
--logdir value, -d value log directory
--loglevel value, -l value defines the log output level from Debug (0) to Quiet (5) (default: 0)
--dry-run, -n do a dry run without starting the server
--client-ip value client IP to connect (default: "localhost")
--client-port value client port to connect (default: "42303")
--help, -h show help
--version, -v print the version
© 2017 quapona technologies GmbH
You want to use the template for your own projects? Great, all code parts which need your interaction are flagged with
TODO comments, like the adaption of the contact details on the help command output:
> rg TODO src/cmd/service/main.go
20: authorName = "Prename Surname" // TODO
21: authorMail = "p.surname@quapona.com" // TODO
22: appUsage = "A microservice template" // TODO
23: appName = "microservice" // TODO
24: appDescription = "This microservice provides..." // TODO
The template structure
The general file structure of the Microservice looks like this:
.
├── ci/ // Continuous Integration related scripts and data
│ └── prepare // - Prepares the test environment
│
├── deploy/ // Continuous Deployment related scripts and data:
│ ├── Dockerfile // - The input for the Docker image build process
│ ├── k8s/ // - Preparation test deployments for Kubernetes
│ │ └── mongo.yml // - Just an example mongodb deployment
│ └── Rkt.acb // - The input for the Rkt image build process
│
├── main.go -> src/cmd/service/main.go // Handy symlink to the main executable command
│
├── Makefile // Wraps the whole build and test targets
│
├── protobuf/ // Protobuf message and service definitions
│ └── microservice.proto // - A hello world example for the microservice
│
├── README.md // The repository README
├── LICENSE.md // The license
│
└── src/ // Every source code goes in here
├── bench_test.go // - Example benchmarks for performance testing
├── client.go // - Client definition to easily connect to this microservice
├── client_test.go // - The unit tests for the client
├── cmd/ // - The main directory for the different executables
│ ├── demo/demo.go // - A demo client implementation
│ └── service/main.go // - The microservice executable
│
├── integration_test.go // - A sample integration test implementation
├── lib.go // - The main library logic
├── lib_test.go // - The unit tests for the main library
├── mock_client/ // - Generated mocking sources from mockgen
│ └── mock_client.go // - Example client connection mocking
├── protobuf/ // - Generated protocol buffer and GRPC output
│ └── microservice.pb.go // - The output from `../../protobuf/microservice.proto`
├── rpc.go // - The actual RPC logic
├── rpc_test.go // - The unit tests for the RPC logic
└── testmain_test.go // - Unit and integration test entry point
Source Code, Continuous Integration and Continuous Deployment scripts are separated to achieve maximum modularity here as well.
Connecting two Microservices
It is no problem to connect multiple Microservices by starting a server, listening on port 42303:
> make && ./deploy/main -p 42303
...
2017/05/26 14:25:06 Microservice - [STATUS] StateRunning
The StateRunning indicates that the server is ready and listens for incoming connections or messages. To connect to
this instance we can simply uncomment the block in lib.go:57-77 that should look like this:
if client.AccessInfo != nil { // The real connection path
service.LogChan - "Connect to microservice"
if err := client.Connect(client.AccessInfo); err != nil {
service.ErrorChan - err
service.StatusChan - server.StateError
service.Stop()
return nil
}
service.LogChan - "Successfully connected to microservice!"
} else if client.MicroServiceClient == nil { // The failure path
service.ErrorChan - fmt.Errorf("Microservice client equals nil")
service.StatusChan - server.StateError
service.Stop()
return nil
} else { // The mocking path
service.LogChan make && ./deploy/main
2017/05/26 14:28:56 GRPC server: Listening on port 42302
2017/05/26 14:28:56 GRPC server: Prepare server options (without TLS)
2017/05/26 14:28:56 GRPC server: Creating new RPC server
2017/05/26 14:28:56 Microservice - [SERVICE] Setting up Service
2017/05/26 14:28:56 Microservice - [LOGGER] Starting Server Logger
2017/05/26 14:28:56 Microservice - [LOGGER] Started Server Logger
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to debug channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to error channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to status channel
2017/05/26 14:28:56 Microservice - [STATUS] StateInitialized
2017/05/26 14:28:56 GRPC client: Initialize connection to grpc server
2017/05/26 14:28:56 GRPC client: Setup connection options
2017/05/26 14:28:56 GRPC client: Connect to server
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to warning channel
2017/05/26 14:28:56 Microservice - [CHANNEL] Listening to log channel
2017/05/26 14:28:56 Microservice - [INFO] Initialized Server
2017/05/26 14:28:56 Microservice - [INFO] Register microservice
2017/05/26 14:28:56 Microservice - [INFO] Connect to microservice
2017/05/26 14:28:56 Microservice - [INFO] Successfully connected to microservice!
2017/05/26 14:28:56 Microservice - [STATUS] StateStarting
2017/05/26 14:28:56 Microservice - [STATUS] StateStarted
2017/05/26 14:28:56 Microservice - [STATUS] StateRunning
The log message Successfully connected to microservice! indicates that we now have a working interconnection of both
Microservices.
A demo client
A demo client in src/cmd/demo/demo.go can be used to test an example GRPC message:
> go run main.go &
> go run ./src/cmd/demo/demo.go
2017/05/26 14:55:44 GRPC client: Initialize connection to grpc server
2017/05/26 14:55:44 GRPC client: Setup connection options
2017/05/26 14:55:44 GRPC client: Connect to server
Sending message:"Hello, world!"
Got response message:"Hello, world!"
Closed connection.
> fg
Great, our Microservice does nothing more than sending the request as a response back to the client. This is exactly the
behavior the RPC implementation in src/rpc.go hints at:
// Hello returns a Response to the given Response
func (service *MicroService) Hello(ctx context.Context, request *protobuf.Request) (*protobuf.Response, error) {
return &protobuf.Response{Message: request.GetMessage()}, nil
}
Building a container with Docker
If you have docker installed, simply run make all docker to build the statically linked executable, pack it
in a small Alpine Linux image in deploy/microservice.tar. This tarball can be loaded with
docker load or simply make dockerload. Then the image should be available within your local docker installation,
which can be verified with docker images:
> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
microservice-template b9c4bbd 8ae4d2eed2ca 18 seconds ago 11.8MB
alpine latest a41a7446062d 7 hours ago 3.97MB
The template is only approximately 12 megabyte large and contains the statically linked executable. This makes it pretty
portable and easy to deploy. Running the image can simply be done via the docker run command:
> docker run -it microservice-template:b9c4bbd
2017/05/26 06:52:37 GRPC server: Listening on port 42302
Testing the Microservice
The unit tests should be able to fully run locally and can be executed with make utest. GoMock is used to simulate
connections to other services by specifying input and output data. make lint uses
gometalinter to apply a lot of code analysis tools to the local source
code. The target make mtest just simulates the setup of the Microservice. You can think of it as a dry-run, which
does not listen to incoming messages.
make itest runs the integration tests. This needs a running Kubernetes cluster and a working
kubectl connection. Kubernetes can be installed locally
for testing by using Minikube or installed remotely via
kubeadm. Explaining Kubernetes is a little bit out of
scope of this post, but there is an extensive documentation what you can do with this software beast. Furthermore a
valid docker registry is needed to make the images available for Kubernetes. The integration tests will run isolated on
a separate Kubernetes DNS namespace on the cluster to allow parallel testing. Integration tests should create all necessary
components in Kubernetes, and then test it with a set of data as input and validate the output. These tests usually are the
hardest to create, but are still needed to do real software quality assurance. The last quality assurance step of the
Microservice will be done in a SCS test, which is not part of the template and will be done in a different repository.
GRPC and Protocol buffers
The connection to other services will be done using a GRPC Client and Server implementation, which uses protocol buffers
(see the *.proto files in the proto folder) and TLS encrypted HTTP/2 to
ensure a secure communication. This functionality is also included within our template. Just a reminder: every component
needs to be exchangeable inside the Microservice as well to ensure the full power of modularity. So we could replace the
GRPC stuff with other technologies like Apache Thrift if we want, which will not influence
the core logic of the Microservice.
Finally the full testing toolchain looks like this:
Here, we execute some build tests in the first step to check if the source code is consistent and complete. Then comes the unit testing stage which should report the test coverage. The third step is the containerization step, which bundles the Microservice together with all needed data. Information like database credentials should stay outside of the container image so that they can be passed inside only when it is really necessary. The module tests can be skipped in some cases and mainly validate the binary within the container image. In the integration test all needed components of the Microservice will be put together and tested with data. This ensures the working interconnection to other dependencies.
Versioning Microservices
Following Semantic Versioning is an important key aspect when developing this way, to ensure stable
connection interfaces between the different Microservices. This means in general that it should be safe to update the MINOR
version of a dependency, when following the format MAJOR.MINOR.PATCH. A microservice should start at version 0.1.0,
whereas everything lower than version 1.0.0 is intended to be unstable. The target is then to develop a stable 1.0.0
version which ensures the overall reliability of the SCS.
The whole Self-Contained System will later be tested in a separate repository and then deployed with the tested combination of Microservices. It should then be possible to reuse the Microservices in other SCS if you succeed in modularity and abstraction within your application logic.
Performance testing
It is important to keep track of the overall performance of a Self-Contained System, especially when the application
grows and the number of Microservices increases over time. Testing the single Microservice template can be done via
make bench, which runs two example tests: A single-threaded and a multi-threaded client server communication. On our
local test machines a client-to-server RPC communication took about 33000 nanoseconds, which results in approximately 30000
RPCs per second. The overall networking overhead is virtually nonexistent in this test since we run the client and the
server locally. Our measurement with real network connection and a single-server instance showed up around 700 RPCs per
second, which sounds more realistic. Kubernetes has a number of features like replication and horizontal autoscaling to
target performance bottlenecks in production environments. The knowledge about the theoretically achievable performance
of the Microservice is important to adjust such settings in a meaningful way. This means the performance of the single
Microservice should be measured continuously to detect performance regressions on a daily basis as early as possible.
To put it all in a nutshell the overall SCS performance should be measured like this as well. This helps to identify the
hot paths within the SCS and get to know where to adjust the right settings.
Als deutsches Forschungs- und Entwicklungsunternehmen unterstützen wir aktiv die Forschungslandschaft in Deutschland. Hierfür wurde quapona technologies mit dem Forschungssiegel des