gomongo part 2

golang microservice on mongo inside docker

Environment used in this example:

  • host: MacOS 10.10.5
  • coreOS: alpha (1000.0.0)
  • Docker: version 1.10.3
  • docker-compose: version 1.6.2

End-to-end example: part2

Simple microservice written in golang, reading data from mongodb, packed into docker and linked used docker-compose.

In this series we will be building a very simple http REST interface on top of a small, filled with geo-data mongodb. You will see how to start thinking about things like automated deployment and continuous delivery in early stages of development. We will also look into docker-compose and how can it help us provide easy integration test sandbox.

For the impatient ones - the final code of these series is available here

git clone git@github.com:envimate/gomongo.git

I will be using tags to indicate step-by-step evolvement of the code.

We will go step by step, building up to an http server serving various requests towards mongodb instance. We’ll package that all into docker. We will run our simple http-call-tests towards a dockerized mongo instance, using docker-compose to describe and run our service together with mongodb instance.

As test data we will use geo data which was put together using http://www.naturalearthdata.com/downloads/ and https://www.maxmind.com/en/open-source-data-and-api-for-ip-geolocation and is available under the repository.

golang http server

In the first part we saw how to build a simple http server in go, write a build script for it and pack it up in a docker image.

the code of the first part is: https://github.com/envimate/gomongo/tree/part1

git checkout tags/part1 -b part1

In this part we will expand the functionality of our http server and also connect the go http server to a mongo instance.

here is the first-step code for the second part: https://github.com/envimate/gomongo/tree/part2_mux

git checkout tags/part2_mux -b part2_mux

Let’s go step by step into how this was developed.

First let’s enable serving different path url’s from our handler. Our goal is to have 2 endpoints - /city serving requests that will go to the city collection and /country - serving countries from the database.

In order to separate the handlers for those 2, we’ll built a small multiplexer - a mapping of url paths to handler functions.


package main

import (
    "log"
    "net/http"
    "flag"
    "fmt"
    "io"
)

var port int

func init() {
    flag.IntVar(&port, "port", 8080, "Port on which to listen")
    flag.Parse()
}

var mux map[string]func(http.ResponseWriter, *http.Request)

func defaultHandler(rw http.ResponseWriter, request *http.Request){
    io.WriteString(rw, "handler called for URI \n")
    io.WriteString(rw, request.RequestURI)
}

func main() {
    log.Println("Starting server on port", port)
    s := &http.Server {
        Addr: fmt.Sprintf(":%d", port),
        Handler: &MongoHandler{},
    }

    mux = make(map[string]func(http.ResponseWriter, *http.Request))

    mux["/"] = defaultHandler

    log.Fatal(s.ListenAndServe())
}

type MongoHandler struct {
}

func (mhandler *MongoHandler) ServeHTTP(rw http.ResponseWriter, request *http.Request) {
    if handler, ok := mux[request.URL.Path]; ok {
        handler(rw, request)
        return
    }
    io.WriteString(rw, "got requestURI "+request.URL.String()+"\n")
}

The highlighted lines give us:

  • a map between a string (path of the request) to the handler function (in this case the defaultHandler)
  • a default handler that prints the request URI into response
  • modified ServeHTTP method that picks the appropriate handler from the map (if present) and passes the processing of the request to it

Go ahead and try to run this.

# ./build.sh
gomongo
# ./bin/gomongo
2016/04/03 23:15:05 Starting server on port 8080

The result is almost the same, except we print it out as the response.


# curl http://localhost:8080/test
got request /test

But now we can go ahead and add a handler for our /city requests and isolate the logic of serving it to a separate method.

adding the handler to the map:


mux["/city"] = cityIdHandler

adding some logic to the city handler:



func cityIdHandler(rw http.ResponseWriter, request *http.Request){
    query := request.URL.Query()
    io.WriteString(rw, query.Get("id")+"\n")
}


The goal here is to be able to answer requests containing query based on the parameters of the City object we have in database.

go to mongo

There is no officially supported golang driver for mongodb, but we are going to use the one listed as a community driver on official mongodb page https://docs.mongodb.org/ecosystem/drivers/go/

You can find details about the project on their website

To start using the library we need to download the dependency. I’ll add the go get for this library directly into our build.sh and run it once


#!/bin/bash
export GOPATH=$(dirname $(readlink -f $0))

go get gopkg.in/mgo.v2
go install -v gomongo

If you noticed, I also change the GOPATH declaration to use dirname to be independant from where it is being run.

As soon as we run the build.sh again you’ll notice that the dependency has now appeared under the src directory


$ tree src/
src/
├── gomongo
│   └── gomongo.go
└── gopkg.in
    └── mgo.v2
        ├── LICENSE
        ├── Makefile
        ...

Now we can go ahead and add an import statement in our gomongo.go file. The whole import statement now looks like this


import (
    "log"
    "net/http"
    "flag"
    "fmt"
    "io"
    "gopkg.in/mgo.v2"
)

We are ready to initialize connections to MongoDB and get our handlers actually bring something from the database.


var mongourl string

func init() {
    flag.IntVar(&port, "port", 8080, "Port on which to listen")
    flag.StringVar(&mongourl, "mongourl", "mongodb://localhost:27017", "the mongodb connection url")
    flag.Parse()
}


func main() {
    log.Println("Starting server on port", port)

    session, err := mgo.Dial(mongourl)
    if err != nil {
        panic(err)
    }
    defer session.Close()

    db := session.DB("geo")

    mh := &MongoHandler{db}
    s := &http.Server {
        Addr: fmt.Sprintf(":%d", port),
        Handler: mh,
    }

    mux = make(map[string]func(http.ResponseWriter, *http.Request))

    mux["/"] = defaultHandler
    mux["/city"] = mh.cityIdHandler

    log.Fatal(s.ListenAndServe())
}

type MongoHandler struct {
    db *mgo.Database
}

You can see that here I added:

  • Mongourl as another parameter in the start
  • Dialing into mongoDB with provided url, deferring the close (this will be called by the end of main() method)
  • Retrieving our geo database and passing it along to MongoHandler
  • A pointer to the database in the MongoHandler struct

In order to get access to the database session, our cityIdHandler would need to be able to use the MongoHandler instance. Hence, I’ve changed the function cityIdHandler into a method of MongoHandler type:


func (mhandler *MongoHandler) cityIdHandler(rw http.ResponseWriter, request *http.Request){
    cityColl := mhandler.db.C("city")

    query := request.URL.Query()
    cityId := query.Get("id")
    io.WriteString(rw, cityId +"\n")

    result := City{}

    id, _ := strconv.ParseInt(cityId, 10, 64)

    err := cityColl.FindId(id).One(&result)

    if err != nil {
        io.WriteString(rw, "City with id "+cityId +" not found\n")
        log.Println(err)
        return
    }

    io.WriteString(rw, "Found city " + result.name +"\n")
}

type City struct {
    Id int64
    Name string
}

Let’s see what we have now:

  • Getting the required collection from the database (in this case city)
  • Parsing the id received in the request query to a 64 bit integer (the _id we have in the database is a NumberLong type)
  • Querying the database with the ID and reading the result

Be careful to name the fields of the struct City with an uppercase, otherwise they are not visible outside the package declared.

docker-compose

Now to run and test this, we’ll need a mongoDB instance running. We’ll introduce a docker-compose file here, which will bind to services together - our service with the mongodb server. You can read more about docker-compose on the official website, where you can also find the detailed installation guide

I’ll place the docker-compose.yml to the docker folder and here is the content of it:


version: '2'
services:
    gomongo:
        build:
            context: ../
            dockerfile: docker/Dockerfile
        ports:
            - "80:8080"
        links:
            - mongodb
        restart: always
        container_name: gomongo
    mongodb:
        image: "mongo"

Let’s go section-by-section:

  • Using version 2 of the docker-compose file. You can read more about the differences here
  • Defining 2 services: gomongo, which is build using the Dockerfile we created previously, maps the 80 port of the host to the port 8080 port of the container just like before, and is linked to
  • mongodb service, that is built from the public mongo image (which will be downloaded by the docker-compose)

The linking of the containers works in the following way - based on the name of the service, it will be added in /etc/hosts file in the linked services. Hence we need to specify the parameter mongourl for our service as mongodb://mongodb:27017 For that, let’s modify the Dockerfile

Instead of pointing the Docker to our executable as an entry point, we’ll run the shell with our executable and the mongourl as an environmental variable


ENTRYPOINT ["/bin/sh", "-c", "exec /go/bin/gomongo -mongourl $MONGO_URL"]

Providing env variables in docker compose is straightforward, just adding this line in the section for our service:


environment:
    - MONGO_URL=mongodb://mongodb:27017

It’s time to run our services! Running docker-compose build and pointing it to our yaml confgiration file:


core@core-01 ~/mongo-go $ docker-compose -f docker/docker-compose.yml build

This will build the Dockerfile for our service, as well as download the image for mongodb.

To run the service group :


core@core-01 ~/mongo-go $ docker-compose -f docker/docker-compose.yml run --service-ports -d gomongo
Creating docker_mongodb_1
docker_gomongo_run_7

core@core-01 ~/mongo-go $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
bf0a16075352        docker_gomongo      "/bin/sh -c 'exec /go"   3 seconds ago       Up 2 seconds        0.0.0.0:80->8080/tcp   docker_gomongo_run_7
bb31d032756d        mongo               "/entrypoint.sh mongo"   3 seconds ago       Up 2 seconds        27017/tcp              docker_mongodb_1

the --service-ports is to expose ports to the host, -d for daemon mode.

now, we can try our requests to city endpoint:


core@core-01 ~/mongo-go $ curl http://localhost/city?id=404611709660041216
404611709660041216
City with id 404611709660041216 not found

importing data

The last bit in the picture is bringing the test-data we have to the mongodb container. For that, we’ll create a small image on it’s own, that will connect to the mongodb image, import the test-data and exit.

Extending Dockerfile from mongo in docker/mongo_Dockerfile:


FROM mongo

ADD mongo-data/dump/geo /initial-data/geo

CMD mongorestore -h mongodb -d geo /initial-data/geo

Adding this to our docker-compose.yml now looks like this:


version: '2'
services:
    gomongo:
        build:
            context: ../
            dockerfile: docker/Dockerfile
        ports:
            - "80:8080"
        depends_on:
            - mongodb_import
        restart: always
        container_name: gomongo
        environment:
            - MONGO_URL=mongodb://mongodb:27017
    mongodb_import:
        build:
            context: ../
            dockerfile: docker/mongo_Dockerfile
        depends_on:
            - mongodb
    mongodb:
        image: "mongo"


Notice, I also changed the linked option to depends_on to indicate the order of services being launched.

building and running the service:


core@core-01 ~/mongo-go $ docker-compose -f docker/docker-compose.yml build gomongo
...

core@core-01 ~/mongo-go $ docker-compose -f docker/docker-compose.yml up -d gomongo

Instead of run with --service-ports, I used the up command.

now trying the request again:


core@core-01 ~/mongo-go $ curl http://localhost/city?id=404611709660041216
404611709660041216
Found city les Escaldes

So, we have improved our http server to reroute requests to their respective handlers, we have successfully connected our service to mongodb, and made first query based on the request parameters. We tested that all using a composition of our service, service populating mongodb with our test-data and a mongodb docker instance.

The final version of what we discussed here can be found under: https://github.com/envimate/gomongo/tree/part2_docker-compose

git checkout tags/part2_docker-compose -b part2_docker-compose
More Reading