Golang basics - fetch JSON from an API

This is a recipe in Golang for making a "GET" request over HTTP to an API on the Internet. We will be querying an endpoint provided for free that tells us how many astronauts are currently in space and what their names are.

If you're already familiar with Go, then you may want to skip the first section.

Pre-requisites

  • Install Go from https://golang.org/dl/
  • Check your Go installation with go version
    • I'm using 1.7.3
  • Set your GOPATH, probably to $HOME/go/
    • export GOPATH=$HOME/go/
    • If you want the change to stick, add the line to .bash_rc or .profile.
    • In Go 1.8 onwards this step will not be needed and is automatic.

We are going to use the Golang standard library which has a number of useful components for HTTP and JSON.

A blank template

I have had good experiences using nano and vim, but Visual Studio Code is probably the easiest to setup with a good Go experience.

If you're using Visual Studio Code, let it install the plugins it suggests such as gofmt because they will ensure your code looks good.

Make yourself a folder for a new project (you can use your Github username here):

mkdir -p $GOPATH/src/github.com/alexellis/blank/  

Now save a new file named app.go:

package main

func main() {

}

This program is effectively hello-world without any print statements. You can type in go run app.go or go build followed by ./blank.

Parsing JSON

Let's go on to parse some JSON, in Go we can turn a JSON document into a struct which is useful for accessing the data in a structured way. If a document doesn't fit into the structure it will throw an error.

Let's take an API from the Open Notify group - it shows the of people in space and their names.

People in Space JSON

It looks a bit like this:

{"people": [{"craft": "ISS", "name": "Sergey Rizhikov"}, {"craft": "ISS", "name": "Andrey Borisenko"}, {"craft": "ISS", "name": "Shane Kimbrough"}, {"craft": "ISS", "name": "Oleg Novitskiy"}, {"craft": "ISS", "name": "Thomas Pesquet"}, {"craft": "ISS", "name": "Peggy Whitson"}], "message": "success", "number": 6}

Create a new project:

mkdir -p $GOPATH/src/github.com/alexellis/json1/  

We will be import the fmt package for the Println function and the encoding/json package so that we can Unmarshal the JSON text into a struct.

If you're coming from Python, Ruby or Node Unmarshal is similar to JSON.parse() but with type checking.

package main

import (  
    "encoding/json"
    "fmt"
)

type people struct {  
    Number int `json:"number"`
}

func main() {  
    text := `{"people": [{"craft": "ISS", "name": "Sergey Rizhikov"}, {"craft": "ISS", "name": "Andrey Borisenko"}, {"craft": "ISS", "name": "Shane Kimbrough"}, {"craft": "ISS", "name": "Oleg Novitskiy"}, {"craft": "ISS", "name": "Thomas Pesquet"}, {"craft": "ISS", "name": "Peggy Whitson"}], "message": "success", "number": 6}`
    textBytes := []byte(text)

    people1 := people{}
    err := json.Unmarshal(textBytes, &people1)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(people1.Number)
}

We have hard-coded the JSON response from Open Notify so that we can work on one chunk of behaviour at a time.

The json.Unmarshall function works with a []byte type instead of a string so we use []byte(stringHere) to create the format we need.

In order to make use of a struct to unmarshal the JSON text we normally need to decorate it with some tags that help the std library understand how to map the properties:

type people struct {  
    Number int `json:"number"`
}

The property names need to begin with a capital letter which marks them as exportable or public. If your struct's property is the same in the JSON you should be able to skip the annotation.

Another thing to notice is that we pass the address of the new / empty people struct into the method. You can try removing the & symbol, but the value will be set in a different copy of the empty struct.

This may seem odd if you are coming from languages that pass parameters by reference.

Putting it together with HTTP

Now we can parse a JSON document matching that of our API, let's go on and write a HTTP client to fetch the text from the Internet.

Go has a built-in HTTP client in the net/http package, but it has a problem with long timeouts and there are some well-known articles recommending that you set a timeout on your request explicitly.

There are more concise ways of creating a HTTP request in Go, but by adding a custom timeout we will harden our application.

Create a new project:

mkdir -p $GOPATH/src/github.com/alexellis/spacecount/  
package main

import (  
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "time"
)

type people struct {  
    Number int `json:"number"`
}

func main() {

    url := "http://api.open-notify.org/astros.json"

    spaceClient := http.Client{
        Timeout: time.Second * 2, // Maximum of 2 secs
    }

    req, err := http.NewRequest(http.MethodGet, url, nil)
    if err != nil {
        log.Fatal(err)
    }

    req.Header.Set("User-Agent", "spacecount-tutorial")

    res, getErr := spaceClient.Do(req)
    if getErr != nil {
        log.Fatal(getErr)
    }

    body, readErr := ioutil.ReadAll(res.Body)
    if readErr != nil {
        log.Fatal(readErr)
    }

    people1 := people{}
    jsonErr := json.Unmarshal(body, &people1)
    if jsonErr != nil {
        log.Fatal(jsonErr)
    }

    fmt.Println(people1.Number)
}

If you're used to coding in Node.js for instance you will be used to most I/O operations returning a nullable error - the same is the case in Go and checking for errors here has made the code quite verbose. You may want to refactor elements of the code into separate methods - one to download and one unmarshal the JSON data.

It may not always make sense to attempt to parse the HTTP response, one case is if the HTTP code returned is a non-200 number. Here's how you can access the HTTP code from the response:

    fmt.Printf("HTTP: %s\n", res.Status)

Being a good citizen on the Internet

There's one more important line I wanted to highlight. I've set a User-Agent in the HTTP request's header. This lets remote servers understand what kind of traffic it is receiving. Some sites will even reject empty or generic User-Agent strings.

    req.Header.Set("User-Agent", "spacecount-tutorial")

Taking it further

I wanted to keep this post focused but do checkout the links for reading up and come back soon for follow-up posts.

Next time

In the next post we'll approach the problem again with TDD (Test Driven Development) and unit testing in mind. We will also build out a Dockerfile using the Official golang Docker image.

Reading up

If you're starting out with Golang and are already familiar with other languages then I'd recommend The Go Programming Language Phrasebook by David Chisnall. If you're hacking on projects already then something like Bill Kennedy's Go in Action will provide a great foundation.

And finally there are the Golang docs, which you may have found already. I personally like having a book but found this site essential.

Feedback

If you have questions or have found an error, please get in touch on Twitter @alexellisuk.

Alex Ellis

Read more posts by this author.

United Kingdom http://alexellis.io/

Subscribe to alex ellis' blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!