Rezha Julio

Hi!
My name is Rezha Julio
I am a chemist graduate from Bandung Institute of Technology. Currently working as Data Engineer at Traveloka.
You can reach me by email:

contact@rezhajulio.id

, @ Q Q

Powered by Hugo

filter by tags

Using BoltDB for your Go Application

time to read 6 min | 1138 words

BoltDB is a pure Go persistence solution that saves data to a memory mapped file. I call it a persistence solution and not a database, because the word database has a lot of baggage associated with it that doesn’t apply to bolt. And that lack of baggage is what makes bolt so awesome.

Bolt is just a Go package. There’s nothing you need to install on the system, no configuration to figure out before you can start coding, nothing. You just go get github.com/boltdb/bolt and then import “github.com/boltdb/bolt”.

All you need to fully use bolt as storage is a file name. This is fantastic from both a developer’s point of view, and a user’s point of view. I don’t know about you, but I’ve spent months of work time over my career configuring and setting up databases and debugging configuration problems, users and permissions and all the other crap you get from more traditional databases like Postgres and Mongo. There’s none of that with bolt. No users, no setup, just a file name. This is also a boon for users of your application, because they don’t have to futz with all that crap either.

Bolt is not a relational database. It’s not even a document store, though you can sort of use it that way. It’s really just a key/value store… but don’t worry if you don’t really know what that means or how you’d use that for storage. It’s super simple and it’s incredibly flexible. Let’s take a look.

Storage in bolt is divided into buckets. A bucket is simply a named collection of key/value pairs, just like Go’s map. The name of the bucket, the keys, and the values are all of type []byte. Buckets can contain other buckets, also keyed by a []byte name.

… that’s it. No, really, that’s it. Bolt is basically a bunch of nested maps. And this simplicity is what makes it so easy to use. There’s no tables to set up, no schemas, no complex querying language to struggle with. Let’s look at a bolt hello world:

package main

import (
    “fmt”
    “log”

    “github.com/boltdb/bolt”
)

var world = []byte(“world”)

func main() {
    db, err := bolt.Open(“/home/rezha/go/bolt.db”, 0644, nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    key := []byte(“hello”)
    value := []byte(“Hello World!”)

    // store some data
    err = db.Update(func(tx *bolt.Tx) error {
        bucket, err := tx.CreateBucketIfNotExists(world)
        if err != nil {
            return err
        }

        err = bucket.Put(key, value)
        if err != nil {
            return err
        }
        return nil
    })

    if err != nil {
        log.Fatal(err)
    }

    // retrieve the data
    err = db.View(func(tx *bolt.Tx) error {
        bucket := tx.Bucket(world)
        if bucket == nil {
            return fmt.Errorf(“Bucket %q not found!”, world)
        }

        val := bucket.Get(key)
        fmt.Println(string(val))

        return nil
    })

    if err != nil {
        log.Fatal(err)
    }
}

// output:
// Hello World!

I know what you’re thinking - that seems kinda long. But keep in mind, I fully handled all errors in at least a semi-proper way, and we’re doing all this:

  1. creating a database
  2. creating some structure (the “world” bucket)
  3. storing data to the structure
  4. retrieving data from the structure.

I think that’s not too bad in 54 lines of code.

So let’s look at what that example is really doing. First we call bolt.Open to get the database. This will create the file if necessary, or open it if it exists.

All reads from or writes to the bolt database must be done within a transaction. You can have as many Readers in read-only transactions at the same time as you want, but only one Writer in a writable transaction at a time (readers maintain a consistent view of the DB while writers are writing).

To begin, we call db.Update, which takes a function to which it’ll pass a bolt.Tx - bolt’s transaction object. We then create a Bucket (since all data in bolt lives in buckets), and add our key/value pair to it. After the write transaction finishes, we start a read- only transaction with DB.View, and get the values back out.

What’s great about bolt’s transaction mechanism is that it’s super simple - the scope of the function is the scope of the transaction. If the function passed to Update returns nil, all updates from the transaction are atomically stored to the database. If the function passed to Update returns an error, the transaction is rolled back. This makes bolt’s transactions completely intuitive from a Go developer’s point of view. You just exit early out of your function by returning an error as usual, and bolt Does The Right Thing. No need to worry about manually rolling back updates or anything, just return an error.

The only other basic thing you may need is to iterate over key/value pairs in a Bucket, in which case, you just call bucket.Cursor(), which returns a Cursor value, which has functions like Next(), Prev() etc that return a key/value pair and work like you’d expect.

There’s a lot more to the bolt API, but most of the rest of it is more about database statistics and some stuff for more advanced usage scenarios… but the above is all you really need to know to start storing data in a bolt database.

For a more complex application, just storing strings in the database may not be sufficient, but that’s ok, Go has your back there, too. You can easily use encoding/json or encoding/gob to serialize structs into the database, keyed by a unique name or id. This is what makes it easy for bolt to go from a key/value store to a document store - just have one bucket per document type. Again, the benefit of bolt is low barrier of entry. You don’t have to figure out a whole database schema or install anything to be able to just start dumping data to disk in a performant and manageable way.

The main drawback of bolt is that there are no queries. You can’t say “give me all foo objects with a name that starts with bar”. You could make your own index in the database and keep it up to date manually. This could be as easy as a slice of IDs serialized into an “indices” bucket for a particular query. Obviously, this is where you start getting into the realm of developing your own relational database, but if you don’t go overboard, it can be nice that all this code is just that - code. It’s not queries in some external DSL, it’s just code like you’d write for an in-memory data store.

Bolt is not for every application. You must understand your application’s needs and if bolt’s key/value style will be sufficient to fulfill those needs. If it is, I think you’ll be very happy to use such a simple data store with so little mental overhead.

Lightning Intro To Go

time to read 13 min | 2602 words

The Go Programming Language

  • Go is a compiled programming language in the tradition of C and C++, with static typing, garbage collection, and unique language features enabling concurrent programming
  • Latest Release: 1.8rc3 (1.8 will be out soon)
  • Developed internally by Google to solve the kind of problems unique to Google (ie, high scale services/systems)
  • Designers/developers of Go have deep ties to C/Unix (Ken Thompson, Rob Pike, Robert Griesemer, et al)

Hello World in Go

package main
import "fmt"

func main() {
    fmt.Println("hello world")
}

Go Language and Runtime Feature Overview

  • Small and powerful standard library
  • Garbage collected
  • Statically compile (or cross-compile!) and deploy almost anywhere
  • Super-fast compiles and single binary deploys
  • Language/standard library are UTF-8 native
  • Design and behavior of language/standard library is opinionated
  • Since v1.5, compiler toolchain is written in Go
  • Built in unit testing
  • Easy integration with C code/libraries
  • Less-is-more!

Installing Go

#!/bin/bash

sudo -s
cd /usr/local/

export GOROOT_BOOTSTRAP=/usr/local/go-1.7.4

git clone https://go.googlesource.com/go
cd go/src && git checkout go1.8rc3
./all.bash

export PATH=$PATH:/usr/local/go/bin

IDEs

  • VSCode - is a new, but strong, cross-platform IDE built by Microsoft, and works well with Go!
  • JetBrains Gogland - new, upcoming JetBrains IDE for Go
  • Plugins available for most other IDEs/editors – Sublime, IntelliJ, etc

Installing VS Code with Go

Run/Build/Install command line example:

  • Run: $ go run hello.go
  • Build and Execute:

    $ go build hello.go
    $ ls
    hello hello.go
    $ ./hello
    hello world
    
  • Install (puts hello in $GOPATH/bin/): $ go install

Packages and go get

  • Go code is grouped in “packages”: a directory containing one or more .go files
  • Packages are retrievable via go get: $ go get -u github.com/knq/baseconv
  • The above will fetch the Go Git repository and store it in $GOPATH/src/$REPO:

    $ cd $GOPATH/src/github.com/knq/baseconv
    $ ls
    baseconv.go  baseconv_test.go  coverage.out  coverage.sh  example  LICENSE  old  README.md
    
  • A package may have any number of sub directories each of which is its own package, ie:

    github.com/knq/baseconv              // would import "baseconv"
    github.com/knq/baseconv/subpackage   // would import "subpackage"
    

Package Imports, and Visibility

  • Packages (ie, libraries) can be imported into the current package: import "github.com/knq/baseconv"
  • Only func’s and type’s defined in that package beginning with a capital letter are visible when imported: func doSomething() {} // not visible to other packages func DoSomething() {} // visible to other packages
  • For example:

    import (
        "fmt"
    )
    fmt.Println("foobar")
    fmt.print("hello") // compiler error
    
  • When in doubt, start the name with a capital

  • You can define an import alias for a package:

    import (
        j "github.com/knq/baseconv" 
    )
    // baseconv's exported funcs/types are now available under 'j':
    j.Decode62()
    
  • Some packages need to be imported for their side-effect:

    import (
        // imports the postgres database driver package 
        _ "github.com/lib/pq"
    )
    

Building, Testing, and Installing a Go package from command line

$ cd $GOPATH/src/github.com/knq/baseconv/
$ go build
$ go test -v
=== RUN   TestErrors
--- PASS: TestErrors (0.00s)
=== RUN   TestConvert
--- PASS: TestConvert (0.00s)
=== RUN   TestEncodeDecode
--- PASS: TestEncodeDecode (0.00s)
PASS
ok      github.com/knq/baseconv 0.002s
$ go install

Some Notes on Go’s Syntax/Design

  • Go designers have purposefully omitted many common features in other languages in the interest of simplicity and readability above almost all else
  • If it can already be done through some other feature available to the language, then there is not a need for a specific language feature

Quick Syntax Primer

  • Go is C-like, but:
  • No semicolons – every line break implies a semicolon
  • Variable names come before the type
  • Braces are required for control statements (for, if, switch, …)
  • Parentheses are not used in control statements
  • Typing is implicit in assignments
  • Unused import or variable is a compiler error
  • Trailing commas are required
  • Standard syntax formatting that is applied automatically with gofmt

C vs Go, a simple comparison

  • Example of printing all command line arguments in C and Go:
#include <stdio.h>

int main(int argc, char **argv) {
    for (int i = 0; i < argc; i++) {
        printf(">> arg %d: %s\n", i, argv[i]);
    }
    return 0;
}
package main

import (
	"fmt"
	"os"
)

func main() {
	for i, a := range os.Args {
		fmt.Printf(">> arg %d: %s\n", i, a)
	}
}

Standard Types (builtin)

var b bool = false 
var s string = "" 
var b byte = 0 // alias for uint8

// -1 0
int   uint   
int8  uint8  
int16 uint16 
int32 uint32 
int64 uint64 

// 0.0  1.0i ...
float32 complex64
float64 complex128

// 'c'
rune // alias for int32

uintptr // an integer type that is large enough to hold the bit pattern of any pointer.

Expressions and Assignments

  • Variable names can include any UTF-8 code point:

    var a = ""                  // variable "a"    is string    value ""
    var 世界 = 15                // variable "世界" is int       value 15
    var f bool                  // variable "f"    is bool      value false
    b := "awesome"              // variable "b"    is string    value "awesome"
    b = "different string"      // variable "b"    is assigned  value "different string"
    
  • Expressions:

    a := b * c // a is assigned the value of b * c
    
  • Supports multiple return values from functions:

    func someFunc() (int, int) { return 7, 10 }
    a, b := someFunc() // a = 7, b = 10
    
  • Special typeless variable _ can be used as placeholder in an assignment:

    a, b, _ = anotherFunc() // the third return value of anotherFunc will be ignored
    

Expressions and Operators

  • Usual operators:

    alt text

  • Note: operators are only are available as a single expression (cannot be inlined), ie:

    // valid
    i++
    j--
    
    // not valid
    j[i++]
    
  • Otherwise, operators work as expected:

    j *= 10
    i = i + 15
    

Constants

  • Go declares constants using the keyword “const”:

    const (
        MyString string = "foobar"
    )
    
  • A const can be any expression:

    const (
        // typed const
        MyConst string = "hello" 
    
        // not typed
        MyOtherConst = 0
    )
    
  • iota is special value for incrementing constants:

    const (
        MyConstA = iota // 0
        MyConstB        // 1
        MyConstC        // 2
    )
    

Slices, Maps, Arrays

  • There are fixed-length arrays, but rarely used:

    var a [8]byte
    
  • A slice provides a dynamic list of any type:

    var a = []int{15, 20, 9}
    for i := range a {
        fmt.Printf(">> %d\n", a[i])
    }
    
  • Maps (dictionaries/hashes in other languages) provides a robust map of key to value pairs for any type:

    var a = map[string]int{
        "foo": 10,
        "bar": 15,
    }
    for k, v := range a {
        fmt.Printf(">> %s: %d\n", k, v)
    }
    

make and new

  • make is used to allocate either a slice, map or channel with a size:

    a := make([]string, 15)              // a has type '[]string' and initial length of 15
    b := make(map[string]interface{}, 0) // b has type 'map[string]interface{}'
    c := make(chan *Point)               // c has type 'chan *Point'
    
  • new allocates a new instance of the type and returns a pointer to the allocated instance:

    b := new(Point) // b has type '*Point'
    p := &Point{}   // more or less the same as new(Point)
    i := new(int)   // i has type '*int'
    

append, len, and reslicing

  • append is used to append to a slice

    b := []string{"foo", "bar"}
    b = append(b, "another string") // b is now []string{"foo", "bar", "another string"}
    
  • len provides the length for slices, maps, or strings:

    a := map[string]int{0, 12}
    b := []int{14, 13, 3}
    len(a)       // 2
    len(b)       // 3
    len("hello") // 5
    
  • Any slice or array can be resliced:

    a := []string{"foo", "bar", "make", "new"}
    b := a[:1]  // slice a from 0 to 1      -- b is []string{"foo"}
    c := a[1:3] // slice a from 1 to 3      -- c is []string{"bar", "make"}
    d := a[1:]  // slice a from 1 to len(a) -- d is []string{"bar", "make", "new"}
    

func

  • Functions are declared with func, and the return type follows the parameter list:

    func empty() {}                                       // no return value
    func doNothing(a string, c int) error { return nil }  // returns error
    
  • A func can be assigned to a variable:

    func someFuncName() error { return nil }
    a := someFuncName // a has type 'func() error'
    
  • A func can also be declared inline:

    func main() {
        g := func() {
            doSomething()
        }
        g()
        func(b int) {
          fmt.Printf("%d\n", b)
        }(10)
    }
    

Control Statements

  • if/else if/else:

    if cond {
        expr
    } else if cond {
        expr
    } else {
        expr
    }
    
  • switch/case/default:

    switch v {
    case "e":
        // something
    default:
        // default
    }
    
  • switch does not require break statements and cases do not automatically fallthrough

Control Statements

  • switch as replacement for complex if/else chains:

    switch {
    case i == 0 && err != nil:
        // something
    case i == 6:
        // something
    case j == 9:
        // something
    default:
        // default
    }
    
  • select is like switch, but waits on a channel:

    select {
    case a := <-c:
        // read a from channel c
    case <-time.After(15*time.Second):
        // a 'timeout' after 15 seconds
    }
    

Control Statements

  • In Go, “while” is spelled “for” – the only provided loop is for:

    for cond {
    }
    
    for {
        if !cond {
            break
        }
    }
    
    loop:
        for i := 0; i < len(someSlice); i++ {
            for {
                if a == 15 {
                    break loop
                }
            }
        }
    
    for key, value := range someSlice {
    }
    

Variadic parameters

  • func can have variable args (variadic)
  • Must be last parameter
  • Special symbol … to indicate expansion:

    func doSomething(prefix string, intList ...int) {
        for m, n := range intList {
            fmt.Printf("> %s (%d): %d\n", prefix, m, n) 
        }
    }
    
  • Can be used also in append statements:

    strList := []string{"bar"}
    j := append([]string{"foo"}, strList...) // j is []string{"foo", "bar"}
    

Type Declaration

  • No classes or objects
  • struct provides compound (“structured”) types:

    type Point struct {
        X, Y float64
    }
    
  • and interface defines a set of func’s:

    type Reader interface {
        Read([]byte) (int, error)
    }
    
  • Can create a type copy for any other type:

    type MyUnsignedInteger uint32
    type MyPoint Point
    type MyReader Reader
    type MyFunc func(string) error
    

Notes on Types

  • Same export / visibility rules apply for struct members:

    type Point struct {
        X, Y float64
        j    int
    }
    
  • Only Go code in the same package as Point can see Point.j

  • Type conversions (casts) are always explicit!

    type MyStr string
    a := MyStr("blah")       // a is of type MyStr and has value "blah"
    var b string = a         // compiler error
    var c string = string(a) // c is of type string and has value "blah"
    

Type Receivers

  • A func can be given a receiver:

    type MyType struct {
        MyValue int
    }
    func (mt MyType) AddOne() int {
        return mt.MyValue+1
    }
    
    type MyString string
    func (ms MyString) String() string {
        return string(ms)
    }
    
  • A func with a pointer receiver, allows the func to modify the value of the receiver:

    // Increment increments MyType's MyValue and returns the result.
    func (mt *MyType) Increment() int {
        mt.MyValue++
        return mt.MyValue
    }
    

About interface

  • Unlike Java or other object-oriented languages, there is no need to explicitly declare a type as having an interface:

    type Reader interface {
        Read([]byte) (int, error)
    }
    
    type MyReader string
    
    // Read satisfies the Reader interface. 
    func (mr MyReader) Read([]byte) (int, error) { /* ... */ }
    
    // DoSomething does something with a Reader.
    func DoSomething(r Reader) { /* ... */ }
    
    func main() {
        s := MyReader("hello")
        DoSomething(s)
    }
    

Pointers, Interfaces, and nil

  • Go pointers are similar to pointers in C/C++ (address to a variable), but there is no pointer math in Go!
  • The . operator is used for both pointer dereference and for accessing a member variable:

    type Point { X, Y int }
    
    a := Point{10, 20}        // a has type 'Point' with value {X: 10, Y: 20}
    b := &Point{X: 30, Y: 40} // b has type '*Point' and **points** to value {X: 30, Y: 40}
    *b = Point{Y: 80}         // b now points to value {X: 0, Y: 80}
        
    // . is used to access struct members for both a and b:
    fmt.Printf("(%d %d) (%d %d)", a.X, a.Y, b.X, b.Y) // prints "(10 20) (0 80)"
    
  • Any type pointer or interface can be assigned the nil value:

    type Reader interface{}
    var a *Point = nil
    var b MyReader = nil
    

Goroutines and Channels

  • Killer features of Go that provides lightweight concurrency in any Go application
  • Any func in Go can be a goroutine:

    func main() {
        for i := 0; i < 10; i++ {
            go func(z int) {
                fmt.Printf(">> %d\n", z)
            }(i)
        }
        time.Sleep(1*time.Second)
    }
    
  • Channels are a unique feature in Go that provides type safe memory sharing

    c := make(chan int)
    c <- 10  // write 10 to c
    j := <-c // read int from c
    
    // channels can be read or write only:
    var c <-chan int // read only chan
    var d chan<- int // write only chan
    

Goroutine and Channel example

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var wg sync.WaitGroup

    c := make(chan int)
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func(z int) {
            defer wg.Done()
            time.Sleep(1 * time.Second)
            c <- z
        }(i)
    }

    wg.Add(1)
    go func() {
        defer wg.Done()
        for {
            select {
            case z := <-c:
                fmt.Printf(">> z: %d\n", z)
            case <-time.After(5 * time.Second):
                return
            }
        }
    }()

    wg.Wait()
    fmt.Println("done")
}

Handling Errors

  • No try/case equivalent
  • Breaking flow should be done by checking for an error:

    func MyFunc() error {
        return errors.New("error encountered")
    }
    
    err := MyFunc()
    if err != nil {
        // handle error
    }
    
  • Utility func in the standard library fmt package:

    fmt.Errorf("encountered: %d at %s", line, str) // returns a error
    

Error Types

  • error is a special Go interface:

    interface error {
        Error() string
    }
    
  • Any type can be an error by satisfying the error interface:

    type MyError struct {}
    func (me *MyError) Error() string {
        return "my error"
    }
    
    func doSomething() error {
        return &MyError{}
    }
    

defer

  • defer is a great feature of Go that allows executes the func when the parent func returns:

    func doSomething() error {
        db, err := sql.Open(/* ... */)
        if err != nil {
            return err
        }
        defer db.Close()
    
        err = db.Exec("DELETE ...")
        /* ... */
    }
    

    panic and recover

  • panic allows immediate halt of the current goroutine:

    panic("some error")
    
  • recover() can only be called in defer’d func’s, but allows recovery after a panic:

    func myFunc() {
        defer func() {
            if e := recover(); e != nil {
                log.Printf("run time panic: %v", e)
            }
        }()
    
        panic("my panic")
    }
    
  • Note: panic’s should not be used unless absolutely necessary.

Quick Overview of the Standard Library

```golang
import (
    "fmt"      // string formatting
    "strings"  // string manipulation
    "strconv"  // string conversion to standard types
    "io"       // system input/output package
    "sync"     // synchronization primitives
    "time"     // robust time handling/formatting
    "net/http" // http package supporting http servers and clients

    "database/sql"  // standardized sql interface
    "encoding/json" // json encoding/decoding (also packages for xml, csv, etc)

    // template libraries for text and html
    "text/template"
    "html/template" 

    // cryptographic libs
    "crypto/rsa"
    "crypto/elliptic"

    "reflect"       // go runtime introspection / reflection
    "regexp"        // regular expressions
)
// And many many many more!
```

What Go doesn’t have (and why this is a good thing)

  • Generics
  • Implicit comparisons
  • Overloading / Inheritance
  • Objects
  • Ternary operator (?:)
  • Miscellany data structures (vector, set, etc)
  • Package manager

Working Example

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
)

func main() {
	// read from web
	res, err := http.Get("http://ifconfig.co/json")
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v", err)
		os.Exit(1)
	}
	defer res.Body.Close()

	// read the body
	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v", err)
		os.Exit(1)
	}

	// decode json
	var vals map[string]interface{}
	err = json.Unmarshal(body, &vals)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v", err)
		os.Exit(1)
	}

	for k, v := range vals {
		fmt.Fprintf(os.Stdout, "%s: %v\n", k, v)
	}
}

RECENT SERIES

  1. java 101 (13):
    Apr 29, 2017 - Translating Scanner tokens into primitive types
  2. python data structure (5):
    May 03, 2017 - Enhance your tuples
  3. python function (2):
    Apr 16, 2017 - Lambda Functions in Python
  4. python generator (4):
    Apr 26, 2017 - Next, Function or Method ?

Friends of Rezha