~5 minutes

Fiddling around with Go and Ruby

Sometimes things are slow in Ruby. I have written about this before.

Last time we ran into this problem, we extracted a service using Martini, well, I don’t want to do that this time. Too much of a deployment hassle.

I want to make a Ruby Gem…with Go inside.

Calling Go from Ruby

To call something else from your source language, you’re looking for an FFI.

Enter Fiddle.

Basically, it is a “libffi” wrapper for Ruby, where “ffi” stands for “foreign function interface” roughly translated as “I want to call something in Go-land from Ruby-land”.

So how to get it to work?

Go-land

Let’s say we have this super hard problem that is computationally intensive that we want to offload to Go: Upcasing Strings!!!

We’ve got an array of strings and we want to convert them all into the uppercase version of themselves. Clearly we need a web-scale language.

Let’s write it in Go!

package main

import (
	"strings"
)

func upcase(things []string) []string {
	upcased := []string{}
	for _, thing := range things {
		upcased = append(upcased, strings.ToUpper(thing))
	}

	return upcased
}

Ok, so it’s not the whole program, but it’s the relevant bits.

Sadly, we can’t use this. Actually, I’m not very sad. It’s pretty gross.

The trouble is, to get Ruby and Go to talk, we do it through C, so stuff like []string just ain’t going to happen. We need to tell the Go code to handle more basic types instead.

Solution: JSON all the things!

package main

import (
	"encoding/json"
	"strings"
)

import "C"

//export upcase
func upcase(data *C.char) *C.char {
	things := []string{}
	err := json.Unmarshal([]byte(C.GoString(data)), &things)
	if err != nil {
		return C.CString(err.Error())
	}
	upperCased := []string{}
	for _, thing := range things {
		upperCased = append(upperCased, strings.ToUpper(thing))
	}

	b, err := json.Marshal(upperCased)
	if err != nil {
		return C.CString("{}")
	}

	return C.CString(string(b))
}

func main() {}

NOTE: Every C.CString above should have a corresponding defer C.free(unsafe.Pointer(strPtr)) with an imported unsafe package.

As you can see, there are a ton of changes.

Main has been stubbed, it’s not going to be used by the Ruby side, so we just have it there for completeness.

The import "C" is the heart of the change and is necessary to convert the Go code into a shared object, via the super fancy c-shared buildmode in Go 1.5. Basically it packs up everything your Go program needs into a binary that we can access later from Ruby. Whee!

There’s also a ton of conversion code there to get from the *C.char type to our native Go types and back. But whatever, it’s the price of doing business I guess.

It’s especially important to point out the comment which exports upcase, without it, the C layer will see nothing.

Ruby-land!

What about the Ruby side of things?

require 'upcaser/version'
require 'json'
require 'fiddle'

module Upcaser
  # where things is an array of strings
  def self.upcase(things)
    dlib = File.expand_path('../ext/libupcase.so', File.dirname(__FILE__))
    libupcase = Fiddle.dlopen(dlib)
    upcase = Fiddle::Function.new(
      libupcase['upcase'],
      [Fiddle::TYPE_VOIDP],
      Fiddle::TYPE_VOIDP
    )
    ptr = upcase.call(JSON.dump(things))
    JSON.load(ptr.to_s)
  end
end

Yeah, it’s pretty crazy. So what is going on here?

First off, we’re telling Fiddle which dynamic library to open. That should be sort of obvious. It gets built via native extensions that you’ll see in the Gem below.

Next, we tell it which function we care about and we indicate its argument and return types, in this case, pointers to stuff. If you didn’t export the function on the Go side, nothing will be found here.

Finally, when we call the upcase function, we pass in our object encoded in JSON and get a pointer back. We access the contents of that pointer via .to_s and convert it back into Ruby objectspace.

It’s pretty complicated, truth be told, but for something super important like upcasing, I’ll make an exception.

It works in Rails!

From the console, for instance:

pry(main)> Upcaser.upcase(["this", "works"])
> ["THIS","WORKS"]

Show me a Gem!

Since there is even more boring glue code to get this to work as a Ruby Gem, I’ll now direct you to a Gem for which I guarantee there will never be a name collision: Yyzyyz

Silly examples get silly gem names, but I’m sure you can find better uses for this. :)

Have fun with Go and Ruby!

Tagged with: programming

My first novel is coming soon-ish!

Check out Singular