Go at Streetspotr

At Streetspotr, we have two services running in production which are a key part of our platform that we’ve written in Go.

Download Service

The first Go project was our download service. It’s a simple service for uploading and downloading files. You upload a file, get back an identifier, and can then download the file via that identifier.

Initially, at the end of 2012, the download service was written in Ruby. It was based on plain Rack on the server side and Net::HTTP on the client side. It worked pretty well, but it had problems dealing with large files in the range of a few gigabytes.

Around half a year later I got interested in Node.js. Its excellent support for non-blocking I/O made Node look like a great platform to write a download server in. I decided to rewrite the server part of the service in Node and had it up and running the same evening. Eventually, the Node version replaced the Ruby server in production.

Node wasn’t the only platform/language I experimented with at that time. I played with Scala, Clojure, Haskell, Erlang, and Go. In July 2013, Brad Fitzpatrick, member of the Go team at Google, gave a talk about dl.google.com being powered by Go. This talk inspired me to start the second rewrite of the download server, and at the end of August, I posted a Tweet that Streetspotr’s download server is now also powered by Go.

Since then, the service has evolved. First, instead of storing metadata of uploaded files in JSON files next to the actual file, the server was updated to use MySQL. Later, we added support to store files on Amazon S3 and moved all files from our servers to S3. Because we had troubles with uploading large files since the beginning, the client was also later rewritten in Go and the Ruby library became a wrapper around the Go command line upload program.

The complete project is about 1400 lines of code. The only imported third-party packages are the official AWS SDK for Go and the MySQL driver for Go. The service handles hundreds of gigabytes each day and we’re very happy with its performance and the little maintenance it needs.

Spotengine

In April 2015, we were experiencing performance problems running a huge amount of geospatial queries on our MySQL database. Back then, the version of MySQL we used didn’t support geospatial indexes, so we looked at other ways to decrease the load on our database server. That was the time when our second Go-powered service was born. We called it Spotengine.

Basically, the Spotengine is an in-memory quadtree that contains points and bounds. On our platform we have spots and there are two types of them: regular spots that are located at a specific location defined by latitude and longitude, and regional spots that cover an area defined by latitude, longitude, and a radius. In the quadtree, these two types are represented by points that have an X and Y coordinate, and by bounds that have two points — the upper left and lower right corner.

Our mobile API — the API our iOS and Android apps talk to and we call MAPI internally — asks the Spotengine to get a list of spots in a certain area (specified by latitude, longitude, radius). To answer the request, the Spotengine queries the quadtree for all points that are inside the requested bounding box to get all regular spots, and it also queries for all bounds that contain the center point of the requested circle to get all regional spots. Because we use rectangles to approximate circles, we calculate the exact distances and drop all items whose distance is greater than the requested radius.

The first version of the service only returned the ID of the spots and the MAPI still had to perform a query to the MySQL database to get the actual spots, but that changed just a few days later to return full spot data. The Spotengine has become a cache, the database load went down dramatically, and the MAPI’s response time for spot searches dropped from single-digit seconds to the range of tens of milliseconds.

Just like the download service, the Spotengine has evolved as well. It now also stores other location-based information we have to query often, and it does requirements and constraints checking (it decides if a spot should be generally available and if a user is allowed to pick up a certain spot). The Spotengine is an intergral part of the Streetspotr platform, answering hundreds of thousands of requests each day, and used by several other services. It’s been a huge success.

The Spotengine’s code base is around 5000 LOC. It imports only three packages that aren’t part of the standard library: my sqlbuilder package to build SQL queries, a client library for exception notification, and — obviously — the MySQL driver.