Go at Streetspotr
At Streetspotr, we have two services running in
production and being a key part of our platform that we’ve written in
The first Go project was our download service. It’s a simple service for
uploading and downloading files. You upload a file, get back an identifier,
and can then download the file via that identifier.
Initially, at the end of 2012, the download service was written in Ruby. It
was based on plain Rack on the server side and
Net::HTTP on the client side.
It worked pretty well, but it had problems dealing with large files in the
range of a few gigabytes.
Around half a year later I got interested in Node.js. Its excellent support
for non-blocking I/O made Node look like a great platform to write a download
server in. I decided to rewrite the server part of the service in Node and had
it up and running the same evening. Eventually, the Node version replaced the
Ruby server in production.
Node wasn’t the only platform/language I experimented with at that time. I
played with Scala, Clojure, Haskell, Erlang, and Go. In July 2013, Brad
Fitzpatrick, member of the Go team at Google, gave a talk about
dl.google.com being powered by Go.
This talk inspired me to start the second rewrite of the download server, and
at the end of August, I posted a Tweet
that Streetspotr’s download server is now also powered by Go.
Since then, the service has evolved. First, instead of storing metadata of
uploaded files in JSON files next to the actual file, the server was updated
to use MySQL. Later, we added support to store files on Amazon S3 and moved
all files from our servers to S3. Because we had troubles with uploading large
files since the beginning, the client was also later rewritten in Go and the
Ruby library became a wrapper around the Go command line upload program.
The complete project is about 1400 lines of code. The only imported
third-party packages are the official AWS SDK for Go and the MySQL driver for
Go. The service handles hundreds of gigabytes each day and we’re very happy
with its performance and the little maintenance it needs.
In April 2015, we were experiencing performance problems running a huge amount
of geospatial queries on our MySQL database. Back then, the version of MySQL
we used didn’t support geospatial indexes, so we looked at other ways to
decrease the load on our database server. That was the time when our second
Go-powered service was born. We called it Spotengine.
Basically, the Spotengine is an in-memory quadtree that
contains points and bounds. On our platform we have spots and there are two
types of them: regular spots that are located at a specific location defined
by latitude and longitude, and regional spots that cover an area defined by
latitude, longitude, and a radius. In the quadtree, these two types are
represented by points that have an X and Y coordinate, and by bounds that have
two points—the upper left and lower right corner.
Our mobile API—the API our iOS and Android apps talk to and we call MAPI
internally—asks the Spotengine to get a list of spots in a certain area
(specified by latitude, longitude, radius). To answer the request, the
Spotengine queries the quadtree for all points that are inside the requested
bounding box to get all regular spots, and it also queries for all bounds that
contain the center point of the requested circle to get all regional spots.
Because we use rectangles to approximate circles, we calculate the exact
distances and drop all items whose distance is greater than the requested
The first version of the service only returned the ID of the spots and the
MAPI still had to perform a query to the MySQL database to get the actual
spots, but that changed just a few days later to return full spot data. The
Spotengine has become a cache, the database load went down dramatically, and
the MAPI’s response time for spot searches dropped from single-digit seconds
to the range of tens of milliseconds.
Just like the download service, the Spotengine has evolved as well. It now
also stores other location-based information we have to query often, and it
does requirements and constraints checking (it decides if a spot should be
generally available and if a user is allowed to pick up a certain spot). The
Spotengine is an intergral part of the Streetspotr platform, answering
hundreds of thousands of requests each day, and used by several other
services. It’s been a huge success.
The Spotengine’s code base is around 5000 LOC. It imports only three packages
that aren’t part of the standard library: my sqlbuilder package to build SQL
queries, a client library for exception notification, and—obviously—the