Athens 101

What is Athens?

Shortly: Athens is a project building on top of vgo (or go1.11+) trying to bring dependencies closer to you so you can count on repeatable builds even at a time when VCS is down.

The big goal of Athens is to provide a new place where dependencies — not code — live. Dependencies are immutable blobs of code and associated metadata that come from Github. They live in storage that Athens controls.

You probably already know what “immutable” means, but let me just point it out again because it’s really important for this whole system. When folks change their packages, iterate, experiment, or whatever else, code on Athens won’t change. If the package author releases a new version, Athens will pull that down and it’ll show up. So if you depend on package M version v1.2.3, it will never change on Athens. Not even after force push, not even after repo cease to exist.

Download protocol

Athens builds on top of Go CLI which specifies a set of endpoints with which it communicates with external proxies providing modules. This set of endpoints we call Download Protocol

The original vgo research paper on Download protocol can be found here: https://research.swtch.com/vgo-module

Each of these endpoints sits on top of a module. Let’s assume module htp authored by acidburn.

So for each of the endpoints mentioned below we will assume address acidburn/htp/@v/{endpoint} (e.g acidburn/htp/@v/list)

In the examples below, $HOST and $PORT are placeholders for the host and port of your Athens server.

List of versions

This endpoint returns a list of versions that Athens knows about for acidburn/htp. The list is just separated by newlines:

GET $HOST:$PORT/github.com/acidburn/htp/@v/list

v0.1.0
v0.1.1
v1.0.0
v1.0.1
v1.2.0

Version info

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.info

This returns JSON with information about v1.0.0. It looks like this:

{
    "Name": "v1.0.0",
    "Short": "v1.0.0",
    "Version": "v1.0.0",
    "Time": "1972-07-18T12:34:56Z"
}

Go.mod file

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.mod

This returns the go.mod file for version v1.0.0. If $HOST:$PORT/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:

module github.com/acidburn/htp

Module sources

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.zip

This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.

Latest

GET $HOST:$PORT/github.com/acidburn/htp/@latest

This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.

Why Does It Matter?

Immutability

The Go community has had lots of problems with libraries disappearing or changing without warning. It’s easy for package maintainers to make changes to their code that can break yours - and much of the time it’s an accident! Could your build break if one of your dependencies did this?

• Commit abdef was deleted
• Tag v0.1.0 was force pushed
• The repository was deleted altogether

Since your app’s dependencies come directly from a VCS (Version Control System, such as GitHub), any of those above cases can happen to you and your builds can break when they do - oh no! Athens solves these problems by copying code from VCS’s into immutable storage.

This way, you don’t need to upload anything manually to Athens storage. The first time Go asks Athens for a dependency, Athens will go get it from VCS (github, bitbucket etc). But once that module has been retrieved, it will be forever persisted in its storage backend and the proxy will never go back to VCS for that same version again. This is how Athens achieves module immutability. Keep in mind, you are in charge of that storage backend.

Logic

The fact that the Go command line can now ping your own server to download dependencies, that means you can program whatever logic you want around providing such dependencies. Things like Access Control (discussed below), adding custom versions, custom forks, and custom packages. For example, Athens provides a Validation Hook that will get called for every module download to determine whether a module should be downloaded or not. Therefore, you can extend Athens with your own logic such as scanning a module path or code for red flags etc.

Performance

Downloading stored dependencies from Athens is significantly faster than downloading dependencies from Version Control Systems. This is because go get by default uses VCS to download modules such as git clone, while go get with GOPROXY enabled will use HTTP to download zip archives. Therefore, depending on your computer and internet connection speed, it takes 10 seconds to download the CockroachDB source tree as a zip file from GitHub but almost four minutes to git clone it.

Access Control

Worse than packages disappearing, packages can be malicious. To make sure no such malicious package is ever installed by your team or company, you can have your proxy server return a 500 when the Go command line asks for an excluded module. This will cause the build to fail because Go expects a 200 HTTP response code. With Athens, you can achieve this through the filter file.

Vendor Directory Becomes Optional

With immutability, performance, and a robust proxy server, there’s no longer an absolute need for each repository to have its vendor directory checked in to its version control. The go.sum file ensures that no package is manipulated after the first install. Furthermore, your CI/CD can install all of your dependencies on every build with little time.

Components

From a very high-level view, there are 3 major components of the system.

Client

The client is a user, powered by go binary with module support. At the moment of writing this document, it is Go v1.12+.

VCS

VCS is an external source of data for Athens. Athens scans various VCSs such as github.com and fetches sources from there.

Proxy

We intend proxies to be deployed primarily inside of enterprises to:

• Host private modules
• Exclude access to public modules
• Store public modules

Importantly, a proxy is not intended to be a complete mirror of an upstream proxy. For public modules, its role is to store and provide access control.