Subsections of Introduction to Athens
Athens 101
What is Athens?
Shortly: Athens is a project building on top of vgo (or go1.11+) trying to bring dependencies closer to you so you can count on repeatable builds even at a time when VCS is down.
The big goal of Athens is to provide a new place where dependencies — not code — live. Dependencies are immutable blobs of code and associated metadata that come from Github. They live in storage that Athens controls.
You probably already know what “immutable” means, but let me just point it out again because it’s really important for this whole system. When folks change their packages, iterate, experiment, or whatever else, code on Athens won’t change. If the package author releases a new version, Athens will pull that down and it’ll show up. So if you depend on package M version v1.2.3, it will never change on Athens. Not even after force push, not even after repo cease to exist.
Download protocol
Athens builds on top of Go CLI which specifies a set of endpoints with which it communicates with external proxies providing modules. This set of endpoints we call Download Protocol
The original vgo research paper on Download protocol can be found here: https://research.swtch.com/vgo-module
Each of these endpoints sits on top of a module. Let’s assume module htp authored by acidburn.
So for each of the endpoints mentioned below we will assume address acidburn/htp/@v/{endpoint} (e.g acidburn/htp/@v/list)
In the examples below, $HOST and $PORT are placeholders for the host and port of your Athens server.
List of versions
This endpoint returns a list of versions that Athens knows about for acidburn/htp. The list is just separated by newlines:
GET $HOST:$PORT/github.com/acidburn/htp/@v/list
v0.1.0
v0.1.1
v1.0.0
v1.0.1
v1.2.0
Version info
GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.info
This returns JSON with information about v1.0.0. It looks like this:
{
"Name": "v1.0.0",
"Short": "v1.0.0",
"Version": "v1.0.0",
"Time": "1972-07-18T12:34:56Z"
}
Go.mod file
GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.mod
This returns the go.mod file for version v1.0.0. If $HOST:$PORT/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:
module github.com/acidburn/htp
Module sources
GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.zip
This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.
Latest
GET $HOST:$PORT/github.com/acidburn/htp/@latest
This endpoint returns the latest version of the module.
If the version does not exist it should retrieve the hash of latest commit.
Why Does It Matter?
Immutability
The Go community has had lots of problems with libraries disappearing or changing without warning. It’s easy for package maintainers to make changes to their code that can break yours - and much of the time it’s an accident! Could your build break if one of your dependencies did this?
- Commit
abdef was deleted
- Tag
v0.1.0 was force pushed
- The repository was deleted altogether
Since your app’s dependencies come directly from a VCS (Version Control System, such as GitHub), any of those above cases can happen to you and your builds can break when they do - oh no! Athens solves these problems by copying code from VCS’s into immutable storage.
This way, you don’t need to upload anything manually to Athens storage. The first time Go asks Athens for a dependency, Athens will go get it from VCS (github, bitbucket etc). But once that module has been retrieved, it will be forever persisted in its storage backend and the proxy will never go back to VCS for that same version again. This is how Athens achieves module immutability. Keep in mind, you are in charge of that storage backend.
Logic
The fact that the Go command line can now ping your own server to download dependencies, that means you can program whatever logic you want around providing such dependencies. Things like Access Control (discussed below), adding custom versions, custom forks, and custom packages. For example, Athens provides a Validation Hook that will get called for every module download to determine whether a module should be downloaded or not. Therefore, you can extend Athens with your own logic such as scanning a module path or code for red flags etc.
Downloading stored dependencies from Athens is significantly faster than downloading dependencies from Version Control Systems. This is because go get by default uses VCS to download modules such as git clone, while go get with GOPROXY enabled will use HTTP to download zip archives. Therefore, depending on your computer and internet connection speed, it takes 10 seconds to download the CockroachDB source tree as a zip file from GitHub but almost four minutes to git clone it.
Access Control
Worse than packages disappearing, packages can be malicious. To make sure no such malicious package is ever installed by your team or company, you can have your proxy server return a 500 when the Go command line asks for an excluded module. This will cause the build to fail because Go expects a 200 HTTP response code. With Athens, you can achieve this through the filter file.
Vendor Directory Becomes Optional
With immutability, performance, and a robust proxy server, there’s no longer an absolute need for each repository to have its vendor directory checked in to its version control. The go.sum file ensures that no package is manipulated after the first install. Furthermore, your CI/CD can install all of your dependencies on every build with little time.
Components
From a very high-level view, there are 3 major components of the system.
Client
The client is a user, powered by go binary with module support. At the moment of writing this document, it is Go v1.12+.
VCS
VCS is an external source of data for Athens. Athens scans various VCSs such as github.com and fetches sources from there.
Proxy
We intend proxies to be deployed primarily inside of enterprises to:
- Host private modules
- Exclude access to public modules
- Store public modules
Importantly, a proxy is not intended to be a complete mirror of an upstream proxy. For public modules, its role is to store and provide access control.
Subsections of Installing Athens
Building a versioned Athens binary from source
You can do that easily with just few commands:
Bash
git clone https://github.com/gomods/athens
cd athens
make build-ver VERSION="0.2.0"
PowerShell
git clone https://github.com/gomods/athens
cd athens
$env:GO111MODULE="on"
$env:GOPROXY="https://proxy.golang.org"
$version = "0.2.0"
$date = (Get-Date).ToUniversalTime()
go build -ldflags "-X github.com/gomods/athens/pkg/build.version=$version -X github.com/gomods/athens/pkg/build.buildDate=$date" -o athens ./cmd/proxy
This will give you a binary named athens. You can print the version and time information by running:
which should return something like:
Build Details:
Version: 0.2.0
Date: 2018-12-13-20:51:06-UTC
Install Athens on Kubernetes
When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. In order to run a more production-like proxy, you may want to run Athens on a Kubernetes cluster. To aid in deployment of the Athens proxy on Kubernetes, a Helm chart has been provided. This guide will walk you through installing Athens on a Kubernetes cluster using Helm.
Prerequisites
In order to install Athens on your Kubernetes cluster, there are a few prerequisites that you must satisfy. If you already have completed the following steps, please continue to configuring helm. This guide assumes you have already created a Kubernetes cluster.
Install the Kubernetes CLI
In order to interact with your Kubernetes Cluster, you will need to install kubectl.
Install The Helm CLI
Helm is a tool for installing pre-configured applications on Kubernetes.
Install helm by running the following command:
MacOS
brew install kubernetes-helm
Windows
- Download the latest Helm release.
- Decompress the tar file.
- Copy helm.exe to a directory on your PATH.
Linux
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
If your cluster has already been configured to use Helm, please continue to deploy Athens.
If not, please read on.
RBAC Cluster
If your cluster has RBAC enabled, you will need to create a ServiceAccount, ClusterRole and ClusterRoleBinding for Helm to use. The following command will create these and initialize Helm.
kubectl create -f https://raw.githubusercontent.com/Azure/helm-charts/master/docs/prerequisities/helm-rbac-config.yaml
helm init --service-account tiller
Non RBAC Cluster
If your cluster has does not have RBAC enabled, you can simply initialize Helm.
Before deploying Athens, you will need to wait for the Tiller pod to become Ready. You can check the status by watching the pods in kube-system:
$ kubectl get pods -n kube-system -w
NAME READY STATUS RESTARTS AGE
tiller-deploy-5456568744-76c6s 1/1 Running 0 5s
Deploy Athens
The fastest way to install Athens using Helm is to deploy it from our public Helm chart repository. First, add the repository with this command:
$ helm repo add gomods https://gomods.github.io/athens-charts
$ helm repo update
Next, install the chart with default values to athens namespace:
$ helm install gomods/athens-proxy -n athens --namespace athens
This will deploy a single Athens instance in the athens namespace with disk storage enabled. Additionally, a ClusterIP service will be created.
Advanced Configuration
Replicas
By default, the chart will install Athens with a replica count of 1. To change this, change the replicaCount value:
helm install gomods/athens-proxy -n athens --namespace athens --set replicaCount=3
Resources
By default, the chart will install Athens without specific resource requests or limits. To change this, change the resources value:
helm install gomods/athens-proxy -n athens --namespace athens \
--set resources.requests.cpu=100m \
--set resources.requests.memory=64Mi \
--set resources.limits.cpu=100m \
--set resources.limits.memory=64Mi
For more information, see Managing Compute Resources for Containers in the Kubernetes documentation.
Give Athens access to private repositories via Github Token (Optional)
- Create a token at https://github.com/settings/tokens
- Provide the token to the Athens proxy either through the config.toml file (the
GithubToken field) or by setting the ATHENS_GITHUB_TOKEN environment variable.
Storage Providers
The Helm chart currently supports running Athens with two different storage providers: disk and mongo. The default behavior is to use the disk storage provider.
Disk Storage Configuration
When using the disk storage provider, you can configure a number of options regarding data persistence. By default, Athens will deploy using an emptyDir volume. This probably isn’t sufficient for production use cases, so the chart also allows you to configure persistence via a PersistentVolumeClaim. The chart currently allows you to set the following values:
persistence:
enabled: false
accessMode: ReadWriteOnce
size: 4Gi
storageClass:
Add it to override-values.yaml file and run:
helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml
enabled is used to turn on the PVC feature of the chart, while the other values relate directly to the values defined in the PersistentVolumeClaim documentation.
Mongo DB Configuration
To use the Mongo DB storage provider, you will first need a MongoDB instance. Once you have deployed MongoDB, you can configure Athens using the connection string via storage.mongo.url. You will also need to set storage.type to “mongo”.
helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=mongo --set storage.mongo.url=<some-mongodb-connection-string>
S3 Configuration
To use S3 storage with Athens, set storage.type to s3 and set storage.s3.region and storage.s3.bucket to the desired AWS region and
S3 bucket name, respectively. By default, Athens will attempt to load AWS credentials using the AWS SDK from the chain of environment
variables, shared credentials files, and EC2 instance credentials. To manually specify AWS credentials, set storage.s3.access_key_id,
storage.s3.secret_access_key, and change storage.s3.useDefaultConfiguration to false.
helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=s3 --set storage.s3.region=<your-aws-region> --set storage.s3.bucket=<your-bucket>
Minio Configuration
To use S3 storage with Athens, set storage.type to minio. You need to set storage.minio.endpoint as the URL of your minio-installation.
This URL can also be an kubernetes-internal one (e.g. something like minio-service.default.svc).
You need to create a bucket inside your minio-installation or use an existing one. The bucket needs to be referenced in storage.minio.bucket.
Last athens need authentication credentials for your minio in storage.minio.accessKey and storage.minio.secretKey.
helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=minio --set storage.minio.endpoint=<your-minio-endpoint> --set storage.minio.bucket=<your-bucket> --set storage.minio.accessKey=<your-minio-access-key> --set storage.minio.secretKey=<your-minio-secret-key>
Google Cloud Storage
To use Google Cloud Storage storage with Athens, set storage.type to gcp. You need to set storage.gcp.projectID and storage.gcp.bucket to the
desired GCP project and bucket name, respectively.
Depending on your deployment environment you will also need to set storage.gcp.serviceAccount to a key which has read/write access to
the GCS bucket. If you are running Athens inside GCP, you will most likely not need this as GCP figures out internal authentication between products for you.
helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=gcp --set storage.gcp.projectID=<your-gcp-project> --set storage.gcp.bucket=<your-bucket>
Kubernetes Service
By default, a Kubernetes ClusterIP service is created for the Athens proxy. “ClusterIP” is sufficient in the case when the Athens proxy will be used from within the cluster. To expose Athens outside of the cluster, consider using a “NodePort” or “LoadBalancer” service. This can be changed by setting the service.type value when installing the chart. For example, to deploy Athens using a NodePort service, the following command could be used:
helm install gomods/athens-proxy -n athens --namespace athens --set service.type=NodePort
Ingress Resource
The chart can optionally create a Kubernetes Ingress Resource for you as well. To enable this feature, set the ingress.enabled resource to true.
helm install gomods/athens-proxy -n athens --namespace athens --set ingress.enabled=true
Further configuration values are available in the values.yaml file:
ingress:
enabled: true
annotations:
certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
kubernetes.io/tls-acme: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
hosts:
- athens.mydomain.com
tls:
- secretName: athens.mydomain.com
hosts:
- "athens.mydomain.com
Example above sets automatic creation/retrieval of TLS certificates from Let’s Encrypt with cert-manager and uses nginx-ingress controller to expose Athens externally to the Internet.
Add it to override-values.yaml file and run:
helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml
Upstream module repository
You can set the URL for the upstream module repository then Athens will try to download modules from the upstream when it doesn’t find them in its own storage.
You have a few good options for what you can set as an upstream:
https://gocenter.io to use JFrog’s GoCenterhttps://proxy.golang.org to use the Go Module Mirror- The URL to any other Athens server
The example below shows you how to set GoCenter up as upstream module repository:
upstreamProxy:
enabled: true
url: "https://gocenter.io"
Add it to override-values.yaml file and run:
helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml
.netrc file support
A .netrc file can be shared as a secret to allow the access to private modules.
The secret must be created from a netrc file using the following command (the name of the file must be netrc):
kubectl create secret generic netrcsecret --from-file=./netrc
In order to instruct athens to fetch and use the secret, netrc.enabled flag must be set to true:
helm install gomods/athens-proxy -n athens --namespace athens --set netrc.enabled=true
gitconfig support
A gitconfig file can be shared as a secret to allow
the access to modules in private git repositories. For example, you can configure access to private repositories via HTTPS
using personal access tokens on GitHub, GitLab and other git services.
First of all, prepare your gitconfig file:
cat << EOF > /tmp/gitconfig
[url "https://user:token@git.example.com/"]
insteadOf = ssh://git@git.example.com/
insteadOf = https://git.example.com/
EOF
Next, create the secret using the file created above:
kubectl create secret generic athens-proxy-gitconfig --from-file=gitconfig=/tmp/gitconfig
In order to instruct athens to use the secret, set appropriate flags (or parameters in values.yaml):
helm install gomods/athens-proxy --name athens --namespace athens \
--set gitconfig.enabled=true \
--set gitconfig.secretName=athens-proxy-gitconfig \
--set gitconfig.secretKey=gitconfig
Using the Athens Docker images
Whether setting Athens up using Kubernetes or using the Walkthrough, you’ll most likely be using one of the images that the Athens project produces. This document details what images are available, and has a recap from the Walkthrough of how to use them on their own.
Available Docker images
The Athens project produces two docker images, available via Docker Hub
- A release version as
gomods/athens, each tag corresponds with an Athens release, e.g. v0.7.1. Additionally, a canary tag is available and tracks each commit to main
- A tip version, as
gomods/athens-dev, tagged with every commit to main, e.g. 1573339
For a detailed tags list, check each image’s Docker Hub
Running Athens as a Docker image
This is a quick recap of the Walkthrough
Using the docker cli
In order to run the Athens Proxy using docker, we need first to create a directory that will store the persitant modules.
In the example below, the new directory is named athens-storage and is located in our userspace (i.e. $HOME).
Then we need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when we run the Docker container.
Bash
export ATHENS_STORAGE=$HOME/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $HOME athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Non-Root User
The Athens docker images comes with a non-root user athens with uid: 1000, gid: 1000 and home directory /home/athens.
In situations where running as root is not permitted, this user can be used instead. In all other instuctions
replace /root/ with /home/athens/ and set the user and group ids in the run environment to 1000.
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
-v "$PWD/gitconfig/.gitconfig:/home/athens/.gitconfig" \
--name athens-proxy \
--restart always \
-p 3000:3000 \
-u 1000:1000 \
gomods/athens:latest
Troubleshooting Athens in Docker
init issues
The Athens docker image uses tini so that defunct processes get reaped.
Docker 1.13 and greater includes tini and lets you enable it by passing the --init flag to docker run or by configuring the docker deamon with "init": true. When running in this mode. you may see a warning like this:
[WARN tini (6)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
This is the “Athens-tini” complaining that it’s not running as PID 1.
There is no harm in that, since the zombie processes will be reaped by the tini included in Docker.
Shared Team Instance
When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. This guide will help you get Athens running in a more suitable manner for scenarios like providing an instance for your development team to share.
We will use Docker to run the Athens proxy, so first make sure you have Docker installed.
Selecting a Storage Provider
Athens currently supports a number of storage drivers. For local, use we recommend starting with the local disk provider. For other providers, please see the Storage Provider documentation.
Running Athens with Local Disk Storage
In order to run Athens with disk storage, you will next need to identify where you would like to persist modules. In the example below, we will create a new directory named athens-storage in our current directory. Now you are ready to run Athens with disk storage enabled. To enable disk storage, you need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when you run the Docker container.
The examples below use the :latest Docker tags for simplicity, however we strongly recommend that after your environment is up and running that you switch to using
an explicit version (for example :v0.3.0).
Bash
export ATHENS_STORAGE=~/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Note: if you have not previously mounted this drive with Docker for Windows, you may be prompted to allow access
Athens should now be running as a Docker container with the local directory, athens-storage mounted as a volume. When Athens retrieves the modules, they will be will be stored in the directory previously created. First, let’s verify that Athens is running:
$ docker ps
CONTAINER ID IMAGE COMMAND PORTS NAMES
f0429b81a4f9 gomods/athens:latest "/bin/app" 0.0.0.0:3000->3000/tcp athens-proxy
Now, we can use Athens from any development machine that has Go v1.12+ installed. To verify this, try the following example:
Bash
$ export GO111MODULE=on
$ export GOPROXY=http://127.0.0.1:3000
$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!
PowerShell
$env:GO111MODULE = "on"
$env:GOPROXY = "http://127.0.0.1:3000"
git clone https://github.com/athens-artifacts/walkthrough.git
cd walkthrough
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!
We can verify that Athens handled this request by examining the Docker logs:
$ docker logs -f athens-proxy
time="2018-08-21T17:28:53Z" level=warning msg="Unless you set SESSION_SECRET env variable, your session storage is not protected!"
time="2018-08-21T17:28:53Z" level=info msg="Starting application at 0.0.0.0:3000"
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.info [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.mod [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.zip [200]
Now, if you view the contents of the athens_storage directory, you will see that you now have additional files representing the samplelib module.
Bash
$ ls -lr $ATHENS_STORAGE/github.com/athens-artifacts/samplelib/v1.0.0/
total 24
-rwxr-xr-x 1 jeremyrickard wheel 50 Aug 21 10:52 v1.0.0.info
-rwxr-xr-x 1 jeremyrickard wheel 2391 Aug 21 10:52 source.zip
-rwxr-xr-x 1 jeremyrickard wheel 45 Aug 21 10:52 go.mod
PowerShell
$ dir $env:ATHENS_STORAGE\github.com\athens-artifacts\samplelib\v1.0.0\
Directory: C:\athens-storage\github.com\athens-artifacts\samplelib\v1.0.0
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 8/21/2018 3:31 PM 45 go.mod
-a---- 8/21/2018 3:31 PM 2391 source.zip
-a---- 8/21/2018 3:31 PM 50 v1.0.0.info
When Athens is restarted, it will serve the module from this location without re-downloading it. To verify that, we need to first remove the Athens container.
docker rm -f athens-proxy
Now, we need to clear the local Go modules storage. This is needed so that your local Go command line tool will re-download the module from Athens. The following commands will clear the local module storage:
Bash
sudo rm -fr "$(go env GOPATH)/pkg/mod"
PowerShell
rm -recurse -force $(go env GOPATH)\pkg\mod
Now, we can re-run the Athens container:
Bash
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
When we re-run our Go example, the Go cli will again download module from Athens. Athens, however, will not need to retrieve the module. It will be served from the Athens on-disk storage.
Bash
$ ls -lr $ATHENS_STORAGE/github.com/athens-artifacts/samplelib/v1.0.0/
total 24
-rwxr-xr-x 1 jeremyrickard wheel 50 Aug 21 10:52 v1.0.0.info
-rwxr-xr-x 1 jeremyrickard wheel 2391 Aug 21 10:52 source.zip
-rwxr-xr-x 1 jeremyrickard wheel 45 Aug 21 10:52 go.mod
PowerShell
$ dir $env:ATHENS_STORAGE\github.com\athens-artifacts\samplelib\v1.0.0\
Directory: C:\athens-storage\github.com\athens-artifacts\samplelib\v1.0.0
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 8/21/2018 3:31 PM 45 go.mod
-a---- 8/21/2018 3:31 PM 2391 source.zip
-a---- 8/21/2018 3:31 PM 50 v1.0.0.info
Notice that the timestamps given have not changed.
Next Steps:
Install on AWS Fargate (ECS)
In this document, we’ll show how to use AWS Fargate (ECS) to run the Athens proxy.
Selecting a Storage Provider
There is documentation about how to use environment variables to configure the various storage providers. However, for
this particular example we will use Amazon S3 Storage (s3).
Before You Begin
This guide assumes you already have an AWS account as well as the necessary authentication and permissions to create
resources in the account.
Whether you choose to create your resources using the awscli or use something like
Terraform, the resources required are the same.
S3 Bucket
In order to persist modules, we will create a s3 bucket for storage.
Below are two examples of creating the s3 bucket using the awscli and Terraform.
awscli:
$ aws s3api create-bucket --bucket athens-proxy-us-east-1-123456789012 --region us-east-1
terraform:
resource "aws_s3_bucket" "cache" {
bucket = "athens-proxy-us-east-1-123456789012"
}
note: it is a good idea to use environment, region, and/or account ID as components to the bucket name due to their
globally unique naming rules.
ECS Task IAM Role
In order for the ECS container instances to use the s3 bucket, we will need to configure the task IAM role to
include the proper allow rules.
Below is a least-privileged policy document in both JSON and Terraform to enable ECS containers s3 bucket access to
store and retrieve cache assets.
json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::athens-proxy-us-east-1-123456789012"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::athens-proxy-us-east-1-123456789012/*"
}
]
}
terraform:
resource "aws_iam_policy" "task_role" {
name = "athens-proxy-task-role"
path = "/"
policy = data.aws_iam_policy_document.task_role_policy.json
}
data "aws_iam_policy_document" "task_role_policy" {
statement {
effect = "Allow"
actions = [
"s3:ListBucket",
"s3:GetBucketLocation"
]
resources = [aws_s3_bucket.cache.arn]
}
statement {
effect = "Allow"
actions = [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
]
resources = ["${aws_s3_bucket.cache.arn}/*"]
}
statement {
effect = "Allow"
actions = [
"sts:AssumeRole",
"sts:TagSession"
]
resources = ["*"]
}
}
ECS Task Definition
In order for Athens to be able to authenticate to the s3 bucket, we will need to configure the storage variables
associated with s3.
Below is an excerpt from a task definition that shows the minimum environment variables needed.
"environment": [
{"name": "AWS_REGION", "value": "us-east-1"},
{"name": "AWS_USE_DEFAULT_CONFIGURATION", "value": "true"},
{"name": "ATHENS_STORAGE_TYPE", "value": "s3"},
{"name": "ATHENS_S3_BUCKET_NAME", "value": "athens-proxy-us-east-1-123456789012"},
]
Install on Azure Container Instances
When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. This guide will help you get Athens running in a more suitable manner for scenarios like providing an instance for your development team to share.
In this document, we’ll show how to use Azure Container Instances (ACI) to run the Athens proxy.
Selecting a Storage Provider
Athens currently supports a number of storage drivers. For quick and easy use on ACI, we recommend using the local disk provider. For more permanent use, we recommend using MongoDB or other more persistent infrastructure. For other providers, please see the storage provider documentation.
Required Environment Variables
Before executing any of the commands below, make sure you have the following environment variables set up on your system:
AZURE_ATHENS_RESOURCE_GROUP - The Azure Resource Group to install the container in. You need to already have one of these before installing Athens
- See here for details on how to create a resource group
AZURE_ATHENS_CONTAINER_NAME - The name of the container. This should be alphanumeric and you can have - and _ charactersLOCATION - The Azure region to install the container in. See the previous link for an exhaustive list, but here’s a useful cheat sheet that you can use immediately, without reading any docs:
- North America:
eastus2
- Europe:
westeurope
- Asia:
southeastasia
AZURE_ATHENS_DNS_NAME - The DNS name to assign to the container. It has to be globally unique inside of the region you set (LOCATION)
Installing with the Disk Storage Driver
az container create \
-g "${AZURE_ATHENS_RESOURCE_GROUP}" \
-n "${AZURE_ATHENS_CONTAINER_NAME}-${LOCATION}" \
--image gomods/athens:v0.3.0 \
-e "ATHENS_STORAGE_TYPE=disk" "ATHENS_DISK_STORAGE_ROOT=/var/lib/athens" \
--ip-address=Public \
--dns-name="${AZURE_ATHENS_DNS_NAME}" \
--ports="3000" \
--location=${LOCATION}
Once you’ve created the ACI container, you’ll see a JSON blob that includes the public IP address of the container. You’ll also see the fully qualified domain name (FQDN) of the running container (it will be prefixed by AZURE_ATHENS_DNS_NAME).
Installing with the MongoDB Storage Driver
First, make sure you have the following environment variable set up:
AZURE_ATHENS_MONGO_URL - The MongoDB connection string. For example: mongodb://username:password@mongo.server.com/?ssl=true
Then run the create command:
az container create \
-g "${AZURE_ATHENS_RESOURCE_GROUP}" \
-n "${AZURE_ATHENS_CONTAINER_NAME}-${LOCATION}" \
--image gomods/athens:v0.3.0 \
-e "ATHENS_STORAGE_TYPE=mongo" "ATHENS_MONGO_STORAGE_URL=${AZURE_ATHENS_MONGO_URL}" \
--ip-address=Public \
--dns-name="${AZURE_ATHENS_DNS_NAME}" \
--ports="3000" \
--location=${LOCATION}
Once you’ve created the ACI container, you’ll see a JSON blob that includes the public IP address of the container. You’ll also see the fully qualified domain name (FQDN) of the running container (it will be prefixed by AZURE_ATHENS_DNS_NAME).
Install on Google Cloud Run
Google Cloud Run is a service that aims to bridge the gap between the maintainance benefits of serverless architecture and the flexibility of Kubernetes. It is built on top of the opensource Knative project. Deploying using Cloud Run is similar to deploying using Google App Engine with the benefits of a free tier and a simpler build process.
Selecting a Storage Provider
There is documentaion about how to use environment variables to configure a large number of storage providers; however, for this prarticular example we will use Google Cloud Storage(GCS) because it fits nicely with Cloud Run.
Before You Begin
This guide assumes you have completed the following tasks:
- Signed up for Google Cloud
- Installed the gcloud command line tool
- Installed the beta plugin for the gcloud command line tool (this is how to set it up)
- Created a (GCS) bucket for your go modules
Setup a GCS Bucket
If you do not already have GCS bucket you can set one up using the gsutil tool.
First select a region you would like to have your storage in. You can then create a bucket in that region using the following command substituting your in your region and bucket name.
$ gsutil mb -l europe-west-4 gs://some-bucket
Setup
Change the values of these environment variables to be appropriate for your application. For GOOGLE_CLOUD_PROJECT, this needs to be the name of the project that has your cloud run deployment in it. ATHENS_REGION should be the region that your cloud run instance will be in, and GCS_BUCKET should be the Google Cloud Storage bucket that Athens will store module code and metadata in..
$ export GOOGLE_CLOUD_PROJECT=your-project
$ export ATHENS_REGION=asia-northeast1
$ export GCS_BUCKET=your-bucket-name
$ gcloud auth login
$ gcloud auth configure-docker
You will then need to push a copy of the Athens docker image to your google cloud container registry.
Below is an example using v0.11.0, for the latest version, check out the latest Athens release
$ docker pull gomods/athens:v0.11.0
$ docker tag gomods/athens:v0.11.0 gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0
$ docker push gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0
Once you have the container image in your registry you can use gcloud to provision your Athens instance.
$ gcloud beta run deploy \
--image gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0 \
--platform managed \
--region $ATHENS_REGION \
--allow-unauthenticated \
--set-env-vars=ATHENS_STORAGE_TYPE=gcp \
--set-env-vars=GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT \
--set-env-vars=ATHENS_STORAGE_GCP_BUCKET=$GCS_BUCKET \
athens
Once this command finishes is will provide a url to your instance, but you can always find this through the cli:
$ gcloud beta run services describe athens --platform managed --region $ATHENS_REGION | grep hostname
Install on Google App Engine
Google App Engine (GAE) is a Google service allows applications to be deployed without provisioning the underlying hardware. It is similar to Azure Container Engine which is covered in a previous section. This guide will demonstrate how you can get Athens running on GAE.
Selecting a Storage Provider
There is documentaion about how to use environment variables to configure a large number of storage providers; however, for this prarticular example we will use Google Cloud Storage(GCS) because it fits nicely with Cloud Run.
Before You Begin
This guide assumes you have completed the following tasks:
- Signed up for Google Cloud
- Installed the gcloud command line tool
Setup a GCS Bucket
If you do not already have GCS bucket you can set one up using the gsutil tool.
First select a global region you would like to have your storage in. You can then create a bucket in that region using the following command substituting your in your region and bucket name.
$ gsutil mb -l europe-west-4 gs://some-bucket
Setup
First clone the Athens repository
$ git clone https://github.com/gomods/athens.git
There is already a Google Application Engine scaffold set up for you. Copy it into a new file and make changes to the environment variables.
$ cd athens
$ cp scripts/gae/app.sample.yaml scripts/gae/app.yaml
$ code scripts/gae/app.yaml
Once you have configured the environment variables you can deploy Athens as a GAE service.
Install Athens with BOSH
Athens can be deployed in many ways. The following guide explains how to use BOSH, a deployment and lifecycle tool, in order to deploy an Athens server on virtual machines (VMs) on any infrastructure as a service (IaaS) that is supported by BOSH.
Prerequisites
Install BOSH
Make sure to have the BOSH CLI installed and set up a BOSH Director on an infrastructure of your choice.
Setup the Infrastructure
If you choose to deploy on a IaaS provider, there are a few prerequisites that need to be set up before starting with the deployment. Depending on which IaaS you will be deploying, you may need to create:
-
Public IP: a public IP address for association with the Athens VM.
-
Firewall Rules: the following ingress ports must be allowed
3000/tcp - Athens proxy port (if you specify a different port than the default port 3000 in the job properties, adapt this rule accordingly).
Egress traffic should be restricted depending on your requirements.
Amazon Web Services (AWS)
AWS requires additional settings that should be added to a credentials.yml file using the following template:
# security group IDs to apply to the VM
athens_security_groups: [sg-0123456abcdefgh]
# VPC subnet to deploy Athens to
athens_subnet_id: subnet-0123456789abcdefgh
# a specific, elastic IP address for the VM
external_ip: 3.123.200.100
The credentials need to be added to the deploy command, i.e.
-o manifests/operations/aws-ops.yml
VirtualBox
The fastest way to install Athens using BOSH is probably a Director VM running on VirtualBox which is sufficient for development or testing purposes. If you follow the bosh-lite installation guide, no further preparation is required to deploy Athens.
Deployment
A deployment manifest contains all the information for managing and updating a BOSH deployment. To aid in the deployment of Athens on BOSH, the athens-bosh-release repository provides manifests for basic deployment configurations inside the manifests directory. For quickly creating a standalone Athens server, clone the release repository and cd into it:
git clone --recursive https://github.com/s4heid/athens-bosh-release.git
cd athens-bosh-release
Once the infrastructure has been prepared and the BOSH Director is running, make sure that a stemcell has been uploaded. If this has not been done yet, choose a stemcell from the stemcells section of bosh.io, and upload it via the command line. Additionally, a cloud config is required for IaaS specific configuration used by the Director and the Athens deployment. The manifests directory also contains an example cloud config, which can be uploaded to the Director via
bosh update-config --type=cloud --name=athens \
--vars-file=credentials.yml manifests/cloud-config.yml
Execute the deploy command which can be extended with ops/vars files depending on which IaaS you will be deploying to.
bosh -d athens deploy manifests/athens.yml # add extra arguments
For example, when using AWS the deploy command for an Athens Proxy with disk storage would look like
bosh -d athens deploy \
-o manifests/operations/aws-ops.yml \
-o manifests/operations/with-persistent-disk.yml \
-v disk_size=1024 \
--vars-file=credentials.yml manifests/athens.yml
This will deploy a single Athens instance in the athens deployment with a persistent disk of 1024MB. The IP address of that instance can be obtained with
which is useful for targeting Athens, e.g. with the GOPROXY variable. You can follow this quickstart guide for more information.
Managing private repos with .netrc files
Authenticate private repositories via .netrc
-
Create a .netrc file that looks like the following:
machine <ip or fqdn>
login <username>
password <user password>
-
Tell Athens through an environment variable the location of that file
ATHENS_NETRC_PATH=<location/to/.netrc>
-
Athens will copy the file into the home directory and override whatever .netrc file is in home directory. Alternatively, if the host of the Athens server already has a .netrc file in the home directory, then authentication should work out of the box.
Authenticate Mercurial private repositories via .hgrc
-
Create a .hgrc file with authentication data
-
Tell Athens through an environment variable the location of that file
ATHENS_HGRC_PATH=<location/to/.hgrc>
-
Athens will copy the file into the home directory and override whatever .hgrc file is in home directory. Alternatively, if the host of the Athens server already has a .hgrc file in the home directory, then authentication should work out of the box.
Subsections of Configuring Athens
The download mode file
Athens accepts an HCL formatted file that has instructions for how it should behave when a module@version isn’t found in its storage. This functionality gives Athens the flexibility configure Athens to fit your organization’s needs. The most popular uses of this download file are:
- Configure Athens to never download or serve a module or group of modules
- Redirect to a different module proxy for a module or group of modules
This document will outline how to use this file - called the download mode file - to accomplish these tasks and more.
Please see the “Use cases” section below for more details on how to enable these behaviors and more.
Configuration
First, once you’ve created your download mode file, you tell Athens to use it by setting the DownloadMode configuration parameter in the config.toml file, or setting the ATHENS_DOWNLOAD_MODE environment variable. You can set this configuration value to one of two values to tell Athens to use your file:
- Set its value to
file:$FILE_PATH, where $FILE_PATH is the path to the HCL file
- Set its value to
custom:$BASE_64 where $BASE_64 is the base64 encoded HCL file
Instead of one of the above two values, you can set this configuration to sync, async, none, redirect, or async_redirect. If you do, the download mode will be set globally rather than for specific sub-groups of modules. See below for what each of these values mean.
Download mode keywords
If Athens receives a request for the module github.com/pkg/errors at version v0.8.1, and it doesn’t have that module and version in its storage, it will consult the download mode file for specific instructions on what action to take:
sync: Synchronously download the module from VCS via go mod download, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.async: Return a 404 to the client, and asynchronously download and persist the module@version to storage.none: Return a 404 and do nothing.redirect: Redirect to an upstream proxy (such as proxy.golang.org) and do nothing after.async_redirect: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage.
Athens expects these keywords to be used in conjunction with module patterns (github.com/pkg/*, for example). You combine the keyword and the pattern to specify behavior for a specific group of modules.
Athens uses the Go path.Match function to parse module patterns.
Below is an example download mode file.
downloadURL = "https://proxy.golang.org"
mode = "async_redirect"
download "github.com/gomods/*" {
mode = "sync"
}
download "golang.org/x/*" {
mode = "none"
}
download "github.com/pkg/*" {
mode = "redirect"
downloadURL = "https://gocenter.io"
}
The first two lines describe the default behavior for all modules. This behavior is overridden for select module groups below. In this case, the default behavior is:
- Immediatley redirect all requests to
https://proxy.golang.org
- In the background, download the module from the version control system (VCS) and store it
The rest of the file contains download blocks. These override the default behavior for specific groups of modules.
The first block specifies that any module matching github.com/gomods/* (such as github.com/gomods/athens) will be downloaded from GitHub, stored, and then returned to the user.
The second block specifies that any module matching golang.org/x/* (such as golang.org/x/text) will always return a HTTP 404 response code. This behavior ensures that Athens will never store or serve any module names starting with golang.org/x.
If a user has their GOPROXY environment variable set with a comma separated list, their go command line tool will always try the option next in the list. For example, if a user has their GOPROXY environment variable set to https://athens.azurefd.net,direct, and then runs go get golang.org/x/text, they will still download golang.org/x/text to their machine. The module just won’t come from Athens.
The last block specifies that any module matching github.com/pkg/* (such as github.com/pkg/errors) will always redirect the go tool to https://gocenter.io. In this case, Athens will never persist the given module to its storage.
Use cases
The download mode file is versatile and allows you to configure Athens in a large variety of different ways. Below are some of the mode common.
Blocking certain modules
If you’re running Athens to serve a team of Go developers, it might be useful to ensure that the team doesn’t use a specific group or groups of modules (for example, because of licensing or security issues).
In this case, you would write this in your file:
download "bad/module/repo/*" {
mode = "none"
}
Preventing storage overflow
If you are running Athens using a storage backend that has limited space, you may want to prevent Athens from storing certain groups of modules that take up a lot of space. To avoid exhausting Athens storage, while still ensuring that the users of your Athens server still get access to the modules you can’t store, you would use a redirect directive, as shown below:
download "very/large/*" {
mode = "redirect"
url = "https://reliable.proxy.com"
}
If you use the redirect mode, make sure that you specify a url value that points to a reliable proxy.
Authentication to private repositories
Authentication
SVN private repositories
-
Subversion creates an authentication structure in
~/.subversion/auth/svn.simple/<hash>
-
In order to properly create the authentication file for your SVN servers you will need to authenticate to them and let svn build out the proper hashed files.
$ svn list http://<domain:port>/svn/<somerepo>
Authentication realm: <http://<domain> Subversion Repository
Username: test
Password for 'test':
-
Once we’ve properly authenticated we want to share the .subversion directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.
Bash
export ATHENS_STORAGE=~/athens-storage
export ATHENS_SVN=~/.subversion
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-v $ATHENS_SVN:/root/.subversion \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
$env:ATHENS_SVN = "$(Join-Path $pwd .subversion)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-v "$($env:ATHENS_SVN):/root/.subversion" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Bazaar(bzr) private repositories
- Bazaar is not supported with the Dockerfile provided by Athens. but the instructions are valid for custom Athens build with bazaar.*
- Bazaaar config files are located in
- There are 3 typical configuration files
- bazaar.conf
- locations.conf
- branch specific overrides and/or settings
- authentication.conf
- credential information for remote servers
- Configuration file syntax
-
Authentication Configuration
Allows one to specify credentials for remote servers.
This can be used for all the supported transports and any part of bzr that requires authentication(smtp for example).
The syntax obeys the same rules as the others except for the option policies which don’t apply.
Example:
[myprojects]
scheme=ftp
host=host.com
user=joe
password=secret
Pet projects on hobby.net
[hobby]
host=r.hobby.net
user=jim
password=obvious1234
Home server
[home]
scheme=https
host=home.net
user=joe
password=lessobV10us
[DEFAULT]
Our local user is barbaz, on all remote sites we’re known as foobar
user=foobar
NOTE: when using sftp the scheme is ssh and a password isn’t supported you should use PPK
[reference code]
scheme=https
host=dev.company.com
path=/dev
user=user1
password=pass1
development branches on dev server
[dev]
scheme=ssh # bzr+ssh and sftp are availablehere
host=dev.company.com
path=/dev/integration
user=user2
#proxy
[proxy]
scheme=http
host=proxy.company.com
port=3128
user=proxyuser1
password=proxypass1
-
Once we’ve properly setup our authentication we want to share the bazaar configuration directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.
Bash
export ATHENS_STORAGE=~/athens-storage
export ATHENS_BZR=~/.bazaar
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-v $ATHENS_BZR:/root/.bazaar \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
$env:ATHENS_BZR = "$(Join-Path $pwd .bazaar)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-v "$($env:ATHENS_BZR):/root/.bazaar" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Atlassian Bitbucket and SSH-secured git VCS’s
This section was originally written to describe configuring the
Athens git client to fetch specific Go imports over SSH instead of
HTTP against an on-prem instance of Atlassian Bitbucket. With some
adjustment it may point the way to configuring the Athens proxy for
authenticated access to hosted Bitbucket and other SSH-secured
VCS’s. If your developer workflow requires that you clone, push,
and pull Git repositories over SSH and you want Athens to perform
the same way, please read on.
As a developer at example.com, assume your application has a
dependency described by this import which is hosted on Bitbucket
import "git.example.com/golibs/logo"
Further, assume that you would manually clone this import like this
$ git clone ssh://git@git.example.com:7999/golibs/logo.git
A go-get client, such as that called by Athens, would begin
resolving this dependency by looking
for a go-import meta tag in this output
$ curl -s https://git.example.com/golibs/logo?go-get=1
<?xml version="1.0"?>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="go-import" content="git.example.com/golibs/logo git https://git.example.com/scm/golibs/logo.git"/>
<body/>
</meta>
</head>
</html>
which says the content of the Go import actually resides at
https://git.example.com/scm/golibs/logo.git. Comparing this URL
to what we would normally use to clone this project over SSH (above)
suggests this global Git config
http to ssh rewrite rule
[url "ssh://git@git.example.com:7999"]
insteadOf = https://git.example.com/scm
So to fetch the git.example.com/golibs/logo dependency over SSH
to populate its storage cache, Athens ultimately calls git, which,
given that rewrite rule, in turn needs an SSH private key matching
a public key bound to the cloning developer or service account on
Bitbucket. This is essentially the github.com SSH model. At a bare
minimum, we need to provide Athens with an SSH private key and the
http to ssh git rewrite rule, mounted inside the Athens container
for use by the root user
$ mkdir -p storage
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
-v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
-v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
-v "$PWD/ssh-keys:/root/.ssh" \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary
$PWD/gitconfig/.gitconfig contains the http to ssh rewrite rule
[url "ssh://git@git.example.com:7999"]
insteadOf = https://git.example.com/scm
$PWD/ssh-keys contains the aforementioned private key and a minimal ssh-config
$ ls ssh-keys/
config id_rsa
We also provide an ssh config to bypass host SSH key verification
and to show how to bind different hosts to different SSH keys
$PWD/ssh-keys/config contains
Host git.example.com
Hostname git.example.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa
Now, builds executed through the Athens proxy should be able to clone the git.example.com/golibs/logo dependency over authenticated SSH.
SSH_AUTH_SOCK and ssh-agent Support
As an alternative to passwordless SSH keys, one can use an ssh-agent.
The ssh-agent-set SSH_AUTH_SOCK environment variable will propagate to
go mod download if it contains a path to a valid unix socket (after
following symlinks).
As a result, if running with a working ssh agent (and a shell with
SSH_AUTH_SOCK set), after setting up a gitconfig as mentioned in the
previous section, one can run athens in docker as such:
$ mkdir -p storage
$ ssh-add .ssh/id_rsa_something
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
-v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
-v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
-v "${SSH_AUTH_SOCK}:/.ssh_agent_sock" \
-e "SSH_AUTH_SOCK=/.ssh_agent_sock" \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary
Configuring Storage
Storage
The Athens proxy supports many storage types:
All of them can be configured using config.toml file. You need to set a valid driver in StorageType value or you can set it in environment variable ATHENS_STORAGE_TYPE on your server.
Also for most of the drivers you need to provide additional configuration data which will be described below.
Memory
This storage doesn’t need any specific configuration and it’s also used by default in the Athens project. It writes all of data into local disk into tmp dir.
This storage type should only be used for development purposes!
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "memory"
Disk
Disk storage allows modules to be stored on a file system. The location on disk where modules will be stored can be configured.
You can pre-fill disk-based storage to enable Athens deployments that have no access to the internet. See here for instructions on how to do that.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "disk"
[Storage]
[Storage.Disk]
RootPath = "/path/on/disk"
where /path/on/disk is your desired location. Also it can be set using ATHENS_DISK_STORAGE_ROOT env
Mongo
This driver uses a Mongo server as data storage. On start this driver will create an athens database and module collection on your Mongo server.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "mongo"
[Storage]
[Storage.Mongo]
# Full URL for mongo storage
# Env override: ATHENS_MONGO_STORAGE_URL
URL = "mongodb://127.0.0.1:27017"
# Not required parameter
# Path to certificate to use for the mongo connection
# Env override: ATHENS_MONGO_CERT_PATH
CertPath = "/path/to/cert/file"
# Not required parameter
# Allows for insecure SSL / http connections to mongo storage
# Should be used for testing or development only
# Env override: ATHENS_MONGO_INSECURE
Insecure = false
# Not required parameter
# Allows for use of custom database
# Env override: ATHENS_MONGO_DEFAULT_DATABASE
DefaultDBName = athens
# Not required parameter
# Allows for use of custom collection
# Env override: ATHENS_MONGO_DEFAULT_COLLECTION
DefaultCollectionName = modules
Google Cloud Storage
This driver uses Google Cloud Storage and assumes that you already have an account and bucket in it.
If you never used Google Cloud Storage there is quick guide
how to create bucket inside it.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "gcp"
[Storage]
[Storage.GCP]
# ProjectID to use for GCP Storage
# Env overide: GOOGLE_CLOUD_PROJECT
ProjectID = "YOUR_GCP_PROJECT_ID"
# Bucket to use for GCP Storage
# Env override: ATHENS_STORAGE_GCP_BUCKET
Bucket = "YOUR_GCP_BUCKET"
AWS S3
This driver is using the AWS S3 and assumes that you already have account and bucket created in it.
If you never used Amazon Web Services there is quick guide how to create bucket inside it.
After this you can pass your credentials inside config.toml file. If the access key ID and secret access key are not specified in config.toml, the driver will attempt to load credentials for the default profile from the AWS CLI configuration file created during installation.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "s3"
[Storage]
[Storage.S3]
### The authentication model is as below for S3 in the following order
### If AWS_CREDENTIALS_ENDPOINT is specified and it returns valid results, then it is used
### If config variables are specified and they are valid, then they return valid results, then it is used
### Otherwise, it will default to default configurations which is as follows
# attempt to find credentials in the environment, in the shared
# configuration (~/.aws/credentials) and from ec2 instance role
# credentials. See
# https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
# and
# https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
# for environment variables that will affect the aws configuration.
# Setting UseDefaultConfiguration would only use default configuration. It will be deprecated in future releases
# and is recommended not to use it.
# Region for S3 storage
# Env override: AWS_REGION
Region = "MY_AWS_REGION"
# Access Key for S3 storage
# Env override: AWS_ACCESS_KEY_ID
Key = "MY_AWS_ACCESS_KEY_ID"
# Secret Key for S3 storage
# Env override: AWS_SECRET_ACCESS_KEY
Secret = "MY_AWS_SECRET_ACCESS_KEY"
# Session Token for S3 storage
# Not required parameter
# Env override: AWS_SESSION_TOKEN
Token = ""
# S3 Bucket to use for storage
# Env override: ATHENS_S3_BUCKET_NAME
Bucket = "MY_S3_BUCKET_NAME"
# If true then path style url for s3 endpoint will be used
# Env override: AWS_FORCE_PATH_STYLE
ForcePathStyle = false
# If true then the default aws configuration will be used. This will
# attempt to find credentials in the environment, in the shared
# configuration (~/.aws/credentials) and from ec2 instance role
# credentials. See
# https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
# and
# https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
# for environment variables that will affect the aws configuration.
# Env override: AWS_USE_DEFAULT_CONFIGURATION
UseDefaultConfiguration = false
# https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/
# Note that this the URI should not end with / when AwsContainerCredentialsRelativeURI is set
# Env override: AWS_CREDENTIALS_ENDPOINT
CredentialsEndpoint = ""
# Env override: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
# If you are planning to use AWS Fargate, please use http://169.254.170.2 for CredentialsEndpoint
# Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html
AwsContainerCredentialsRelativeURI = ""
# An optional endpoint URL (hostname only or fully qualified URI)
# that overrides the default generated endpoint for S3 storage client.
#
# You must still provide a `Region` value when specifying an endpoint.
# Env override: AWS_ENDPOINT
Endpoint = ""
Minio
Minio is an open source object storage server that provides an interface for S3 compatible block storages. If you have never used minio, you can read this quick start guide. Any S3 compatible object storage is supported by Athens through the minio interface. Below, you can find different configuration options we provide for Minio. Example configuration for Digital Ocean and Alibaba OSS block storages are provided below.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Endpoint for Minio storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "127.0.0.1:9001"
# Access Key for Minio storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_MINIO_SECRET_KEY"
# Secret Key for Minio storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_MINIO_SECRET_KEY"
# Enable SSL for Minio connections
# Defaults to true
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = false
# Minio Bucket to use for storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "gomods"
DigitalOcean Spaces
For Athens to communicate with DigitalOcean Spaces, we are using Minio driver because DO Spaces tries to be fully compatible with it.
Also configuration for this storage looks almost the same in our proxy as for Minio.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Address of DO Spaces storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "YOUR_ADDRESS.digitaloceanspaces.com"
# Access Key for DO Spaces storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_DO_SPACE_KEY_ID"
# Secret Key for DO Spaces storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_DO_SPACE_SECRET_KEY"
# Enable SSL
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = true
# Space name in your DO Spaces storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "YOUR_DO_SPACE_NAME"
# Region for DO Spaces storage
# Env override: ATHENS_MINIO_REGION
Region = "YOUR_DO_SPACE_REGION"
Alibaba OSS
For Athens to communicate with Alibaba Cloud Object Storage Service, we are using Minio driver.
Also configuration for this storage looks almost the same in our proxy as for Minio.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Address of Alibaba OSS storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "YOUR_ADDRESS.aliyuncs.com"
# Access Key for Minio storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_OSS_KEY_ID"
# Secret Key for Alibaba OSS storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_OSS_SECRET_KEY"
# Enable SSL
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = true
# Parent folder in your Alibaba OSS storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "YOUR_OSS_FOLDER_PREFIX"
Azure Blob Storage
This driver uses Azure Blob Storage
If you never used Azure Blog Storage, here is a quickstart
It assumes that you already have the following:
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "azureblob"
[Storage]
[Storage.AzureBlob]
# Storage Account name for Azure Blob
# Env override: ATHENS_AZURE_ACCOUNT_NAME
AccountName = "MY_AZURE_BLOB_ACCOUNT_NAME"
# Account Key to use with the storage account
# Env override: ATHENS_AZURE_ACCOUNT_KEY
AccountKey = "MY_AZURE_BLOB_ACCOUNT_KEY"
# Name of container in the blob storage
# Env override: ATHENS_AZURE_CONTAINER_NAME
ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"
External Storage
External storage lets Athens connect to your own implementation of a storage backend.
All you have to do is implement the storage.Backend interface and run it behind an http server.
Once you implement the backend server, you must then configure Athens to use that storage backend as such:
Configuration:
# Env override: ATHENS_STORAGE_TYPE
StorageType = "external"
[Storage]
[Storage.External]
# Env override: ATHENS_EXTERNAL_STORAGE_URL
URL = "http://localhost:9090"
Athens provides a convenience wrapper that lets you implement a storage backend with ease. See the following example:
package main
import (
"github.com/gomods/athens/pkg/storage"
"github.com/gomods/athens/pkg/storage/external"
)
// TODO: implement storage.Backend
type myCustomStorage struct {
storage.Backend
}
func main() {
handler := external.NewServer(&myCustomStorage{})
http.ListenAndServe(":9090", handler)
}
Running multiple Athens pointed at the same storage
Athens has the ability to run concurrently pointed at the same storage medium, using
a distributed locking mechanism called “single flight”.
By default, Athens is configured to use the memory single flight, which
stores locks in local memory. This works when running a single Athens instance, given
the process has access to it’s own memory. However, when running multiple Athens instances
pointed at the same storage, a distributed locking mechansism is required.
Athens supports several distributed locking mechanisms:
etcdredisredis-sentinelgcp (available when using the gcp storage type)azureblob (available when using the azureblob storage type)
Setting the SingleFlightType (or ATHENS_SINGLE_FLIGHT TYPE in the environment) configuration
value will enable usage of one of the above mechanisms. The azureblob and gcp types require
no extra configuration.
Using etcd as the single flight mechanism
Using the etcd mechanism is very simple, just a comma separated list of etcd endpoints.
The recommend configuration is 3 endpoints, however, more can be used.
SingleFlightType = "etcd"
[SingleFlight]
[SingleFlight.Etcd]
# Env override: ATHENS_ETCD_ENDPOINTS
Endpoints = "localhost:2379,localhost:22379,localhost:32379"
Using redis as the single flight mechanism
Athens supports two mechanisms of communicating with redis: direct connection, and
connecting via redis sentinels.
Direct connection to redis
Using a direct connection to redis is simple, and only requires a single redis-server.
You can also optionally specify a password to connect to the redis server with
SingleFlightType = "redis"
[SingleFlight]
[SingleFlight.Redis]
# Endpoint is the redis endpoint for the single flight mechanism
# Env override: ATHENS_REDIS_ENDPOINT
Endpoint = "127.0.0.1:6379"
# Password is the password for the redis instance
# Env override: ATHENS_REDIS_PASSWORD
Password = ""
Customizing lock configurations:
If you would like to customize the distributed lock options then you can optionally override the default lock config to better suit your use-case:
[SingleFlight.Redis]
...
[SingleFlight.Redis.LockConfig]
# TTL for the lock in seconds. Defaults to 900 seconds (15 minutes).
# Env override: ATHENS_REDIS_LOCK_TTL
TTL = 900
# Timeout for acquiring the lock in seconds. Defaults to 15 seconds.
# Env override: ATHENS_REDIS_LOCK_TIMEOUT
Timeout = 15
# Max retries while acquiring the lock. Defaults to 10.
# Env override: ATHENS_REDIS_LOCK_MAX_RETRIES
MaxRetries = 10
Customizations may be required in some cases for eg, you can set a higher TTL if it usually takes longer than 5 mins to fetch the modules in your case.
Connecting to redis via redis sentinel
NOTE: redis-sentinel requires a working knowledge of redis and is not recommended for
everyone.
redis sentinel is a high-availability set up for redis, it provides automated monitoring, replication,
failover and configuration of multiple redis servers in a leader-follower setup. It is more
complex than running a single redis server and requires multiple disperate instances of redis
running distributed across nodes.
For more details on redis-sentinel, check out the documentation
As redis-sentinel is a more complex set up of redis, it requires more configuration than standard redis.
Required configuration:
Endpoints is a list of redis-sentinel endpoints to connect to, typically 3, but more can be usedMasterName is the named master instance, as configured in the redis-sentinel configuration
Optionally, like redis, you can also specify a password to connect to the redis-sentinel endpoints with
SingleFlightType = "redis-sentinel"
[SingleFlight]
[SingleFlight.RedisSentinel]
# Endpoints is the redis sentinel endpoints to discover a redis
# master for a SingleFlight lock.
# Env override: ATHENS_REDIS_SENTINEL_ENDPOINTS
Endpoints = ["127.0.0.1:26379"]
# MasterName is the redis sentinel master name to use to discover
# the master for a SingleFlight lock
MasterName = "redis-1"
# SentinelPassword is an optional password for authenticating with
# redis sentinel
SentinelPassword = "sekret"
Distributed lock options can be customised for redis sentinal as well, in a similar manner as described above for redis.
Proxying a checksum database API
If you run go get github.com/mycompany/secret-repo@v1.0.0 and that module version is not yet in your go.sum file, Go will by default send a request to https://sum.golang.org/lookup/github.com/mycompany/secret-repo@v1.0.0. That request will fail because the Go tool requires a checksum, but sum.golang.org doesn’t have access to your private code.
The result is that (1) your build will fail, and (2) your private module names have been sent over the internet to an opaque public server that you don’t control.
You can read more about this sum.golang.org service here
Proxying a checksum DB
Many companies use Athens to host their private code, but Athens is not only a module proxy. It’s also a checksum database proxy. That means that anyone inside of your company can configure go to send these checksum requests to Athens instead of the public sum.golang.org server.
If the Athens server is configured with checksum filters, then you can prevent these problems.
If you run the below command using Go 1.13 or later:
$ GOPROXY=<athens-url> go build .
… then the Go tool will automatically send all checksum requests to <athens-url>/sumdb/sum.golang.org instead of https://sum.golang.org.
By default, when Athens receives a /sumdb/... request, it automatically proxies it to https://sum.golang.org, even if it’s a private module that sum.golang.org doesn’t and can’t know about. So if you are working with private modules, you’ll want to change the default behavior.
If you want Athens to not send some module names up to the global checksum database, set those module names in the NoSumPatterns value in config.toml or using the ATHENS_GONOSUM_PATTERNS environment variable.
The following sections will go into more detail on how checksum databases work, how Athens fits in, and how this all impacts your workflow.
How to set this all up
Before you begin, you’ll need to run Athens with configuration values that tell it to not proxy certain modules. If you’re using config.toml, use this configuration:
NoSumPatterns = ["github.com/mycompany/*", "github.com/secret/*"]
And if you’re using an environment variable, use this configuration:
$ export ATHENS_GONOSUM_PATTERNS="github.com/mycompany/*,github.com/secret/*"
You can use any string compatible with path.Match in these environment variables
After you start Athens up with this configuration, all checksum requests for modules that start with github.com/mycompany or github.com/secret will not be forwarded, and Athens will return an error to the go CLI tool.
This behavior will ensure that none of your private module names leak to the public internet, but your builds will still fail. To fix that problem, set another environment variable on your machine (that you run your go commands)
$ export GONOSUMDB="github.com/mycompany/*,github.com/secret/*"
Now, your builds will work and you won’t be sending information about your private codebase to the internet.
I’m confused, why is this hard?
When the Go tool has to download new code that isn’t currently in the project’s go.sum file, it tries its hardest to get a checksum from a server it trusts, and compare it to the checksum in the actual code it downloads. It does all of this to ensure provenance. That is, to ensure that the code you just downloaded wasn’t tampered with.
The trusted checksums are all stored in sum.golang.org, and that server is centrally controlled.
These build failures and potential privacy leaks can only happen when you try to get a module version that is not already in your go.sum file.
Athens does its best to respect and use the trusted checksums while also ensuring that your private names don’t get leaked to the public server. In some cases, it has to choose whether to fail your build or leak information, so it chooses to fail your build. That’s why everybody using that Athens server needs to set up their GONOSUMDB environment variable.
We believe that along with good documentation - which we hope this is! - we have struck the right balance between convenience and privacy.
Pre-filling disk storage
One of the popular features of Athens is that it can be run completely cut off from the internet. In this case, though, it can’t reach out to an upstream (e.g. a VCS or another module proxy) to fetch modules that it doesn’t have in storage. So, we need to manually fill up the disk partition that Athens uses with the dependencies that we need.
This document will guide you through packaging up a single module called github.com/my/module, and inserting it into the Athens disk storage.
You’ll need to produce the following assets from module source code:
source.zip - just the Go source code, packaged in a zip filego.mod - just the go.mod file from the module$VERSION.info - metadata about the module
The source.zip file has a specific directory structure and the $VERSION.info has a JSON structure, both of which you’ll need to get right in order for Athens to serve up the right dependency formats that the Go toolchain will accept.
We don’t recommend that you create these assets yourself. Instead, use pacmod or gopack
Using pacmod
To install the pacmod tool, run go get like this:
$ go get github.com/plexsystems/pacmod@v0.4.0
This command will install the pacmod binary to your $GOPATH/bin/pacmod directory, so make sure that is in your $PATH.
Next, run pacmod to create assets
After you have pacmod, you’ll need the module source code that you want to package. Before you run the command, set the VERSION variable in your environment to the version of the module you want to generate assets for.
Below is an example for how to configure it.
$ export VERSION="v1.0.0"
Note: make sure your VERSION variable starts with a v
Next, navigate to the top-level directory of the module source code, and run pacmod like this:
$ pacmod pack github.com/my/module $VERSION .
Once this command is done, you’ll notice three new files in the same same directory you ran the command from:
go.mod$VERSION.info$VERSION.zip
Using gopack
To use this method you need docker-compose installed.
Fork gopack project and clone it to your local machine (or just download files to your computer)
Edit goget.sh with a list of go modules you want to download:
#!/bin/bash
go get github.com/my/module1;
go get github.com/my/module2;
Run
docker-compose up --abort-on-container-exit
Once this command is done, you’ll notice in the ATHENS_STORAGE folder all modules ready to be moved to your Athens disk storage.
Next, move assets into Athens storage directory
Now that you have assets built, you need to move them into the location of the Athens disk storage. In the below commands, we’ll assume $STORAGE_ROOT is the environment variable that points to the top-level directory that Athens uses for its on-disk.
If you set up Athens with the $ATHENS_DISK_STORAGE_ROOT environment variable, the root of this storage location is the value of this environment variable. Use export STORAGE_ROOT=$ATHENS_DISK_STORAGE_ROOT to prepare your environment for the below commands.
First create the subdirectory into which you’ll move the assets you created:
$ mkdir -p $STORAGE_ROOT/github.com/my/module/$VERSION
Finally, make sure that you’re still in the module source repository root directory (the same as you were in when you ran the pacmod command), and move your three new files into the new directory you just created:
$ mv go.mod $STORAGE_ROOT/github.com/my/module/$VERSION/go.mod
$ mv $VERSION.info $STORAGE_ROOT/github.com/my/module/$VERSION/$VERSION.info
$ mv $VERSION.zip $STORAGE_ROOT/github.com/my/module/$VERSION/source.zip
Note that we’ve changed the name of the .zip file
Finally, test your setup
At this point, your Athens server should have its disk-based cache filled with the github.com/my/module module at version $VERSION. Next time you request this module, Athens will find it in its disk storage and will not try to fetch it from an upstream source.
You can quickly test this behavior by running below curl command, assuming your Athens server is running on http://localhost:3000 and is already configured to use the same disk storage that you pre-filled above.
$ curl localhost:3000/github.com/my/module/@v/$VERSION.info
When you run this command, Athens should immediately return, without contacting any other network services.
Filtering modules (deprecated)
Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to filter modules in Athens.
The proxy supports the following three use cases
- Fetches a module directly from the source (upstream proxy)
- Exclude a particular module
- Include a module in the local proxy.
These settings can be done by creating a configuration file which can be pointed by setting either
FilterFile in config.dev.toml or setting ATHENS_FILTER_FILE as an environment variable.
Writing the configuration file
Every line of the configuration can start either with a
+ denoting that the module has to be included by the proxyD denoting that the module has to be fetched directly from an upstream proxy and not stored locally- denoting that the module is excluded and will not be fetched into the proxy or from the upstream proxy
It allows for # to add comments and new lines are skipped. Anything else would result in an error
Sample configuration file
# This is a comment
- github.com/azure
+ github.com/azure/azure-sdk-for-go
# get golang tools directly
D golang.org/x/tools
In the above example, golang.org/x/tools is fetched directly from the upstream proxy. All the modules from github.com/azure are excluded except github.com/azure/azure-sdk-for-go
Adding a default mode
The list of modules can grow quickly in size and sometimes may want to specify configuration for a handful of modules. In this case, they can set a default mode for all the modules and add specific rules to certain modules that they want to apply to. The default rule is specified at the beginning of the file. It can be an either +, - or D
An example default mode is
D
- github.com/manugupt1/athens
+ github.com/gomods/athens
In the above example, all the modules are fetched directly from the source. github.com/manugupt1/athens is excluded and github.com/gomods/athens is stored in the proxy storage.
Adding versions to the filter
Using an “approved list” is a common practice that requires each minor or patch version to be approved before it is allowed in the codebase. This is accomplished by adding a list of version patterns to the rule. These version patterns are comma-separated and prefix-matching, so v2 and v2.3.* both match the requested version 2.3.5.
An example version filter is
-
# use internal github enterprise server directly
D enterprise.github.com/company
# external dependency approved list
+ github.com/gomods/athens v0.1,v0.2,v0.4.1
In the above example, any module not in the rules will be excluded. All modules from enterprise.github.com/company are fetched directly from the source. The github.com/gomods/athens module will be stored in the proxy storage, but only for version v0.4.1 and any patch versions under v0.1 and v0.2 minor versions.
Versions Filter Modifiers
Athens provides advanced filter modifiers to cover cases such as API compatibility or when a given dependency changes its license from a given versions. The modifiers are intended to be used in the pattern list of the filter file.
-
# external dependency approved list
+ github.com/gomods/athens
The currently supported modifiers are
-
~1.2.3 will enable all patch versions from 1.2.3 and above (e.g. 1.2.3, 1.2.4, 1.2.5)
- Formally,
1.2.x where x >= 3
-
^1.2.3 will enable all patch and minor versions from 1.2.3 and above (e.g. 1.2.4, 1.3.0 and 1.4.5)
- Formally,
1.x.y where x >= 2 and y >= 3
-
<1.2.3 will enable all versions lower than 1.2.3 (e.g. 1.2.2, 1.0.0 and 0.58.9)
- Formally,
x.y.z where x <= 1, y < = 2 and z < 3
This kind of modifiers will work only if a three parts semantic version is specified. For example, ~4.5.6 will work while ~4.5 won’t.
Using an upstream Go modules repository (deprecated)
Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to set upstream repositories in Athens.
By default, Athens fetches module code from an upstream version control system (VCS) like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.
-
Create a filter file (e.g /usr/local/lib/FilterForGoCenter) with letter D (stands for “direct access”) in first line. For more details, please refer to documentation on - Filtering Modules
# FilterFile for fetching modules directly from upstream
D
-
If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment).
Additionally in the current or new config file, set the following parameters as suggested:
FilterFile = "/usr/local/lib/FilterForGoCenter"
GlobalEndpoint = "https://<url_to_upstream>"
# To use GoCenter for example, replace <url_to_upstream> with gocenter.io
# You can also use https://proxy.golang.org to use the Go Module mirror
-
Restart Athens specifying the updated current or new config file.
<path_to_athens>/proxy -config_file <path-to updated current or new configfile>
-
Verify the new configuration using the steps mentioned in “Try out Athens” document, and go through the same walkthrough example.
Home template configuration
As of v0.14.0 Athens ships with a default, minimal HTML home page that advises users on how to connect to the proxy. It factors in whether GoNoSumPatterns is configured, and attempts
to build configuration for GO_PROXY. It relies on the users request Host header (on HTTP 1.1) or the Authority header (on HTTP 2) as well as whether the request was over TLS to advise
on configuring GO_PROXY. Lastly, the homepage provides a quick guide on how users can leverage the Athens API.
Of course, not all instructions will be this simple. Some installations may be reachable at different addresses in CI than for desktop users. In this case, and others where the default
home page does not make sense it is possible to override the template.
Do so by configuring HomeTemplatePath via the config or ATHENS_HOME_TEMPLATE_PATH to a location on disk with a Go HTML template or placing a template file at /var/lib/athens/home.html.
Athens automatically injects the following variables in templates:
| Setting |
Source |
Host |
Built from the request Host (HTTP1) or Authority (HTTP2) header and presence of TLS. Includes ports. |
NoSumPatterns |
Comes directly from the configuration. |
Using these values is done by wrapping them in bracers with a dot prepended. Example: {{ .Host }}
For more advanced formatting read more about Go HTML templates.
<!DOCTYPE html>
<html>
<head>
<title>Athens</title>
<style>
body {
font-family: Arial, sans-serif;
margin: 20px;
}
pre {
background-color: #f4f4f4;
padding: 5px;
border-radius: 5px;
width: fit-content;
padding: 10px;
}
code {
background-color: #f4f4f4;
padding: 5px;
border-radius: 5px;
}
</style>
</head>
<body>
<h1>Welcome to Athens</h1>
<h2>Configuring your client</h2>
<pre>GOPROXY={{ .Host }},direct</pre>
{{ if .NoSumPatterns }}
<h3>Excluding checksum database</h3>
<p>Use the following GONOSUM environment variable to exclude checksum database:</p>
<pre>GONOSUM={{ .NoSumPatterns }}</pre>
{{ end }}
<h2>How to use the Athens API</h2>
<p>Use the <a href="/catalog">catalog</a> endpoint to get a list of all modules in the proxy</p>
<h3>List of versions</h3>
<p>This endpoint returns a list of versions that Athens knows about for <code>acidburn/htp</code>:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/list</pre>
<h3>Version info</h3>
<p>This endpoint returns information about a specific version of a module:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.info</pre>
<p>This returns JSON with information about v1.0.0. It looks like this:
<pre>{
"Name": "v1.0.0",
"Short": "v1.0.0",
"Version": "v1.0.0",
"Time": "1972-07-18T12:34:56Z"
}</pre>
<h3>go.mod file</h3>
<p>This endpoint returns the go.mod file for a specific version of a module:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.mod</pre>
<p>This returns the go.mod file for version v1.0.0. If {{ .Host }}/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:</p>
<pre>module github.com/acidburn/htp</pre>
<h3>Module sources</h3>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.zip</pre>
<p>This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.</p>
<h3>Latest</h3>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@latest</pre>
<p>This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.</p>
</body>
</html>
Logging
Athens is designed to support a myriad of logging scenarios.
Standard
The standard structured logger can be configured in plain or json formatting via LogFormat or ATHENS_LOG_FORMAT. Additionally, verbosity can be controlled by setting LogLevel or ATHENS_LOG_LEVEL. In order for the standard structured logger to work, CloudRuntime and ATHENS_CLOUD_RUNTIME should not be set to a valid value.
Runtimes
Athens can be configured according to certain cloud provider specific runtimes. The GCP runtime configures Athens to rename certain logging fields that could be dropped or overriden when running in a GCP logging environment. This runtime can be used with LogLevel or ATHENS_LOG_LEVEL to control the verbosity of logs.
Subsections of The Design of Athens
Proxy
The Athens Proxy
The Athens proxy has two primary use cases:
- Internal deployments
- Public mirror deployments
This document details features of the Athens proxy that you can use to achieve either use case.
The Role of the Athens proxy
We intend proxies to be deployed primarily inside of enterprises to:
- Host private modules
- Exclude access to public modules
- Store public modules
Importantly, a proxy is not intended to be a complete mirror of an upstream proxy. For public modules, its role is to store the modules locally and provide access control.
What happens when a public module is not stored?
When a user requests a module MxV1 from a proxy and the Athens proxy doesn’t have MxV1 in its store, it first determines whether MxV1 is private or not private.
If it’s private, it immediately stores the module into the proxy storage from the internal VCS.
If it’s not private, the Athens proxy consults its exclude list for non-private modules (see below). If MxV1 is on the exclude list, the Athens proxy returns 404 and does nothing else. If MxV1 is not on the exclude list, the Athens proxy executes the following algorithm:
upstreamDetails := lookUpstream(MxV1)
if upstreamDetails == nil {
return 404 // if the upstream doesn't have the thing, just bail out
}
return upstreamDetails.baseURL
The important part of this algorithm is lookUpstream. That function queries an endpoint on the upstream proxy that either:
- Returns 404 if it doesn’t have
MxV1 in its storage
- Returns the base URL for MxV1 if it has
MxV1 in its storage
In a later version of the project, we may implement an event stream on proxies that any other proxy can subscribe to and listen for deletions/deprecations on modules that it cares about
Exclude Lists and Private Module Filters
To accommodate private (i.e. enterprise) deployments, the Athens proxy maintains two important access control mechanisms:
- Private module filters
- Exclude lists for public modules
Private Module Filters
Private module filters are string globs that tell the Athens proxy what is a private module. For example, the string github.internal.com/** tells the Athens proxy:
- To never make requests to the public internet (i.e. to upstream proxies) regarding this module
- To download module code (in its store mechanism) from the VCS at
github.internal.com
Exclude Lists for Public Modules
Exclude lists for public modules are also globs that tell the Athens proxy what modules it should never download from any upstream proxy. For example, the string github.com/arschles/** tells the Athens proxy to always return 404 Not Found to clients.
Catalog Endpoint
The proxy provides a /catalog service endpoint to fetch all the modules and their versions contained in the local storage. The endpoint accepts a continuation token and a page size parameter in order to provide paginated results.
A query is of the form
https://proxyurl/catalog?token=foo&pagesize=47
Where token is an optional continuation token and pagesize is the desired size of the returned page.
The token parameter is not required for the first call and it’s needed for handling paginated results.
The result is a json with the following structure:
{"modules": [{"module":"github.com/athens-artifacts/no-tags","version":"v1.0.0"}],
"next":""}'
If a next token is not returned, then it means that no more pages are available. The default page size is 1000.
Communication
Communication flow
This is the story of a time long ago. A time of myth and legend when the ancient Gods were petty and cruel and they plagued build with irreproducibility.
Only one project dared to challenge their power…Athens. Athens possessed a strength the world had never seen. Filling its storage.
Clean plate
At the beginning, there’s theoretical state when a storage of the Athens proxy is empty.
When User makes a request at this ancient time, it works as described on the flow below.
- User runs
go get to acquire new module.
- Go CLI contacts the Athens proxy asking for module M, version v1.0
- The Athens proxy checks whether or not it has this module in its storage. It does not.
- The Athens proxy downloads code from the underlying VCS and converts it into the Go Module format.
- After it receives all the bits, it stores it into its own storage and serves it to the User.
- User receives module and is happy.
The process from the user using go get all the way to the user downloading a module is synchronous.
Happy path
Now that the Athens proxy is aware of module M at version v1.0, it can serve that module immediately to the user, without fetching it from the VCS.
From VCS to the User
You read about proxy, communication and then opened a codebase and thought to yourself: This is not as simple as described in the docs.
Athens has a set of architectural components that handle Go modules from their journey from the VCS into storage and down to the user. If you feel lost on how all these pieces work, read on my friend!
From communication, you know that when a module is not backed up in the storage it gets downloaded from VCS (such as github.com) and then it is served to the user. You also know that this whole process is synchronous. But when you read a code you see module fetchers and download protocol stashers and you struggle to figure out what’s what and how they differ. It might seem complicated, but this document will help explain everything that’s going on.
Components
Let’s start with describing all the components you will see along the way. There’s no better way to get a clear picture of everything than with a diagram.
As you can see, there are a lot of layers and wrappers. The first two pieces you’ll see in the code are the Storage and the Fetcher. We’ll start our journey there.
Storage
Storage is what it sounds like. Storage instance created in proxy/storage.go’s GetStorage function.
Based on storage type passed as an ENV variable it will create in-memory, filesystem, mongo… storage.
This is where modules live. Once there, always there.
Fetcher
Fetcher is the first component on our way. As we can guess from the name, Fetcher (pkg/module/fetcher.go) is responsible for fetching the sources from VCS.
For this, it needs two things: the go binary and afero.FileSystem. Path to binary and filesystems are passed to Fetcher during initialization.
mf, err := module.NewGoGetFetcher(goBin, fs)
if err != nil {
return err
}
app_proxy.go
When a request for a new module comes, the Fetch function is invoked.
Fetch(ctx context.Context, mod, ver string) (*storage.Version, error)
fetch function
Then the Fetcher:
- creates a temp directory using an injected
FileSystem
- in this temp dir, it constructs dummy go project consisting of simple
main.go and go.mod so the go CLI can be used.
- invokes
go mod download -json {module}
This command downloads the module into the storage. Once the download is completed,
- the
Fetch function reads the module bits from storage and returns them to the caller.
- The exact path of module files is returned by
go mod as part of JSON response.
Stash
As it is important for us to keep components small and readable, we did not want to bloat Fetcher with storing functionality. For storing modules into a storage we use Stasher. This is the single responsibility of a simple Stasher.
We think it’s important to keep components small and orthogonal, so the Fetcher and the storage.Backend don’t interact. Instead, the Stasher composes them together and orchestrates the process of fetching code and then storing it.
The New method accepts the Fetcher and the storage.Backend with a set of wrappers (explained later).
New(f module.Fetcher, s storage.Backend, wrappers ...Wrapper) Stasher
stasher.go
The code in pkg/stash/stasher.go isn’t complex, but it’s important:
I think this does two things:
- invokes
Fetcher to get module bits
- stores the bits using a
storage
If you read carefully you noticed wrappers passed into a basic Stasher implementation.
These wrappers add more advanced logic and help to keep components clean.
The new method then returns a Stasher which is a result of wrapping basic Stasher with wrappers.
for _, w := range wrappers {
st = w(st)
}
stasher.go
Stash wrapper - Pool
As downloading a module is resource heavy (memory) operation, Pool (pkg/stash/with_pool.go) helps us to control simultaneous downloads.
It uses N-worker pattern which spins up the specified number of workers which then waits for a job to complete. Once they complete their job, they return the result and are ready for the next one.
A job, in this case, is a call to Stash function on a backing Stasher.
Stash wrapper - SingleFlight
We know that module fetching is a resource-heavy operation and we just put a limit on a number of parallel downloads. To help us save more resources we wanted to avoid processing the same module multiple times.
SingleFlight wrapper (pkg/stash/with_singleflight.go) takes care of that.
Internally it keeps track of currently running downloads using a map.
If a job arrives and map[moduleVersion] is empty, it initiates it with a callback channel and invokes a job on a backing Stasher.
s.subs[mv] = []chan error{subCh}
go s.process(ctx, mod, ver)
if there is an entry for the requested module, SingleFlight will subscribe for a result
s.subs[mv] = append(s.subs[mv], subCh)
and once the job is complete, the module is served one level up to the download protocol (or to wrapping stasher possibly)
Download protocol
The outer most level is the download protocol.
dpOpts := &download.Opts{
Storage: s,
Stasher: st,
Lister: lister,
}
dp := download.New(dpOpts, addons.WithPool(protocolWorkers))
It contains two components we already mentioned: Storage, Stasher
and one more additional: Lister.
Lister is used in List and Latest funcs to look upstream for the list of versions available.
Storage is here again. We saw it in a Stasher before, used for saving.
In Download protocol it is used to check whether or not the module is already present. If it is, it is served directly from storage.
Otherwise, Download protocol uses Stasher to download module, store it into a storage and then it serves it back to the user.
You can also see addons.WithPool in a code snippet above. This addon is something similar to Stash wrapper - Pool. It controls the number of concurrent requests proxy can handle.
FAQ
Is Athens Just a Proxy? A Registry?
TL;DR “Registry” doesn’t describe what Athens is trying to do here. That implies that there’s only one service in the world that can serve Go modules to everyone. Athens isn’t trying to be that. Instead, Athens is trying to be part of a federated group of module proxies.
A registry is generally run by one entity, is one logical server that provides authentication (and provenance sometimes), and is pretty much the de-facto only source of dependencies. Sometimes it’s run by a for-profit company.
That’s most definitely not what we in the Athens community are going for, and that would harm our community if we did go down that path.
First and foremost, Athens is an implementation of the Go Modules download API. Not only does the standard Go toolchain support any implementation of that API, the Athens proxy is designed to talk to any other server that implements that API as well. That allows Athens to talk to other proxies in the community.
Finally, we’re purposefully building this project - and working with the toolchain folks - in a way that everyone who wants to write a proxy can participate.
Athens is currently supported by the Go v1.12+ toolchain via the download protocol.
For the TL;DR of the protocol, it’s a REST API that lets the go toolchain (i.e. go get) see lists of versions and fetch source code for a specific version.
Athens is a server that implements the protocol. Both it, the protocol and the toolchain (as you almost certainly know) is open source.
Are the packages served by Athens immutable?
TL;DR Athens does store code in CDNs and has the option to store code in other persistent datastores.
The longer version:
It’s virtually impossible to ensure immutable builds when source code comes from Github. We have been annoyed by that problem for a long time. The Go modules download protocol is a great opportunity to solve this issue. The Athens proxy works pretty simply at a high level:
go get github.com/my/module@v1 happens- Athens looks in its datastore, it’s missing
- Athens downloads
github.com/my/module@v1 from Github (it uses go get on the backend too)
- Athens stores the module in its datastore
- Athens serves
github.com/my/module@v1 from its datastore forever
To repeat, “datastore” means a CDN (we currently have support for Google Cloud Storage, Azure Blob Storage and AWS S3) or another datastore (we have support for MongoDB, disk and some others).
Can the Athens proxy authenticate to private repositories?
TL;DR: yes, with proper authentication configuration defined on the Athens proxy host.
When the GOPROXY environment variable is set on the client-side, the Go 1.11+ cli
does not attempt to request the meta tags, via a request that looks like https://example.org/pkg/foo?go-get=1.
Internally Athens uses go get under the hood (go mod download to be exact)
without the GOPROXY environment variable set so that go will in turn request
the meta tags using the standard authentication mechanisms supported by go.
Therefore, if go before v1.11 worked for you, then go 1.11+ with GOPROXY
should work as well, provided that the Athens proxy host is configured with the
proper authentication.
Can I exclude a module completely?
Yes, this is possible. The proxy provides a configuration file that will allow users to specify which modules that should not be fetched at all. The filtering modules configuration provides details about the configuration file and how to exclude certain modules.
Can I specify that a module is fetched from an upstream proxy and not stored locally?
Yes, this is possible. Refer to the filtering modules configuration provides details about the configuration file and how to exclude certain modules.
Is there support for monitoring and observability for Proxy?
Right now, we have structured logs for proxy. Along with that, we have added tracing to help developers identify critical code paths and debug latency issues. While there is no setup required for logs, tracing requires some installation. We currently support exporting traces with Jaeger, GCP Stackdriver & Datadog (untested). Further support for other exporters is in progress.
To try out tracing with Jaeger, do the following:
-
Set the environment to development (otherwise traces will be sampled)
-
Run docker-compose up -d that is found in the athens source root directory to initialize the services required
-
Run the walkthrough tutorial
-
Open http://localhost:16686/search
Observability is not a hard requirement for the Athens proxy. So, if the infrastructure is not properly set up, it will fail with an information log. For example, if Jaeger is not running or if the wrong URL to the exporter is provided, the proxy will continue to run. However, it will not collect any traces or metrics while the exporter backend is unavailable.
What VCS servers does Athens support?
Athens uses go mod download under the hood, so it supports anything go mod suppports.
Which currently includes:
When should I use a vendor directory, and when should I use Athens?
The Go community has used vendor directories for a long time before module proxies like Athens came along, so naturally each group collaborating on code should decide for themselves whether they want to use a vendor directory, use Athens, or do both!
Using a vendor directory (without a proxy) is valuable when:
- CI/CD systems don’t have access to an Athens (even if it’s internal)
- When the vendor directory is so small that it is still faster to check it out from a repo than it is to pull zip files from the server
- If you’re coming from glide/dep or another dependency management system that leveraged the vendor directory
Athens (without a vendor directory) is valuable when:
- You have a new project
- You are upgrading a Go project to use Go modules
- Your team requires that you use Athens (i.e. for isolation or dependency auditing)
- Your vendor directory is large and causing slow checkouts and downloading from Athens speeds the build up
- For developers slow checkouts will not be as much of a problem as for ci tools which frequently need to checkout fresh copies of the project
- You want to remove the vendor directory from your project to:
- Reduce noise in pull requests
- Reduce difficulty doing fuzzy file searching in your project
Walkthrough
First, make sure you have Go v1.12+ installed and that GOPATH/bin is on your path.
Without the Athens proxy
Let’s review what everything looks like in Go without the Athens proxy in the picture:
Bash
$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ GO111MODULE=on go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!
PowerShell
$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ $env:GO111MODULE = "on"
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!
The end result of running this command is that Go downloaded the package source and packaged
it into a module, saving it in the Go Modules local storage.
Now that we have seen Go Modules in action without the Athens proxy, let’s take a look at
how the Athens proxy changes the workflow and the output.
With the Athens proxy
Using the most simple installation possible, let’s walk through how to use the
Athens proxy, and figure out what is happening at each step.
Before moving on, let’s clear our Go Modules local files so that we can see the Athens proxy
in action without any modules locally populated:
Bash
sudo rm -fr $(go env GOPATH)/pkg/mod
PowerShell
rm -recurse -force "$(go env GOPATH)\pkg\mod"
Now run the Athens proxy in a background process:
Bash
$ mkdir -p $(go env GOPATH)/src/github.com/gomods
$ cd $(go env GOPATH)/src/github.com/gomods
$ git clone https://github.com/gomods/athens.git
$ cd athens
$ GO111MODULE=on go run ./cmd/proxy -config_file=./config.dev.toml &
[1] 25243
INFO[0000] Starting application at 127.0.0.1:3000
PowerShell
$ mkdir "$(go env GOPATH)\src\github.com\gomods"
$ cd "$(go env GOPATH)\src\github.com\gomods"
$ git clone https://github.com/gomods/athens.git
$ cd athens
$ $env:GO111MODULE = "on"
$ $env:GOPROXY = "https://proxy.golang.org"
$ Start-Process -NoNewWindow go 'run .\cmd\proxy -config_file=".\config.dev.toml"'
[1] 25243
INFO[0000] Starting application at 127.0.0.1:3000
The Athens proxy is now running in the background and is listening for requests
from localhost (127.0.0.1) on port 3000.
Since we didn’t provide any specific configuration
the Athens proxy is using in-memory storage, which is only suitable for trying out the Athens proxy
for a short period of time, as you will quickly run out of memory and the storage
doesn’t persist between restarts.
With Docker
For more details on running Athens in docker, take a look at the install documentation
In order to run the Athens Proxy using docker, we need first to create a directory that will store the persitant modules.
In the example below, the new directory is named athens-storage and is located in our userspace (i.e. $HOME).
Then we need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when we run the Docker container.
Bash
export ATHENS_STORAGE=$HOME/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $HOME athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Next, you will need to enable the Go Modules
feature and configure Go to use the Athens proxy!
Using the Athens proxy
Bash
export GO111MODULE=on
export GOPROXY=http://127.0.0.1:3000
PowerShell
$env:GO111MODULE = "on"
$env:GOPROXY = "http://127.0.0.1:3000"
The GO111MODULE environment variable controls the Go Modules feature in Go 1.11 only.
Possible values are:
on: Always use Go Modulesauto (default): Only use Go Modules when a go.mod file is present, or the go command is run from outside the GOPATHoff: Never use Go Modules
The GOPROXY environment variable tells the go binary that instead of talking to
the version control system, such as github.com, directly when resolving your package
dependencies, instead it should communicate with a proxy. The Athens proxy implements
the Go Download Protocol, and is responsible for listing available
versions for a package in addition to providing a zip of particular package versions.
Now, you when you build and run this example application, go will fetch dependencies via Athens!
$ cd ../walkthrough
$ go run .
go: finding github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.info [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.mod [200]
go: downloading github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.zip [200]
The 🦁 says rawr!
The output from go run . includes attempts to find the github.com/athens-artifacts/samplelib dependency. Since the
proxy was run in the background, you should also see output from Athens indicating that it is handling requests for the dependency.
Let’s break down what is happening here:
-
Before Go runs our code, it detects that our code depends on the github.com/athens-artifacts/samplelib package
which is not present in the Go Modules local storage.
-
At this point the Go Modules feature comes into play because we have it enabled.
Instead of looking in the GOPATH for the package, Go reads our go.mod file
and sees that we want a particular version of that package, v1.0.0.
module github.com/athens-artifacts/walkthrough
require github.com/athens-artifacts/samplelib v1.0.0
-
Go first checks for github.com/athens-artifacts/samplelib@v1.0.0 in the Go Modules local storage,
located in GOPATH/pkg/mod. If that version of the package is already local storage,
then Go will use it and stop looking. But since this is our first time
running this, our local storage is empty and Go keeps looking.
-
Go requests github.com/athens-artifacts/samplelib@v1.0.0 from our proxy because
it is set in the GOPROXY environment variable.
-
The Athens proxy checks its own storage (in this case is in-memory) for the package and doesn’t find it. So it
retrieves it from github.com and then saves it for subsequent requests.
-
Go downloads the module zip and puts it in the Go Modules local storage
GOPATH/pkg/mod.
-
Go will use the module and build our application!
Subsequent calls to go run . will be much less verbose:
$ go run .
The 🦁 says rawr!
No additional output is printed because Go found github.com/athens-artifacts/samplelib@v1.0.0 in the Go Module
local storage and did not need to request it from the Athens proxy.
Lastly, quitting from the Athens proxy. This cannot be done directly because we are starting the Athens proxy in the background, thus we must kill it by finding it’s process ID and killing it manually.
Bash
lsof -i @localhost:3000
kill -9 <<PID>>
PowerShell
netstat -ano | findstr :3000 (local host Port number)
taskkill /PID typeyourPIDhere /F
Next Steps
Now that you have seen Athens in Action:
