Subsections of Configuring Athens
The download mode file
Athens accepts an HCL formatted file that has instructions for how it should behave when a module@version isn’t found in its storage. This functionality gives Athens the flexibility configure Athens to fit your organization’s needs. The most popular uses of this download file are:
- Configure Athens to never download or serve a module or group of modules
- Redirect to a different module proxy for a module or group of modules
This document will outline how to use this file - called the download mode file - to accomplish these tasks and more.
Please see the “Use cases” section below for more details on how to enable these behaviors and more.
Configuration
First, once you’ve created your download mode file, you tell Athens to use it by setting the DownloadMode
configuration parameter in the config.toml
file, or setting the ATHENS_DOWNLOAD_MODE
environment variable. You can set this configuration value to one of two values to tell Athens to use your file:
- Set its value to
file:$FILE_PATH
, where $FILE_PATH
is the path to the HCL file
- Set its value to
custom:$BASE_64
where $BASE_64
is the base64 encoded HCL file
Instead of one of the above two values, you can set this configuration to sync
, async
, none
, redirect
, or async_redirect
. If you do, the download mode will be set globally rather than for specific sub-groups of modules. See below for what each of these values mean.
Download mode keywords
If Athens receives a request for the module github.com/pkg/errors
at version v0.8.1
, and it doesn’t have that module and version in its storage, it will consult the download mode file for specific instructions on what action to take:
sync
: Synchronously download the module from VCS via go mod download
, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.
async
: Return a 404 to the client, and asynchronously download and persist the module@version to storage.
none
: Return a 404 and do nothing.
redirect
: Redirect to an upstream proxy (such as proxy.golang.org) and do nothing after.
async_redirect
: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage.
Athens expects these keywords to be used in conjunction with module patterns (github.com/pkg/*
, for example). You combine the keyword and the pattern to specify behavior for a specific group of modules.
Athens uses the Go path.Match function to parse module patterns.
Below is an example download mode file.
downloadURL = "https://proxy.golang.org"
mode = "async_redirect"
download "github.com/gomods/*" {
mode = "sync"
}
download "golang.org/x/*" {
mode = "none"
}
download "github.com/pkg/*" {
mode = "redirect"
downloadURL = "https://gocenter.io"
}
The first two lines describe the default behavior for all modules. This behavior is overridden for select module groups below. In this case, the default behavior is:
- Immediatley redirect all requests to
https://proxy.golang.org
- In the background, download the module from the version control system (VCS) and store it
The rest of the file contains download
blocks. These override the default behavior for specific groups of modules.
The first block specifies that any module matching github.com/gomods/*
(such as github.com/gomods/athens
) will be downloaded from GitHub, stored, and then returned to the user.
The second block specifies that any module matching golang.org/x/*
(such as golang.org/x/text
) will always return a HTTP 404 response code. This behavior ensures that Athens will never store or serve any module names starting with golang.org/x
.
If a user has their GOPROXY
environment variable set with a comma separated list, their go
command line tool will always try the option next in the list. For example, if a user has their GOPROXY
environment variable set to https://athens.azurefd.net,direct
, and then runs go get golang.org/x/text
, they will still download golang.org/x/text
to their machine. The module just won’t come from Athens.
The last block specifies that any module matching github.com/pkg/*
(such as github.com/pkg/errors
) will always redirect the go
tool to https://gocenter.io. In this case, Athens will never persist the given module to its storage.
Use cases
The download mode file is versatile and allows you to configure Athens in a large variety of different ways. Below are some of the mode common.
Blocking certain modules
If you’re running Athens to serve a team of Go developers, it might be useful to ensure that the team doesn’t use a specific group or groups of modules (for example, because of licensing or security issues).
In this case, you would write this in your file:
download "bad/module/repo/*" {
mode = "none"
}
Preventing storage overflow
If you are running Athens using a storage backend that has limited space, you may want to prevent Athens from storing certain groups of modules that take up a lot of space. To avoid exhausting Athens storage, while still ensuring that the users of your Athens server still get access to the modules you can’t store, you would use a redirect
directive, as shown below:
download "very/large/*" {
mode = "redirect"
url = "https://reliable.proxy.com"
}
If you use the redirect
mode, make sure that you specify a url
value that points to a reliable proxy.
Authentication to private repositories
Authentication
SVN private repositories
-
Subversion creates an authentication structure in
~/.subversion/auth/svn.simple/<hash>
-
In order to properly create the authentication file for your SVN servers you will need to authenticate to them and let svn build out the proper hashed files.
$ svn list http://<domain:port>/svn/<somerepo>
Authentication realm: <http://<domain> Subversion Repository
Username: test
Password for 'test':
-
Once we’ve properly authenticated we want to share the .subversion directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.
Bash
export ATHENS_STORAGE=~/athens-storage
export ATHENS_SVN=~/.subversion
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-v $ATHENS_SVN:/root/.subversion \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
$env:ATHENS_SVN = "$(Join-Path $pwd .subversion)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-v "$($env:ATHENS_SVN):/root/.subversion" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Bazaar(bzr) private repositories
- Bazaar is not supported with the Dockerfile provided by Athens. but the instructions are valid for custom Athens build with bazaar.*
- Bazaaar config files are located in
- There are 3 typical configuration files
- bazaar.conf
- locations.conf
- branch specific overrides and/or settings
- authentication.conf
- credential information for remote servers
- Configuration file syntax
-
Authentication Configuration
Allows one to specify credentials for remote servers.
This can be used for all the supported transports and any part of bzr that requires authentication(smtp for example).
The syntax obeys the same rules as the others except for the option policies which don’t apply.
Example:
[myprojects]
scheme=ftp
host=host.com
user=joe
password=secret
Pet projects on hobby.net
[hobby]
host=r.hobby.net
user=jim
password=obvious1234
Home server
[home]
scheme=https
host=home.net
user=joe
password=lessobV10us
[DEFAULT]
Our local user is barbaz, on all remote sites we’re known as foobar
user=foobar
NOTE: when using sftp the scheme is ssh and a password isn’t supported you should use PPK
[reference code]
scheme=https
host=dev.company.com
path=/dev
user=user1
password=pass1
development branches on dev server
[dev]
scheme=ssh # bzr+ssh and sftp are availablehere
host=dev.company.com
path=/dev/integration
user=user2
#proxy
[proxy]
scheme=http
host=proxy.company.com
port=3128
user=proxyuser1
password=proxypass1
-
Once we’ve properly setup our authentication we want to share the bazaar configuration directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.
Bash
export ATHENS_STORAGE=~/athens-storage
export ATHENS_BZR=~/.bazaar
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
-v $ATHENS_BZR:/root/.bazaar \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
-e ATHENS_STORAGE_TYPE=disk \
--name athens-proxy \
--restart always \
-p 3000:3000 \
gomods/athens:latest
PowerShell
$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
$env:ATHENS_BZR = "$(Join-Path $pwd .bazaar)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
-v "$($env:ATHENS_BZR):/root/.bazaar" `
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
-e ATHENS_STORAGE_TYPE=disk `
--name athens-proxy `
--restart always `
-p 3000:3000 `
gomods/athens:latest
Atlassian Bitbucket and SSH-secured git VCS’s
This section was originally written to describe configuring the
Athens git client to fetch specific Go imports over SSH instead of
HTTP against an on-prem instance of Atlassian Bitbucket. With some
adjustment it may point the way to configuring the Athens proxy for
authenticated access to hosted Bitbucket and other SSH-secured
VCS’s. If your developer workflow requires that you clone, push,
and pull Git repositories over SSH and you want Athens to perform
the same way, please read on.
As a developer at example.com, assume your application has a
dependency described by this import which is hosted on Bitbucket
import "git.example.com/golibs/logo"
Further, assume that you would manually clone this import like this
$ git clone ssh://git@git.example.com:7999/golibs/logo.git
A go-get
client, such as that called by Athens, would begin
resolving this dependency by looking
for a go-import
meta tag in this output
$ curl -s https://git.example.com/golibs/logo?go-get=1
<?xml version="1.0"?>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="go-import" content="git.example.com/golibs/logo git https://git.example.com/scm/golibs/logo.git"/>
<body/>
</meta>
</head>
</html>
which says the content of the Go import actually resides at
https://git.example.com/scm/golibs/logo.git
. Comparing this URL
to what we would normally use to clone this project over SSH (above)
suggests this global Git config
http to ssh rewrite rule
[url "ssh://git@git.example.com:7999"]
insteadOf = https://git.example.com/scm
So to fetch the git.example.com/golibs/logo
dependency over SSH
to populate its storage cache, Athens ultimately calls git, which,
given that rewrite rule, in turn needs an SSH private key matching
a public key bound to the cloning developer or service account on
Bitbucket. This is essentially the github.com SSH model. At a bare
minimum, we need to provide Athens with an SSH private key and the
http to ssh git rewrite rule, mounted inside the Athens container
for use by the root user
$ mkdir -p storage
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
-v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
-v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
-v "$PWD/ssh-keys:/root/.ssh" \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary
$PWD/gitconfig/.gitconfig
contains the http to ssh rewrite rule
[url "ssh://git@git.example.com:7999"]
insteadOf = https://git.example.com/scm
$PWD/ssh-keys
contains the aforementioned private key and a minimal ssh-config
$ ls ssh-keys/
config id_rsa
We also provide an ssh config to bypass host SSH key verification
and to show how to bind different hosts to different SSH keys
$PWD/ssh-keys/config
contains
Host git.example.com
Hostname git.example.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa
Now, builds executed through the Athens proxy should be able to clone the git.example.com/golibs/logo
dependency over authenticated SSH.
SSH_AUTH_SOCK
and ssh-agent
Support
As an alternative to passwordless SSH keys, one can use an ssh-agent
.
The ssh-agent
-set SSH_AUTH_SOCK
environment variable will propagate to
go mod download
if it contains a path to a valid unix socket (after
following symlinks).
As a result, if running with a working ssh agent (and a shell with
SSH_AUTH_SOCK
set), after setting up a gitconfig
as mentioned in the
previous section, one can run athens in docker as such:
$ mkdir -p storage
$ ssh-add .ssh/id_rsa_something
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
-v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
-v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
-v "${SSH_AUTH_SOCK}:/.ssh_agent_sock" \
-e "SSH_AUTH_SOCK=/.ssh_agent_sock" \
-e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary
Configuring Storage
Storage
The Athens proxy supports many storage types:
All of them can be configured using config.toml
file. You need to set a valid driver in StorageType
value or you can set it in environment variable ATHENS_STORAGE_TYPE
on your server.
Also for most of the drivers you need to provide additional configuration data which will be described below.
Memory
This storage doesn’t need any specific configuration and it’s also used by default in the Athens project. It writes all of data into local disk into tmp
dir.
This storage type should only be used for development purposes!
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "memory"
Disk
Disk storage allows modules to be stored on a file system. The location on disk where modules will be stored can be configured.
You can pre-fill disk-based storage to enable Athens deployments that have no access to the internet. See here for instructions on how to do that.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "disk"
[Storage]
[Storage.Disk]
RootPath = "/path/on/disk"
where /path/on/disk
is your desired location. Also it can be set using ATHENS_DISK_STORAGE_ROOT
env
Mongo
This driver uses a Mongo server as data storage. On start this driver will create an athens
database and module
collection on your Mongo server.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "mongo"
[Storage]
[Storage.Mongo]
# Full URL for mongo storage
# Env override: ATHENS_MONGO_STORAGE_URL
URL = "mongodb://127.0.0.1:27017"
# Not required parameter
# Path to certificate to use for the mongo connection
# Env override: ATHENS_MONGO_CERT_PATH
CertPath = "/path/to/cert/file"
# Not required parameter
# Allows for insecure SSL / http connections to mongo storage
# Should be used for testing or development only
# Env override: ATHENS_MONGO_INSECURE
Insecure = false
# Not required parameter
# Allows for use of custom database
# Env override: ATHENS_MONGO_DEFAULT_DATABASE
DefaultDBName = athens
# Not required parameter
# Allows for use of custom collection
# Env override: ATHENS_MONGO_DEFAULT_COLLECTION
DefaultCollectionName = modules
Google Cloud Storage
This driver uses Google Cloud Storage and assumes that you already have an account
and bucket
in it.
If you never used Google Cloud Storage there is quick guide
how to create bucket
inside it.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "gcp"
[Storage]
[Storage.GCP]
# ProjectID to use for GCP Storage
# Env overide: GOOGLE_CLOUD_PROJECT
ProjectID = "YOUR_GCP_PROJECT_ID"
# Bucket to use for GCP Storage
# Env override: ATHENS_STORAGE_GCP_BUCKET
Bucket = "YOUR_GCP_BUCKET"
AWS S3
This driver is using the AWS S3 and assumes that you already have account
and bucket
created in it.
If you never used Amazon Web Services there is quick guide how to create bucket
inside it.
After this you can pass your credentials inside config.toml
file. If the access key ID and secret access key are not specified in config.toml
, the driver will attempt to load credentials for the default profile from the AWS CLI configuration file created during installation.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "s3"
[Storage]
[Storage.S3]
### The authentication model is as below for S3 in the following order
### If AWS_CREDENTIALS_ENDPOINT is specified and it returns valid results, then it is used
### If config variables are specified and they are valid, then they return valid results, then it is used
### Otherwise, it will default to default configurations which is as follows
# attempt to find credentials in the environment, in the shared
# configuration (~/.aws/credentials) and from ec2 instance role
# credentials. See
# https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
# and
# https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
# for environment variables that will affect the aws configuration.
# Setting UseDefaultConfiguration would only use default configuration. It will be deprecated in future releases
# and is recommended not to use it.
# Region for S3 storage
# Env override: AWS_REGION
Region = "MY_AWS_REGION"
# Access Key for S3 storage
# Env override: AWS_ACCESS_KEY_ID
Key = "MY_AWS_ACCESS_KEY_ID"
# Secret Key for S3 storage
# Env override: AWS_SECRET_ACCESS_KEY
Secret = "MY_AWS_SECRET_ACCESS_KEY"
# Session Token for S3 storage
# Not required parameter
# Env override: AWS_SESSION_TOKEN
Token = ""
# S3 Bucket to use for storage
# Env override: ATHENS_S3_BUCKET_NAME
Bucket = "MY_S3_BUCKET_NAME"
# If true then path style url for s3 endpoint will be used
# Env override: AWS_FORCE_PATH_STYLE
ForcePathStyle = false
# If true then the default aws configuration will be used. This will
# attempt to find credentials in the environment, in the shared
# configuration (~/.aws/credentials) and from ec2 instance role
# credentials. See
# https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
# and
# https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
# for environment variables that will affect the aws configuration.
# Env override: AWS_USE_DEFAULT_CONFIGURATION
UseDefaultConfiguration = false
# https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/
# Note that this the URI should not end with / when AwsContainerCredentialsRelativeURI is set
# Env override: AWS_CREDENTIALS_ENDPOINT
CredentialsEndpoint = ""
# Env override: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
# If you are planning to use AWS Fargate, please use http://169.254.170.2 for CredentialsEndpoint
# Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html
AwsContainerCredentialsRelativeURI = ""
# An optional endpoint URL (hostname only or fully qualified URI)
# that overrides the default generated endpoint for S3 storage client.
#
# You must still provide a `Region` value when specifying an endpoint.
# Env override: AWS_ENDPOINT
Endpoint = ""
Minio
Minio is an open source object storage server that provides an interface for S3 compatible block storages. If you have never used minio, you can read this quick start guide. Any S3 compatible object storage is supported by Athens through the minio interface. Below, you can find different configuration options we provide for Minio. Example configuration for Digital Ocean and Alibaba OSS block storages are provided below.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Endpoint for Minio storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "127.0.0.1:9001"
# Access Key for Minio storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_MINIO_SECRET_KEY"
# Secret Key for Minio storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_MINIO_SECRET_KEY"
# Enable SSL for Minio connections
# Defaults to true
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = false
# Minio Bucket to use for storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "gomods"
DigitalOcean Spaces
For Athens to communicate with DigitalOcean Spaces, we are using Minio driver because DO Spaces tries to be fully compatible with it.
Also configuration for this storage looks almost the same in our proxy as for Minio.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Address of DO Spaces storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "YOUR_ADDRESS.digitaloceanspaces.com"
# Access Key for DO Spaces storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_DO_SPACE_KEY_ID"
# Secret Key for DO Spaces storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_DO_SPACE_SECRET_KEY"
# Enable SSL
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = true
# Space name in your DO Spaces storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "YOUR_DO_SPACE_NAME"
# Region for DO Spaces storage
# Env override: ATHENS_MINIO_REGION
Region = "YOUR_DO_SPACE_REGION"
Alibaba OSS
For Athens to communicate with Alibaba Cloud Object Storage Service, we are using Minio driver.
Also configuration for this storage looks almost the same in our proxy as for Minio.
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"
[Storage]
[Storage.Minio]
# Address of Alibaba OSS storage
# Env override: ATHENS_MINIO_ENDPOINT
Endpoint = "YOUR_ADDRESS.aliyuncs.com"
# Access Key for Minio storage
# Env override: ATHENS_MINIO_ACCESS_KEY_ID
Key = "YOUR_OSS_KEY_ID"
# Secret Key for Alibaba OSS storage
# Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
Secret = "YOUR_OSS_SECRET_KEY"
# Enable SSL
# Env override: ATHENS_MINIO_USE_SSL
EnableSSL = true
# Parent folder in your Alibaba OSS storage
# Env override: ATHENS_MINIO_BUCKET_NAME
Bucket = "YOUR_OSS_FOLDER_PREFIX"
Azure Blob Storage
This driver uses Azure Blob Storage
If you never used Azure Blog Storage, here is a quickstart
It assumes that you already have the following:
Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "azureblob"
[Storage]
[Storage.AzureBlob]
# Storage Account name for Azure Blob
# Env override: ATHENS_AZURE_ACCOUNT_NAME
AccountName = "MY_AZURE_BLOB_ACCOUNT_NAME"
# Account Key to use with the storage account
# Env override: ATHENS_AZURE_ACCOUNT_KEY
AccountKey = "MY_AZURE_BLOB_ACCOUNT_KEY"
# Managed Identity Resource Id to use with the storage account
# Env override: ATHENS_AZURE_MANAGED_IDENTITY_RESOURCE_ID
ManagedIdentityResourceId = "MY_AZURE_MANAGED_IDENTITY_RESOURCE_ID"
# Storage Resource to use with the storage account
# Env override: ATHENS_AZURE_STORAGE_RESOURCE
StorageResource = "MY_AZURE_STORAGE_RESOURCE"
# Name of container in the blob storage
# Env override: ATHENS_AZURE_CONTAINER_NAME
ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"
External Storage
External storage lets Athens connect to your own implementation of a storage backend.
All you have to do is implement the storage.Backend interface and run it behind an http server.
Once you implement the backend server, you must then configure Athens to use that storage backend as such:
Configuration:
# Env override: ATHENS_STORAGE_TYPE
StorageType = "external"
[Storage]
[Storage.External]
# Env override: ATHENS_EXTERNAL_STORAGE_URL
URL = "http://localhost:9090"
Athens provides a convenience wrapper that lets you implement a storage backend with ease. See the following example:
package main
import (
"github.com/gomods/athens/pkg/storage"
"github.com/gomods/athens/pkg/storage/external"
)
// TODO: implement storage.Backend
type myCustomStorage struct {
storage.Backend
}
func main() {
handler := external.NewServer(&myCustomStorage{})
http.ListenAndServe(":9090", handler)
}
Running multiple Athens pointed at the same storage
Athens has the ability to run concurrently pointed at the same storage medium, using
a distributed locking mechanism called “single flight”.
By default, Athens is configured to use the memory
single flight, which
stores locks in local memory. This works when running a single Athens instance, given
the process has access to it’s own memory. However, when running multiple Athens instances
pointed at the same storage, a distributed locking mechansism is required.
Athens supports several distributed locking mechanisms:
etcd
redis
redis-sentinel
gcp
(available when using the gcp
storage type)
azureblob
(available when using the azureblob
storage type)
Setting the SingleFlightType
(or ATHENS_SINGLE_FLIGHT TYPE
in the environment) configuration
value will enable usage of one of the above mechanisms. The azureblob
and gcp
types require
no extra configuration.
Using etcd as the single flight mechanism
Using the etcd
mechanism is very simple, just a comma separated list of etcd endpoints.
The recommend configuration is 3 endpoints, however, more can be used.
SingleFlightType = "etcd"
[SingleFlight]
[SingleFlight.Etcd]
# Env override: ATHENS_ETCD_ENDPOINTS
Endpoints = "localhost:2379,localhost:22379,localhost:32379"
Using redis as the single flight mechanism
Athens supports two mechanisms of communicating with redis: direct connection, and
connecting via redis sentinels.
Direct connection to redis
Using a direct connection to redis is simple, and only requires a single redis-server
.
You can also optionally specify a password to connect to the redis server with
SingleFlightType = "redis"
[SingleFlight]
[SingleFlight.Redis]
# Endpoint is the redis endpoint for the single flight mechanism
# Env override: ATHENS_REDIS_ENDPOINT
Endpoint = "127.0.0.1:6379"
# Password is the password for the redis instance
# Env override: ATHENS_REDIS_PASSWORD
Password = ""
Customizing lock configurations:
If you would like to customize the distributed lock options then you can optionally override the default lock config to better suit your use-case:
[SingleFlight.Redis]
...
[SingleFlight.Redis.LockConfig]
# TTL for the lock in seconds. Defaults to 900 seconds (15 minutes).
# Env override: ATHENS_REDIS_LOCK_TTL
TTL = 900
# Timeout for acquiring the lock in seconds. Defaults to 15 seconds.
# Env override: ATHENS_REDIS_LOCK_TIMEOUT
Timeout = 15
# Max retries while acquiring the lock. Defaults to 10.
# Env override: ATHENS_REDIS_LOCK_MAX_RETRIES
MaxRetries = 10
Customizations may be required in some cases for eg, you can set a higher TTL if it usually takes longer than 5 mins to fetch the modules in your case.
Connecting to redis via redis sentinel
NOTE: redis-sentinel requires a working knowledge of redis and is not recommended for
everyone.
redis sentinel is a high-availability set up for redis, it provides automated monitoring, replication,
failover and configuration of multiple redis servers in a leader-follower setup. It is more
complex than running a single redis server and requires multiple disperate instances of redis
running distributed across nodes.
For more details on redis-sentinel, check out the documentation
As redis-sentinel is a more complex set up of redis, it requires more configuration than standard redis.
Required configuration:
Endpoints
is a list of redis-sentinel endpoints to connect to, typically 3, but more can be used
MasterName
is the named master instance, as configured in the redis-sentinel
configuration
Optionally, like redis
, you can also specify a password to connect to the redis-sentinel
endpoints with
SingleFlightType = "redis-sentinel"
[SingleFlight]
[SingleFlight.RedisSentinel]
# Endpoints is the redis sentinel endpoints to discover a redis
# master for a SingleFlight lock.
# Env override: ATHENS_REDIS_SENTINEL_ENDPOINTS
Endpoints = ["127.0.0.1:26379"]
# MasterName is the redis sentinel master name to use to discover
# the master for a SingleFlight lock
MasterName = "redis-1"
# SentinelPassword is an optional password for authenticating with
# redis sentinel
SentinelPassword = "sekret"
Distributed lock options can be customised for redis sentinal as well, in a similar manner as described above for redis.
Using GCP as a singleflight mechanism
The GCP singleflight mechanism does not required configuration, and works out of the box. It has a
single option with which it can be customized:
[SingleFlight.GCP]
# Threshold for how long to wait in seconds for an in-progress GCP upload to
# be considered to have failed to unlock.
StaleThreshold = 120
Proxying a checksum database API
If you run go get github.com/mycompany/secret-repo@v1.0.0
and that module version is not yet in your go.sum
file, Go will by default send a request to https://sum.golang.org/lookup/github.com/mycompany/secret-repo@v1.0.0
. That request will fail because the Go tool requires a checksum, but sum.golang.org
doesn’t have access to your private code.
The result is that (1) your build will fail, and (2) your private module names have been sent over the internet to an opaque public server that you don’t control.
You can read more about this sum.golang.org
service here
Proxying a checksum DB
Many companies use Athens to host their private code, but Athens is not only a module proxy. It’s also a checksum database proxy. That means that anyone inside of your company can configure go
to send these checksum requests to Athens instead of the public sum.golang.org
server.
If the Athens server is configured with checksum filters, then you can prevent these problems.
If you run the below command using Go 1.13 or later:
$ GOPROXY=<athens-url> go build .
… then the Go tool will automatically send all checksum requests to <athens-url>/sumdb/sum.golang.org
instead of https://sum.golang.org
.
By default, when Athens receives a /sumdb/...
request, it automatically proxies it to https://sum.golang.org
, even if it’s a private module that sum.golang.org
doesn’t and can’t know about. So if you are working with private modules, you’ll want to change the default behavior.
If you want Athens to not send some module names up to the global checksum database, set those module names in the NoSumPatterns
value in config.toml
or using the ATHENS_GONOSUM_PATTERNS
environment variable.
The following sections will go into more detail on how checksum databases work, how Athens fits in, and how this all impacts your workflow.
How to set this all up
Before you begin, you’ll need to run Athens with configuration values that tell it to not proxy certain modules. If you’re using config.toml
, use this configuration:
NoSumPatterns = ["github.com/mycompany/*", "github.com/secret/*"]
And if you’re using an environment variable, use this configuration:
$ export ATHENS_GONOSUM_PATTERNS="github.com/mycompany/*,github.com/secret/*"
You can use any string compatible with path.Match
in these environment variables
After you start Athens up with this configuration, all checksum requests for modules that start with github.com/mycompany
or github.com/secret
will not be forwarded, and Athens will return an error to the go
CLI tool.
This behavior will ensure that none of your private module names leak to the public internet, but your builds will still fail. To fix that problem, set another environment variable on your machine (that you run your go
commands)
$ export GONOSUMDB="github.com/mycompany/*,github.com/secret/*"
Now, your builds will work and you won’t be sending information about your private codebase to the internet.
I’m confused, why is this hard?
When the Go tool has to download new code that isn’t currently in the project’s go.sum
file, it tries its hardest to get a checksum from a server it trusts, and compare it to the checksum in the actual code it downloads. It does all of this to ensure provenance. That is, to ensure that the code you just downloaded wasn’t tampered with.
The trusted checksums are all stored in sum.golang.org
, and that server is centrally controlled.
These build failures and potential privacy leaks can only happen when you try to get a module version that is not already in your go.sum
file.
Athens does its best to respect and use the trusted checksums while also ensuring that your private names don’t get leaked to the public server. In some cases, it has to choose whether to fail your build or leak information, so it chooses to fail your build. That’s why everybody using that Athens server needs to set up their GONOSUMDB
environment variable.
We believe that along with good documentation - which we hope this is! - we have struck the right balance between convenience and privacy.
Pre-filling disk storage
One of the popular features of Athens is that it can be run completely cut off from the internet. In this case, though, it can’t reach out to an upstream (e.g. a VCS or another module proxy) to fetch modules that it doesn’t have in storage. So, we need to manually fill up the disk partition that Athens uses with the dependencies that we need.
This document will guide you through packaging up a single module called github.com/my/module
, and inserting it into the Athens disk storage.
You’ll need to produce the following assets from module source code:
source.zip
- just the Go source code, packaged in a zip file
go.mod
- just the go.mod
file from the module
$VERSION.info
- metadata about the module
The source.zip
file has a specific directory structure and the $VERSION.info
has a JSON structure, both of which you’ll need to get right in order for Athens to serve up the right dependency formats that the Go toolchain will accept.
We don’t recommend that you create these assets yourself. Instead, use pacmod or gopack
Using pacmod
To install the pacmod
tool, run go get
like this:
$ go get github.com/plexsystems/pacmod@v0.4.0
This command will install the pacmod
binary to your $GOPATH/bin/pacmod
directory, so make sure that is in your $PATH
.
Next, run pacmod
to create assets
After you have pacmod
, you’ll need the module source code that you want to package. Before you run the command, set the VERSION
variable in your environment to the version of the module you want to generate assets for.
Below is an example for how to configure it.
$ export VERSION="v1.0.0"
Note: make sure your VERSION
variable starts with a v
Next, navigate to the top-level directory of the module source code, and run pacmod
like this:
$ pacmod pack github.com/my/module $VERSION .
Once this command is done, you’ll notice three new files in the same same directory you ran the command from:
go.mod
$VERSION.info
$VERSION.zip
Using gopack
To use this method you need docker-compose installed.
Fork gopack project and clone it to your local machine (or just download files to your computer)
Edit goget.sh
with a list of go modules you want to download:
#!/bin/bash
go get github.com/my/module1;
go get github.com/my/module2;
Run
docker-compose up --abort-on-container-exit
Once this command is done, you’ll notice in the ATHENS_STORAGE folder all modules ready to be moved to your Athens disk storage.
Next, move assets into Athens storage directory
Now that you have assets built, you need to move them into the location of the Athens disk storage. In the below commands, we’ll assume $STORAGE_ROOT
is the environment variable that points to the top-level directory that Athens uses for its on-disk.
If you set up Athens with the $ATHENS_DISK_STORAGE_ROOT
environment variable, the root of this storage location is the value of this environment variable. Use export STORAGE_ROOT=$ATHENS_DISK_STORAGE_ROOT
to prepare your environment for the below commands.
First create the subdirectory into which you’ll move the assets you created:
$ mkdir -p $STORAGE_ROOT/github.com/my/module/$VERSION
Finally, make sure that you’re still in the module source repository root directory (the same as you were in when you ran the pacmod
command), and move your three new files into the new directory you just created:
$ mv go.mod $STORAGE_ROOT/github.com/my/module/$VERSION/go.mod
$ mv $VERSION.info $STORAGE_ROOT/github.com/my/module/$VERSION/$VERSION.info
$ mv $VERSION.zip $STORAGE_ROOT/github.com/my/module/$VERSION/source.zip
Note that we’ve changed the name of the .zip
file
Finally, test your setup
At this point, your Athens server should have its disk-based cache filled with the github.com/my/module
module at version $VERSION
. Next time you request this module, Athens will find it in its disk storage and will not try to fetch it from an upstream source.
You can quickly test this behavior by running below curl
command, assuming your Athens server is running on http://localhost:3000
and is already configured to use the same disk storage that you pre-filled above.
$ curl localhost:3000/github.com/my/module/@v/$VERSION.info
When you run this command, Athens should immediately return, without contacting any other network services.
Filtering modules (deprecated)
Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to filter modules in Athens.
The proxy supports the following three use cases
- Fetches a module directly from the source (upstream proxy)
- Exclude a particular module
- Include a module in the local proxy.
These settings can be done by creating a configuration file which can be pointed by setting either
FilterFile
in config.dev.toml
or setting ATHENS_FILTER_FILE
as an environment variable.
Writing the configuration file
Every line of the configuration can start either with a
+
denoting that the module has to be included by the proxy
D
denoting that the module has to be fetched directly from an upstream proxy and not stored locally
-
denoting that the module is excluded and will not be fetched into the proxy or from the upstream proxy
It allows for #
to add comments and new lines are skipped. Anything else would result in an error
Sample configuration file
# This is a comment
- github.com/azure
+ github.com/azure/azure-sdk-for-go
# get golang tools directly
D golang.org/x/tools
In the above example, golang.org/x/tools
is fetched directly from the upstream proxy. All the modules from github.com/azure
are excluded except github.com/azure/azure-sdk-for-go
Adding a default mode
The list of modules can grow quickly in size and sometimes may want to specify configuration for a handful of modules. In this case, they can set a default mode for all the modules and add specific rules to certain modules that they want to apply to. The default rule is specified at the beginning of the file. It can be an either +
, -
or D
An example default mode is
D
- github.com/manugupt1/athens
+ github.com/gomods/athens
In the above example, all the modules are fetched directly from the source. github.com/manugupt1/athens
is excluded and github.com/gomods/athens
is stored in the proxy storage.
Adding versions to the filter
Using an “approved list” is a common practice that requires each minor or patch version to be approved before it is allowed in the codebase. This is accomplished by adding a list of version patterns to the rule. These version patterns are comma-separated and prefix-matching, so v2
and v2.3.*
both match the requested version 2.3.5
.
An example version filter is
-
# use internal github enterprise server directly
D enterprise.github.com/company
# external dependency approved list
+ github.com/gomods/athens v0.1,v0.2,v0.4.1
In the above example, any module not in the rules will be excluded. All modules from enterprise.github.com/company
are fetched directly from the source. The github.com/gomods/athens
module will be stored in the proxy storage, but only for version v0.4.1
and any patch versions under v0.1
and v0.2
minor versions.
Versions Filter Modifiers
Athens provides advanced filter modifiers to cover cases such as API compatibility or when a given dependency changes its license from a given versions. The modifiers are intended to be used in the pattern list of the filter file.
-
# external dependency approved list
+ github.com/gomods/athens
The currently supported modifiers are
-
~1.2.3
will enable all patch versions from 1.2.3 and above (e.g. 1.2.3, 1.2.4, 1.2.5)
- Formally,
1.2.x
where x >= 3
-
^1.2.3
will enable all patch and minor versions from 1.2.3 and above (e.g. 1.2.4, 1.3.0 and 1.4.5)
- Formally,
1.x.y
where x >= 2
and y >= 3
-
<1.2.3
will enable all versions lower than 1.2.3 (e.g. 1.2.2, 1.0.0 and 0.58.9)
- Formally,
x.y.z
where x <= 1
, y < = 2
and z < 3
This kind of modifiers will work only if a three parts semantic version is specified. For example, ~4.5.6
will work while ~4.5
won’t.
Using an upstream Go modules repository (deprecated)
Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to set upstream repositories in Athens.
By default, Athens fetches module code from an upstream version control system (VCS) like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.
-
Create a filter file (e.g /usr/local/lib/FilterForGoCenter
) with letter D
(stands for “direct access”) in first line. For more details, please refer to documentation on - Filtering Modules
# FilterFile for fetching modules directly from upstream
D
-
If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment).
Additionally in the current or new config file, set the following parameters as suggested:
FilterFile = "/usr/local/lib/FilterForGoCenter"
GlobalEndpoint = "https://<url_to_upstream>"
# To use GoCenter for example, replace <url_to_upstream> with gocenter.io
# You can also use https://proxy.golang.org to use the Go Module mirror
-
Restart Athens specifying the updated current or new config file.
<path_to_athens>/proxy -config_file <path-to updated current or new configfile>
-
Verify the new configuration using the steps mentioned in “Try out Athens” document, and go through the same walkthrough example.
Home template configuration
As of v0.14.0 Athens ships with a default, minimal HTML home page that advises users on how to connect to the proxy. It factors in whether GoNoSumPatterns
is configured, and attempts
to build configuration for GO_PROXY
. It relies on the users request Host header (on HTTP 1.1) or the Authority header (on HTTP 2) as well as whether the request was over TLS to advise
on configuring GO_PROXY
. Lastly, the homepage provides a quick guide on how users can leverage the Athens API.
Of course, not all instructions will be this simple. Some installations may be reachable at different addresses in CI than for desktop users. In this case, and others where the default
home page does not make sense it is possible to override the template.
Do so by configuring HomeTemplatePath
via the config or ATHENS_HOME_TEMPLATE_PATH
to a location on disk with a Go HTML template or placing a template file at /var/lib/athens/home.html
.
Athens automatically injects the following variables in templates:
Setting |
Source |
Host |
Built from the request Host (HTTP1) or Authority (HTTP2) header and presence of TLS. Includes ports. |
NoSumPatterns |
Comes directly from the configuration. |
Using these values is done by wrapping them in bracers with a dot prepended. Example: {{ .Host }}
For more advanced formatting read more about Go HTML templates.
<!DOCTYPE html>
<html>
<head>
<title>Athens</title>
<style>
body {
font-family: Arial, sans-serif;
margin: 20px;
}
pre {
background-color: #f4f4f4;
padding: 5px;
border-radius: 5px;
width: fit-content;
padding: 10px;
}
code {
background-color: #f4f4f4;
padding: 5px;
border-radius: 5px;
}
</style>
</head>
<body>
<h1>Welcome to Athens</h1>
<h2>Configuring your client</h2>
<pre>GOPROXY={{ .Host }},direct</pre>
{{ if .NoSumPatterns }}
<h3>Excluding checksum database</h3>
<p>Use the following GONOSUM environment variable to exclude checksum database:</p>
<pre>GONOSUM={{ .NoSumPatterns }}</pre>
{{ end }}
<h2>How to use the Athens API</h2>
<p>Use the <a href="/catalog">catalog</a> endpoint to get a list of all modules in the proxy</p>
<h3>List of versions</h3>
<p>This endpoint returns a list of versions that Athens knows about for <code>acidburn/htp</code>:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/list</pre>
<h3>Version info</h3>
<p>This endpoint returns information about a specific version of a module:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.info</pre>
<p>This returns JSON with information about v1.0.0. It looks like this:
<pre>{
"Name": "v1.0.0",
"Short": "v1.0.0",
"Version": "v1.0.0",
"Time": "1972-07-18T12:34:56Z"
}</pre>
<h3>go.mod file</h3>
<p>This endpoint returns the go.mod file for a specific version of a module:</p>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.mod</pre>
<p>This returns the go.mod file for version v1.0.0. If {{ .Host }}/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:</p>
<pre>module github.com/acidburn/htp</pre>
<h3>Module sources</h3>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.zip</pre>
<p>This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.</p>
<h3>Latest</h3>
<pre>GET {{ .Host }}/github.com/acidburn/htp/@latest</pre>
<p>This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.</p>
</body>
</html>
Logging
Athens is designed to support a myriad of logging scenarios.
Standard
The standard structured logger can be configured in plain
or json
formatting via LogFormat
or ATHENS_LOG_FORMAT
. Additionally, verbosity can be controlled by setting LogLevel
or ATHENS_LOG_LEVEL
. In order for the standard structured logger to work, CloudRuntime
and ATHENS_CLOUD_RUNTIME
should not be set to a valid value.
The logging is via Logrus, so the allowed values for logging config options are determined by that project. For example, ATHENS_LOG_LEVEL
can be debug
, info
, warn
/warning
, error
, etc.
Runtimes
Athens can be configured according to certain cloud provider specific runtimes. The GCP runtime configures Athens to rename certain logging fields that could be dropped or overriden when running in a GCP logging environment. This runtime can be used with LogLevel
or ATHENS_LOG_LEVEL
to control the verbosity of logs.