The download mode file

Athens accepts an HCL formatted file that has instructions for how it should behave when a module@version isn’t found in its storage. This functionality gives Athens the flexibility configure Athens to fit your organization’s needs. The most popular uses of this download file are:

• Configure Athens to never download or serve a module or group of modules
• Redirect to a different module proxy for a module or group of modules

This document will outline how to use this file - called the download mode file - to accomplish these tasks and more.

Please see the “Use cases” section below for more details on how to enable these behaviors and more.

Configuration

First, once you’ve created your download mode file, you tell Athens to use it by setting the DownloadMode configuration parameter in the config.toml file, or setting the ATHENS_DOWNLOAD_MODE environment variable. You can set this configuration value to one of two values to tell Athens to use your file:

1. Set its value to file:$FILE_PATH, where $FILE_PATH is the path to the HCL file
2. Set its value to custom:$BASE_64 where $BASE_64 is the base64 encoded HCL file

Instead of one of the above two values, you can set this configuration to sync, async, none, redirect, or async_redirect. If you do, the download mode will be set globally rather than for specific sub-groups of modules. See below for what each of these values mean.

Download mode keywords

If Athens receives a request for the module github.com/pkg/errors at version v0.8.1, and it doesn’t have that module and version in its storage, it will consult the download mode file for specific instructions on what action to take:

1. sync: Synchronously download the module from VCS via go mod download, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.
2. async: Return a 404 to the client, and asynchronously download and persist the module@version to storage.
3. none: Return a 404 and do nothing.
4. redirect: Redirect to an upstream proxy (such as proxy.golang.org) and do nothing after.
5. async_redirect: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage.

Athens expects these keywords to be used in conjunction with module patterns (github.com/pkg/*, for example). You combine the keyword and the pattern to specify behavior for a specific group of modules.

Athens uses the Go path.Match function to parse module patterns.

Below is an example download mode file.

downloadURL = "https://proxy.golang.org"

mode = "async_redirect"

download "github.com/gomods/*" {
    mode = "sync"
}

download "golang.org/x/*" {
    mode = "none"
}

download "github.com/pkg/*" {
    mode = "redirect"
    downloadURL = "https://gocenter.io"
}

The first two lines describe the default behavior for all modules. This behavior is overridden for select module groups below. In this case, the default behavior is:

• Immediatley redirect all requests to https://proxy.golang.org
• In the background, download the module from the version control system (VCS) and store it

The rest of the file contains download blocks. These override the default behavior for specific groups of modules.

The first block specifies that any module matching github.com/gomods/* (such as github.com/gomods/athens) will be downloaded from GitHub, stored, and then returned to the user.

The second block specifies that any module matching golang.org/x/* (such as golang.org/x/text) will always return a HTTP 404 response code. This behavior ensures that Athens will never store or serve any module names starting with golang.org/x.

If a user has their GOPROXY environment variable set with a comma separated list, their go command line tool will always try the option next in the list. For example, if a user has their GOPROXY environment variable set to https://athens.azurefd.net,direct, and then runs go get golang.org/x/text, they will still download golang.org/x/text to their machine. The module just won’t come from Athens.

The last block specifies that any module matching github.com/pkg/* (such as github.com/pkg/errors) will always redirect the go tool to https://gocenter.io. In this case, Athens will never persist the given module to its storage.

Use cases

The download mode file is versatile and allows you to configure Athens in a large variety of different ways. Below are some of the mode common.

Blocking certain modules

If you’re running Athens to serve a team of Go developers, it might be useful to ensure that the team doesn’t use a specific group or groups of modules (for example, because of licensing or security issues).

In this case, you would write this in your file:

download "bad/module/repo/*" {
    mode = "none"
}

Preventing storage overflow

If you are running Athens using a storage backend that has limited space, you may want to prevent Athens from storing certain groups of modules that take up a lot of space. To avoid exhausting Athens storage, while still ensuring that the users of your Athens server still get access to the modules you can’t store, you would use a redirect directive, as shown below:

download "very/large/*" {
    mode = "redirect"
    url = "https://reliable.proxy.com"
}

If you use the redirect mode, make sure that you specify a url value that points to a reliable proxy.

Authentication to private repositories

Authentication

SVN private repositories

1. Subversion creates an authentication structure in

   ~/.subversion/auth/svn.simple/<hash>
   

2. In order to properly create the authentication file for your SVN servers you will need to authenticate to them and let svn build out the proper hashed files.

   $ svn list http://<domain:port>/svn/<somerepo>
   Authentication realm: <http://<domain> Subversion Repository
   Username: test
   Password for 'test':
   

3. Once we’ve properly authenticated we want to share the .subversion directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.

   Bash

   export ATHENS_STORAGE=~/athens-storage
   export ATHENS_SVN=~/.subversion
   mkdir -p $ATHENS_STORAGE
   docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
     -v $ATHENS_SVN:/root/.subversion \
     -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
     -e ATHENS_STORAGE_TYPE=disk \
     --name athens-proxy \
     --restart always \
     -p 3000:3000 \
     gomods/athens:latest

   PowerShell

   $env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
   $env:ATHENS_SVN = "$(Join-Path $pwd .subversion)"
   md -Path $env:ATHENS_STORAGE
   docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
     -v "$($env:ATHENS_SVN):/root/.subversion" `
     -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
     -e ATHENS_STORAGE_TYPE=disk `
     --name athens-proxy `
     --restart always `
     -p 3000:3000 `
     gomods/athens:latest

Bazaar(bzr) private repositories

• Bazaar is not supported with the Dockerfile provided by Athens. but the instructions are valid for custom Athens build with bazaar.*

1. Bazaaar config files are located in

• Unix

  ~/.bazaar/
  

• Windows

  C:\Documents and Settings\<username>\Application Data\Bazaar\2.0
  

• You can check your location using

  bzr version
  

2. There are 3 typical configuration files

• bazaar.conf
  • default config options
• locations.conf
  • branch specific overrides and/or settings
• authentication.conf
  • credential information for remote servers

3. Configuration file syntax

• # this is a comment

• [header] this denotes a section header

• section options reside in a header section and contain an option name an equals sign and a value

  • EXAMPLE:

    [DEFAULT]
    email = John Doe <jdoe@isp.com>
    

4. Authentication Configuration

   Allows one to specify credentials for remote servers. This can be used for all the supported transports and any part of bzr that requires authentication(smtp for example). The syntax obeys the same rules as the others except for the option policies which don’t apply.

   Example:

   [myprojects] scheme=ftp host=host.com user=joe password=secret

   Pet projects on hobby.net

   [hobby] host=r.hobby.net user=jim password=obvious1234

   Home server

   [home] scheme=https host=home.net user=joe password=lessobV10us

   [DEFAULT]

   Our local user is barbaz, on all remote sites we’re known as foobar

   user=foobar

   NOTE: when using sftp the scheme is ssh and a password isn’t supported you should use PPK

   [reference code] scheme=https host=dev.company.com path=/dev user=user1 password=pass1

   development branches on dev server

   [dev] scheme=ssh # bzr+ssh and sftp are availablehere host=dev.company.com path=/dev/integration user=user2

   #proxy [proxy] scheme=http host=proxy.company.com port=3128 user=proxyuser1 password=proxypass1

5. Once we’ve properly setup our authentication we want to share the bazaar configuration directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.

   Bash

   export ATHENS_STORAGE=~/athens-storage
   export ATHENS_BZR=~/.bazaar
   mkdir -p $ATHENS_STORAGE
   docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
     -v $ATHENS_BZR:/root/.bazaar \
     -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
     -e ATHENS_STORAGE_TYPE=disk \
     --name athens-proxy \
     --restart always \
     -p 3000:3000 \
     gomods/athens:latest

   PowerShell

   $env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
   $env:ATHENS_BZR = "$(Join-Path $pwd .bazaar)"
   md -Path $env:ATHENS_STORAGE
   docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
     -v "$($env:ATHENS_BZR):/root/.bazaar" `
     -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
     -e ATHENS_STORAGE_TYPE=disk `
     --name athens-proxy `
     --restart always `
     -p 3000:3000 `
     gomods/athens:latest

Atlassian Bitbucket and SSH-secured git VCS’s

This section was originally written to describe configuring the Athens git client to fetch specific Go imports over SSH instead of HTTP against an on-prem instance of Atlassian Bitbucket. With some adjustment it may point the way to configuring the Athens proxy for authenticated access to hosted Bitbucket and other SSH-secured VCS’s. If your developer workflow requires that you clone, push, and pull Git repositories over SSH and you want Athens to perform the same way, please read on.

As a developer at example.com, assume your application has a dependency described by this import which is hosted on Bitbucket

import "git.example.com/golibs/logo"

Further, assume that you would manually clone this import like this

$ git clone ssh://git@git.example.com:7999/golibs/logo.git

A go-get client, such as that called by Athens, would begin resolving this dependency by looking for a go-import meta tag in this output

$ curl -s https://git.example.com/golibs/logo?go-get=1
<?xml version="1.0"?>
<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="utf-8">
         <meta name="go-import" content="git.example.com/golibs/logo git https://git.example.com/scm/golibs/logo.git"/>
         <body/>
      </meta>
   </head>
</html>

which says the content of the Go import actually resides at https://git.example.com/scm/golibs/logo.git. Comparing this URL to what we would normally use to clone this project over SSH (above) suggests this global Git config http to ssh rewrite rule

[url "ssh://git@git.example.com:7999"]
	insteadOf = https://git.example.com/scm

So to fetch the git.example.com/golibs/logo dependency over SSH to populate its storage cache, Athens ultimately calls git, which, given that rewrite rule, in turn needs an SSH private key matching a public key bound to the cloning developer or service account on Bitbucket. This is essentially the github.com SSH model. At a bare minimum, we need to provide Athens with an SSH private key and the http to ssh git rewrite rule, mounted inside the Athens container for use by the root user

$ mkdir -p storage
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
    -v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
    -v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
    -v "$PWD/ssh-keys:/root/.ssh" \
    -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary

$PWD/gitconfig/.gitconfig contains the http to ssh rewrite rule

[url "ssh://git@git.example.com:7999"]
	insteadOf = https://git.example.com/scm

$PWD/ssh-keys contains the aforementioned private key and a minimal ssh-config

$ ls ssh-keys/
config		id_rsa

We also provide an ssh config to bypass host SSH key verification and to show how to bind different hosts to different SSH keys

$PWD/ssh-keys/config contains

Host git.example.com
Hostname git.example.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa

Now, builds executed through the Athens proxy should be able to clone the git.example.com/golibs/logo dependency over authenticated SSH.

SSH_AUTH_SOCK and ssh-agent Support

As an alternative to passwordless SSH keys, one can use an ssh-agent. The ssh-agent-set SSH_AUTH_SOCK environment variable will propagate to go mod download if it contains a path to a valid unix socket (after following symlinks).

As a result, if running with a working ssh agent (and a shell with SSH_AUTH_SOCK set), after setting up a gitconfig as mentioned in the previous section, one can run athens in docker as such:

$ mkdir -p storage
$ ssh-add .ssh/id_rsa_something
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
    -v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
    -v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
    -v "${SSH_AUTH_SOCK}:/.ssh_agent_sock" \
    -e "SSH_AUTH_SOCK=/.ssh_agent_sock" \
    -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary

Configuring Storage

Storage

The Athens proxy supports many storage types:

• Storage
• Memory - Configuration:
• Disk - Configuration:
• Mongo - Configuration:
• Google Cloud Storage - Configuration:
• AWS S3 - Configuration:
• Minio - Configuration:
  • DigitalOcean Spaces
    • Configuration:
  • Alibaba OSS
    • Configuration:
• Azure Blob Storage - Configuration:
• External Storage - Configuration:
• Running multiple Athens pointed at the same storage
  • Using etcd as the single flight mechanism
  • Using redis as the single flight mechanism
    • Direct connection to redis
    • Connecting to redis via redis sentinel

All of them can be configured using config.toml file. You need to set a valid driver in StorageType value or you can set it in environment variable ATHENS_STORAGE_TYPE on your server. Also for most of the drivers you need to provide additional configuration data which will be described below.

Memory

This storage doesn’t need any specific configuration and it’s also used by default in the Athens project. It writes all of data into local disk into tmp dir.

This storage type should only be used for development purposes!

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "memory"

Disk

Disk storage allows modules to be stored on a file system. The location on disk where modules will be stored can be configured.

You can pre-fill disk-based storage to enable Athens deployments that have no access to the internet. See here for instructions on how to do that.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "disk"

[Storage]
    [Storage.Disk]
        RootPath = "/path/on/disk"

where /path/on/disk is your desired location. Also it can be set using ATHENS_DISK_STORAGE_ROOT env

Mongo

This driver uses a Mongo server as data storage. On start this driver will create an athens database and module collection on your Mongo server.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "mongo"

[Storage]
    [Storage.Mongo]
        # Full URL for mongo storage
        # Env override: ATHENS_MONGO_STORAGE_URL
        URL = "mongodb://127.0.0.1:27017"

        # Not required parameter
        # Path to certificate to use for the mongo connection
        # Env override: ATHENS_MONGO_CERT_PATH
        CertPath = "/path/to/cert/file"

        # Not required parameter
        # Allows for insecure SSL / http connections to mongo storage
        # Should be used for testing or development only
        # Env override: ATHENS_MONGO_INSECURE
        Insecure = false

        # Not required parameter
        # Allows for use of custom database 
        # Env override: ATHENS_MONGO_DEFAULT_DATABASE
        DefaultDBName = athens

        # Not required parameter
        # Allows for use of custom collection 
        # Env override: ATHENS_MONGO_DEFAULT_COLLECTION
        DefaultCollectionName = modules

Google Cloud Storage

This driver uses Google Cloud Storage and assumes that you already have an account and bucket in it. If you never used Google Cloud Storage there is quick guide how to create bucket inside it.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "gcp"

[Storage]
    [Storage.GCP]
        # ProjectID to use for GCP Storage
        # Env overide: GOOGLE_CLOUD_PROJECT
        ProjectID = "YOUR_GCP_PROJECT_ID"

        # Bucket to use for GCP Storage
        # Env override: ATHENS_STORAGE_GCP_BUCKET
        Bucket = "YOUR_GCP_BUCKET"

AWS S3

This driver is using the AWS S3 and assumes that you already have account and bucket created in it. If you never used Amazon Web Services there is quick guide how to create bucket inside it. After this you can pass your credentials inside config.toml file. If the access key ID and secret access key are not specified in config.toml, the driver will attempt to load credentials for the default profile from the AWS CLI configuration file created during installation.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "s3"

[Storage]
        [Storage.S3]
        ### The authentication model is as below for S3 in the following order
        ### If AWS_CREDENTIALS_ENDPOINT is specified and it returns valid results, then it is used
        ### If config variables are specified and they are valid, then they return valid results, then it is used
        ### Otherwise, it will default to default configurations which is as follows
        # attempt to find credentials in the environment, in the shared
        # configuration (~/.aws/credentials) and from ec2 instance role
        # credentials. See
        # https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
        # and
        # https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
        # for environment variables that will affect the aws configuration.
        # Setting UseDefaultConfiguration would only use default configuration. It will be deprecated in future releases 
        # and is recommended not to use it.

        # Region for S3 storage
        # Env override: AWS_REGION
        Region = "MY_AWS_REGION"

        # Access Key for S3 storage
        # Env override: AWS_ACCESS_KEY_ID
        Key = "MY_AWS_ACCESS_KEY_ID"

        # Secret Key for S3 storage
        # Env override: AWS_SECRET_ACCESS_KEY
        Secret = "MY_AWS_SECRET_ACCESS_KEY"

        # Session Token for S3 storage
        # Not required parameter
        # Env override: AWS_SESSION_TOKEN
        Token = ""

        # S3 Bucket to use for storage
        # Env override: ATHENS_S3_BUCKET_NAME
        Bucket = "MY_S3_BUCKET_NAME"
        
        # If true then path style url for s3 endpoint will be used
        # Env override: AWS_FORCE_PATH_STYLE
        ForcePathStyle = false

        # If true then the default aws configuration will be used. This will
        # attempt to find credentials in the environment, in the shared
        # configuration (~/.aws/credentials) and from ec2 instance role
        # credentials. See
        # https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
        # and
        # https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
        # for environment variables that will affect the aws configuration.
        # Env override: AWS_USE_DEFAULT_CONFIGURATION
        UseDefaultConfiguration = false

        # https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/
        # Note that this the URI should not end with / when AwsContainerCredentialsRelativeURI is set
        # Env override: AWS_CREDENTIALS_ENDPOINT
        CredentialsEndpoint = ""

        # Env override: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
        # If you are planning to use AWS Fargate, please use http://169.254.170.2 for CredentialsEndpoint
        # Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html
        AwsContainerCredentialsRelativeURI = ""

        # An optional endpoint URL (hostname only or fully qualified URI)
        # that overrides the default generated endpoint for S3 storage client.
        #
        # You must still provide a `Region` value when specifying an endpoint.
        # Env override: AWS_ENDPOINT
        Endpoint = ""

Minio

Minio is an open source object storage server that provides an interface for S3 compatible block storages. If you have never used minio, you can read this quick start guide. Any S3 compatible object storage is supported by Athens through the minio interface. Below, you can find different configuration options we provide for Minio. Example configuration for Digital Ocean and Alibaba OSS block storages are provided below.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Endpoint for Minio storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "127.0.0.1:9001"

        # Access Key for Minio storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_MINIO_SECRET_KEY"

        # Secret Key for Minio storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_MINIO_SECRET_KEY"

        # Enable SSL for Minio connections
        # Defaults to true
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = false

        # Minio Bucket to use for storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "gomods"

DigitalOcean Spaces

For Athens to communicate with DigitalOcean Spaces, we are using Minio driver because DO Spaces tries to be fully compatible with it. Also configuration for this storage looks almost the same in our proxy as for Minio.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Address of DO Spaces storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "YOUR_ADDRESS.digitaloceanspaces.com"

        # Access Key for DO Spaces storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_DO_SPACE_KEY_ID"

        # Secret Key for DO Spaces storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_DO_SPACE_SECRET_KEY"

        # Enable SSL
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = true

        # Space name in your DO Spaces storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "YOUR_DO_SPACE_NAME"

        # Region for DO Spaces storage
        # Env override: ATHENS_MINIO_REGION
        Region = "YOUR_DO_SPACE_REGION"

Alibaba OSS

For Athens to communicate with Alibaba Cloud Object Storage Service, we are using Minio driver. Also configuration for this storage looks almost the same in our proxy as for Minio.

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Address of Alibaba OSS storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "YOUR_ADDRESS.aliyuncs.com"

        # Access Key for Minio storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_OSS_KEY_ID"

        # Secret Key for Alibaba OSS storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_OSS_SECRET_KEY"

        # Enable SSL
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = true

        # Parent folder in your Alibaba OSS storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "YOUR_OSS_FOLDER_PREFIX"

Azure Blob Storage

This driver uses Azure Blob Storage

If you never used Azure Blog Storage, here is a quickstart

It assumes that you already have the following:

• An Azure storage account
• The credentials (storage account key)
• A container (to store blobs)

Configuration:

# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "azureblob"

[Storage]
    [Storage.AzureBlob]
        # Storage Account name for Azure Blob
        # Env override: ATHENS_AZURE_ACCOUNT_NAME
        AccountName = "MY_AZURE_BLOB_ACCOUNT_NAME"

        # Account Key to use with the storage account
        # Env override: ATHENS_AZURE_ACCOUNT_KEY
        AccountKey = "MY_AZURE_BLOB_ACCOUNT_KEY"

        # Name of container in the blob storage
        # Env override: ATHENS_AZURE_CONTAINER_NAME
        ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"

External Storage

External storage lets Athens connect to your own implementation of a storage backend. All you have to do is implement the storage.Backend interface and run it behind an http server.

Once you implement the backend server, you must then configure Athens to use that storage backend as such:

Configuration:

# Env override: ATHENS_STORAGE_TYPE
StorageType = "external"

[Storage]
    [Storage.External]
        # Env override: ATHENS_EXTERNAL_STORAGE_URL
        URL = "http://localhost:9090"

Athens provides a convenience wrapper that lets you implement a storage backend with ease. See the following example:

package main

import (
    "github.com/gomods/athens/pkg/storage"
    "github.com/gomods/athens/pkg/storage/external"
)

// TODO: implement storage.Backend
type myCustomStorage struct {
    storage.Backend
}

func main() {
    handler := external.NewServer(&myCustomStorage{})
    http.ListenAndServe(":9090", handler)
}

Running multiple Athens pointed at the same storage

Athens has the ability to run concurrently pointed at the same storage medium, using a distributed locking mechanism called “single flight”.

By default, Athens is configured to use the memory single flight, which stores locks in local memory. This works when running a single Athens instance, given the process has access to it’s own memory. However, when running multiple Athens instances pointed at the same storage, a distributed locking mechansism is required.

Athens supports several distributed locking mechanisms:

• etcd
• redis
• redis-sentinel
• gcp (available when using the gcp storage type)
• azureblob (available when using the azureblob storage type)

Setting the SingleFlightType (or ATHENS_SINGLE_FLIGHT TYPE in the environment) configuration value will enable usage of one of the above mechanisms. The azureblob and gcp types require no extra configuration.

Using etcd as the single flight mechanism

Using the etcd mechanism is very simple, just a comma separated list of etcd endpoints. The recommend configuration is 3 endpoints, however, more can be used.

SingleFlightType = "etcd"

[SingleFlight]
    [SingleFlight.Etcd]
        # Env override: ATHENS_ETCD_ENDPOINTS
        Endpoints = "localhost:2379,localhost:22379,localhost:32379"

Using redis as the single flight mechanism

Athens supports two mechanisms of communicating with redis: direct connection, and connecting via redis sentinels.

Direct connection to redis

Using a direct connection to redis is simple, and only requires a single redis-server. You can also optionally specify a password to connect to the redis server with

SingleFlightType = "redis"

[SingleFlight]
    [SingleFlight.Redis]
        # Endpoint is the redis endpoint for the single flight mechanism
        # Env override: ATHENS_REDIS_ENDPOINT
        Endpoint = "127.0.0.1:6379"

        # Password is the password for the redis instance
        # Env override: ATHENS_REDIS_PASSWORD
        Password = ""

Customizing lock configurations:

If you would like to customize the distributed lock options then you can optionally override the default lock config to better suit your use-case:

[SingleFlight.Redis]
    ...
    [SingleFlight.Redis.LockConfig]
        # TTL for the lock in seconds. Defaults to 900 seconds (15 minutes).
        # Env override: ATHENS_REDIS_LOCK_TTL
        TTL = 900
        # Timeout for acquiring the lock in seconds. Defaults to 15 seconds.
        # Env override: ATHENS_REDIS_LOCK_TIMEOUT
        Timeout = 15
        # Max retries while acquiring the lock. Defaults to 10.
        # Env override: ATHENS_REDIS_LOCK_MAX_RETRIES
        MaxRetries = 10

Customizations may be required in some cases for eg, you can set a higher TTL if it usually takes longer than 5 mins to fetch the modules in your case.

Connecting to redis via redis sentinel

NOTE: redis-sentinel requires a working knowledge of redis and is not recommended for everyone.

redis sentinel is a high-availability set up for redis, it provides automated monitoring, replication, failover and configuration of multiple redis servers in a leader-follower setup. It is more complex than running a single redis server and requires multiple disperate instances of redis running distributed across nodes.

For more details on redis-sentinel, check out the documentation

As redis-sentinel is a more complex set up of redis, it requires more configuration than standard redis.

Required configuration:

• Endpoints is a list of redis-sentinel endpoints to connect to, typically 3, but more can be used
• MasterName is the named master instance, as configured in the redis-sentinel configuration

Optionally, like redis, you can also specify a password to connect to the redis-sentinel endpoints with

SingleFlightType = "redis-sentinel"

[SingleFlight]
  [SingleFlight.RedisSentinel]
      # Endpoints is the redis sentinel endpoints to discover a redis
      # master for a SingleFlight lock.
      # Env override: ATHENS_REDIS_SENTINEL_ENDPOINTS
      Endpoints = ["127.0.0.1:26379"]
      # MasterName is the redis sentinel master name to use to discover
      # the master for a SingleFlight lock
      MasterName = "redis-1"
      # SentinelPassword is an optional password for authenticating with
      # redis sentinel
      SentinelPassword = "sekret"

Distributed lock options can be customised for redis sentinal as well, in a similar manner as described above for redis.

Proxying a checksum database API

If you run go get github.com/mycompany/secret-repo@v1.0.0 and that module version is not yet in your go.sum file, Go will by default send a request to https://sum.golang.org/lookup/github.com/mycompany/secret-repo@v1.0.0. That request will fail because the Go tool requires a checksum, but sum.golang.org doesn’t have access to your private code.

The result is that (1) your build will fail, and (2) your private module names have been sent over the internet to an opaque public server that you don’t control.

You can read more about this sum.golang.org service here

Proxying a checksum DB

Many companies use Athens to host their private code, but Athens is not only a module proxy. It’s also a checksum database proxy. That means that anyone inside of your company can configure go to send these checksum requests to Athens instead of the public sum.golang.org server.

If the Athens server is configured with checksum filters, then you can prevent these problems.

If you run the below command using Go 1.13 or later:

$ GOPROXY=<athens-url> go build .

… then the Go tool will automatically send all checksum requests to <athens-url>/sumdb/sum.golang.org instead of https://sum.golang.org.

By default, when Athens receives a /sumdb/... request, it automatically proxies it to https://sum.golang.org, even if it’s a private module that sum.golang.org doesn’t and can’t know about. So if you are working with private modules, you’ll want to change the default behavior.

If you want Athens to not send some module names up to the global checksum database, set those module names in the NoSumPatterns value in config.toml or using the ATHENS_GONOSUM_PATTERNS environment variable.

The following sections will go into more detail on how checksum databases work, how Athens fits in, and how this all impacts your workflow.

How to set this all up

Before you begin, you’ll need to run Athens with configuration values that tell it to not proxy certain modules. If you’re using config.toml, use this configuration:

NoSumPatterns = ["github.com/mycompany/*", "github.com/secret/*"]

And if you’re using an environment variable, use this configuration:

$ export ATHENS_GONOSUM_PATTERNS="github.com/mycompany/*,github.com/secret/*"

You can use any string compatible with path.Match in these environment variables

After you start Athens up with this configuration, all checksum requests for modules that start with github.com/mycompany or github.com/secret will not be forwarded, and Athens will return an error to the go CLI tool.

This behavior will ensure that none of your private module names leak to the public internet, but your builds will still fail. To fix that problem, set another environment variable on your machine (that you run your go commands)

$ export GONOSUMDB="github.com/mycompany/*,github.com/secret/*"

Now, your builds will work and you won’t be sending information about your private codebase to the internet.

I’m confused, why is this hard?

When the Go tool has to download new code that isn’t currently in the project’s go.sum file, it tries its hardest to get a checksum from a server it trusts, and compare it to the checksum in the actual code it downloads. It does all of this to ensure provenance. That is, to ensure that the code you just downloaded wasn’t tampered with.

The trusted checksums are all stored in sum.golang.org, and that server is centrally controlled.

These build failures and potential privacy leaks can only happen when you try to get a module version that is not already in your go.sum file.

Athens does its best to respect and use the trusted checksums while also ensuring that your private names don’t get leaked to the public server. In some cases, it has to choose whether to fail your build or leak information, so it chooses to fail your build. That’s why everybody using that Athens server needs to set up their GONOSUMDB environment variable.

We believe that along with good documentation - which we hope this is! - we have struck the right balance between convenience and privacy.

Pre-filling disk storage

One of the popular features of Athens is that it can be run completely cut off from the internet. In this case, though, it can’t reach out to an upstream (e.g. a VCS or another module proxy) to fetch modules that it doesn’t have in storage. So, we need to manually fill up the disk partition that Athens uses with the dependencies that we need.

This document will guide you through packaging up a single module called github.com/my/module, and inserting it into the Athens disk storage.

First, get the tools

You’ll need to produce the following assets from module source code:

• source.zip - just the Go source code, packaged in a zip file
• go.mod - just the go.mod file from the module
• $VERSION.info - metadata about the module

The source.zip file has a specific directory structure and the $VERSION.info has a JSON structure, both of which you’ll need to get right in order for Athens to serve up the right dependency formats that the Go toolchain will accept.

We don’t recommend that you create these assets yourself. Instead, use pacmod or gopack

Using pacmod

To install the pacmod tool, run go get like this:

$ go get github.com/plexsystems/pacmod@v0.4.0

This command will install the pacmod binary to your $GOPATH/bin/pacmod directory, so make sure that is in your $PATH.

Next, run pacmod to create assets

After you have pacmod, you’ll need the module source code that you want to package. Before you run the command, set the VERSION variable in your environment to the version of the module you want to generate assets for.

Below is an example for how to configure it.

$ export VERSION="v1.0.0"

Note: make sure your VERSION variable starts with a v

Next, navigate to the top-level directory of the module source code, and run pacmod like this:

$ pacmod pack github.com/my/module $VERSION .

Once this command is done, you’ll notice three new files in the same same directory you ran the command from:

• go.mod
• $VERSION.info
• $VERSION.zip

Using gopack

To use this method you need docker-compose installed.

Fork gopack project and clone it to your local machine (or just download files to your computer)

Edit goget.sh with a list of go modules you want to download:

#!/bin/bash
go get github.com/my/module1;
go get github.com/my/module2;

Run

docker-compose up --abort-on-container-exit

Once this command is done, you’ll notice in the ATHENS_STORAGE folder all modules ready to be moved to your Athens disk storage.

Next, move assets into Athens storage directory

Now that you have assets built, you need to move them into the location of the Athens disk storage. In the below commands, we’ll assume $STORAGE_ROOT is the environment variable that points to the top-level directory that Athens uses for its on-disk.

If you set up Athens with the $ATHENS_DISK_STORAGE_ROOT environment variable, the root of this storage location is the value of this environment variable. Use export STORAGE_ROOT=$ATHENS_DISK_STORAGE_ROOT to prepare your environment for the below commands.

First create the subdirectory into which you’ll move the assets you created:

$ mkdir -p $STORAGE_ROOT/github.com/my/module/$VERSION

Finally, make sure that you’re still in the module source repository root directory (the same as you were in when you ran the pacmod command), and move your three new files into the new directory you just created:

$ mv go.mod $STORAGE_ROOT/github.com/my/module/$VERSION/go.mod
$ mv $VERSION.info $STORAGE_ROOT/github.com/my/module/$VERSION/$VERSION.info
$ mv $VERSION.zip $STORAGE_ROOT/github.com/my/module/$VERSION/source.zip

Note that we’ve changed the name of the .zip file

Finally, test your setup

At this point, your Athens server should have its disk-based cache filled with the github.com/my/module module at version $VERSION. Next time you request this module, Athens will find it in its disk storage and will not try to fetch it from an upstream source.

You can quickly test this behavior by running below curl command, assuming your Athens server is running on http://localhost:3000 and is already configured to use the same disk storage that you pre-filled above.

$ curl localhost:3000/github.com/my/module/@v/$VERSION.info

When you run this command, Athens should immediately return, without contacting any other network services.

Filtering modules (deprecated)

Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to filter modules in Athens.

The proxy supports the following three use cases

1. Fetches a module directly from the source (upstream proxy)
2. Exclude a particular module
3. Include a module in the local proxy.

These settings can be done by creating a configuration file which can be pointed by setting either FilterFile in config.dev.toml or setting ATHENS_FILTER_FILE as an environment variable.

Writing the configuration file

Every line of the configuration can start either with a

• + denoting that the module has to be included by the proxy
• D denoting that the module has to be fetched directly from an upstream proxy and not stored locally
• - denoting that the module is excluded and will not be fetched into the proxy or from the upstream proxy

It allows for # to add comments and new lines are skipped. Anything else would result in an error

Sample configuration file

# This is a comment


- github.com/azure
+ github.com/azure/azure-sdk-for-go

# get golang tools directly
D golang.org/x/tools

In the above example, golang.org/x/tools is fetched directly from the upstream proxy. All the modules from github.com/azure are excluded except github.com/azure/azure-sdk-for-go

Adding a default mode

The list of modules can grow quickly in size and sometimes may want to specify configuration for a handful of modules. In this case, they can set a default mode for all the modules and add specific rules to certain modules that they want to apply to. The default rule is specified at the beginning of the file. It can be an either +, - or D

An example default mode is

D
- github.com/manugupt1/athens
+ github.com/gomods/athens

In the above example, all the modules are fetched directly from the source. github.com/manugupt1/athens is excluded and github.com/gomods/athens is stored in the proxy storage.

Adding versions to the filter

Using an “approved list” is a common practice that requires each minor or patch version to be approved before it is allowed in the codebase. This is accomplished by adding a list of version patterns to the rule. These version patterns are comma-separated and prefix-matching, so v2 and v2.3.* both match the requested version 2.3.5.

An example version filter is

-
# use internal github enterprise server directly
D enterprise.github.com/company

# external dependency approved list
+ github.com/gomods/athens v0.1,v0.2,v0.4.1

In the above example, any module not in the rules will be excluded. All modules from enterprise.github.com/company are fetched directly from the source. The github.com/gomods/athens module will be stored in the proxy storage, but only for version v0.4.1 and any patch versions under v0.1 and v0.2 minor versions.

Versions Filter Modifiers

Athens provides advanced filter modifiers to cover cases such as API compatibility or when a given dependency changes its license from a given versions. The modifiers are intended to be used in the pattern list of the filter file.

-
# external dependency approved list
+ github.com/gomods/athens 
The currently supported modifiers are


~1.2.3 will enable all patch versions from 1.2.3 and above (e.g. 1.2.3, 1.2.4, 1.2.5)

Formally, 1.2.x where x >= 3



^1.2.3 will enable all patch and minor versions from 1.2.3 and above (e.g. 1.2.4, 1.3.0 and 1.4.5)

Formally, 1.x.y where x >= 2 and y >= 3



<1.2.3 will enable all versions lower than 1.2.3 (e.g. 1.2.2, 1.0.0 and 0.58.9)

Formally, x.y.z where x <= 1, y < = 2 and z < 3



This kind of modifiers will work only if a three parts semantic version is specified. For example, ~4.5.6 will work while ~4.5 won’t.

            
          

Using an upstream Go modules repository (deprecated)

Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to set upstream repositories in Athens.

By default, Athens fetches module code from an upstream version control system (VCS) like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.

1. Create a filter file (e.g /usr/local/lib/FilterForGoCenter) with letter D (stands for “direct access”) in first line. For more details, please refer to documentation on - Filtering Modules

   # FilterFile for fetching modules directly from upstream
   D

2. If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment). Additionally in the current or new config file, set the following parameters as suggested:

   FilterFile = "/usr/local/lib/FilterForGoCenter"
   GlobalEndpoint = "https://<url_to_upstream>"
   # To use GoCenter for example, replace <url_to_upstream> with gocenter.io
   # You can also use https://proxy.golang.org to use the Go Module mirror

3. Restart Athens specifying the updated current or new config file.

    <path_to_athens>/proxy  -config_file <path-to updated  current or new configfile>

4. Verify the new configuration using the steps mentioned in “Try out Athens” document, and go through the same walkthrough example.

Home template configuration

As of v0.14.0 Athens ships with a default, minimal HTML home page that advises users on how to connect to the proxy. It factors in whether GoNoSumPatterns is configured, and attempts to build configuration for GO_PROXY. It relies on the users request Host header (on HTTP 1.1) or the Authority header (on HTTP 2) as well as whether the request was over TLS to advise on configuring GO_PROXY. Lastly, the homepage provides a quick guide on how users can leverage the Athens API.

Of course, not all instructions will be this simple. Some installations may be reachable at different addresses in CI than for desktop users. In this case, and others where the default home page does not make sense it is possible to override the template.

Do so by configuring HomeTemplatePath via the config or ATHENS_HOME_TEMPLATE_PATH to a location on disk with a Go HTML template or placing a template file at /var/lib/athens/home.html.

Athens automatically injects the following variables in templates:

Using these values is done by wrapping them in bracers with a dot prepended. Example: {{ .Host }}

For more advanced formatting read more about Go HTML templates.

<!DOCTYPE html>
<html>
<head>
	<title>Athens</title>
	<style>
		body {
			font-family: Arial, sans-serif;
			margin: 20px;
		}

		pre {
				background-color: #f4f4f4;
				padding: 5px;
				border-radius: 5px;
				width: fit-content;
  				padding: 10px;
		}


		code {
			background-color: #f4f4f4;
			padding: 5px;
			border-radius: 5px;
		}

	</style>
</head>
<body>
	
	<h1>Welcome to Athens</h1>

	<h2>Configuring your client</h2>
	<pre>GOPROXY={{ .Host }},direct</pre>
	{{ if .NoSumPatterns }}
	<h3>Excluding checksum database</h3>
	<p>Use the following GONOSUM environment variable to exclude checksum database:</p>
	<pre>GONOSUM={{ .NoSumPatterns }}</pre>
	{{ end }}

	<h2>How to use the Athens API</h2>
	<p>Use the <a href="/catalog">catalog</a> endpoint to get a list of all modules in the proxy</p>

	<h3>List of versions</h3>
	<p>This endpoint returns a list of versions that Athens knows about for <code>acidburn/htp</code>:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/list</pre>

	<h3>Version info</h3>
	<p>This endpoint returns information about a specific version of a module:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.info</pre>
	<p>This returns JSON with information about v1.0.0. It looks like this:
	<pre>{
	"Name": "v1.0.0",
	"Short": "v1.0.0",
	"Version": "v1.0.0",
	"Time": "1972-07-18T12:34:56Z"
}</pre>

	<h3>go.mod file</h3>
	<p>This endpoint returns the go.mod file for a specific version of a module:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.mod</pre>
	<p>This returns the go.mod file for version v1.0.0. If {{ .Host }}/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:</p>
	<pre>module github.com/acidburn/htp</pre>

	<h3>Module sources</h3>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.zip</pre>
	<p>This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.</p>

	<h3>Latest</h3>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@latest</pre>
	<p>This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.</p>

</body>
</html>

Logging

Athens is designed to support a myriad of logging scenarios.

Standard

The standard structured logger can be configured in plain or json formatting via LogFormat or ATHENS_LOG_FORMAT. Additionally, verbosity can be controlled by setting LogLevel or ATHENS_LOG_LEVEL. In order for the standard structured logger to work, CloudRuntime and ATHENS_CLOUD_RUNTIME should not be set to a valid value.

Runtimes

Athens can be configured according to certain cloud provider specific runtimes. The GCP runtime configures Athens to rename certain logging fields that could be dropped or overriden when running in a GCP logging environment. This runtime can be used with LogLevel or ATHENS_LOG_LEVEL to control the verbosity of logs.