Application Configuration - Practical Go Lessons https://www.practical-go-lessons.

com/chap-33-application-configuration

Chapter 33: Application Configuration

1 What will you learn in this chapter?

• How to configure an application?

• How to add a command-line option to a program?

• How to create and parse environment variables?

2 Technical concepts covered

• Environment variable

• Unmarshal

• Command-line options

3 Introduction
When we write programs for ourselves, we are (often) writing directly in the source code some important options, like, for instance, the
port on which our server will listen, the name of the database we are using... But what if you attempt to share your program with others?
There is a significant probability that your users will not use the same database name as you are using in the development phase; they will
also want to use another port for their server!

And what about security! Do you want everybody to know your database password? You should not commit credentials in your code.

In this section, we will cover how you can configure your applications.

The paper and the digital edition of this book are available here. ×
I also filmed a video course to build a real world project with Go.

4 What is it?
An application is configurable when it offers users the possibility of control the aspects of execution [@rabkin2011static]. For instance,
users can define the port on which a server will listen, the database credentials, the duration of the timeout of HTTP requests...

5 How to do it?
In many open-source applications, the configuration is handled via key-value data structure [@rabkin2011static]. Options are often
denoted by a unique name. Unix systems use strings as option names. Windows use a tree structure.

In a real-world application, you often find a class (or a type struct) that exposes the configuration options. This class often parses a file to

1 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

attribute values to each option. Applications often propose command-line options to adjust configurations.

6 Command-line options
To define a named command-line option, we can use the standard package flag .

Let’s take an example. You are building a REST API. Your application will expose a web server; you want the listening port to be configurable
by the user.

The first step is to define the new flag :

// configuration/cli/main.go

port := flag.Int("port", 4242, "the port on which the server will listen")

The first argument is the flag name; the second one is the default value; the third is the help text. The returned variable will be a pointer to
an integer.

There is an alternative syntax available. You first define a variable, then you pass a pointer to that variable as argument of the IntVar
function :

var port int

flag.IntVar(&port, "port", 4242, "the port on which the server will listen")

The next step is to parse the flags :

flag.Parse()

The function Parse has to be called before using the variables that you defined. Internally it will iterate over all the arguments given in
the command line and assign values to the flags you have defined.

In the first version ( flag.Int ) the variable will be a pointer to an integer ( *int ). In the second version ( flag.Intvar ) will be of type
integer ( int ). This specificity changes the way you use the variable afterward:

// version 1 : Int

port := flag.Int("port", 4242, "the port on which the server will listen")
flag.Parse()
fmt.Printf("%d",*port)

// version 2 : IntVar

var port2 int flag.IntVar(&port2, "port2", 4242, "the port on which the server will listen") flag.Parse()
fmt.Printf("%d\n",port2)

6.0.1 Other types

The flag package expose functions to add flags that have different types. Here is a list of the types on the left and the flag package’s
corresponding functions.

Int64, Int64Var

String, StringVar

Uint, UintVar

Uint64, Uint64Var

Float64, Float64Var

Duration, DurationVar

Bool, BoolVar

6.0.2 Usage in the command line

To pass the specified flags to your program you have two options :

$ myCompiledGoProgram -port=4242

2 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

or

$ myCompiledGoProgram -port 4242

This last usage is forbidden for boolean flags. Instead, the user of your application can set the flag to true by simply writing :

$ myCompiledGoProgram -myBoolFlag

By doing so, the variable behind the flag will be set to true.

To get a view of all flags available, you can use the help flag (which is added by default to your program !)

$ ./myCompiledGoProgram -help
Usage of ./myCompiledProgram:
-myBool
a test boolean
-port int
the port on which the server will listen (default 4242)
-port2 int
the port on which the server will listen (default 4242)

7 Environment variables
Some cloud services support configuration via environment variables. Environment variables are easy to set with the export command. To
create an environment variable named MYVAR, just type the following command in your terminal1 :

$ export MYVAR=value

If you want to pass environment variables to your application, use the following syntax

$ export MYVAR=test && export MYVAR2=test2 && ./myCompiledProgram

Here we are setting two environment variables, and then we launch the program myCompiledProgram .

7.0.1 How to get an environment variable value

To retrieve the value of an environment variable, you can use the os standard package. The package exposes a function named Getenv .
This function takes the name of the variable as argument and returns its value (as a string) :

// configuration/env/main.go
myvar := os.Getenv("MYVAR")
myvar2 := os.Getenv("MYVAR2")
fmt.Printf("myvar : '%s'\n", myvar)
fmt.Printf("myvar2 :'%s'\n", myvar2)

Another method allows you to retrieve environment variables : LookupEnv . It’s better than the previous one because it will inform you if
the variable is not present :

// configuration/env/main.go
port, found := os.LookupEnv("DB_PORT")
if !found {
log.Fatal("impossible to start up, DB_PORT env var is mandatory")
}
portParsed, err := strconv.ParseUint(port, 10, 8)
if err != nil {
log.Fatalf("impossible to parse db port: %s", err)
}
log.Println(portParsed)

LookupEnv will check if the environment variable exists

• If it does not exists, the second result ( found ) will be equal to false

• In the previous example, we parse the port retrieved into an uint16 (base 10) with strconv.ParseUint

7.0.2 How to get ALL environment variables

You can also get all the environment variables with the function Environ(), which will return a slice of strings :

3 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

fmt.Println(os.Environ())

The returned slice is composed of all the variables in the format “key=value”. Each element of the slice is a key-value pair. The package
does not isolate the value from the key. You have to parse the slice values.

7.0.3 How to set an environment variable

The os package exposes the os.Setenv function :

err := os.Setenv("MYVAR3","test3")
if err != nil {
panic(err)
}

os.Setenv takes two arguments :

• the name of the variable to set

• its value.

Note that this can generate an error. You have to handle that error.

The paper and the digital edition of this book are available here. ×
I also filmed a video course to build a real world project with Go.

8 File-based configuration
Configuration can also be saved in a file and parsed automatically by the application.

The format of the file depends on the needs of the application. If you need a complex configuration with levels and hierarchy (in the
fashion of the Windows Registry), then you might need to think about formats like JSON, YAML, or XML.

Let’s take the example of a JSON configuration file :

{
"server": {
"host": "localhost",
"port": 80
},
"database": {
"host": "localhost",
"username": "myUsername",
"password": "abcdefgh"
}
}

We will place this file somewhere in the machine that hosts the application. We have to parse this file inside the application to get those
configuration values.

To parse it, we first have to create a type struct in our program :

// configuration/file/main.go
type Configuration struct {
Server Server `json:"server"`
Database DB `json:"database"`
}
type Server struct {
Host string `json:"host"`
Port int `json:"port"`
}
type DB struct {
Host string `json:"host"`
Username string `json:"username"`
Password string `json:"password"`
}

We create three types struct :

4 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

• Configuration that will group the two other types :

• Server that holds the properties of the JSON object server

• DB that store the properties of the JSON object database

Those type structs will allow you to unmarshal the JSON configuration file. The next step is to open the file and read its content :

// configuration/file/main.go

confFile, err := os.Open("myConf.json")

if err != nil {
panic(err)
}
defer confFile.Close()
conf, err := ioutil.ReadAll(confFile)
if err != nil {
panic(err)
}

Here we are calling os.Open to open the file "myConf.json" located for the example in the same directory as the executable. In real life,
you should upload this file to an appropriate location on the machine.

We get the whole content of the file by using ioutil.ReadAll . This function returns a slice of bytes. The next step is to use this slice of
raw bytes with json.Unmarshal :

// configuration/file/main.go

myConf := Configuration{}
err = json.Unmarshal(conf, &myConf)
if err != nil {
panic(err)
}
fmt.Printf("%+v",myConf)

We create a variable myConf that is initialized with an empty struct Configuration .Then we pass to json.Unmarshal the variable conf
that contains the slice of bytes extracted from the file, and (as the second argument) a pointer to myConf .

In the end, our program outputs the following :

{Server:{Host:localhost Port:80} Database:{Host:localhost Username:myUsername Password:abcdefgh}}

You can then share this variable with the whole application.

9 The github.com/spf13/viper module

The module github.com/spf13/viper is at the time of writing very popular among the Go community. It has more than 14.000 stars and
more than 1.3k forks2.

This package allows you to simply define a configuration for your application from files, environment variables, command line, buffers... It
also supports an interesting feature: if your configuration changes during your application’s lifetime, it will be reloaded.

9.1 Read from a file

Let’s give it a try :

// configuration/viper/main.go
//...
// import "github.com/spf13/viper"
viper.SetConfigName("myConf")
viper.(".")
err := viper.ReadInConfig()
if err != nil {
log.Panicf("Fatal error config file: %s\n", err)
}

First, you have to define the name of the config file you want to load : viper.SetConfigName("myConf") . Here we are loading the file
named myConf . Note that we do not give viper the extension name of the file. It will retrieve the correct encoding and parse the file.

5 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

In the next line, we have to tell viper were to search with the function AddConfigPath . The point means “look in the current directory” for a
file named myConf. The supported extension at the time of writing are :

• json,

• toml,

• yaml,

• yml,

• properties,

• props,

• prop,

• hcl

Note that we could have added another path to search for config. The AddConfigPath function will append the path to a slice.

When we call ReadInConfig , the package will look for the file specified.

You can then use the configuration loaded :

fmt.Println(viper.AllKeys())
//[database.username database.password server.host server.port database.host]
fmt.Println(viper.GetString("database.username"))
// myUsername

With the AllKeys function, you can retrieve all the existing configuration keys of your application. To read a specific configuration
variable, you can use the function GetString .

The GetString function takes as parameter the key of the configuration option. The key is a reflection of the configuration file hierarchy.
Here "database.username" means that we access the value of the property username from the object database.

The database object also has the property host. To access it, the key is simply "database.host" .

Viper exposes GetXXX functions for the types bool , float64 , int , string , map[string]interface{} , map[string]string , slice of
strings, time, and duration. It’s also possible to simply use Get to retrieve a configuration value. The returning type of this last function is
the empty interface. I do not recommend it because it can lead to type errors. In our case, we are expecting a string, but if a user puts an
integer into the config file, it will return an int. Your program will not work as expected and might panic.

9.2 Read from environment variables

Viper can parse environment variables automatically with a prefix :

viper.SetEnvPrefix("myapp")
viper.AutomaticEnv()

With those two lines, each time, you will call a getter method ( GetString , GetBool , ...) viper will look for environment variables with the
prefix MYAPP_ :

fmt.Println(viper.GetInt("timeout"))

will look for the value of an environment variable named MYAPP_TIMEOUT .

10 Problems and how to avoid them

10.1 Missing documentation
Many projects are suffering from a lack of documentation concerning their configuration options.

By documenting every single option available in a separate file, you are lowering your software’s adoption barrier.

Inside a company, you might not be the person who will handle your application’s deployment. Hence you have to include in your
estimates the documentation time. Better documentation means a reduced time to market for your solution.

The interesting study of Ariel Rabkin and Randy Katz from the Berkeley University [@rabkin2011static] demonstrated that even for large
open-source projects (they studied seven large projects like Cassandra, Hadoop, HBase...) the documentation of configuration was

6 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

inaccurate :

• There are mentions of configuration options that do not exist in the application’s source code. They noted that sometimes the
options are simply commented into the application source code.

• Some options exist in the application source code that are not even documented.

The solution is easy: create precise documentation of your configuration options and keep it up to date when the project evolves.

10.2 Configuration mistakes

During a study [@nagaraja2004understanding] technical operators were asked to perform operations on a realistic internet system (three-
tier auction application). It has been observed that 50% of errors were configuration errors!

This figure is impressive and also very interesting. Systems can fail because of configuration mistakes. You cannot protect yourself against
users that put the wrong port number in the configuration. But your application can protect itself from wrong configuration.

• By detecting it and alerting the user.

• By not replacing a fraudulent value with the default one.

// configuration/error-detection/main.go
//...
dbPortRaw := os.Getenv("DATABASE_PORT")
dbPort, err := strconv.ParseUint(port, 10, 16)
if err != nil {
log.Panicf("Impossible to parse database port number '%s'. Please double check the env variable DATABASE_PORT",dbPortRaw)
}

In the previous code listing, we are retrieving the value of the environment variable DATABASE_PORT . We then attempt to convert it to an
uint16 . If an error occurs, we refuse to start the application and inform the user about what’s wrong.

Note here that we tell the user what to do to fix this error. This is a good practice, it only takes you 10 seconds to write, but it might save 1
hour of research to your user.

Double-check also that the program effectively uses all your configuration variables.

11 Security
Configuration variables are often composed of secrets: passwords, access tokens, encryption keys... A potential leak of those secrets may
cause harm. None of the solutions above are perfect to store and secure production secrets. Rotation of those secrets is also not easy with
the solution evoked before.

Some open-source solutions exist to handle this very specific issue. They offer a large set of features to protect, monitor, and audit the
usage of your secrets. At the time of writing, it seems that Vault, edited by Hashicorp, is the most popular.3 And it’s written in Go!

The paper and the digital edition of this book are available here. ×
I also filmed a video course to build a real world project with Go.

12 Test yourself
12.1 Questions
1. How to add a command-line option of type string to a program?

2. How to check if an environment variable exists & get its value in the same call?

3. How to set an environment variable?

4. How to configure your application with a file?

12.2 Answers
1. How to add a command-line option of type string to a program?

1. var password stringflag.StringVar(&password,"password", "default","the db password")// define other flags// parse

input flagsflag.Parse()

2. password2 := flag.String(“password2”,“default”, “the db password”)

7 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

2. How to check if an environment variable exists & get its value in the same call?

1. With the function LookupEnv from the package os

2. port, ok := os.LookupEnv(“DB_PORT”) if !ok { log.Fatal(“impossible to start up, DB_PORT env var is mandatory”) }

3. How to set an environment variable?

1. With the function SetEnv from the package os

2. err := os.Setenv(“MYVAR3”, “test3”) if err != nil { panic(err) }

4. How to configure your application with a JSON file?

1. Create a specific type with all your config variables.

2. At bootstrap, open the JSON file

3. Unmarshal the file.

13 Key Takeaways
• Application configuration can be done via command-line options.

• The standard package flag provide methods to add such options.

var password stringflag.StringVar(&password,"password", "default","the db password")// define other flags// parse input
flagsflag.Parse()

• The configuration options are given at program startup via the command line : $ myCompiledGoProgram -password insecuredPass

• The configuration is generally loaded at application startup (in the main)

• Configuration of an application can also be done via environment variables

• To retrieve an environment variable, you can use the os package

◦ dbHost := os.GetEnv(“DB_HOST”)

◦ dbHost, found := os.LookupEnv(“DB_HOST”)

• You can also handle configuration via a file (JSON, YAML,.…). The idea is to create a type struct & unmarshal the file contents.

• The module github.com/spf13/viper is a popular reference to handle application configuration.

• Configuration options should be documented carefully by developers. An up to date documentation is a competitive advantage.

• A dedicated secret management solution should handle application secrets.

1. In Unix systems, environment variables are local to the process on which they are defined↩

2. Checked on 02/22/21↩

3. Source: https://github.com/search?o=desc&q=secrets+management&s=stars&type=Repositories last checked 02/22/21↩

Bibliography
• [rabkin2011static] Rabkin, Ariel, and Randy Katz. 2011. “Static Extraction of Program Configuration Options.” In Proceedings of the
33rd International Conference on Software Engineering, 131–40. ACM.
• [rabkin2011static] Rabkin, Ariel, and Randy Katz. 2011. “Static Extraction of Program Configuration Options.” In Proceedings of the
33rd International Conference on Software Engineering, 131–40. ACM.
• [rabkin2011static] Rabkin, Ariel, and Randy Katz. 2011. “Static Extraction of Program Configuration Options.” In Proceedings of the
33rd International Conference on Software Engineering, 131–40. ACM.
• [nagaraja2004understanding] Nagaraja, Kiran, Fábio Oliveira, Ricardo Bianchini, Richard P Martin, and Thu D Nguyen. 2004.
“Understanding and Dealing with Operator Mistakes in Internet Services.” In OSDI, 4:61–76.

Previous Next

Templates Benchmarks

8 of 9 02/01/2023, 02:19
Application Configuration - Practical Go Lessons https://www.practical-go-lessons.com/chap-33-application-configuration

Table of contents

Did you spot an error ? Want to give me feedback ? Here is the feedback page! ×

Newsletter:
Like what you read ? Subscribe to the newsletter.

I will keep you informed about the book updates.

@ [email protected]

Practical Go Lessons
By Maximilien Andile
Copyright (c) 2023
Follow me Contents
Posts
Book
Support the author Video Tutorial

About
The author
Legal Notice
Feedback
Buy paper or digital copy
Terms and Conditions

9 of 9 02/01/2023, 02:19