During my internship period at WSO2, Inc I worked on a project to develop a CI/CD pipeline for WSO2 API Manager. The tooling was mostly done in Golang.
When we were developing the tooling we wanted to have a project initialization phase through the CLI tool. That indeed involves a lot of code generation.
In the beginning, everything was fine, we had few files and we kept them as slices of bytes and accessed them, which was totally fine.
Nightmare of backticks `````
The project was scaling up and content we wanted to store was growing too. It was very hard to manage these contents as they were basically in Go packages and had to manually edit them each and every time.
One day I wanted to use Markdown for our README file which generated by the tool in the initialization phase of a project, we were generating a simple Text file until then.
Markdown uses backticks for code blocks and Go uses backticks for Raw Strings, I could not find a better way to escape them. It was a total disaster !.
We can’t ship these files separately as this was focused to be a small CLI tool, so I kept looking methods to embed resources into Go.
Yes, there are existing solutions for the exact problem like Go.rice But for my task, I thought they are more complex and bloated. All I wanted to do is convert my resources into Go files and access them.
Go generate for rescue
Go is awesome, it comes batteries included. The language is capable of code generation by default. So why not using it. If you are not familiar with what go generate is read the official blog. Go generate uses a special magic string to identify files that need to be generated
When you execute
go generate filename.go the compiler will check for the magic string and executes the command provided.
go generate ./... will run the command through an entire project looking for files with magic string and executing the command you provided.
Go generate runs the given command relative to the directory containing the file with magic string.
Since we need to make our code generation universal, why shouldn’t we use go itself for code generation?
My main requirement was to storing files as bytes and accessing them in my go files. To fulfil the same requirement I created a package called
box. It was acting as a proxy between consumers and data source. We store our all files inside a special directory called resources inside the box package. It can contain directories as well to organize content.
This makes editing content much more easier and much more clean as they are just files in the system.
This is a very simple go file, all it does is wrapping our map inside a nice
ResourceBox exposing all methods at package level so you can invoke them using
box.Get('filename') in anywhere, super cool.
resource object plays a key role in the package acting as the singleton to hold all the resources for our box.
Now here is the magic, it has a special comment with
//go:generate go run gen.go , This is where magic is happening.
When you are invoking
go generate ./… this file will be detected and it will run another go program.
Now the fun part, we want to read our resource directory and store the content of files in a
ResourceBox. For that we are running another go file called
gen.go which is in the same
package box .
As I described above, the go generate is smart enough to run the given program relative to the directory it found the magic string, so
go run gen.go would run inside the box directory. It means we can easily read all the files in the resources directory without a problem.
We are going to create a file called
blob.go with all the content we need inside the same package.
Since gen.go is a program we need to tell go that this is not a part of our build. Otherwise, it will complain about declaring multiple packages in the same directory.
So just add,
on the top of the file to inform the compiler to ignore this on the build. So it won’t complain.
Now we need to walk through all the files in the resources directory and convert them to go files.
A part of
gen.go which walks the file tree and reading each file and storing them
Go provides a nice way to handle this using
filepath.Walk method. We are simply iterating all the files inside the resources directory(including all the subdirectories as well) and storing them in a map called resources. We are going to access files using the relative path to the resources directory.
For example to access a file in resources/init/sample.yaml we can use
box.Get('/init/sample.yaml') in a very intuitive way. But what if we run the generation in a Windows machine?
To overcome this we are going to use a feature called
ToSlash in the
filepath package, which will transform the platform native path separator to a slash.
Next, we need to create
blob.go with all the data we have, we just need to add all the data to the resources object in the box package.
We just simply need to call
resources.Add('/file/path', data) , what we can do is putting this in the
init() method on the blob.go which will be initialized for the whole package at once.
To do this we simply use Go Templates as following,
It simply iterates and creates the actual file containing the name and data for the file, later we will write this to the disk allowing access within the code.
You can see there is a function map passed to the template engine. I used this to create the byte representation of the slice as a string and use it within the template. It’s a very simple function as follows. It takes a slice of byte and returns a comma-separated value string containing all the bytes as numbers.
If you had any back-tick now they all have been converted to some numbers, so no worries
Now we write the file to the disk and we are ready to Go
But wait, my linters are failing
This is because the code is not properly formatted, you can do it yourself after generation.
Nope, we are not gonna do that, Go itself provides the formatter for formatting any go code, we can just invoke it before saving the file to the disk
Simple as that.
Now just run
go generate ./… within your application and you can find a file called
box/blob.go , it will contain all the data for resources in your
The generated blob file
Now all you have to do is invoking
box.Get('/sample/sample_config.yaml') to get the data from your file. Now it is embedded in your Go binary.
There were many existing solutions out there for solving this problem of embedding with advanced features. But I wanted to implement a simple solution for solving the problem we had.
Go is a powerful language which has many features that every programmer wants in their tool belt. It has templates, code generation and many more within the standard library.
This solution separates content from the code and makes them intuitive and more easy to manage.
For example, if a content writer wanted to correct some issues in your README file they can simply edit that file and build the program without thinking of the implementation. This is a win/win for both parties.
Code for the entire project: wso2/product-apim-tooling
Author Kasun Vithanage
LastMod 2020-01-16 (1111753)