IS COMPILING THE NEW SCRIPTING?

You may have read my friend and colleague Peters post recently on why laziness, impatience and hubris drives developers to script.
It’s a great read and this post follows on from what Peter has to say. If you know me or follow me on medium or twitter, you’ll no doubt have seen my recent posts on Golang, more commonly known as Go. I’m attempting to learn the language and what better way to learn that to have an actual example/challenge to run with. So, before you go any further, have a read of Peters post first to get context, and then come back here.

Sorted? Good. So a long time ago I used to code a lot in C , and I like the C similarities that are apparent in Go. I figured I’d have a go at this log parsing example and see how the Go version performs.

The code looks like this:

  • Lines 75–82: We define our main function and use the Go flag library to setup what command line parameters that we anticipate being passed to the executable. These take the format: variable := flag.type(“parameter”, default, “message to user”), where type is the type of the parameter (bool, int, string), default is the default value and the message to the user is the prompt they will see on the command line. In most cases, we default our expected inputs to true, and set a default of 1 (second) for the threshold value.
  • The flag.parse() command on line 82 basically means that you are happy with the parameters you have setup beforehand and that anything else coming in on the command line after these parameters have been parsed should be captured into a slice (a dynamic array) that we can access with flag.Args(). This allows us to iterate over the log file paths/names (testdata/*.log) and pass these in for processing when we run grep_elb -b -i -r -t=5 -v testdata/*.log <- this bit goes into flag.Args()
  • Lines 84:88: If the debug flag ‘d’ is true, print out some additional information on what arguments are set and the file paths.
  • Lines 91–93: Pass in each log file to the parseELBLogfile() function, along with any command line parameters, such as thresholds etc.
  • Lines 31–71: This is where the magic happens. parseELBLogfile() basically opens each file, and reads all the lines in the file into a slice. For each line, it splits it into separate array items by using a space ‘ ‘ as a delimiter. We then evaluate the status of our flags that we accepted via the command line. If i, b or r are set, then we want those timings to be compared against the threshold parameter value ‘t’, and if they are greater than the threshold, output them.
  • Lines 15–29: We basically read in the lines of the file here. Notice the defer statement. This is kinda like a ‘finally’ block in a try-catch-finally statement. It makes sure the file gets closed before exiting the function.
  • Lines 13, 71, 92, 96 — you might be wondering what those wg.x() statements are for? Well they are signifying that we want to wait until the concurrent function calls (goroutines) are complete before we exit the program. We define the waitGroup on line 13, and then on line 92 we increment a waitGroup counter to show that we are running a function concurrently, i.e. we prefix it with the ‘Go’ keyword.
  • When we are finished within a function, we signify this with a wg.Done() (line71) and we wait until all goroutines are complete with a wg.Wait() (line 96).

So whats it all look like then? Well, I ran the application against the same log data (5 files with 411,000 lines in total) that Peter has used, on my ‘manager’ Macbook [1].

  • PERL: perl grep_elb.pl -t 5 testdata/*.log 6.56s user 0.19s system 95% cpu 7.067 total
  • GO: grep_elb -t=5 testdata/*.log 1.00s user 0.17s system 92% cpu 1.263 total. Note, this version excluded goroutines.

I actually tried again with goroutines and the timing ended up around 1.56s or so. I suspect that the IO operations of reading the file and writing to stdout are the bottlenecks. One final thing — Peter ran this against the full set of 24 hours worth of ELB log files for a large service. It has 1589 log files that contain 5,159,601 lines and are 1.6Gb. Perl took 114 seconds. The Go version? 26.39 seconds.

Now the title of this post is asking if compiling is the new scripting? Possibly. I wrote this little app in Sublime Text with the GoSublime plugin. Compilation took 0.902 seconds, and I get autocomplete, formatting and REPL from within the environment. Definitely a nice way to knock up a simple ‘script’.

So — do you fancy having a go yourself with another language? Let us know how you get on.


[1] MacBook, Early 2015 (MacBook 8,1). 1.3 Ghz, 8Gb RAM 1600 Mhz DDR3, 256Gb SSD (FileVault enabled), Intel HD Graphics 5300 1536Mb. El Capitan, 10.11.3

Setting up a Go IDE

I want to learn Go, and to do that I want to have a nice IDE that allows me to execute a build, run some tests and see what the errors are, all from within the IDE.

Doesn’t sound like much of an ask though, does it? Well there’s a bit of a learning curve out there with regard to setting up an Go environment. I’ve managed to pull something together that suits me, and maybe it’ll suit you too. First off, there are a few options* that you can follow:

1) Use VIM — the following gives you some idea on how to setup VIM for Go. I didn’t follow this approach as I’m not as great with VIM as I could be, and I don’t want to have the frustrations of VIM put me off from learning Go.

2) Use ATOM — ATOM is an IDE from Github, and go-plus provides an improved Go experience for the Atom Editor. This looks like a nice extension, but I want to execute my app as well as having the nice features that go-plus gives you, such as autocomplete, formatting, code quality checks etc. Atom & Go Plus may very well work for you, and if it does please add some tips to the comments section of this doc — sharing is good.

3) Use Sublime Text — This is the approach I have followed, principally because I use Sublime a lot and have a license, but also because I can get the nice formatting and validation features combined with the ability to execute my app from within the IDE as well. Below is what I’ve ended up with:

To get to this point, I followed the following steps:

  1. Install Go — I downloaded the Go Tools from the golang site
  2. Set your GOPATH and GOROOT. Given that I am on OSX, I set it from the terminal by adding the following to the ~/.bash_profile
  3. 1. export GOPATH=/Users/davidmcg/Dropbox/Github/gocode
  4. 2. export GOROOT=/usr/local/go
  5. Add $GOPATH/bin:$GOROOT/bin to your PATH variable — again I added this to ~/.bash_profile
  6. Install Sublime Text — you can easily get Sublime Text from here
  7. Install the package control for Sublime Text by following these steps
  8. Once you have package control installed, you’ll want to install 2 packages:
  • GoSublime — Provides code completion and other IDE-like features such as syntax checking as you type, code formatting and removing unused imports.
  • Sublime-build — The official Sublime Text package for Go build system integration. Golang Build is a Sublime Text package for compiling Go projects. It provides integration between Sublime Text and the command line go tool.

9. To install these packages, run the following bring up the command palette and start typing Package Control: Install Package then press return or click on that option to activate it.

You will be presented with a new Quick Panel with the list of available packages. Type GoSublime or sublime-build and press return to install each package individually.

Configuring GoSublime

To configure GoSublime, you need to configure Sublime Text to run a number of commands on the save of the .go file. The commands will check the code quality, and apply formatting. First, you need to access the settings file for your user, within Sublime Text:

Once you access this section, you’ll need to paste in the following. Note — you’ll need to change your path to match your own system. [CREDIT– I got this from Jonathan Gautheron’s excellent medium post ]

{
 “env”: {
 “GOPATH”: “YOUR GO PATH GOES HERE”,
“PATH”: YOUR PATH GOES HERE”
 },
 “fmt_cmd”: [“goimports”],
 “on_save”: [{
 “cmd”: “gs9o_open”, “args”: {
 “run”: [
 “sh”,
 “go build . errors && go test -i && go test && go vet && golint .”
 ],
 “focus_view”: false
 }
 }],
 “autocomplete_closures”: true,
 “complete_builtins”: true
}

Additionally, you will need to install the golint and goimports tools. Golint prints out style mistakes, and goimports removes unneeded import statements from your code. To install these tools, type the following in a terminal:

go get -u github.com/golang/lint/golint
go get golang.org/x/tools/cmd/goimports

Configuring Sublime-build

So, we have source formatting occurring, but the next step is to be able to actually run our app from within Sublime Text. Thats where sublime build comes in. You should have installed the sublime build package in step 4.2 above. Once you’ve done that, you’ll need to select it as the build system within Sublime Text, as follows:

And once you have done that, then set the ‘Tools | Build With’ value to be ‘Go Run’ And thats you setup.

Once you have done these steps, you can hit the following commands:

  • ⌘ S (Save) — will run the GoSublime on save commands and format your code, check syntax and make sure the imports are correct.
  • ⌘ B (Build) — will run your app for you, with output in the Sublime console window. Note, you can set this value to be go run, go test, go install from the Build With setting. Each time you press ⌘B, it will execute that command.

One final thing — as per Jonathan Gautheron’s medium post, I have also set Sublime Text to use the excellent Predawn theme. The following providesinstructions to set this up for yourself.

Please feel free to point out anything that I may have missed/could have done better. Good luck and happy learning!

* There are other Go Editors out there — the following provides some more details on these other editors. I just wanted to use Sublime Text :-)