Authentication fun with the Box.com API and Go

Box, if you haven’t used it, is a secure online collaboration site that allows you to upload, view and collaborate on files. Recently I deployed an Enterprise version of Box for a customer, and had a need to generate a report that showed when files were renamed by a specific user. This report needed to be run on a schedule, so the out-of-the-box reports, whilst useful, would become tedious to run all the time as they require a person to run them.

To make this report happen, I looked into using the Box api from Go. This post covers off what I’ve learned from the process, whilst also giving you some code to have a look through as well.

Getting Started

To get started, I would recommend that you have a read through these following links to get an idea of whats involved.

Authentication Types

So Box talks about using two authentication types depending on whether you want to build a Box Integration, or whether you want to access the Box Platform. The first authentication approach uses standard oAuth 2, sometimes known the 3-legged-version authentication, whereby you receive a web-page back from Box and then you click on a button to authorise an applications access to you Box account.

The second approach is server-to-server authentication using JSON Web Tokens (JWT) which removes the need for the manual, button clicking interaction. I wanted to go with this approach as we don’t want use interaction in the app.

Here’s some details of the approaches from the Box documentation:

  • OAuth 2 requires a user to log in to Box and grant your application permission to access files and folders.The OAuth 2 standard defines athree-legged authentication process; Box conforms to this process. The Box implementation of OAuth 2 is designed to be used with managed users and external users. An external user is an account created by a Box user for that person’s own use.
  • OAuth 2 with JSON Web Tokens (JWT) enables an application to connect directly to Box and obtain authorisation to access files and folders without requiring users to log in. Using OAuth 2 with JSON Web Tokens an application can provide Box features without users even being aware that Box exists. Instead of requiring the user to log in to Box, the application generates JSON Web Token (JWT) verified by an RSA key pair. If this authentication succeeds then the application obtains an access token that grants authorisation to operate on Box files and folders. This machine-to-machine authentication replaces the first leg of the three-legged authentication process defined by OAuth 2, and enables users of your application to work with Box content without seeing Box login requests.

For the purposes of not wasting your time, it appears that the 3-legged oAuth approach is the only show in town that will give you full functionality. to access Box events. Needless to say, I’d progressed quite a bit with the JWT approach before understanding this condition.

Unfortunately the above constraint does inflict upon you the unique scenario whereby you have to authenticate your app manually, and then rely on refresh tokens to ensure that you can still access your account via your app. When I say manually, I mean that you have to acknowledge the standard oAuth webpage you get sent back when your app tries to access an oAuth protected site. This is a bit crap.

If you’re still interested in the JWT approach (and a look at the code I’ve created), keep reading, otherwise I would recommend you have a look at the Box SDK, specifically around the refresh token approach.

JWT With Go for Box.com

If you want to use JSON Web Tokens with Box, then the first thing you need to be aware of is the need to generate private and public keys. I’d been doing this on a mac, which actually by default has an older version of openSSL. If you run the openSSL version command, by default you’ll get the following:

Default OpenSSL for Mac (El Capitan 10.11.3)

This caused me problems as the certs didn’t appear to generate correctly as this version of openSSL is from last year. A quick jump to the OpenSSL site gets you a new version. Pull down the source, and when you get it extracted, you’ll need to run the following commands:

$ ./config
$ make
$ make test
$ make install

When it finishes you’ll get a brand new version of OpenSSL. In my case, I didn’t overwrite or upgrade the default mac version as I only needed to upgrade the certs. If it all goes good, you’ll end up with a new OpenSSL installed into /usr/local/ssl/bin

New OpenSSL install

Generating your keys

To add a public key to Box, you need to generate a public and private key pair. I used the following commands from the /usr/local/ssl/bin directory

./openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048
./openssl rsa -pubout -in private_key.pem -out public_key.pem

Keep your private key safe. Note, I didn’t add a password to the private key, but this can be done with the -pass argument.

Show me the code

To get a look at the code, check out the following linkhttps://github.com/daveym/box-reporter.

The main things to note is that this is a command line app that uses theviper and cobra frameworks for parameter parsing. The following screen shots give you a bit of a run down on how things are constructed within the go app. (As an aside, I’ve switched from Sublime to VSCode for Go work — its come on so much and is really useful now).

Reporter.go is the main entry point to the application, and basically executes and sets up command line parsing. Under the cmd folder, you’ll see a root.go and an auth.go file. Root will always get executed, and auth.go corresponds to passing in an auth parameter to the reporter executable, i.e.

./reporter auth
reporter.go —main entry point for the app

Below is a view of the root file, as you will see, in the execute method, I get the Public KeyId, ClientID and Client Secret — these correspond to the values you’ll get from the Box application settings, usually under:

https://[your app]/developers/services

root.go — retrieving configuration data

And here’s the corresponding box configuration screen:

box configuration, under https://[your app]/developers/services

I’ve constructed the app to have an API for its own use, that pretty much corresponds to each of the parameters that you can pass in. In this case, the auth parameter will correspond to an auth.go file in the api folder.

Auth.go is where the crux of the work happens. In this case we go through 4 steps to get our JSON Web Token, namely:

  1. Create the JWT, passing in the public key ID, Client ID, Claim Sub and whether this is an enterprise token or an app user token (* see below). At this point I use a Go Wrapper I’ve created for the Box api, under the box api folder. The main points to this are the the Authenticate method takes in a box client. This can either be an instance of the client itself, or a mock version that can be used for unit testing. The mock client must have the same signature as the main client for it to be passed in correctly.
  2. Once you get the token back, I build an oAuth request, that again passes in the clientId, Secret and the JWT we got in step one.
  3. This returns an enterprise access token that we can use to create an App User. This is where it gets a bit interesting, because when you get to it, and App user cannot get access to the any events in Box, despite previous documentation implying this. Remember this screenshot? Well if you select App users, you explicitly cannot manage enterprise events. You have to use a standard box user type which is the crux of this post, sadly.
User types — App Users can’t actually manage the enterprise.

Creating an App user is pretty straightforward in that you only need to specify a couple of parameters and make a subsequent post. Here’s the snippet:

req.IsPlatformAccess = true
req.Name = “your app name”
jsonStr, _ := json.Marshal(req)
err := postJSON(“POST”, JWTUSERURL, jsonStr, newUser, EnterpriseAccessToken)
 if err != nil {
 fmt.Println(err.Error())
 }

4. Finally, when you get an App User back, you need to create an access token on behalf of that user. This is where I am reusing the method to create the JWT, except in this case instead of creating an enterprise token, we create an App User token.

At this point, I’ve decided to hold off on further Go work against Box as we needed to progress quickly, but hopefully you’ll find this useful if you want to use Go yourself with Box. On another note, I did speak to Box directly about the confusion in their documentation, and they had responded that their enterprise API is undergoing some work at present, and to use the standard oAuth rather than server-to-server validation.

A colleague of mine is progressing things with the Java SDK and manual oAuth and I’ll see if I can coax him to share his thoughts on it as well.

All for now,

Davey

IS COMPILING THE NEW SCRIPTING?

You may have read my friend and colleague Peters post recently on why laziness, impatience and hubris drives developers to script.
It’s a great read and this post follows on from what Peter has to say. If you know me or follow me on medium or twitter, you’ll no doubt have seen my recent posts on Golang, more commonly known as Go. I’m attempting to learn the language and what better way to learn that to have an actual example/challenge to run with. So, before you go any further, have a read of Peters post first to get context, and then come back here.

Sorted? Good. So a long time ago I used to code a lot in C , and I like the C similarities that are apparent in Go. I figured I’d have a go at this log parsing example and see how the Go version performs.

The code looks like this:

  • Lines 75–82: We define our main function and use the Go flag library to setup what command line parameters that we anticipate being passed to the executable. These take the format: variable := flag.type(“parameter”, default, “message to user”), where type is the type of the parameter (bool, int, string), default is the default value and the message to the user is the prompt they will see on the command line. In most cases, we default our expected inputs to true, and set a default of 1 (second) for the threshold value.
  • The flag.parse() command on line 82 basically means that you are happy with the parameters you have setup beforehand and that anything else coming in on the command line after these parameters have been parsed should be captured into a slice (a dynamic array) that we can access with flag.Args(). This allows us to iterate over the log file paths/names (testdata/*.log) and pass these in for processing when we run grep_elb -b -i -r -t=5 -v testdata/*.log <- this bit goes into flag.Args()
  • Lines 84:88: If the debug flag ‘d’ is true, print out some additional information on what arguments are set and the file paths.
  • Lines 91–93: Pass in each log file to the parseELBLogfile() function, along with any command line parameters, such as thresholds etc.
  • Lines 31–71: This is where the magic happens. parseELBLogfile() basically opens each file, and reads all the lines in the file into a slice. For each line, it splits it into separate array items by using a space ‘ ‘ as a delimiter. We then evaluate the status of our flags that we accepted via the command line. If i, b or r are set, then we want those timings to be compared against the threshold parameter value ‘t’, and if they are greater than the threshold, output them.
  • Lines 15–29: We basically read in the lines of the file here. Notice the defer statement. This is kinda like a ‘finally’ block in a try-catch-finally statement. It makes sure the file gets closed before exiting the function.
  • Lines 13, 71, 92, 96 — you might be wondering what those wg.x() statements are for? Well they are signifying that we want to wait until the concurrent function calls (goroutines) are complete before we exit the program. We define the waitGroup on line 13, and then on line 92 we increment a waitGroup counter to show that we are running a function concurrently, i.e. we prefix it with the ‘Go’ keyword.
  • When we are finished within a function, we signify this with a wg.Done() (line71) and we wait until all goroutines are complete with a wg.Wait() (line 96).

So whats it all look like then? Well, I ran the application against the same log data (5 files with 411,000 lines in total) that Peter has used, on my ‘manager’ Macbook [1].

  • PERL: perl grep_elb.pl -t 5 testdata/*.log 6.56s user 0.19s system 95% cpu 7.067 total
  • GO: grep_elb -t=5 testdata/*.log 1.00s user 0.17s system 92% cpu 1.263 total. Note, this version excluded goroutines.

I actually tried again with goroutines and the timing ended up around 1.56s or so. I suspect that the IO operations of reading the file and writing to stdout are the bottlenecks. One final thing — Peter ran this against the full set of 24 hours worth of ELB log files for a large service. It has 1589 log files that contain 5,159,601 lines and are 1.6Gb. Perl took 114 seconds. The Go version? 26.39 seconds.

Now the title of this post is asking if compiling is the new scripting? Possibly. I wrote this little app in Sublime Text with the GoSublime plugin. Compilation took 0.902 seconds, and I get autocomplete, formatting and REPL from within the environment. Definitely a nice way to knock up a simple ‘script’.

So — do you fancy having a go yourself with another language? Let us know how you get on.


[1] MacBook, Early 2015 (MacBook 8,1). 1.3 Ghz, 8Gb RAM 1600 Mhz DDR3, 256Gb SSD (FileVault enabled), Intel HD Graphics 5300 1536Mb. El Capitan, 10.11.3