• Open Policy Agent is now a graduated CNCF project.
• Styra has been listed as one of the best workplaces of 2021
• …named as a cool vendor by Gartner.
• …has raised $40 million in series B funding.

To call the last six months eventful would be an understatement! While I’d like to believe I’ve played at least a tiny part in the bigger events, this also feels like a good time to note down some of my own work and experiences working with OPA at Styra for the last half of a year.

Community

One thing that wasn’t new to me was the fantastic OPA community. In fact it alone had been a major driver in my decision to work full time on OPA. Getting to work not just on open source, but one of the coolest projects around, was just insanely awesome.. and six months later, it still is. There’s just so much cool going on around OPA that I can barely wait to see where we are just six months or a year from now.

Needless to say I’m somewhat biased at this point, but I believe the clear separation between OPA the open source project and Styra the company has really been well managed by Styra, and while the Styra DAS is a kick-ass control plane for managing OPA at scale, I appreciate having a clear line drawn between community work and Styra the product. I’ve come to understand that this is something many developer advocates struggle with at other companies and communities.

Team and Colleagues

Having worked in and around the OPA community for quite some time, and for a Styra customer even, I knew at least a handful of colleagues before joining. Though I don’t work with everyone in the company I have probably had some interactions with most by now, and I’ve gotta say that Styra has really managed to collect some of the most competent and kind people I’ve had the pleasure of working with in my career so far. Working with so many smart people has been a humbling experience, but since they seem to appreciate what I’m doing I guess I must be doing something good, or they’re not that smart after all :)

Coding

Already before starting at Styra I had contributed quite a few commits into OPA and the surrounding ecosystem. While I still try to write at least some code each day, I find myself prioritizing other tasks, like blogging, higher. Working closely with people who are way better coders than I am helps some with that too. Not in the sense that I don’t feel like I can contribute, but I find it pretty hard to maintain a balance between my responsibilities if I let myself get too involved in bigger coding tasks or PR discussions. While I enjoy coding on OPA I’ve had just as fun working with side projects in the ecosystem, such as the Clojure OPA middleware. Need to remind myself to work more on projects like that!

Working From Home and Timezones

While I, like most of the people in tech during the pandemic, worked from home long before joining Styra, this is my first job where I know there won’t be an office to go to later. While I feel more productive at home most days, having to share an “office” with a newborn kid has been a challenge at times. Having a real office to go to for presentations, meetups and webinars would definitely help, and after the summer and my parental leave is over I’ll look into renting something like a shared workspace with private rooms for that purpose.

With the majority of my colleagues in the US west coast, the timezone difference has been a bit of a problem at times. There’s only really an hour or so of natural overlap in office hours between Stockholm and Pacific Time, but with most communication taking place on Slack or over email, there hasn’t been a need for that many meetings.

Timezone wise I expect things to normalize somewhat as Styra grows in Europe and more colleagues will work closer geographically. Secondly, working from home becomes truly awesome when combined with occasional traveling, as that pretty much provides the best of both worlds, and I am really looking forward able to go to non-virtual conferences, meetups and events soon.

Blogging

While I did write the occasional internal blog at Bisnode, and certainly my fair share of wiki pages, documentation, and so on.. I’ve really gotten the chance to ramp up my blogging at Styra. While writing posts for blogs that are not only public, but often have quite a large number of readers, can feel quite intimidating at times, it is also extremely rewarding!

Having people, from both my own team and from Styra’s fantastic marketing team, review my blog posts before they are published and provide corrections, edits and feedback has really been invaluable. I feel like I’ve learnt so much here these past six months and yet I feel like I’ve only just started. Really looking forward to keep learning in this space! Here’s the blogs I’ve authored so far. Some for the OPA blog, some for Styra and some for external publications:

• Open Policy Agent 2020, year in review
• Integrating Identity: OAuth2 and OpenID Connect in Open Policy Agent
• 5 OPA Deployment Performance Models for Microservices later republished on the Styra blog
• The Kubernetes Authorization Webhook
• Linting Rego with… Rego!
• Dynamic Policy Composition with OPA
• WTF is policy as code
• Community Spotlight, Grant Shively
• Getting Open Policy Agent Up and Running
• What is Open Policy Agent?

Meetups and Conferences

Besides blogging, presenting OPA at meetups and conferences has really been the main activity on my list of priorities. While this wasn’t something new to me, and I’d much rather meet people in person than the online events we’ve all had to attend this last year and a half, I must say I really enjoyed this.

I’ve talked at a couple of conferences and more than a dozen public webinars and meetups. On top of that probably as many or more internal meetups at companies. I’ve come to think that the meetup format has been a better fit for online/video than the (especially big) conferences have been. Another observation is that although there is nothing stopping anyone from joining local community meetups online, they have remained surprisingly… local. Presenting for a small local community of peers where almost everyone knows each other by name, that feeling really comes across even online.

Twitter

Having worked mainly with internal development communities at companies in the past, I hadn’t really had much use for Twitter. Now working full time with the larger OPA community made Twitter a great choice for interacting with the community just as much as for promoting blogs, events and meetups to people outside of it. In fact Twitter was so engaging that I eventually had to remove the app from my phone, and I’m now only checking in from my computer a couple of times per day. I’ve already made some great connections there and I’m looking forward to growing my network. Find me there, and if we haven’t already I’d be happy to connect!

Summary

Though I, as I guess most people do, sometimes feel I could have have done more, looking back at the past six months makes me realize I’ve really done a lot! Most importantly, I’ve had a lot of fun and learnt so much along the way. Exciting times, and I’m really looking forward to growing in my role as a developer advocate here, just as much as I enjoy being part of the bigger Styra journey. If you’d like to join me on that, Styra is hiring!

Bisnode historically had worked more or less like a big company with independent entities in many European countries with little or no coordination between the units. This worked well for creating products and offerings tailor made for local markets, but obviously made less sense for competing on an increasingly international market. So a few years back Bisnode changed direction towards a model named “One Bisnode” which tried to address this by developing a series of products that would work across borders and markets. From a tech perspective this meant not just merging products and platforms but also that development teams scattered across thirteen countries - all used to working more or less in isolation - now had to work together. Having someone from outside of the teams with both development experience and at least some degree of social skills would surely be useful in that coordination effort. That’s how the role of developer advocate came to be, and I started working in August the same year.

As pretty much all development teams worked in or at least aimed to work on the new platform Bisnode was building, it made sense that I’d be a part of the platform team. While I wasn’t initially that interested in the infrastructure side of things, the enthusiasm and skills of the team quickly had me engaged in various infra, ops and devops issues whenever I got some time outside of my responsibilities as a developer advocate. This paid off immediately not just for my own enjoyment but just as much when getting new teams on-boarded to the platform. Unsurprisingly people tend to listen to someone who knows what they’re talking about, and occasionally I do. That fall I went on a tour through Europe to meet as many Bisnode developers that I possibly could. To learn who they were and what they were working on to some extent, but just as much to talk about and demonstrate the new common systems, platforms and tools. I loved every second. Talking to smart people about code, tech, design, methodologies and whatnot - all things I do whenever I get a chance, and here I was having someone actually pay me to do this? Spending days in front of a whiteboard leading presentations and workshops, often with discussions that continued long into the evening at some local restaurant or bar. Good times!

One of the trips I made wasn’t to one of our offices but to Barcelona together with my platform team. In a huge conference center we spent almost a week listening to some of the brightest (and as is commonly the case, some of the not as bright) minds in the kubernetes community talk about their projects, products and ideas. Some of the talks we attended were about a policy engine one could plug into kubernetes to put guard rails around ones resources - things like “if this application being deployed has too many replicas it will be too expensive and should thus be denied”. Not super interesting to me at the time, but the idea of having policies separate from application code to govern access control and being used as the basis for other types of enforcement - this brought me back to a lot of the problems I faced and ultimately left unsolved when working on the identity side of things. Like, I’ve built all these things to prove who someone is - now what do we do with that information? That’s exactly where Open Policy Agent (OPA) entered the room. Using OPA and its policy language Rego there was finally a way to make use of the identity to make informed decisions in all of these distributed systems, and it didn’t take long before OPA was “everywhere” at Bisnode - mainly because we just had so much fun implementing it in some product or domain that once we were done we just kept on going. Several developers and teams joined in and soon after OPA was officially adopted as the way to work with authorization and other types of policy within the Bisnode tech organization.

Having many of my plans and ambitions around advocacy cancelled due to the ongoing pandemic was disappointing, but cutting down on some of the social activities also meant that I got to direct more focus to.. well, tech. This past year I have tried to spend about half the time contributing to the platform team’s projects and the rest of the time focused on getting OPA out into the hands of our development teams and hopefully have them experience the same joy working with this piece of technology as I have myself. To help me out in this effort a team from Warsaw have acted as my closest allies since the start of the year, and I’m truly proud of what we’ve accomplished together. Not only have we introduced OPA to Bisnode, but we have tried our best to just as much be an active member of the OPA open source community with all that entails - contributing code, participating in meetups and online forums, advocating OPA in public channels, and so on. Working like this was new to all of us, and we’ve learnt more than I could ever try and summarize here, but most of all we’ve just had so much fun. Having worked in the OPA community for this long also opened a lot of doors and opportunities, including a good relationship with Styra, the commercial backer of OPA. Getting to work close to them as a business partner to Bisnode I got to know a whole bunch of enthusiastic and interesting personalities all sharing my interest for this emerging domain and technology. So while I’ve had the best of times at Bisnode and forever will be grateful for the opportunities they offered me, I eventually realized this was too good of an opportunity to pass up on.

Beginning December I will start my new position as the first developer advocate of Styra. What exactly awaits me remains to be seen, but from what I’ve experienced so far I know I will be in the best company possible. Getting to work on - and spread my enthusiasm for the things I enjoy doing the most, together with some of the best people in this space - I can’t beliebe my luck.

I wish I could thank all the people I’ve worked with at Bisnode for the last couple of years, but the list would just be way too long. Thank you Bisnode, and thank you all smart, kindhearted people who have been there with me along the way. You’ve easily made this time the best in my career so far, and many of you will remain some of my best friends. Keep up the good work, and I hope to see you soon.

A new journey awaits, and I can’t wait to get started. Styra, here I come!

As the cluster is shared by many teams and products the provisioning of each database follows a quite traditional pattern. A user notifies the platform/database team who runs a script to setup the database and user, stores the credentials somewhere safe and then notifies the user who ordered it. This script is run as the root user of the cluster, and that user is the only one allowed access across databases. So when assigned the task of deleting a few misplaced rows in one these databases, this was naturally done as the root user. Here’s where the problems starts:

postgres=> \c example
You are now connected to database "example" as user "root".
example=> DELETE FROM sometable WHERE somedata = 'shouldnotbehere';
ERROR:  permission denied for relation sometable

OK, so that’s unusual. Logging in as the root user would mean “anything goes” in pretty much any system I’ve worked with, so what’s going on here? Turns out that the RDS Aurora clusters have a somewhat different idea on that. Since “anything goes” includes things such as configuring (and possibly breaking) replication - and this is a managed product after all - the root user has been severly limited in what it is allowed to do. Checking the available roles with \du confirms this:

                                                                  List of roles
       Role name       |                         Attributes                         |                          Member of
-----------------------+------------------------------------------------------------+-------------------------------------------------------------
 rds_superuser         | Cannot login                                               | {pg_monitor,pg_signal_backend,rds_replication,rds_password}
 rdsadmin              | Superuser, Create role, Create DB, Replication, Bypass RLS+| {}
                       | Password valid until infinity                              |
 root                  | Create role, Create DB                                    +| {rds_superuser}
                       | Password valid until infinity                              |

Good news first - at least we can use the root role to login! The only attributes assigned to the role however is Create role and Create DB. Alright, I suppose we can live with that. What doesn’t really follow though is how the same root user is unable to modify, or even read, database resources itself created. Looking at the database and user creation script it all looked pretty standard. As the root user something like the below got executed:

CREATE DATABASE example
CREATE USER someuser WITH ENCRYPTED PASSWORD 'somepassword'
GRANT ALL PRIVILEGES ON DATABASE example TO someuser

Surely the root user creating both the user and database here would be granted access to the very same resource? Turns out, well yes and no. The root user has access to the database, and may list tables, etc. It is however not allowed acceess to resources later created by the someuser user in the example database, even if it created both of them. Trying to grant these privileges to the root user, as the root user also turned out to be pointless - the root user would still be denied access. Only by logging in as the user just created, and having that user grant table level privileges to the root account would allow the root user subsequent access to the tables there:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO root
ALTER DEFAULT PRIVILEGES FOR USER someuser IN SCHEMA public GRANT ALL ON example TO root

The last ALTER statement is crucial - granting privileges in postgres only applies to existing resources, not those created at some point in the future. So unless you want to repeat the process every now and then you’ll need to ensure that the default privileges are altered as well. Running this procedure for the database in need of maintenance and updating the user and database provisioning scripts means the problem is solved for now and the ticket could be closed. The root (hah!) of the problem however remains - the AWS Aurora Postgres root user is still a wimp!

The pre-commit framework

Running hooks in response to events is functionality built into git, but having to manually manage shell scripts for each repository being worked on quickly becomes a real chore, and the idea here is to make us more effective after all. The pre-commit framework is a “framework for managing and maintaining multi-language pre-commit hooks” and does just that. Though it’s called a “framework” it’s dead simple to use, simply by following these three steps:

1. Install pre-commit.
2. Place a config file (.pre-commit-config.yaml) in your project’s root directory, pointing out any pre-commit hooks you want to include. There’s a long list of ready-made hooks available, so for any language or technology you’re working with, chances are good that there’s one for your needs.
3. Run pre-commit install to install the pre-commit hooks configured in the step above.

That’s it! Anytime you run git commit from now on, the hooks will be called and any errors reported in hooks like test runners, linters or formatters will now abort the commit, let you fix the issues before re-adding the files to the staging area and reattempt the commit.

Pre-commit hooks for OPA

Having spent considerable time with Open Policy Agent (OPA) in the last year or so, one thing I’ve really come to miss when authoring and testing policies is pre-commit hooks. OPA comes with a few command line tools to help out during developoment, such as opa check to verify syntax, opa fmt to format rego policies, and opa test to run unit tests. Not having these run by default before commiting has occasionally meant forgetting about it, only to be reminded either by the CI/CD server or a colleague reviewing the policy. We can’t have that! So a couple of weeks ago I looked into writing my hooks for this purpose, and it turned out to be dead simple. Adding some fixes based off of real usage in my OPA repos since, and I think they’re now in a state where they’re good enough for general use. I hope you’ll find the pre-commit-opa hooks useful in your OPA projects.

Sometimes you’ll want/need to use TLS for connections to internal kubernetes services where TLS isn’t supported or used by the original application, or where you’d normally terminate TLS in the ingress but don’t want to have internal kubernetes traffic routed through that. Cases like that are a great fit for the sidecar pattern, injecting a small proxy inside a pod to terminate TLS and forward the requests to the original application. Having configured a few of these sidecars for various projects, I started to compile a common configuration for nginx which I could reuse for this purpose. Hence the nginx-tls-terminator project was born.

From the project’s decription:

Single-purpose TLS terminating nginx proxy.

Primary function is to run as a sidecar container in kubernetes pods where the app running in the main container either does not support TLS, or it’s inconvenient to add it - such as for legacy apps or where the code is not under your control. Another common use case is when external traffic has TLS terminated by the ingress controller, but some internal services need to reach the same services from the inside on the same URL. One example of this is the issuer URL provided in OAuth2 and OpenID Connect, where both external and internal applications will need to query the same endpoints over a secure channel.

Features:

• Small single-purpose container at ~9 MB with minimal configuration needed.
• Does not require root privileges and runs as non-root user by default. Runs with strictest securityContext configured on the container.
• Running with a read-only root filesystem as easy as mounting a volume on /tmp.
• Exposed TLS port as well as target upstream port configurable through environment variables.

Check it out on Github and Docker Hub.

Incremental rule - sets

A common pattern when writing Rego policies is to use incremental rules to build a set of values. This is tremendously useful for e.g. returning a message for each violation of a defined policy:

deny[reason] {
    input.role != "admin"
    reason = "User not an admin"
}

deny[reason] {
    time.weekday(time.now_ns()) == "Sunday"
    reason = "Access not allowed on Sundays"
}

Any non-admin trying to access the requested resource on a Sunday would now not only be denied, but could also be given an explanation as to why:

"deny": [
    "Access not allowed on Sundays",
    "User not an admin"
]

See full example on the Rego Playground.

Incremental rules - maps

Not as commonly used, but certainly just as useful - incremental rules can also be used to build up maps. This could be used similarily to build up key/value pairs of reasons or violations, but might just as well be used for data transformations. Consider the following input provided to OPA:

{
    "street": "High street 32",
    "zip": "11766"
}

We could create an incremental map rule to gradually build a structured address object with values in a normalized format:

address["zip"] = zip_numeric {
    zip_numeric = to_number(input.zip)
}

address["street_name"] = streetname {
    streetname = trim(concat(" ", regex.find_n(`[^\d]+`, input.street, -1)), " ")
}

address["street_number"] = streetnumber {
    numbers := regex.find_n(`\d+$`, input.street, 1)
    streetnumber = to_number(numbers[0])
}

Which would increment the address rule map as follows:

{
    "address": {
        "street_name": "High street",
        "street_number": 32,
        "zip": 11766
    }
}

Again, see full example on the Rego Playground.

From a tech perspective this blog is powered by Jekyll and hosted as a “GitHub Pages” page - pretty much just markdown files and some stylesheets that Jekyll is kind enough to transform into something resembling a blog. The (ruby based) tech behind it is old and uninteresting enough that I might just get time for writing rather than trying to hack the actual blogging software, something that has stopped at least a handful of previous attempts to start blogging.

About Jekyll being old - I was especially amused by the documentation section on creating posts where the example post is named “2011-12-31-new-years-eve-is-awesome.md” - that’s almost a decade ago now! Just as GitHub repositories with a “last modified” dating a few years back in time, that would have been a huge warning flag to me had you asked me just a few years ago. Now I consider “slow moving, old and boring” a feature, at least for software like this where it’s not the actual code itself that’s of interest but rather something I approach as an end user.

Hoping to focus on writing down my thoughts rather than coding the blog itself, but given this is all very new there are a few things that I’ll need to look into before I can do so:

• Should poke around in themes and stylesheets and pick a theme for the blog as the minima theme certainly was minimal.
• Analytics. There’s Google Analytics of course, but I’d rather use something that respects user privacy, doesn’t do tracking outside of the blog, etc. Piwik/Matomo seems to be a contender here, but not sure yet how to use it with a statically generated blog.
• Setup something for post comments. Unsurprisingly this is a little trickier to pull off on a statically generated site. Disqus seemed to have ruled this space for long, but their popularity has declined since they introduced (allegedly, annoying) ads in their free model. Staticman seems to be a great free alternative - will check it out as soon as possible.

Will look into my options for any of the above as soon as I get an opportunity to do so, and perhaps I’ll write some about it here as well. For now just happy to finally get something out the door!