Skip to main content

Open Policy Agent v0.19 Release

Torin Sandall
Torin Sandall

Open Policy Agent v0.19 release announcement banner

Last week we released OPA v0.19, containing 63 commits from 12 contributors (of which, 9 were external.) This release includes many important fixes and enhancements, as well as a new Rego parser written in Go that speeds up parsing time by ~100x in most cases. You can find more details on the GitHub releases page.

Community Updates​

Since many in-person events have gone virtual due to the COVID-19 crisis, there have been several virtual events, webinars and podcasts featuring OPA over the last few weeks. Here's a quick roundup:

Since KubeCon 2019 in Barcelona, we have asked users to post Q&A style queries on Stack Overflow instead of slack.openpolicyagent.org. The reason is that most answers posted on Slack are not discoverable! If you have Q&A style questions (e.g., "How to test not deny?"), try posting on Stack Overflow and tagging with open-policy-agent.

Faster Parsing & Better Errors​

The largest change in v0.19 is the new Rego parser, which is written from scratch in Go. Previously, OPA relied on a generated parser that was defined using PEG (Parsing Expression Grammar [wikipedia]). Over the years, as the grammar has grown, and larger inputs have been thrown at it, the generated parser became a bottleneck (e.g., it could take about 10x longer to parse an input than compile and evaluate the query.

Inside OPA we were able to workaround the performance problems with caching, using Go's "encoding/json" package and manually converting to AST ("Abstract Syntax Tree") values when possible, etc. However, new users embedding OPA as a library would (understandably) make mistakes and wonder why performance was poor. The majority of the performance problems in the generated parser were due to a significant amount of heap allocations required to parse any input.

In addition to performance, we also struggled with usability around parser error messages. If the parser was not able to match an input, you would be presented with an error like "policy.rego:19: no match found". No match? Tell me more!

Rather than attempt to continue incrementally improving the existing parser, we decided to rewrite it from scratch in Go. The result is a new parser that allocates significantly less memory (which improves performance by approximately 100x in most cases) and has better error messages. One important requirement for the new parser was backwards compatibility — the new parser could not break existing policies OR programs that embed OPA as a library (e.g., the parser APIs and the AST types also had to remain the same). To ensure we did not break existing (valid) policies, we checked for differences in the output of the old and new parser for hundreds of thousands of Rego snippets (which deserves another blog post in the future.) Lastly, we also applied the wonderful go-fuzz project to the parser to help catch crashes and other bugs.

Since we no longer have a declarative representation of the language grammar in Go, please refer to the ENBF grammar in the OPA documentation as the authoritative source.

The chart below shows the difference in performance between the old (v0.18 and earlier) and new (v0.19 and later) parser (log scale):

The chart below shows the difference in performance between the old (v0.18 and earlier) and new (v0.19 and later) parser (log scale)

Overall, we are happy with the process. In the future we plan to continue optimizing performance in the parser and looking for ways to improve error messaging and usability.

man(1) pages, http.send, and Emacs support​

In addition to the new parser, v0.19 includes dozens of bugfixes and feature enhancements. @olivierlemasle contributed code to generate OPA man pages from the OPA CLI definitions. The man pages are automatically available if you:

brew install opa

Install OPA with homebrew and use 'man opa' to learn about it.

Install OPA with homebrew and use 'man opa' to learn about it.

@jpeach submitted a number of patches that improve testing and support for the http.send built-in function. For example, policies can now explicitly set TLS server names as well as certificates and keys when invoking the built-in function (previously they could only come from the environment or local files). This is useful if you want to specify those values in data or as local variables inside the policy itself.

Lastly, the release also includes a pointer to the new rego-mode Emacs package developed by @psibi. The package provides syntax highlighting, formatting and more. In the future, the package could be extended to support many of the same features as the OPA extension for VS Code.

WebAssembly Update​

At KubeCon 2019 in San Diego we announced support for compiling OPA policies into WebAssembly (Wasm). Wasm enables OPA policies to execute in new environments like CDNs, service proxies and more without requiring an out-of-process RPC call to query OPA.

This week we are excited to release further support for Wasm in OPA with the new golang-opa-wasm project! This project wraps the wasmerio/go-ext-wasm runtime library to provide convenient APIs for policy execution and more. The golang-opa-wasm SDK is still work-in-progress but feedback and contributions are welcome.