Skip to main content
View all authors

OPA 1.0 is coming. Here's what you need to know.

Open Policy Agent Logo with "1.0" written below Towards Open Policy Agent 1.0

December 28th marked the 8th anniversary of the first commit in the Open Policy Agent project. 5000+ commits from more than 400 contributors later, we're starting to prepare for OPA 1.0.

Following the rules of semantic versioning, one would be excused to think of 1.0 as the first "stable" version. That's not really the case for this project. Since the first public releases of OPA, great care has been taken to ensure new changes don't break existing policy. Thousands of organizations have come to rely on OPA for policy enforcement across the whole stack — often for critical production use cases.

While many features have been added to OPA and the Rego language since 2015, few have ever been removed. All this means that almost any policy written eight, five or three years ago still evaluates using the very latest version OPA, just as it did when it was written! But to keep adding features without ever being able to remove things that might not have worked out as we imagined — or simply didn't age well — also comes at a price. An increased cost of maintenance for sure, but more importantly, old ideas, language constructs and built-in functions all add to the cost of learning Rego, compared to having policy look and act consistently using only modern equivalent features.

With OPA 1.0, we're aiming to fix this.

TL;DR

If you have a busy day ahead and want to get right to something actionable — here's what you can do. From OPA v0.59.0 and onwards, you can start to prepare for the changes in the upcoming 1.0 release following these steps:

  • Use import rego.v1 in each of your Rego files. This replaces all future.keywords imports from previous versions, and is all you need to import until OPA 1.0. Use opa fmt --rego-v1 to format your policy with automatic additions of OPA v1.0 constructs, like if, contains and more. This will also replace any future.keywords imports with the rego.v1 import.
  • Use opa check --rego-v1 to ensure your policy is compatible with "Rego 1.0" mode.

We'll get back to practical concerns by the end of the blog, but before that, let's see what changes are planned for the first major OPA version.

Changes to Rego coming in OPA 1.0

Note: Below follows a non-comprehensive list subject to change. While these features are planned — and most of them even implemented already! — updates may be made before the final 1.0 release. I'll do my best to keep this blog post up to date, as we will with the OPA 1.0 documentation.

The future is now — no more import future.keywords

Several keywords (in, every, if and contains*)* have been added to OPA since the start. Introducing new keywords means there's always a risk that existing policy might break in case identifiers, like rule names or variables, have been named in a way that clashes with the new keywords. In order to prevent this, access to these keywords have required an import of future.keywords. We now live in that future.

That was a problem for future me, and now I am future me.

OPA v1.0 makes import future.keywords a no-op, as all keywords are now made available everywhere. In the time before OPA 1.0 is released, the new rego.v1 should be used in place of future.keywords imports.

The if keyword made mandatory

The if keyword helps explain the "inverted if … then" nature of Rego rules, and makes rules easier to read. Additionally, any rule body with only a single expression, like:

allow {
"admin" in user.roles
}

May with the help of if be expressed as a one-liner, with the curly brackets removed:

allow if "admin" in user.roles

OPA 1.0 makes if a natural — and mandatory — part of every rule's anatomy.

Tip: Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your rules with if added to the rule head.

The contains keyword made mandatory

The contains keyword helps express multi-value (or as they're often called, partial) rules — i.e. rules that build a set of values. It also helps avoid ambiguities around certain classes of rules, like the fairly recently introduced "nested" rule type:

users.names contains name if {
# ...
name := sprintf("%s, %s", [first_name, last_name])
}

The contains keyword was already mandatory for nested rules. OPA v1.0 makes the use of this consistent across all multi-value rules by making contains a requirement.

Tip: Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your multi-value rules with contains added to the rule head.

Strict mode made (mostly) the default

Most of the rules that have existed in OPA strict mode will be made the default in OPA 1.0. This will help users catch mistakes early, and have them fixed right away. If you have been running opa check --strict as part of your policy build pipeline, you're already in the clear here.

The rules from strict mode that will be made default in OPA 1.0 are:

No duplicate imports

Duplicate imports should realistically not be a problem in any repo, and the fix is simply to have them removed.

package policy

import data.authz
import data.authz # this is now an error

No deprecated built-in functions

Deprecated built-in functions will be removed in OPA 1.0, and most of them are trivial to replace using a single line of Rego, or a different built-in function.

package policy

# simply change to use `true in {input.foo, input.bar}`
# using `in` additionally has the benefit that it can be used
# to check for any type of value, and not just boolean "true"
a := any([input.foo, input.bar])

# change to use `every` keyword, e.g.
# every x in [input.foo, input.bar] {
# x == true
# }
# just like `in` may be used for much more, `every` can be
# used to evaluate complex expressions
e := all([input.foo, input.bar])

# simply use the minus (`-`) operator instead, e.g.
# s3 := s1 - s2
s3 := set_diff(s1, s2)

# simply change to use regex.match instead
r := re_match(..)

# simply change to use net.cidr_intersects
n := net.cidr_overlap(..)

# cast_array, cast_set, cast_string, cast_boolean, cast_null, cast_object
# use the "is_x" equivalent built-in function in their place
s := is_string("yes")

input and data now reserved keywords

OPA 1.0 prohibits the use of input or data as identifiers:

# this is not allowed in OPA 1.0
input := "overloaded"

# and neither is this
data := {}

Overloading input has mostly been common in tests. Do note that with input as {..} remains valid. If you're using assignment to input however, (input := {..}), you'll just need to change the name to something like inp instead.

Other changes coming in OPA 1.0

See the 1.0 tag in the OPA backlog for a list of all issues related to OPA 1.0. Do note though that not all issues marked 1.0 might be picked for inclusion, and new issues may pop up before the release!

Other notable changes include:

import rego.v1

As previously mentioned — beginning with OPA v0.59.0 a new handy import to help with the 1.0 transition is made available. By adding import rego.v1 to a Rego policy, you can tell OPA to treat the policy just as it will handle it once version 1.0 is released.

  • Since rego.v1 implies all the (no longer) future keywords, the importing future.keywords is no longer needed when import rego.v1 is present, and will in fact be an error.
  • Just as in OPA 1.0, the use of if and contains will be enforced
  • The strict mode requirements brought in OPA 1.0 will be checked automatically

Bind server to localhost interface by default

In OPA 1.0, the server will bind to the localhost interface by default, and not 0.0.0.0 (all interfaces). This change is needed in order to avoid accidentally exposing OPA to the internet, which while uncommon (as OPA normally runs behind firewalls and gateways) still happens, and we should aim to provide a secure default. Should you still want to bind against 0.0.0.0, or some other interface, you can use the --addr flag of the opa run command, like opa run --server --addr 0.0.0.0:8181. The impact of this change is expected to be small, but good to keep in mind.

Documentation

The OPA docs have been updated to cover much of what's mentioned in this blog in greater detail. It also covers more of the technical reasons some of these changes are needed. See the docs on OPA 1.0 for more information.

How to prepare

As we covered in the TL;DR section, we're providing a number of tools to help with the transition starting from OPA v0.59.0 already. These tools will likely be extended and improved in following releases, but starting to use them today will ensure as smooth transition as possible.

To summarize

  • OPA 1.0 planned for release this year, including some backwards incompatible changes
  • Starting now, you should use import rego.v1 in all of your policies (this replaces future.keywords) imports
  • Use opa check --rego-v1 for testing compliance against 1.0
  • Use opa fmt --rego-v1 to have your Rego code updated for 1.0 compliance
  • Run the OPA server with the --v1-compatible flag for OPA 1.0 compliance

Also worth pointing out — following guides like the Rego Style Guide, and using tools like Regal, is an excellent way to ensure not just compliance with future changes to Rego, but that your current policy repo is continuously kept in the best possible condition.

If you have any questions, concerns or would like to provide feedback around the upcoming 1.0 release, or the tools made available to help you transition smoothly — don't hesitate to reach out using any of the below channels:

Open Policy Agent 2023, Year in Review

Banner image for Open Policy Agent 2023 year in review post

As 2023 draws to a close, the time has come to reflect on another important year for Open Policy Agent (OPA). Now more than two years deep into CNCF Graduated status, OPA continues to see accelerated growth in production deployments — and across a diverse range of use cases. Such use cases demand both performance and stability, while user base growth depends on learning resources and ease of use. This year, the OPA community has worked hard and delivered on all fronts, for new and experienced users alike. This post takes time to share how this was achieved; highlight prominent events and updates; celebrate input from the wider community and set the scene for a historic year of OPA in 2024.

OPA 'Away From Keyboard'

While OPA users and maintainers predominantly collaborate online, there were a good number of occasions where OPA existed very much in the physical realm this year too.

KubeCon EU enabled a few OPA events in Amsterdam early this summer. For the first time ever, an OPA-themed ContribFest session was held, where OPA, Conftest and OPA Gatekeeper maintainers worked with new contributors to the different OPA projects. In Amsterdam we also saw an OPA meet-up where speakers from Miro, Bankdata and Styra presented. At this KubeCon EU there were four OPA talks:

Contribfest session at KubeCon EU in Amsterdam

OPA Meetup in Amsterdam hosted by Miro

Rolling forward a few months, OPA also had a strong presence in Chicago at KubeCon NA. KubeCon is a huge event and it was great to get so many eyes on OPA as part of the graduated projects update in the keynote session. On top of that, the OPA kiosk in the project pavilion was an important meeting place for maintainers and users at the event. Discussions covered all sorts of use cases from authorization of applications, Kubernetes admission, IAC policy and beyond. Don't forget to check out the OPA project update from the conference's maintainer track.

OPA Update on the big stage

Open Policy Agent kiosk in the project pavilion

From Strength to Strength

OPA grows in so many different ways each year it's sometimes hard to know how to quantify it. Here are some highlighted figures which illustrate OPA's trajectory as we enter 2024.

2700 Contributors. Nearly 3000 people have helped make OPA into the project it is today. Contributors help make OPA better by making changes to docs and code; by participating in GitHub discussions and by filing bugs. What's equally impressive is how these contributors are from over 450 different companies. OPA is a general purpose, domain agnostic policy engine so it's vital the project is guided by such a varied contributor base.

9 years spent by users reading the documentation on the OPA website. This year OPA contributors worked hard and made over 160 updates to the docs; and so it's reassuring to look back at the end of the year and see just how many users benefited from the hard work.

2000 Go repositories build on OPA. Integrating with OPA has always been a priority so it's fantastic to see that just so many different projects are adding policy functionality in this way. With the OPA SDK, it's possible to bring all the best parts of OPA right into your Go application making it a powerful tool when standardizing your policy as code stack.

1.5 million Playground Runs. The Rego Playground is for every OPA user, it's there as a learning tool, as a collaborative scratch pad and now also integrates the output from Regal, the new linter for Rego. On average, every 20s someone clicks the 'Evaluate' button on the playground, all day long, all year long. One of the major uses of the playground is for users and maintainers collaborating on support in the OPA Slack, if you're interested in getting help within your team or on the Slack, creating a minimal example on the playground is a place to start.

It's not just OPA's community that's moving forward in leaps and bounds, OPA itself has been keeping pace and has received loads of great updates this year too. Let's dig into that now.

New Features

General references in rule heads

The single most important addition to Rego this year was arguably general references in rule heads. Simply put, it is now possible to include variables in rule names (or "references"), making it possible to build complex, nested map structures which would previously require multiple rules distributed over several packages.

Example using dynamic policy composition to collect informative notices from all "rules" policies, and have them organized by category and title.

grouped_notices[category][title] contains notice if {
some category, title
rules_to_run[category][title]

some notice in data.rules[category][title].notices
}

Output would be a nested structure, as expected:

{
"grouped_notices": {
"testing": {
"file-missing-test-suffix": ["ignored"]
},
"custom": {
"naming-convention": ["ignored"],
"one-liner-rule": ["obsolete", "ignored"]
}
}
}

For more examples and information, see the OPA docs on the topic.

Default keyword on functions

The default keyword has been around since forever, and is considered idiomatic for scenarios where a "fallback" value is needed, should rule evaluation fail in other rules sharing the same name. A long requested feature has been to extend support for default to cover custom functions, and 2023 was the year it happened.

package functions

default first_name(_) := "unknown"

first_name(full_name) := split(full_name, " ")[0]

New built-in functions

Seven new built-in functions were added to Rego this year. The json.verify_schema and json.match_schema functions are both recent additions for evaluating policy against JSON schemas — a use case that's been increasingly common in recent times. The time.format function will help policy authors present dates and time using either a custom format, or one of the supported constants for datetime formats that were also added this year. Three new crypto functions were added: crypto.hmac.equal, crypto.x509.parse_keypair, and crypto.parse_private_keys. Finally, the new numbers.range_step function, which works just as numbers.range, but with a configurable step value.

Package scoped annotations

A metadata annotation using the scope of package is now truly scoped to the package, and not just the file in which it is declared. This allows for some interesting opportunities to separate metadata declarations from "implementing" packages, and build things like lightweight frameworks leveraging Rego's metadata annotations. Additionally, it'll allow defining package scoped annotations for "packages" created via general references in rule heads, where the package scope isn't directly allowed on the rule itself.

Debugging

The Swiss army knife of OPA also known as "opa eval" got a new flag to help debugging policy this year. Using the --show-builtin-errors flag, policy authors may now get a list of all errors produced by built-in functions as part of evaluation, making it much faster to identify certain types of problems.

Performance

OPA keeps getting faster, and in 2023 we saw some great improvements in this area. The json.patch built-in function was remodeled entirely and now performs extremely well even when provided with a huge list of changes. Performance isn't entirely in the hands of OPA though. Some optimizations can only be performed at the level of an actual Rego policy, and OPA provides several tools to help policy authors with this. The profiler (opa eval --profile) is one such tool, and from this year it'll now also include the number of generated expressions in evaluation. This helps policy authors better understand why some expressions are evaluated more times than what one might expect.

Server

Several improvements to the server (and by extension, the OPA SDK) landed in OPA this year. Bundle fetching now works with AWS Signing Version 4A, allowing bundles hosted on AWS to be distributed across different geographical regions. Also, a new shorthand format for quickly running the server pointed at a remote bundle was introduced. The OCI downloader saw several new authentication methods added. Finally, instance labels may now be added via discovery allowing for greater flexibility in runtime re-configuration of long running OPAs.

Monitoring

Given the large number of OPA instances running in production, having a good story around monitoring is essential. The status API provides a way for any OPA deployed to report its current status to a centralized control plane or monitoring system. In 2023, several new metrics got added to the status reports, including most notably the request count for unauthorized calls to the OPA REST API when an authentication/authorization policy is in use, as well as errors that might have happened in decision logging.

Security

2023 was the year the OPA docker images finally made rootless the default, and the special "-rootless" images that previously existed for this purpose are now obsolete. If you're still using them, make sure to remove the suffix from the image on your next version upgrade!

Ecosystem

Gatekeeper

The OPA Gatekeeper project had a busy 2023, with many improvements landing this year. The external data feature allows users to connect with external data sources as part of policy evaluation. This year it gained support for caching of responses from external data providers for both audit and admission. A new AssignImage mutator which enables mutation of image registry or tag was also made available. The new PubSub feature (currently in alpha) enables users to subscribe to pubsub services to consume a large number of audit violations. Additionally, observability statistics for admission, audit and gator CLI are now available!

Speaking of the Gator CLI — the tool now prints violating object names on test output, and additionally supports trace and image flags. It may now also be provided an AdmissionReview object for verification.

Using the new (experimental) Kubernetes Native Validation feature, users can now write CEL (Common Expression Language) based rules in addition to Rego rules in constraint templates, similar to Kubernetes ValidatingAdmissionPolicy. Finally, the ExpansionTemplate feature, which enables validation of workload resources, has graduated to beta.

Conftest

The Conftest project saw many improvements around tooling this year. A new --strict flag was added to the verify and test commands, which will enforce additional safety checks on the policies such as unused arguments, duplicate imports, and more. Two more flags got added: the --quiet flag to the verify command which will silence success notifications and only show errors, and a --config flag which allows users to specify where the config file to be tested lives. Test results may now also be emitted in a format compatible with Azure DevOps. On the topic of formats, a new input format was added to the already long list of supported ones, and textproto files may now be targeted for policy evaluation too. Finally, the Confest Docker images now also support both the linux/amd64 and linux/arm64 platforms.

OPA Ecosystem

One goal of the OPA project is to build a domain agnostic policy engine. Being domain agnostic is achieved by simultaneously building generic core policy functionality, while also supporting a range of out-of-the-box integrations for different use cases. This year, the wider OPA community has wholeheartedly delivered on the latter and listed 22 new integrations on the website. The OPA Ecosystem also has a new home as a top level page, where integrations can be browsed by category, check it out!

New OPA Ecosystem Showcase

Most new ecosystem additions this year have been with other open source tools, generally adding policy functionality to a larger tool or leaning on Rego to provide a solid foundation for a domain-specific policy tool. Some notable examples include:

  • Source Code Management: Reposaur, a repository compliance tool; and Legitify, a repository security configuration scanner.
  • Supply Chain Security: dependency-management-data, helps understand software dependency posture; and Enterprise Contract verifies supply chain security artifacts with Rego policy.
  • Infrastructure CD checks: Torque, Spinnaker integrate Rego-based checks for continuous deployment while ccbr and ​​BrainIAC support a range of checks on existing IAC codebases.
  • Extending Authorization with OPA: The data orchestration tool Alluxio now also supports delegation of permissions to OPA.

Digger, an open source CI/CD orchestrator for Terraform both integrates OPA for user RBAC and leaning into existing tooling by leveraging OPA project conftest for IAC policy.

Meanwhile, other integrations went deeper and applied Rego in previously unexplored ways. regocpp is a cutting-edge project from collaborators at Microsoft that aims to bring Rego to other environments, natively. Based on C++, regocpp supports a number of Rego built-ins and the grammar as of v0.55.0.

The aforementioned linter, Regal also pushes the boundaries of where Rego can be used to write policies. Using the JSON representation of the Rego abstract syntax tree, this project implements a range of linting rules… in Rego! Regal has already been deployed by a number of open source Rego policy libraries and now supports over 60 rules. Integrated with the Rego Playground the linter is already available to everyone. There's no doubt that this will be a great tool for OPA learners and long-timers alike while continuing to help scale the OPA community.

If you're interested in listing your OPA integration or project, please see the instructions or stop by the #ecosystem channel in the OPA slack if you have any questions.

Thanks

2023 was an exciting year for OPA and its community. With so many projects using, integrating or extending OPA for all sorts of use cases — and so many users helping to contribute in all sorts of ways — this community is truly a great place to be. Thank you all who helped make it so! Your efforts are seen and appreciated.

There's a lot of great stuff lined up for next year already, so buckle up, and let's import future.2024!

Special thanks to Charlie Egan, Rita Zhang and John Reese for having helped contribute to this blog.

Open Policy Agent 2022, Year in Review

Banner image for OPA 2022 year in review blog post

It's a new year, and once again it's time to look back and reflect on the year that passed in the world of Open Policy Agent! 2022 was OPA's first full year as a CNCF graduated project, and while it would be easy to think that things would slow down after reaching that point of maturity and recognition, things have rather sped up. In community growth as well as pace of development — both of which we'll take a closer look at here.

This year, tech communities like ours enjoyed finally getting to meet in person again, and major conferences in our space, like KubeCon / CloudNativeCon, saw thousands of attendees in both Europe and North America. We also saw meetups, hackathons and other smaller gatherings move back from virtual to in-person events. On the topic of events — let's start our review of the year 2022 for Open Policy Agent there.

Notable Events

2022 User Survey

While we queried our nearest OPA instance for policy decisions, we also queried the OPA community to learn about their experience of interacting with the project, documentation and tooling. The survey results provided us valuable insights into where we might want to spend some extra time and effort in 2023 — richer documentation with more examples was requested by many. We're also seeing a need to better highlight the benefits of the various management capabilities in OPA, like decision logging and monitoring. An interesting trend which continues from 2022 is the increasing number of OPA use cases inside organizations. While infrastructure policies (including Kubernetes) still come out on top, we're seeing more organizations embrace OPA for policy across their whole stack. A standardized way of working with policy was always a goal of the project, so it's really exciting to see more organizations seeing the benefits of this approach!

Open Policy Day with OPA

This year we were excited to see an entire event dedicated to OPA — the Open Policy Day with OPA co-located with KubeCon North America. During the course of the day, attendees got to hear end-user stories on using OPA in production, with speakers from organizations like Nvidia, T-Mobile. Capital One, Chime and Snowflake. If you couldn't attend in person, all the talks are up on YouTube!

Conferences and Meetups

In the cloud-native space, OPA was represented in talks at both KubeCon / CloudNativeCon Europe as well as North America. As the interest in policy as code grew over the year, we saw a number of talks, workshops and events focused on the topic, and many discovered the benefits of unified policy management across tech stacks and organizations.

Additionally — while OPA has been, and continues to be, a popular topic at cloud-native, security and DevOps themed meetups, it's great to see new meetup groups dedicated entirely to the topic of OPA. Several OPA meetups took place in 2022, and we'd love to see that trend continue in the new year! If you'd like to host your own, let us know.

It was great to see so much buzz around OPA at these events, and much of that is thanks to the amazing work going on within the OPA projects — let's look into what's been going on there next!

New Features

OPA and Rego saw a record number of new features added in 2022.

New Keywords

A few new keywords made a big difference both to the aesthetics of Rego, as well as its functionality. First off, the new every keyword elegantly helps solve the problem of expressing "for all" type of queries:

import future.keywords.every

only_dev_servers {
some site in sites
site.name == "dev"

every server in site.servers {
endswith(server.name, "-dev")
}
}

Next, the new if keyword helps policy authors express their rules in the same way as they normally should be read — as conditional assignments: "allow is true if conditions x, y and z are true". As an added bonus, the if keyword allows skipping the braces around single-line rule bodies, leading to rule constructs that read more like plain English:

allow {
"admin" in input.user.roles
}

May now be written as:

allow if "admin" in input.user.roles

Similarly, the new contains keyword allows set-building partial rules to be expressed as "the set contains x if conditions x, y and z are true":

deny[message] {
input.user.security_clearance_level < 2
message := "Security clearance level 2 or higher required"
}

May now be written as:

deny contains message if {
input.user.security_clearance_level < 2
message := "Security clearance level 2 or higher required"
}

Refs in Rule Heads

It is said that one of the hardest things in computer science is naming things… "refs in rule heads" seems to support that claim! Don't let the name intimidate you though, this is a great addition to Rego! Dynamically creating deeply nested objects would previously often require the use of nested packages, with each package provided in its own file. This is no longer the case, as nesting can now be expressed in one place:

package policy

claims.user.name := concat(" ", [input.user.first_name, input.user.last_name])

request.method := input.request.method
request.valid := validate(input.request)

Evaluating the policy package will now provide something like the below result:

{
"claims": {
"user": {
"name": "John Doe"
}
},
"request": {
"method": "PUT",
"valid": true
}
}

Built-in Function Metadata and Metadata Introspection

Another exciting addition to Rego this year was the introduction of metadata annotations. Previously, policy authors would have to devise custom methods and formats for annotating their policies, packages and rules with metadata in the form of comments. Native support for annotations now provides a unified way not only for authoring structured annotations, but also for parsing them using either the "opa inspect" command, programmatically from Go, or even from Rego policies themselves via the new built-in rego.metadata functions. Having a standardized way of annotating packages and rules with metadata should benefit both human consumers as well as tooling.

Testing

A major advantage of treating policy as code is that code is testable. Unit testing provides effective guardrails around policies and rules, and allows frequent updates without the risk of breaking things. While test-driven development has been considered a best practice for Rego since the start, one feature that many found missing was the ability to mock functions. Function mocking allows replacing built-in functions, like http.send, or custom functions, with different implementations, commonly free of side-effects, all using the same with keyword familiar to test authors.

Policy and Data Distribution

Policy is commonly only half of the equation when OPA makes decisions — having access to up-to-date data related to users, endpoints or resources is often just as important. Data tends to be more dynamic in nature than policy, and certain types of deployments require a lot of data. The combination of huge datasets and frequently changing data would previously require continuous transfer of all required data in a bundle — even when only a few attributes had been updated. The introduction of delta bundles solves this by providing a new bundle type containing only the changes to data made since the last fetch, i.e. the delta. A much-awaited feature, and one that will help ensure OPA covers even the most complex of use cases going forward.

Another introduction this year was support for OCI bundle registries. In an increasingly containerized world, distribution of applications is no longer the only use case for containers, and providing policy and data using the same channels as you would for e.g. Docker images, simplify things considerably in many environments.

Disk Storage

While delta bundles may solve the problem of distributing large volumes of data, that data must eventually still be stored somewhere for OPA to make use of it. Up until this year, only a single option for storage was provided: in-memory. While this is normally the best place to store it for fast access, large datasets distributed in-memory across a large number of running instances quickly adds up, and many are those who learnt the hard way that the old slogan of "memory is cheap" isn't always true at scale. The disk based storage option now provides a balance between performance and costs, and is a welcome addition to OPA in many types of integrations.

Compiler Strict mode

The Topdown compiler in OPA was enhanced with a new strict mode option, allowing policy authors to catch common mistakes before deploying their policy to production. Unused imports and variables, or use of deprecated built-in functions — now all flagged by the compiler with strict mode turned on. Together with JSON schema-based type checking, developers are provided some powerful tools to ensure the robustness of their policy. Guard rails around the guard rails!

Intermediate Representation (IR)

The OPA project has the ambitious goal of standardizing policy across the full stack, because of this it's sometimes necessary to consider alternative deployment models to the traditional standalone service. This year we saw OPA provide a new intermediate representation format, allowing custom implementations to parse and execute evaluation plans. An implementation for the JVM and Javascript has already been made available with the Jarl project, and hopefully we'll see more to follow in 2023!

Built-in functions

20 new built-in functions got added in 2022 — more than any year before! We saw new functions in almost every existing category, and OPA's exciting new GraphQL capabilities create a category of their own.

GraphQL: graphql.is_valid, graphql.parse, graphql.parse_and_verify, graphql.parse_query, graphql.parse_schema, graphql.schema_is_valid

Strings: indexof_n, regex.replace, strings.any_prefix_match, strings.any_suffix_match

Objects: object.subset, object.union_n, object.keys

Crypto: crypto.hmac.md5, crypto.hmac.sha1, crypto.hmac.sha256, crypto.hmac.sha512

Misc: providers.aws.sign_req, net.cidr_is_valid, graph.reachable_paths

Performance Improvements

Some improvements are more understated, though no less noteworthy. As a mature software project, deployed in production in thousands of organizations across the world, OPA needs to be both robust and performant. This year saw the following exciting improvements in OPA performance:

  • The — optimize flag now works for more commands (previously only available for opa build)
  • Lazy objects optimization allows delaying evaluation of attributes until needed
  • Built-in function optimizations for object.get, in, object.union_n, and others
  • Two new highly optimized built-in functions for doing prefix and suffix matching en masse: strings.any_prefix_match and strings.any_suffix_match
  • Internal optimizations to set element addition, object insertion and set union.

Ecosystem and integrations

Gatekeeper

A whole lot of great things landed in the Gatekeeper project this year! Following recent developments, Gatekeeper was made compatible with the Kubernetes v1.25 shift from Pod Security Policies to Pod Security Admission. On the topic of Kubernetes workloads, the new Validation of Workload Resources feature allows writing rules that apply to any Pod spec, whether deployed as a standalone pod or embedded in a parent resource, like a Deployment. Similarly, Gatekeeper now also allows validating subresources. The external data feature, which, as the name implies, allows Gatekeeper to interface with various external data sources for validation and mutation, moved to beta this year, and the next feature to do so is Gator, which allows testing of Gatekeeper ConstraintTemplates and Constraints in a local environment. Finally, the mutation feature is now considered stable.

In addition to all the features listed above, Gatekeeper is now faster than ever before! Some of the most notable improvements include reduced time for template compilation, adding and evaluating constraints, a whopping ~20X reduction in persistent audit memory usage, reduced request duration for policies with replicated data and reduced CPU time when adding data to OPA storage.

The ecosystem around Gatekeeper also saw improvements this year, where the most notable ones were the new Gatekeeper Policies website, and the inclusion of Gatekeeper policies on ArtifactHub.

Conftest

Conftest saw a rapid pace of development this year, with 12 releases pushed — from version 0.29.0, and ending in version 0.37.0! The project added support for policy authoring using a number of additional file types — like env, hcl, jsonc, CycloneDX, and SPDX — as input. Possibly even more exciting is the addition of several new built-in functions exclusive to Conftest, like parse_config, parse_config_file, and parse_combined_config. These functions all allow policy authors to pull in configuration to test from inside of a policy, allowing a greater deal of flexibility in how config tests are executed.

The tooling around Conftest improved as well: when using the --version flag, the version of OPA used by Conftest will now also be displayed. Additionally, the new --quiet flag allows excluding anything but errors in the output, which should help in quickly identifying issues.

Integrations

OPA would not be what it is without its massive ecosystem of tools, integrations and useful and fun projects. The year started out with some great news in the infrastructure space, with AWS opening up for the possibility of externalizing compliance checks of CloudFormation templates via hooks, and it did not take long for the AWS CloudFormation hook for OPA to arrive on the scene. Later this year, Hashicorp announced support for OPA in their Terraform Cloud offering. A Pulumi integration was also added to the ecosystem. The message seems clear — the tool to use for infrastructure as code (IaC) compliance is OPA, and the language to define IaC policies is Rego!

Outside of the infrastructure space, we saw a number of interesting integrations being built by the community, like Alfred, a Rego Playground you can self host, a CircleCI integration for CI/CD pipeline policies, fig support for command line auto-completion goodness, self-sovereign identity (SSI) integrations, and even policy-driven access to remote systems via SansShell. OPA-powered policy enforcement even made it to the desktop this year, with the CISA-developed ScubaGear project using Rego for validating M365 tenant configurations!

Finally, a much awaited addition to the OPA ecosystem — the Rego Style Guide now offers policy authors a comprehensive set of rules and best practices for authoring Rego.

For a more comprehensive list of OPA integrations, check out the OPA ecosystem page, and the Awesome OPA list.

Credits

None of the above would have been made possible without the amazing community around OPA. In 2022, we saw an incredible number of people contribute to the project in all imaginable ways — code, documentation, bug reports, support discussions, integrations and tools. OPA is being used more and more widely around the world, and in different domains. With this growth, it'd be easy to overlook the huge effort in growing a community to support the project. We know how hard the community has worked to get to where we are and for that we are immensely grateful. Thank you all for getting us to where we are today and laying the foundation for another fantastic year with Open Policy Agent.

I have a plan! Exploring the OPA Intermediate Representation (IR) format

Diagram illustrating the OPA intermediate representation format

It isn't an overstatement to say that the versatility of Open Policy Agent (OPA) is a key factor in its success. As a general purpose policy engine, OPA needs to handle inputs from a disparate set of systems — Terraform, Kubernetes, CI/CD pipelines or custom applications, to name a few — and deliver decisions in a format understood by that particular system. Providing an agnostic approach to input and output data — anything that is or can be modeled hierarchically in a JSON or YAML document is potentially subject to policy — allows integrating OPA with all kinds of systems, applications and tech stacks.

But how does the data get passed between client application and OPA? Running OPA as a standalone service, and querying OPA for decisions over its REST API, is by far the most common way to integrate OPA, and for good reasons! Running OPA as a separate component provides a nice, unified interface for communication, which commonly involves writing only a few lines of code in most modern programming languages. Additionally, many technologies allow extending the built-in functionality for e.g. authorization by utilizing webhooks, which are commonly REST requests with a JSON-encoded payload. Perfect integration point for OPA!

There are however some scenarios where the standalone REST model proves to be challenging:

  • Resource-constrained environments like embedded systems. While OPA is fairly light-weight, some environments simply don't have the resources required to run a standalone OPA service, or networking capabilities for querying.
  • Distributed deployments with tight latency budgets, where every millisecond counts.
  • Environments constrained by other limitations on what type of software can be run, like web browsers.

To accommodate these requirements, OPA provides a few alternatives to the standalone service model:

  • Applications written in Go may integrate directly with the OPA Go API, or through the high-level Go SDK, alleviating the need for OPA to run as a separate service.
  • Policy may be compiled into Wasm modules, which can then be evaluated in any Wasm runtime. While the most famous Wasm runtime may be that included in web browsers, most programming languages today offer integrations with a Wasm runtime, allowing (at least the policy evaluation parts of) OPA to run "inside" of the application rather than outside of it. Additionally, OPA itself ships with a Wasm runtime, which makes it possible to have OPA pull down bundles including Wasm compiled policy for faster evaluation, and potentially other benefits.

Wasm

Rego policies compiled to Wasm modules offers a flexible, highly performant alternative to "regular" policy evaluation, with runtimes available for a wide array of languages, frameworks and platforms. As such, it should be considered an option for any OPA integration where the standalone server model falls short of the requirements. However, as ubiquitous as Wasm runtimes may be, they are not available everywhere. Embedded environments, exotic architectures or specialized hardware all constitute examples of environments where we're unlikely to encounter a Wasm runtime. But even with one available, Wasm itself is not without limitations, even by design!

With the goals of providing a safe, sandboxed environment, originally targeting web browsers, Wasm has several restrictions on what can and can't be done in the confines of the runtime. Interacting with the host system, or for that matter, other host systems — whether through system calls, network requests, or file system operations, is generally prohibited. While the WebAssembly System Interface (WASI) aims to offer an API for this exact purpose, and could potentially be used for certain features of Rego (like the http.send built-in function) in the future, relying on WASI means that a policy evaluated in one runtime might not work in another, as currently only a certain subset of the WASI API is implemented in any given runtime, and Wasm runtimes like those provided by web browsers likely have no interest in supporting interactions with the host system at all. Last, while some great progress has been made around WASI recently, it is still nowhere near the maturity of Wasm.

Intermediate Representation (IR)

OPA v0.37.0, released early 2022, brought two major enhancements to OPA: compiler strict mode and delta bundles. While those two features might have stolen the show of the release, the changelog additionally provides us with this:

The compile package and the opa build command support a new output format: "plan". It represents a query plan, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.

Interesting! Now, what does it mean? As alluded to in the last sentence, the low-level building blocks, or the evaluation plan, that eventually becomes Wasm, is now made available for consumption by other implementations. What would another implementation look like? That's up to you! While OPA may provide us with the low-level, step-by-step plan, for the evaluation of a query, it'll be on us to parse and evaluate that plan. Ever wanted to have your policy decisions served right inside of your Python app? Doable. Can't run the OPA server on your tiny microcontroller? You no longer need to. No runtime, no restrictions. What's the catch?

Bring Your Own OPA

Using your programming language of choice to implement the full set of instructions included in the intermediate representation format isn't something you'd pull off in just a few hours. A robust implementation is likely going to necessitate quite some effort, and even if you decide to invest the days — or possibly, weeks — required for a greenfield implementation, implementing the IR instructions is only half of the story. OPA provides an impressive number of built-in functions, requiring corresponding implementations in the platform you choose to target. You probably won't need every single built-in to accommodate your use case though, so starting with the ones known to be relevant for you is likely a smart idea. Rego modules compiled to Wasm, on the other hand, ship with native implementations of many of the built-ins. If you plan to build an IR compiler or evaluator in C or C++, leveraging those would give you a head start!

Another aspect to consider is the management features provided by OPA. Similarly to Wasm, the scope of the IR format is limited to policy evaluation. Fetching bundles from remote endpoints, sending decision logs, or providing metrics and status reports is left as an exercise to the implementation. However, just like with the built-in functions, you likely won't need to support the full set of management capabilities shipped with OPA, but can pick and choose the parts that make sense to you. More interestingly, you're free to implement your own management features. Rather than pulling bundles from an S3 bucket, why not stream your permissions data from a Kafka topic? Or build a direct integration against that tool your organization uses for health checks, and so on. As laborious as a custom implementation may be, it opens up for some very interesting opportunities!

We're getting ahead of ourselves though. Before we dash off to write our own, next generation, OPA implementation in whatever the hottest programming language is these days, we should probably start by getting familiar with the IR format, and how to make sense of evaluation plans.

Making Plans

Let's create a simple policy, and build a plan from that. The below policy contains two rules — is_admin to check if the "admin" role is included in the list of roles provided in the input for a user, and allow, which in this case simply is true if is_admin is true, but presumably would be extended to include more checks in future iterations of our policy.

package policy

import future.keywords.if

import future.keywords.in

allow if is_admin

is_admin if "admin" in input.user.roles

Simple enough, right? Let's see what a plan might look like! In order to build one, we'll use the aptly named opa build command. This command is used to build bundles, and the --target flag allows us to say that rather than just copying Rego and data files into the bundle, we want OPA to compile either a plan, or a wasm and put that in the bundle for us too. When building a plan, we'll additionally need to provide an entrypoint — this would be the path to either a package or a rule, from which the plan should be built. The path to the entry points (more than one is allowed) will later be used to query an implementation capable of parsing and evaluating our plan. Let's build a bundle with the plan target, and the entrypoint set to that of our allow rule:

opa build --target plan --entrypoint policy/allow .

This will create a bundle.tar.gz file in the current directory, with our plan inside of it. Since we're only interested in the plan for now, let's extract it from the bundle:

tar -zxvf bundle.tar.gz /plan.json

The plan.json file

We now have a plan to work with! Let's see what's in that plan.json file. The first thing you'll notice is that the plan file contains three top level attributes — static, plans and funcs. The static object is fairly straightforward:

{
"static": {
"strings": [{ "value": "result" }, { "value": "user" }, { "value": "roles" }, { "value": "admin" }],
"builtin_funcs": [
{
"name": "internal.member_2",
"decl": {
"args": [{ "type": "any" }, { "type": "any" }],
"result": { "type": "boolean" },
"type": "function"
}
}
],
"files": [{ "value": "policy.rego" }]
}
}

The strings array contains references to all strings included in the plan, and will be referenced whenever needed in evaluation. We'll recognize the "user", "roles" and "admin" strings from our policy, while the "result" string has been added by the plan builder, to be used as a key in the result set from plan evaluation. Thanks, plan builder! The builtin_funcs array provides a list of all the built-in functions used in our policy, along with the types expected for their arguments and return values. While "internal.member_2" might look unfamiliar, it's the internal name used for the built-in function representing the in operator used in our policy! Finally, the files array contains a list of all files used to build the plan, which in our case is only policy.rego.

The next attribute is the actual plans, and here's where things start to turn a bit cryptic. But don't worry, I'll walk you through it!

{
"plans": {
"plans": [
{
"name": "policy/allow",
"blocks": [
{
"stmts": [
{
"type": "CallStmt",
"stmt": {
"func": "g0.data.policy.allow",
"args": [{ "type": "local", "value": 0 }, { "type": "local", "value": 1 }],
"result": 2
}
},
{
"type": "AssignVarStmt",
"stmt": {
"source": { "type": "local", "value": 2 },
"target": 3
}
},
{
"type": "MakeObjectStmt",
"stmt": {
"target": 4
}
},
{
"type": "ObjectInsertStmt",
"stmt": {
"key": { "type": "string_index", "value": 0 },
"value": { "type": "local", "value": 3 },
"object": 4
}
},
{
"type": "ResultSetAddStmt",
"stmt": {
"value": 4
}
}
]
}
]
}
]
}
}

Functions, statements, blocks

For each entrypoint provided, we'll find a corresponding plan, which represents the planned evaluation path for that entrypoint. The name of the plan is our entrypoint ("policy/allow") and the blocks attribute contains the statements to be evaluated in order to "run" the plan. Quite literally in order too, as each statement block and statement will be evaluated in an entirely procedural fashion. Quite a contrast to the Rego code that produced it!

The first statement is a CallStmt, which means we'll need to evaluate the function (i.e. func) corresponding to the provided name in the funcs object — in our case this has been named "g0.data.policy.allow" (mapped from our allow rule) — and we'll take a closer look at the funcs part in a minute. The args provided to the function is "local" value 0, and "local" value 1. These represent the global input and data variables that you're likely familiar with from your Rego policies, and "value" in this case is rather the "name" — or pointer — to the values in the local scope. Who said naming was one of the hardest problems in computer science? Not the OPA plan compiler!

Where the Locals Go

Since you'll see a lot of references to local throughout compiled plans, learning how it's used is imperative to understanding the steps involved in plan execution. When a function is invoked, like our "g0.data.policy.allow" above, a local object is created to represent the inputs to that function. The statements that comprise the function may in turn both read from the local object, as well as write to it, effectively making it a bearer of local state.

If a statement inside of a function involves calling another function, a new, "inner" local object will be created for the scope of that function, and the result of the function evaluation will be stored in the "outer" local, and so on.

Back to our CallStmt! We now know that it'll be invoked with the input (local 0) and data (local 1) as its arguments. The next attribute in the statement simply says "return": 2, meaning that whatever value is returned by the function should be stored in the next position — i.e. 2 — in the local state. Next up, the AssignVarStmt is used to assign the value at position 2 — that's our return value — to a new local value at position 3 (as defined by the "target" attribute). Moving on, we'll see a MakeObjectStmt used to create an empty object, which is placed at local value 4. In the next step, a key value pair is inserted into the object (ObjectInsertStmt) where the key is the "string_index" at position 0. Remember the strings attribute from the static object from before? This is it. The first item in that array is "result", so it looks like we're building a result set over here! The value associated with the result is good ol' local value 3, which we might recall was the result of invoking the "g0.data.policy.allow" function. Finally ResultSetAddStmt signals that we're done here. We have a result from the plan, and we're now ready to return it.

Following Procedure

What about "g0.data.policy.allow" then? I promised we'd get back to the "funcs" object in a minute, and wow, time really flies when describing procedural instructions of an evaluation plan! A quick glance at the funcs object reveals that it contains not just the "allow" function, but also our "is_admin" function, here in the form of "g0.data.policy.is_admin". Since the allow function merely mirrors the result of is_admin, let's zoom in on the latter to learn how an implementation would evaluate the statements step by step, and how the state of the local object is updated in (almost) each step along the way. Rather than describing the steps in words, let's use a table for demonstration. In the left column you'll see a simplified version of each statement called, and in the right you'll see the local state after the statement has been applied. Note how each step procedurally builds up the final state, which is eventually returned to the caller. Beautiful, isn't it?

Evaluation plan statements and local state

Evaluation plan statements and local state. As Medium does not have embeddable tables, you may find the spreadsheet from the image above in this Google Sheets document.

To learn more about the different statements an evaluator implementation may encounter, consult the OPA docs on the topic.

Planning Ahead

Every aspect and instruction of the IR format would be too much for a single blog to cover, but the process of untangling an evaluation plan has hopefully been made clearer by now. While creating a full-fledged "OPA" native to your language or platform of choice might be a huge undertaking, even a basic implementation, with only a handful of built-in functions implemented, gets you surprisingly far towards something that actually feels usable. Open source implementations of course would have the benefit of others being able to contribute the parts that make the project usable to them. On the topic of open source implementations, are there any of those out there yet? In fact, there is!

Introducing Jarl

For the past few months, fellow OPA maintainer Johan Fylling and I have spent some of our spare time hacking away on an IR implementation for the Java Virtual Machine (JVM) called Jarl. A jarl was a chieftain in the age of vikings, and given the viking theme of OPA itself, we figured it would be a good name for the project. And of course, it has that "J" in there too, which seems almost mandatory for JVM-based software.

We chose to use Clojure for our implementation, and while we are both rather novice Clojure coders, it's been a lot of fun to work with! Not only that, but using Clojure means getting access to the broader JVM ecosystem, both in terms of libraries available, and that applications written in Java, Kotlin or Scala will be able to use Jarl eventually. As an added bonus, Clojurescript allows the library to be compiled into Javascript as well, allowing us to target deployments in Node, or web browsers. Quite a versatile platform to build on!

While we still have a long way to go before Jarl is anywhere near production readiness, we're already at a point where it's usable for evaluation of many types of common policies. As we wanted to ensure conformance against OPA from the start, we ported the OPA compliance test suite to the IR format, which has proven to be tremendously useful for testing not just plan evaluation, but also the behavior of built-in function implementations. This code should be useful for anyone building their own implementation, so if that's you, make sure to check it out. As for the built-in functions, we currently have most of them ported, but some work remains to be done before we're able to have all tests from OPA pass. Then awaits the management features…

If the project sounds interesting to you, we'd love to hear from you! Reach out on the OPA Slack, or just try it out and report back on any issues, feature requests or ideas you might have. Or if you'd rather work on your own implementation, we'd love to help you get started.

Wrapping Up

With the introduction of the intermediate representation format, another integration option for OPA has been made available. While it might be a bit of a niche — and certainly not the first choice to consider for most applications — it opens the door for many new and interesting use cases where an OPA integration using the standalone server model might not have been the best fit, or possible at all. Although perhaps not a well-known option until now, I hope this blog may contribute to changing that, and I'm looking forward to seeing how this alternative is leveraged in the future. Interesting times ahead!

Introducing the OPA print function

Banner image for introducing the OPA print function post

One of the key takeaways from the Open Policy Agent 2021 Survey, was the need to improve the OPA debugging experience. Simply put, we need to make it easier to know what's going on when policies and rules are evaluated.

However, whenever someone talks about an "experience," it's rarely a small task and a checkbox to be checked once completed. Rather, it's all the little things that when combined provide a great improvement to the greater goal. If the OPA project used JIRA, it would probably be a safe bet to classify the "improve debugging experience" story as an "epic." With some improvements made, many new ideas and feature requests are likely to emerge along the way, and it would be rather optimistic to think that such a story ever got done, in the sense that no new improvements could be made.

To make things more complicated — and certainly more interesting — the OPA debugging experience isn't isolated to OPA itself. Improving the debugging experience for OPA entails not just looking at where things can be made better in OPA, but just as much in the tools commonly used when authoring Rego policies. These include tools like VS Code, IntelliJ IDEA and all the other editors commonly used for policy authoring.

So, where do we start?

Debugging with OPA eval

Evaluating rules and variables has traditionally been done using the aptly named opa eval command. Commonly referred to as the "Swiss army knife of OPA," opa eval allows a policy author to quickly evaluate either standalone expressions like:

opa eval --format raw 1+1

Or, more commonly, with policy, data and input provided through command line arguments, and the path to the rule or variable of interest.

opa eval --data policy.rego --input input.json data.policy.main

While truly a versatile tool, debugging with opa eval has a couple of drawbacks. Having to create a new file to provide input might feel a little clunky, but hardly a terrible experience. But what if you want to evaluate the value of some variable inside of a rule, a test or a comprehension?

The trace built-in has to some extent been used for this purpose, but requires additional parameters passed to OPA to actually print something, and while occasionally useful, it was often perceived as somewhat clunky for the purpose of simply printing something.

Another problem frequently mentioned in the context of debugging is how sometimes opa eval, or the trace built-in comes back with just… nothing.

This brings us right into another topic often considered tricky with regards to debugging — OPA's handling of undefined. When OPA encounters undefined values, policy evaluation normally halts. Considering how Rego is a declarative query language, there isn't a whole lot more to do when a query comes back with nothing, just like a query language like SQL wouldn't have a whole lot more to do with an empty resultset. Since Rego rules are often compositions of many statements or other rules, it can sometimes be pretty difficult to tell where the undefined value that halted policy evaluation was introduced.

What to do?

Introducing the print built-in

To tackle this, OPA v0.34.0 introduces a new print function to its ever growing list of built-ins. The print function does exactly what you'd expect it to do — prints any provided values to the console. Consider a rule like the one below.

allow {
print("Entering allow")
role := input.user.roles[_]
print("Found role", role)
role == "admin"
}

Running OPA eval would produce the following result.

$ opa eval -f raw -d policy.rego -i input.json 'data.policy.allow'
Entering allow
Found role developer
Found role sysadmin
Found role dba
Found role admin
true

The print function takes any number of arguments (static values, variables, input, data, etc) and prints each one (separated by whitespace) to the console.

While simple on the surface, a whole lot of thought has been put into its design, and unlike other built-in functions (which are often trivial to add into OPA) the print function required changes to the internal compiler. How come?

Varargs

One of the design goals of the new print function was to allow a variable number of values or variables (i.e. varargs) to be passed as arguments, without resorting to the use of an array for the arguments, as is done by sprintf and other built-ins. Simply put we wanted something intuitive like:

print("x", input.x, "y", input.y)

To work just as expected. Sounds easy, right? Well, not really.

One of the more obscure (and hence, not encouraged) features of Rego can be traced back to its Datalog roots. Any built-in function can have it's return value expressed as the last argument to the function. Meaning that:

x := concat(".", ["a", "b", "c"])

Could alternatively be written as:

concat(".", ["a", "b", "c"], x)

With the last argument "reserved" for the return value, how would an implementation of varargs work? The answer was a new type of void function, where there simply is no return value to take into account. Since Rego functions are generally free from side effects, a void type of function hasn't really made sense previously, but with print having no purpose other than the desired side effect of printing to the console, adding a void type made sense.

Printing undefined

The next problem to tackle in order for print to work nicely as a debugging tool was how to deal with undefined. Since we can expect print to be used to debug variables from input and data that might not be defined, it would be kind of a bummer if calling print itself halted policy evaluation! Ideally we'd be able to call the print function and have it print something even if some of the arguments provided pointed at undefined values. That way we could use the function to try and help also with the problem of identifying where in a policy undefined values have been introduced.

This requirement meant some internal assumptions of how Rego is parsed had to change, and the end result is a print function that prints undefined values as <undefined>, without halting policy evaluation.

allow {
print(input.user.email, input.user.roles)
input.user.roles[_] == "admin"
endswith(input.user.email, "@acmecorp.com")
}

Evaluating the above allow rule with user.email missing from the input would now output something like this to the console:

$ opa eval -f raw -d policy.rego -i input.json data.policy.allow
<undefined> ["developer", "admin"]

Using print

OPA supports many different modes of operation, from opa eval and opa test, to the OPA REPL and of course running as a standalone server. Both opa eval and the REPL will always print to the console (stderr, specifically) as expected. When running as a server, OPA will print any output from print function calls at the info log level. This makes print useful for debugging at the default info level or below. When configured to run with log level error (the generally recommended log level for production), OPA erases any calls to print from policies as they are loaded. Print calls left in the policy at that point will thus not impact performance whatsoever.

When running opa test, the print function by default will print to the console on test failures. Should you want to print output also for successful tests, the — verbose (short form -v) will do the trick. One case I've found particularly useful in tests is to use print in combination with the with … as mocking construct, to quickly see what exactly the result of a rule evaluation returns, like:

test_decison_allowed {
result := decision with input as {
"user": {
"id": "abc123"
},
"request": {
"method": "POST",
"path": "/users"
}
}
print(result)
print(expectedResult)

result == expectedResult
}

decision {
[_, payload, _] := io.jwt.decode(input.user.token)
print(payload)
...
}

While printing the outcome of rule evaluation in tests like this is valuable, the decision rule itself might be composed of multiple rules, and being able to add a few print lines in those to understand why our result isn't what we expect can help us quickly pinpoint the problem.

Wrapping up

Rego as a policy language isn't a general purpose programming language, and shouldn't be treated as one. However, making the OPA and Rego debugging experience as smooth as possible means we sometimes might need to adapt concepts familiar from the programming languages policy authors normally work with. That the print function pull request added almost 2000 lines of code — half of them however from added test cases! — to the codebase is an interesting case study in how all the "little" details — like backwards compatibility, usability and performance — need to be considered when adding new functionality to a mature open source project like OPA.

I hope that you'll find the print function a useful addition to OPA, and a small improvement to the OPA/Rego debugging experience. Expect a lot more to come out in this space in future releases, and as always, make your voice heard in the OPA Slack if you have ideas, questions or feature requests you'd like to see incorporated into OPA!

Community Spotlight — Grant Shively

Portrait photo of Grant Shively, GoDaddy principal software engineer

"Community Spotlight" is a new series of blogs where we talk to people in and around the OPA community: users, integrators and contributors. Our first "spotlight" is Grant Shively, principal software engineer at GoDaddy.

Okay, so let's start with an introduction.

Sure! My name is Grant Shively and I've been working at GoDaddy for the last 13 years. For the last six years or so I've been a principal engineer on what we call our Care Platform, which is essentially our CRM system.

On the infrastructure side, we've moved from bare metal legacy servers with big monolithic ASP.NET web applications to a cloud-native, microservice architecture. Meanwhile we've transitioned much of the application platform towards Node.js and .NET Core. Recently, I've been doing a lot of work migrating things from our internal cloud to AWS.

For many years I've been involved in projects where we've needed complex authorization policies. And so we've built all these systems to deal with that, but there's never been a company-wide solution for that kind of thing.

I've been pushing for an authorization platform for the company for a few years, and last year it finally gained traction. We completed two company-wide initiatives with leaders from each of the major organizations where we all got together to talk about how we wanted to solve for authorization holistically at GoDaddy.

During the initiatives, we did a buy vs. build analysis where we evaluated different vendors, and open source projects. And that's when I discovered OPA. I watched the Netflix talk, which showed how they used OPA to solve many of the issues we are currently experiencing.

And so we did a proof of concept at the end of last year to prove the value and efficacy of the project using OPA, and we got buy-in on it, and now we're here, building out the final multi-tenant solution! We are hoping to have the next team running in production by the end of Q2, and we already have a number of teams lined up to onboard after that.

That was a great introduction! So, OPA adoption starting from a platform team, and growing from there?

I've definitely evangelized OPA within the company, since I'm a big fan of the project. There's someone from another team helping to contribute to this platform we're building so I guess that makes us two teams at the moment. My team is focused on the systems that support our customer service guides, while the other team is working on our domain control center. Initially, they're looking to implement this authorization platform for some of the high risk scenarios we have around domains, such as transfer of ownership and things like that.

You said you were working mainly with .NET and Node.js?

Yep. As a company we work in a number of languages, including Python, Go, Node.js, .NET Core, and Java… I think those are most of the blessed languages, but there are probably some other languages used for one-offs as well.

My team, we were traditionally C#, and then we brought in Node.js during our transition to microservices. With the arrival of .NET Core, we were comfortable with continuing to support C# as well. Personally, after working with both languages for a number of years, I prefer Node.js for implementing our small, single-purpose APIs.

You've made quite a few contributions to OPA, which is written in Go. Did you have experience working with that before?

No, I hadn't touched it before! Go is definitely approachable. And I think the OPA code base is very clean and logical to explore. You don't have to look too much to figure out where something's at, once you get used to it.

Go feels like a language that minimizes syntactic sugar. There's really only a couple of "right" ways to do things, I feel. Sometimes it's frustrating too, like with the lack of map/filter/reduce operations. Coming from more functional paradigms it feels frustrating at times.

Agreed. In this process of finding OPA, what other alternatives did you consider? Building your own? Some commercial options?

We looked at Athenz. One of our senior architects had worked on that project and it looked interesting. It did some of the stuff we wanted, but not all. And then we looked at a whole bunch of products from various vendors. We briefly thought about building our own until we ran into OPA. OPA completely negated any reason for us to try and build our own because it did exactly what we needed, and in such an elegant way that it would be very difficult for us to replicate.

Compared to the other options you considered, what was the main appeal of OPA? What was it that won you over?

One of the things we needed was the ability for multiple engineering teams to work with policy, in a self-service manner, while at the same time leveraging globally-managed policy and data, like authentication token expiration rules and identity attributes.

Like everybody else, we have a few home-grown systems, and we definitely needed to be able to integrate with those. Also, we have an extremely distributed architecture. Hundreds of AWS accounts, and whatever solution we found was going to need to run in pretty much all of them.

I knew that there were going to be integration points that would be very difficult for most vendor products. One thing we really didn't want was some sort of centralized authorization API for the whole company. We've been bitten by such architectures in the past. Our apps are highly distributed, and we need our authorization decision points to be highly distributed too, while still being able to manage and distribute policy from a central location. And a lot of the bigger vendors had these products that just felt a little too heavy for large-scale decentralization, you know?.

Right.

And we really liked Rego. We've talked about this before, but a lot of the vendors are based on XACML, which is just really tedious to work with. Compared to that, Rego was a breath of fresh air. Another factor we considered was the buy-in we saw from some of the bigger companies like Netflix, etc. Rapid adoption of a project is usually a great sign.

I'm sure I'll have more opinions on Rego once we try and onboard more teams onto our platform. Certain teams at GoDaddy have more operational experience than others, and we'd really like to build a simplified UI policy builder on top of our platform and Rego. That way, they can just fill out some simple input boxes, hit enter, and it publishes a policy for them. That kind of thing.

Interesting!

Yeah, I'm both looking and not looking forward to tackling that problem, haha!

Maybe you don't remember anymore, but did you have any such moments where you got stuck while learning Rego? I know one thing that felt a little odd to me when starting out was how Rego handled undefined values. How rule evaluation just stops when they are encountered.

Some of the syntax around iteration was a little confusing at first. I think it's one of those things that makes total sense once you understand it, but it wasn't like anything I'd worked with before.

The biggest issue though — and we still kind of have it—is understanding how a decision was made, and how to surface that in logs. We have a couple of open issues about that, and we have been running OPA internally with a few patches we want to upstream that help address some of this.

It's really important to our security and engineering teams that we can look at a somewhat human readable, but hopefully not super verbose, way of saying that "this authorization request was approved or denied because of these reasons".

Another related issue would be how to best propagate obligations up through layers of policy. Some obligations can come from a really low level, like policy checking the claims of a JWT, and if some property doesn't exist, then we want to propagate an obligation up from that lower level rule. Solving some of those problems has felt a little clunky.

On the other side, what did you really like?

I really liked that you could create custom built-in functions and other things to extend OPAs functionality. That has been really useful to us as we've built things around our requirements, like custom key signing, custom decision logging, and so on.

There have been a few cases where we needed to extend portions of the OPA code, only to find that they were hardcoded around a particular implementation, without interfaces for us to leverage. I've been very pleased with the responsiveness of and guidance from the OPA maintainers. We've been able to contribute a number of small changes to make the OPA code more flexible for our extensions.

On the topic of flexibility, one of the things that really appealed to me was the number of integrations available. Even if you're never going to use all of them, it really shows what the general purpose nature of OPA enables.

Yeah, that brings to mind another consideration. Netflix talked about how they used OPA for authorization across their entire stack—infrastructure, forward facing code, back end. When I saw OPA, I knew our platform could grow to be much more than just, you know, a simple authorization engine—there are many other policy-based scenarios that don't necessarily have to do with authorization. So that may be a far future thing, but that was also very enticing about OPA.

I think some of the flexibility really pays off at scale. Having one unified language to describe policy across the stack, having one place to go for decision logs, and so on.

Yep. We're definitely looking at increasing the number of integrations. So many of the systems deployed today work only with Active Directory, and AD group-based authorization, so it'll be interesting to see how we can integrate with systems like that.

About integrations. Are there any of the existing ones in the ecosystem you are using or are planning to use?

I'm only familiar with the Envoy one. Oh, and Kubernetes. What else is there? We're just starting to use Envoy, so I think there will likely be some places where we use that integration. And, we'll probably want to look into a Kubernetes integration too at some point, if AWS EKS supports that.

Looking at the list now… Wow, there's really a bunch of them here! I'll need to look into some of these. What we're really doing a lot of is AWS integrations. Kind of in the same vein as the Envoy integration, extending OPA and all that, you know? We're doing the same thing with OPA and AWS, and I hope we'll be able to eventually open source that.

Things like key signing using the AWS Key Management Service (KMS) or shipping decision logs directly to AWS Kinesis without having to go through HTTP endpoints in between.

I know you've mentioned AWS Lambda functions too.

Yeah. A lot of our APIs are lambda-based, and we'll obviously want to use this platform we're building for that too. We're currently trying to figure out how to make OPA work as well for on-demand serverless functions as it does for traditional compute environments.

Definitely a hot topic! I've seen some questions on that on the OPA Slack as well. What are the challenges there?

So, the way OPA currently works is that many of the internal "plugins", like the client downloading bundles, or the one uploading decision logs-they all work on a time-based loop. So you configure them to upload or download or to do whatever they are meant to do at a certain interval, like every thirty seconds or something.

This doesn't really work for serverless functions though as they don't have continuous compute available; functions freeze after serving a response, so nothing can run in the background. What we'd need in this context is rather something based on other types of triggers. We're waiting for a feature called AWS Lambda Extensions, which should reach general availability in May. They should make it possible to use OPA really efficiently even in a lambda context.

Anything you have found missing from OPA or would like to see on the roadmap?

I mentioned it before, but yeah, the big thing would be figuring out how to do the tracing in a more efficient way. I have come to understand that the way "full" tracing is done today is apparently pretty expensive. We just need something like rule level tracing, which has been discussed a bit in GitHub issues in the past. With that in OPA proper we wouldn't have a reason to run our own modifications at all, except for plugins.

Awesome! Finally, what are your future plans for OPA at GoDaddy?

What we're starting out with is authorization for internal systems. Once we nail that and it's working well we're looking to expand it to authorization for customer systems, and at some point possibly our infrastructure too.

With what OPA can do, I feel like the sky's the limit in terms of what you want to throw into it. We just have to make sure we fully support the platform we're building as we continue growing. I have high hopes for it!

Open Policy Agent 2020, year in review

Open Policy Agent 2020 year in review banner image

Introduction

To call 2020 an eventful year would be an understatement. With a pandemic raging globally, we saw the tech community quickly adapt by moving largely online. Working from home became the new norm as meetings, conferences and other events also moved to digital channels. With so much of our lives now taking place online, the importance of our online platforms became overwhelmingly apparent. As businesses and organizations worked hard to accommodate an ever increasing number of users, both interest and investments in securing their platforms followed.

2020 saw a big uptake in the adoption of OPA. As the de facto standard for cloud-native authorization, OPA has also found its way into many new and interesting domains. The list of integrations has been constantly growing during the year, and many more are being worked on. Hundreds of new open source projects, businesses and organizations have come to rely on OPA as their open source unified policy framework across the stack. And, with the increasing adoption of OPA, we've seen the community grow with it — from the number of contributors to active users on Slack. The trajectory for next year has been set, and it truly looks great — for OPA and its community. Before we look ahead though, let's take some time to look back at what we achieved in 2020.

OPA 2020 in numbers

  • ~29 million downloads
  • ~1500 new Github stars
  • ~1700 new Slack users, doubled in size!
  • ~750 commits
  • 90 contributors
  • 9 major versions released, 17 point releases
  • 100th release pushed!
  • 1000th issue closed!
  • 10000 VS Code OPA plug-in installs!

Notable features and enhancements

Management

Several management capabilities were added. The bundle signing feature allows verifying the integrity of any bundle processed by OPA. Moreover, the new bundle persistence feature enables OPA to store a local copy of bundles received, which may be used in case the server is forced to restart while the configured bundle endpoints are unreachable. To authenticate at remote endpoints, several new credential providers were added, including:

For example, the following configuration instructs OPA to use the GCP Metadata Server to obtain credentials for downloading bundles from a remote endpoint (which happens to be hosted on Cloud Run):

services:
cloudrun:
url: <BUNDLE_SERVICE_URL>
response_header_timeout_seconds: 5
credentials:
gcp_metadata:
audience: <BUNDLE_SERVICE_URL>
bundles:
authz:
service: cloudrun
resource: bundles/http/example/authz.tar.gz
polling:
min_delay_seconds: 60
max_delay_seconds: 120

Lastly, the decision logger now supports mutating masks so that administrators can inject or overwrite (as opposed to just removing) fields inside decision logs.

Tooling

The OPA binary had several new flags and features added. The most notable additions are probably the remodeled opa build command for creating bundles and the new opa bench/opa test --bench for benchmarking policy evaluation.

WebAssembly (Wasm)

Many improvements for Wasm were incorporated this year. Among the bigger ones we saw the opa build command get support for building Wasm modules from Rego policies, and the addition of an SDK for Javascript. Plenty of built-ins were implemented natively, and the status for Wasm support is now listed for each on the policy reference page.

Performance

In the performance department we saw many improvements. The comprehension indexing, which allows O(n) runtime complexity on "group-by" operations, is probably the one to stand out the most. Less visible, but of no less importance, a new parser was introduced, improving the internals of OPA and resulting in a 100x speedup on most .rego files!

Rego & Built-in Functions

Built-in functions error handling was remodeled to allow policy authors to gracefully deal with errors like garbled input, which would previously halt policy evaluation in certain cases.

The new caching options for http.send introduced this year vastly increases its utility, allowing it to be used for things like OAuth2 and OpenID Connect metadata retrieval. The raise_error option allows policy authors to deal with errors in communication rather than having policy evaluation halt. Given the number of improvements made to the HTTP client this year — and the number of policies in where it is used — it makes the previous "experimental" label it carried less relevant, and the feature should be considered stable moving forward.

More than 30 new built-ins were added to OPA in 2020. These include functions for:

  • JSON manipulation: json.patch, json.remove.
  • Bitwise operations: bits.or, bits.and, bits.negate, bits.xor, bits.lsh, bits.rsh.
  • Hex encoding: hex.encode, hex.decode.
  • Hashing: crypto.md5, crypto.sha1, crypto.sha256.
  • And a whole bunch of functions for validating various formats: json.is_valid, yaml.is_valid, base64.is_valid, regex.is_valid, semver.is_valid.
  • Other useful built-ins added include graph.reachable, object.get, numbers.range and semver.compare.

The wide variety of these help push OPA as a true general purpose policy engine, further increasing the number of domains where Rego may be used. In addition to all the above, hundreds of bugs were fixed and tons of improvements to documentation were made. It's been an incredible year for the project.

Ecosystem and integrations

Gatekeeper

The Gatekeeper project took some great leaps forward this year. In terms of new features this meant, among other things, granular namespace exclusions (narrowing the scope of resources to present for audit, webhooks and sync), Helm 3 support and the Gatekeeper Pod Security Policies being referenced as a serious contender to the Kubernetes provided PSPs.

In terms of stability, Gatekeeper gained support for multi-pod deployments, completed a CNCF security review and had their first stable non-beta release pushed. Other notable enhancements include semantic logging (getting cluster wide resources in violation of policy from logs), standalone auditing and the dependency on finalizers removed.

Still in early alpha, we're seeing much anticipated support for mutating webhooks. Surely one thing to look out for in 2021!

If you haven't already, make sure to check out the gatekeeper-library repo, which was moved out from the Gatekeeper repository this year and has seen continuous additions and improvements since then.

Conftest

2020 was an exciting year in the development of the Conftest project. Having previously been maintained independently, this year we saw the project included in the larger OPA family of projects.

As for features, the already versatile tool gained support for a number of new input formats (VCL, XML, EDN, TOML, HOCON, Jsonnet) and a new output format (JUnit). Using conftest for Terraform policies became a common use case, and many improvements for Terraform and the HCL2 format got implemented this year. A new system for native plug-ins was added, as was support for loading data files alongside policies. Keeping conftest up to date on Linux was made easier than ever with both RPM and deb packages being made available. A new documentation site was launched at https://www.conftest.dev/

OPA Envoy plugin

The OPA Envoy plugin now also supports the v3 transport API, as well as decoding gRPC payloads. Oh, and the project changed its name too — while Istio is very much still supported it is not limited to that implementation, and OPA Envoy plugin was deemed a better name than the OPA Istio plugin.

IntelliJ IDEA plugin

Of all new integrations and projects worked on in the larger OPA ecosystem, the OPA plug-in for IntelliJ IDEA was probably the most anticipated one. After some time in development, a first release was announced this fall and proved to be well worth waiting for. Not only does it cover the necessities, like syntax highlighting, but integrates features for both evaluating rules as well as for running tests right from inside of the policy editor. If you haven't already, make sure to check it out in the IntelliJ plug-in marketplace or at the project GitHub page.

OPA Plugin for IntelliJ IDEA

Credits

To all who engaged with or contributed to OPA and its ecosystem in 2020 — whether as maintainers, contributors, integrators, adopters, users, or by helping others on Slack: You have all contributed not only to OPA but just as much to making this community be such a fun and rewarding place to be. Who knows what 2021 has in store for us? One thing however is for certain — with all the knowledge, skills and creative energy found in the OPA community, it will be no less eventful.

Thank you!