If you run into problems getting OPA to enforce admission control policies in Kubernetes there are a few things you can check to make sure everything is configured correctly. If none of these tips work, feel free to join slack.openpolicyagent.org and ask for help.
The tips below cover the OPA-Kubernetes integration that uses kube-mgmt. The OPA Gatekeeper version has its own docs.
Check for the openpolicyagent.org/policy-status
annotation on ConfigMaps containing policies
If you are loading policies into OPA via
kube-mgmt you can check the
openpolicyagent.org/policy-status
annotation on ConfigMaps that contain your
policies. The annotation should be set to "ok"
if the policy was loaded
successfully. If errors occurred during loading (e.g., because the policy
contained a syntax error) the cause will be reported here.
If the annotation is
missing entirely, check the kube-mgmt
container logs for connection errors
between the container and the Kubernetes API server.
Check the kube-mgmt
container logs for error messages
When kube-mgmt
is healthy, the container logs will be quiet/empty. If you are
trying to enforce policies based on Kubernetes context (e.g., to check for
ingress conflicts) then you need to make sure that kube-mgmt
can replicate
Kubernetes objects into OPA. If kube-mgmt
is unable to list/watch resources in
the Kubernetes API server, they will not be replicated into OPA and the policy
will not get enforced.
Check the opa
container logs for TLS errors
Communication between the Kubernetes API server and OPA is secured with TLS. If the CA bundle specified in the webhook configuration is out-of-sync with the server certificate that OPA is configured with, OPA will log errors indicating a TLS issue. Verify that the CA bundle specified in the validating or mutating webhook configurations matches the server certificate you configured OPA to use.
Check for POST requests in the opa
container logs
When the Kubernetes API server queries OPA for admission control decisions, it
sends HTTP POST
requests. If there are no POST
requests contained in the
opa
container logs, it indicates that the webhook configuration is wrong or
there is a network connectivity problem between the Kubernetes API server and
OPA.
- If you have access to the Kubernetes API server logs, review them to see if they indicate the cause.
- If you are running on AWS EKS make sure your security group settings allow traffic from Kubernetes “master” nodes to the node(s) where OPA is running.
Ensure the webhook is configured for the proper namespaces
When you create the webhook according to the installation instructions, it includes a namespaceSelector so that you can decide which namespaces to ignore.
namespaceSelector:
matchExpressions:
- key: openpolicyagent.org/webhook
operator: NotIn
values:
- ignore
If OPA seems to not be making the decisions you expect, check if the namespace
is using the label openpolicyagent.org/webhook: ignore
.
If OPA is making decision on namespaces (like kube-system
) that you would
prefer OPA would ignore, assign the namespace the label
openpolicyagent.org/webhook: ignore
.
Ensure mutating policies construct JSON Patches correctly
If you are using OPA to enforce mutating admission policies you must ensure the
JSON Patch objects you generate escape “/” characters in the JSON Pointer. For
example, if you are generating a JSON Patch that sets annotations like
acmecorp.com/myannotation
you need to escape the “/” character in the
annotation name using ~1
(per RFC
6901).
Correct:
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
Incorrect:
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com/myannotation",
"value": "somevalue"
}
In addition, when your policy generates the response for the Kubernetes API
server, you must use the base64.encode
built-in function to encode the JSON
Patch objects. DO NOT use the base64url.encode
function because the Kubernetes
API server will not process it:
Correct:
xxxxxxxxxx
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
response := {
"allowed": true,
"patchType": "JSONPatch",
"patch": base64.encode(json.marshal(patches)) # <-- GOOD: uses base64.encode
}
patches := [
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
]
Incorrect:
xxxxxxxxxx
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
response := {
"allowed": true,
"patchType": "JSONPatch",
"patch": base64url.encode(json.marshal(patches)) # <-- BAD: uses base64url.encode
}
patches := [
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
]
Also, for more examples of how to construct mutating policies and integrating them with validating policies, see these examples in https://github.com/open-policy-agent/library.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.