Upgrade guide
0.27.0
Breaking
Deprecated JWT endpoint
The /.pomerium/jwt endpoint is now deprecated and disabled by default. (To temporarily opt out of this deprecation, set the runtime flag pomerium_jwt_endpoint
to true. This flag will be removed in a future release.)
This endpoint was originally added for single-page web apps to get information about the currently signed-in user, but for this use case it is not necessary to receive this information as a signed JWT.
Furthermore, this endpoint is incompatible with the desired security properties for the Pomerium JWT. We intend for the Pomerium JWT to represent that a specific request to an upstream service was duly authorized by Pomerium. The JWTs issued by the /.pomerium/jwt endpoint do not satisfy this property.
There is a new /.pomerium/user endpoint to provide the same user data, but as a plaintext JSON response. If you are using the Pomerium JavaScript SDK, version 1.1.0 includes a new getBrowserUser()
method to replace the existing verifyBrowserUser()
method.
Upgrading Pomerium Zero deployments in Kubernetes
For Pomerium Zero deployments in Kubernetes, we updated the Kubernetes manifest to use a Deployment instead of a StatefulSet. Before you upgrade, you need to delete your existing StatefulSet with a command like:
kubectl delete statefulset/pomerium -n pomerium-zero
Once you’ve removed your StatefulSet, run the following command to update Pomerium in Kubernetes:
kubectl apply -k github.com/pomerium/pomerium/k8s/zero
0.26.0
Routes port matching
Pomerium’s route matching behavior has changed with regards to port numbers in incoming requests. Previously, when matching an incoming request against the defined routes, Pomerium would require that the request’s Host
(or :authority
) header match the route’s from
URL including any port number. This can cause problems in deployments with a NAT with port mapping in front of Pomerium.
As of v0.26, if a Pomerium route’s from
URL does not include an explicit port number, the matching behavior is more lenient: the route will match an incoming request with any port number. For example, take a route with the from
URL https://app.example.com
. Incoming requests with a host header of app.example.com
, app.example.com:443
, and app.example.com:1234
would all match this route.
However, if you specify a port number explicitly in the from
URL, then incoming requests must include the same port number in the host header in order to match that route.
You can temporarily revert this change in behavior by setting the runtime flag match_any_incoming_port
to false.
Host header rewrite behavior
Pomerium will now consistently rewrite the host header of an incoming request to match the host and port specified in the route to
URL. (Previously Pomerium would never include a port number even if specified, and Pomerium would not rewrite the host header for any to
URLs with a host of localhost
or an IP address.) The new behavior is intended to be more consistent and predictable.
Please set the Preserve Host Header option for any routes where Pomerium should not rewrite the host header.
Improved session refresh reliability
We’ve updated the way Pomerium refreshes OAuth access tokens in order to improve reliability. Previously, Pomerium could fall behind on access token refresh, leading to users being prompted to sign in again before their Pomerium session should have expired. This may result in a higher rate of requests to your configured identity provider.
If you suspect this is causing any problems for your deployment, you can temporarily revert to the previous implementation by setting the runtime flag legacy_identity_manager
to true.
Deprecations
Support for the deprecated client_ca
config file key (and CLIENT_CA
environment variable) is now removed. Please update any remaining usage to downstream_mtls.ca
or the DOWNSTREAM_MTLS_CA
environment variable. See https://www.pomerium.com/docs/reference/downstream-mtls-settings#ca for more information about this option.
0.25.0
Breaking
Base64-encoded Certificates
Previously, the certificates
key supported base64-encoded certificates as a value (this option was not documented). We've removed support for base64-encoded certificates for this setting and now require that you only use the certificate file location. To avoid incompatibilities between versions, please update these values accordingly.
Note: The certificates
key is distinct from the certificate
key. The certificate
key setting still supports base64-encoded certificates; the certificates
list does not.
Remove Debug Option
We've removed support for the Debug setting, which changed the format of logs from JSON to a pretty-print format. If you prefer to review logs in a pretty-print format, you can use a command-line processing tool like jq
.
New
Authentication Flows
In v0.21, we modified the Core authentication flow to support the Hosted Authenticate service. However, this flow posed some limitations for self-hosted deployments (see #4819 for more details). In v0.25, we’ve updated the Core authentication flow so that it’s more versatile: Deployments configured to use the Hosted Authenticate service will use the newer authentication flow introduced in v0.21, while self-hosted deployments will use the older flow.
0.24.0
Breaking
Set Authorization Header
The deprecated set_authorization_header
configuration setting has been removed. You can use the Set Request Headers setting to pass IdP tokens to upstream services in any header.
Base64-encoded Certificates
Previously, the certificates
key supported base64-encoded certificates as a value (this option was not documented). We've removed support for base64-encoded certificates for this setting and now require that you only use the relative file location.
Note: The certificates
key is distinct from the certificate
key. The certificate
key setting still supports base64-encoded certificates; the certificates
list does not.
See Certificates for more information.
Redis Storage Backend
PostgreSQL has been the recommended databroker storage backend since v0.18. Support for Redis has now been removed.
See Persistence for more information.
Performance
v0.24.0 includes several performance enhancements for Pomerium Core. See the sections below for more information.
Policy evaluator reuse
The authorize service maintains one policy evaluator object for each route, which is responsible for all authorization decisions for that route.
Previously, the authorize service would recreate all policy evaluator objects in response to a configuration change.
Now, it will avoid recreating policy evaluator objects for any policies that have not changed, provided that the overall settings are compatible between the old and new configuration. (Some settings affect all routes, e.g. the downstream_mtls options, and changes to these options will still require all policy evaluators to be recreated.)
Parallelization
Core will now build route configuration objects and policy evaluator objects in parallel, each using up to half the number of available CPU cores.
0.23.0
New
Logging Configurations
The new Access Log Fields and Authorize Log Fields settings allow you to customize the fields logged in the access and authorize logs. You can now opt to log ID tokens or specific ID token claims, custom request headers, and the request query params. See the linked reference pages for details.
How the new Logging Configurations work
1. Configure logs
Specify which fields you want to log (omitting the setting will display all the default fields):
# Access logs from Proxy service
access_log_fields:
- authority
- path
# Authorize logs from Authorize service
authorize_log_fields:
- request-id
- method
- path
2. Access a route
For example, Pomerium’s Verify service:
routes:
- from: https://verify.localhost.pomerium.io
to: http://verify:8000
3. Find logs
After you access a route, filter your logs by searching for “http-request”
and “authorize check”
:
// Search for “message”: “http-request”
{
"level": "info",
"service": "envoy",
"authority": "verify.pomerium.com",
"path": "/img/json.svg",
"time": "2023-08-04T12:12:35-04:00",
"message": "http-request"
}
// Search for “message”: “authorize check”
{
"level": "info",
"service": "authorize",
"request-id": "c9afae5a-ec5a-4242-864f-df4189f20e99",
"method": "GET",
"path": "/index.css",
"allow": true,
"allow-why-true": ["domain-ok"],
"deny": false,
"deny-why-false": [],
"time": "2023-08-07T10:26:33-04:00",
"message": "authorize check"
}
New Downstream mTLS Settings
Downstream mTLS refers to the requirement that end users connecting to Pomerium-managed routes must present a trusted client certificate. The options for configuring downstream mTLS have been expanded and moved to a new settings group, under a new downstream_mtls
configuration file key.
The existing Certificate Authority setting has moved from client_ca
to downstream_mtls.ca
. The client_ca
configuration file key will continue to function as an alias for the new setting (but will be removed in a future release).
Support for certificate revocation via CRLs is newly introduced. Please see the CRL reference for more information and some important limitations.
A new Enforcement Mode option has been added, to control the behavior when a client does not present a trusted certificate. The default setting preserves the behavior of previous Pomerium releases: client certificates are required only for user-configured Pomerium routes, and Pomerium will serve an HTML error page for requests without a trusted certificate. The new reject_connection
setting allows for stricter client certificate enforcement: in this mode any attempt to make a TLS connection without a trusted client certificate will be rejected. Naturally, this means that client certificates will be required not only for user-configured Pomerium routes, but also for internal Pomerium routes. This mode allows you to use mTLS as an isolated security layer, entirely separate from Pomerium policy enforcement. Please review the reference page carefully before enabling this mode.
To give further control over which specific client certificates are allowed, Pomerium now also offers a Match Subject Alt Names setting. This allows you to trust only those client certificates containing a Subject Alternative Name (SAN) of a specific type, matching a particular regular expression.
When the new Max Verify Depth option is set, Pomerium will consider client-supplied intermediate CA certificates when verifying a client certificate. The default setting preserves the behavior of previous Pomerium releases: all client certificates must be issued directly by a certificate authority included in the CA setting (no client-supplied intermediate CA certificates are allowed). This default behavior may change in a future release.
Certificate Matcher PPL Criteria (beta)
The new Certificate Matcher can be used with the new PPL criterion client_certificate
to build policies that grant or deny access based on the client certificate’s fingerprint or Subject Public Key Info (SPKI) hash. Both of these options allow you to create an allowlist or denylist of specific certificates.
See the Certificate Matcher policy page for more information and examples.
Set Request Headers options
You can configure the Set Request Headers setting to send the client certificate fingerprint (downstream mTLS must be enabled) to the upstream application or service. The fingerprint can be built into your authorization policy with the new Certificate Matcher (beta) PPL criteria to grant or deny users based on the fingerprint’s value.
See the Set Request Headers settings page for more information and examples.
Deprecated
Set Authorization Header
The Set Authorization Header option is deprecated in favor of the new variable substitution support in the Set Request Headers option. This new support allows you to pass IdP tokens to upstream services in any header, not just the Authorization
header.
The Set Authorization Header option will be removed in a future release.
TLS Downstream Client Certificate Authority
The TLS Downstream Client Certificate Authority option is deprecated, and will be removed in a future release.
If you previously used this setting to require client certificates only on certain routes, you can achieve this same behavior by setting the new downstream mTLS Enforcement Mode option to the value policy
and adding a policy deny rule with the invalid_client_certificate
criterion on all routes that should require client certificates.
If you want to enforce an allowlist or denylist of specific certificates on a particular route, you can use the new client_certificate
policy criterion.
If you do need to set completely different trusted client CAs for different routes, we recommend running separate Pomerium clusters for each set of trusted client CAs.
Breaking
Set Request Headers options
To prevent a ‘$’ character from being treated as the start of a variable substitution, you may need to replace it with ‘$$’.
0.22.0
New
Hosted Authenticate Service
- Hosted Authenticate Service will now be used by default to handle single-sign-on. Pomerium hosts this service as a convenience to its users; no identity provider configuration or authenticate service URL needs to be specified if the hosted authenticate service is used. The Self-Hosted Authenticate Service, which requires a self-hosted authenticate service URL and identity provider, is still supported for users that prefer to host these services themselves.
Wildcard From Routes
- Wildcard From Routes is a Beta support feature that allows you to define a wildcard route that points matching external routes to a single destination.
Better Memory performance
- Internal RDS changes reduce memory consumption, especially for environments where configuration changes rapidly.
0.21.0
Upgrading
There are several data model changes in this release that are not backward compatible. Please make sure you back up your Postgres database before performing an upgrade.
Breaking
Devices need to be re-enrolled
The Device Identity (beta) data model had an internal change that is not forward compatible. Your enrolled devices will need be re-registered. Your existing policies may need to be updated.
Forward Auth (deprecated, removed in this release)
Forward auth was introduced in early versions of Pomerium to provide a gradual migration path for users of other reverse proxies to Pomerium. Since then, Pomerium has come a long way - it is now based around first class reverse proxy core (Envoy) and has been battle tested for many years. Unfortunately, supporting forward authentication mode provides a subpar experience in security (cookies cannot be stripped from upstream requests), configuration (misconfiguration issues are common and hard to troubleshoot), and it is not compatible with many of Pomerium's newer features and deployment scenarios.
New
Bastion Host support for TCP routes
See Bastion Host
Internal TLS by default
If you run Pomerium Enterprise, you may set up a secure HTTPS connection between Pomerium Core and Enterprise without need to explicitly supply certificates. See tls_derive
0.20.0
Breaking
IdP Groups Policy
A deprecated routes.allowed_groups
and groups
PPL criteria were removed.
For Open Source, please use IdP Claims passed by your IdP.
- Please visit your IdP provider admin console to adjust group membership propagation to Claims.
- You may need adjust requested scopes via
idp_scopes
config option. - visit your authenticate endpoint
/.pomerium
route to check the group claims are passed by your IdP. - use
claim/
PPL criteria
routes:
- from: https://httpbin.localhost.pomerium.io
to: https://httpbin.org
pass_identity_headers: true
policy:
allow:
and:
- claim/groups: admins
For Enterprise, use PPL Builder
IdP Directory Sync
IdP directory sync has been moved to https://github.com/pomerium/datasource and becomes part of the External Data Sources integration, in order to provide unification with other external data sources, consolidate job scheduling and monitoring.
Setting the below options in Pomerium config file would now result in an error. In Pomerium Enterprise Console, please navigate to Settings > Identity Provider and configure directory sync there.
idp_service_account
: use IdP provider specific options in the UI.idp_refresh_directory_timeout
: use Polling Min Delay.idp_refresh_directory_interval
: replaced by Polling Max Delay.idp_qps
: not required, IdP providers adjust their qps rate.
Pomerium Core would only perform user authentication and session refresh with the IdP provider, and would not try to synchronize user details and groups, which is now part of External Data Sources.
0.19.0
No changes required to upgrade
- This release has no breaking changes. Review the v19 Changelog for more information.
0.18.0
No changes required to upgrade
- This release has no breaking changes. Review the v18 Changelog for more information.
0.17.0
New
Per Route OIDC Credentials
This release of Pomerium adds the ability to bind a route to unique OIDC credentials. This allows Identity Provider administrators to view Pomerium protected applications individually rather than as a single shared application.
See idp_client_id and idp_client_secret for configuration details.
Updated User Info Page
The .pomerium
user info page has been redesigned to better structure data around user identity, group, and device information.
External Google Groups
Pomerium policy now supports group members from outside of your organization.
0.16.0
New
Policy for Device Identity
This release of Pomerium adds the ability to set policy based on system registration via WebAuthN.
See Device Identity for more details.
HTTP PPL Criteria
http_path
and http_method
are now supported for matching HTTP requests in policies. See Pomerium Policy Language for more details.
Breaking
Self-signed fallback certificates
When selecting a TLS certificate for a listener, Pomerium attempts to locate one by iterating through the provided certs and searching for a SAN match. This applies to all listeners, including internal service URLs like databroker_service_url
and public endpoints like authenticate.example.com
.
Previously, when no match was found, Pomerium would select the "first" certificate in the list. However, the definition of "first" might change based on runtime configuration, so the certificate selection was non-deterministic.
Starting in v0.16, Pomerium will instead generate a self-signed certificate if it cannot locate an appropriate certificate from the provided configuration or system key/trust store. If you discover that you are receiving a self-signed certificate rather than a certificate from certificate
/certificates
/certificate_file
or the trust store, you have a mismatch between your service URL and the names covered in your certificates.
OIDC flow no longer sets default uri params
Previously, Pomerium would default to setting the uri param access_type
to offline
for all OpenID Connect based identity providers. However, using uri params to ensure offline access (e.g. refresh_tokens
used to keep user's sessions alive) is unique to Google. Those query params will now only be set for Google. Other OIDC based IdP's should continue to work using OIDC's offline_access
scope.
Removed options
The deprecated headers
option has been removed. Use set_response_headers
instead.
The signing_key_algorithm
option has been removed and will now be inferred from signing_key
.
Changed GitHub Team IDs
To improve performance, IdP directory synchronization for GitHub now uses the GraphQL API. This API returns the same information as the REST API, except that the GraphQL node IDs are different. Where we previously used the team integer ID from the REST API, we now use the team slug instead. Most policies should already use the team slug for group based rules, which should continue to work. However, if the integer ID is used it will no longer work. Update those policies to use the team slug instead.
CLI Source and Packaging Update
pomerium-cli
has been factored out of the core repository and now resides at https://github.com/pomerium/cli. If you currently install the CLI tool from Packages or Homebrew, no changes should be required to your process. However, users of docker images or direct github release downloads will need to update their references.
Please see the updated install instructions for additional details.
0.15.0
Breaking
Removed options
The unused grpc_server_max_connection_age
, grpc_server_max_connection_age_grace
and refresh_cooldown
options were removed.
Removed support for Ed25519 Signing Keys
Ed25519 is no longer supported for signing_key
since OPA Rego only supports ECDSA and RSA.
New
Updated and expanded policy syntax
Routes and policies may now be configured under a new top level key - routes
- This more closely aligns to how policies and routes are conceptually related
- The
routes
block supports a more powerful syntax for defining policies with conditionals and various criteria
Support environmental proxy settings
pomerium-cli
now respects proxy related environmental variables.
0.14.0
New
Ping Identity
Ping Identity is supported as a directory provider. See the documentation for details.
Customized Identity Headers
With the v0.14 release, the names of X-Pomerium-Claim-{Name}
headers can now be customized. This enables broader 3rd party application support for Pomerium's identity headers.
Redis High Availability
Databroker now supports redis sentinel and cluster for increased availability. See the databroker documentation for details.