HashiCorp Vault Missing Release of Memory after Effective Lifetime vulnerability

@@ -9,23 +9,38 @@ description: |-
  
 Vault can be configured to log completed requests using the `log_requests_level` configuration parameter.
  
-## Activating the Log Completed Requests
+## Activating Logging Completed Requests
  
-By default, logging completed requests is disabled. To activate the requests logging, set the `log_requests_level`
+By default, logging completed requests is disabled. To activate request logging, set the `log_requests_level`
 configuration option in the Vault server configuration to the desired logging level. The acceptable logging levels are
-`error`, `warn`, `info`, `debug`, and `trace`.
-If the vault server is already running, you can still configure the parameter in the Vault server configuration,
-and then send an `SIGHUP` signal to the vault process.
+`error`, `warn`, `info`, `debug`, `trace`, and `off`, which is the default.
+ 
+Completed requests will be logged at the configured level, if Vault's log level includes this level of logs. For
+example, if `log_level` is set to `debug`, but `log_requests_level` is set to `trace`, completed requests will
+not be logged.
+ 
+If the Vault server is already running, you can still configure the parameter in the Vault server configuration,
+and then send a `SIGHUP` signal to the Vault process.
  
 ```hcl
-log_requests_level = "trace"
+log_requests_level = "debug"
+log_level = "debug"
  
 listener "tcp" {
   # ...
 }
 ```
  
-## Deactivating the Log Completed Requests
+## Deactivating Logging Completed Requests
+ 
+To deactivate logging completed requests, simply remove the `log_requests_level` configuration parameter
+from the Vault server configuration, or set it to `off`, and send a `SIGHUP` signal to the Vault process.
+ 
+```hcl
+log_requests_level = "off"
+log_level = "debug"
  
-To deactivate logging completed requests, simply remove the `log_requests_level`
-configuration parameter from the vault server configuration, and send a `SIGHUP` signal to the vault process.
+listener "tcp" {
+  # ...
+}
+```
 <82990506+hc-github-team-secure-vault-core@users.noreply.github.com>
 d8b811d8da56589ad1de0396d8d09a07c88fdb10 (#21385)
 .../docs/enterprise/license/utilization-reporting.mdx    | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

@@ -1,84 +1,64 @@
-### update-primary can lead to data loss
- 
-It's possible to lose data from a Vault cluster given a particular configuration
-and sequence of steps.  This page describes two paths to data loss, both
-associated with the use of
-[update-primary](/vault/api-docs/system/replication/replication-performance#update-performance-secondary-s-primary).
- 
-Normally update-primary does not need to be used.  However, there are a few cases
-where it's needed, e.g. when the known primary cluster addresses of a secondary
-don't contain any of the correct addresses.  But update-primary does more than
-you might think: it does almost everything that enabling a secondary does,
-except that it doesn't wipe storage.  One of the steps that it takes is to
-temporarily remove most of the mount table records: it removes all mount entries
-except for those that are managed automatically by vault, e.g. identity mounts.
- 
-This update-primary behaviour is unintended and we'll be reworking it in an upcoming release.
-Once it lands the changelog entry will be "Fix a race condition with update-primary that could
-result in data loss after a DR failover."
- 
-#### update-primary with local data in shared mounts
- 
-If update-primary is done on a PR secondary with shared mounts containing
-[local data](/vault/docs/enterprise/replication#replicated-data) (e.g. pki certs,
-approle secretids), the merkle tree on the PR secondary may get corrupted due to
-a timing race.
- 
-When this happens, the PR secondary still contains all the stored data, e.g. listing
-local certs from PKI mounts will return the correct results.  However, because the
-merkle tree has been corrupted, a downstream DR secondary will not receive the local
-data, and will delete it if it already had it.  If the PR secondary's DR secondary is promoted before
-the PR secondary is repaired, the newly promoted PR secondary will not contain the local
-data it ought to.  If the former PR secondary is lost or destroyed, the missing data
-will not be recoverable other than via a snapshot restore.
- 
-#### Detection and remediation
- 
-If the TRACE level log line `"cleaning key in merkle tree"` appears immediately subsequent
-to an update-primary on a PR secondary, that's an indicator that the timing race was lost
-and that the merkle tree may be corrupt.
- 
-Repairing the corrupt merkle tree is done by issuing a
+### API calls to update-primary may lead to data loss ((#update-primary-data-loss))
+ 
+#### Affected versions
+ 
+- All current versions of Vault
+ 
+<Tip title="We are actively working on the underlying issue">
+ 
+Look for **Fix a race condition with update-primary that could result in data
+loss after a DR failover.** in a future changelog for the resolution.
+ 
+</Tip>
+ 
+#### Issue
+ 
+The [update-primary](/vault/api-docs/system/replication/replication-performance#update-performance-secondary-s-primary)
+endpoint temporarily removes all mount entries except for those that are managed
+automatically by vault (e.g. identity mounts). In certain situations, a race
+condition between mount table truncation replication repairs may lead to data
+loss when updating secondary replication clusters. 
+ 
+Situations where the race condition may occur:
+ 
+- **When the cluster has local data (e.g., PKI certificates, app role secret IDs)
+  in shared mounts**.
+  Calling `update-primary` on a performance secondary with local data in shared
+  mounts may corrupt the merkle tree on the secondary. The secondary still
+  contains all the previously stored data, but the corruption means that
+  downstream secondaries will not receive the shared data and will interpret the
+  update as a request to delete the information. If the downstream secondary is
+  promoted before the merkle tree is repaired, the newly promoted secondary will
+  not contain the expected local data. The missing data may be unrecoverable if
+  the original secondary is is lost or destroyed.
+- **When the cluster has an `Allow` paths defined.**
+  As of Vault 1.0.3.1, startup, unseal, and calling `update-primary` all trigger a
+  background job that looks at the current mount data and removes invalid entries
+  based on path filters. When a secondary has `Allow` path filters, the cleanup
+  code may misfire in the windown of time after update-primary truncats the mount
+  tables but before the mount tables are rewritten by replication. The cleanup
+  code deletes data associated with the missing mount entries but does not modify
+  the merkle tree. Because the merkle tree remains unchanged, replication will not
+  know that the data is missing and needs to be repaired.
+ 
+#### Workaround 1: PR secondary with local data in shared mounts
+ 
+Watch for `cleaning key in merkle tree` in the TRACE log immediately after an
+update-primary call on a PR secondary to indicate the merkle tree may be
+corrupt. Repair the merkle tree by issuing a
 [replication reindex request](/vault/api-docs/system/replication#reindex-replication)
 to the PR secondary.
  
-If logs are no longer present (the update-primary was done some time in the past), it's
-probably best to reindex the PR secondary pre-emptively as a precaution.
- 
-#### update-primary with "Allow" path filters
- 
-There is a further path to data loss associated update-primary.
-This issue requires that the PR secondary receiving an update-primary request has an
-associated `Allow` path filter defined for it. Like the first issue, this one too has
-a timing aspect: the problem may or may not manifest, depending on how
-quickly the mount tables truncated by update-primary get repaired by replication.
- 
-At startup/unseal (and after an update-primary), Vault runs a background job that looks
-at the mount data it has stored and tries to delete any that doesn't belong there, based
-on path filters.  This behaviour was introduced in 1.0.3.1 to recover from a regression
-that allowed for inappropriate filtering of data: we needed to ensure that any previously
-unfiltered data got cleaned up on secondaries that ought not have it.
- 
-If a performance secondary has an associated Allow path filter, this cleanup code can
-misfire during the interval between when the truncated mount tables are written by
-update-primary and the time when they get rewritten by replication.  The cleanup code
-will delete the data associated with the missing mount entries.  The cleanup code
-doesn't modify the merkle tree, and as a result this deleted data won't be discovered
-as missing and repaired by replication.
- 
-#### Detection and remediation
- 
-When the cleanup code fires it logs the INFO level message `"deleted mistakenly stored
-mount entry from backend"`.  This is a reliable indicator that the bug was hit.
- 
-If logs aren't available, the other indicator that this problem has manifested is to
-query the shared mount in question. The secondary won't have any of the data that
-the primary does, e.g. roles and configuration will be absent.
+If TRACE logs are no longer available, we recommend pre-emptively reindexing the
+PR secondary as a precaution.
  
-Reindexing the performance secondary will update the merkle tree to reflect the missing
-storage entries and allow missing shared data to be replaced by replication.  **However,
-any local data on shared mounts (such as PKI certs) will not be recoverable.**
+#### Workaround 2: PR secondary with "Allow" path filters
  
-#### Impacted Versions
+Watch for `deleted mistakenly stored mount entry from backend` in the INFO log.
+Reindex the performance secondary to update the merkle tree with the missing
+data and allow replication to disseminate the changes. **You will not be able to
+recover local data on shared mounts (e.g., PKI certificates)**.
  
-Affects all current versions of Vault.
+If INFO logs are no longer available, query the shared mount in question to
+confirm whether your role and configuration data are present on the primary but
+missing from the secondary.

@@ -1,64 +1,64 @@
-### API calls to update-primary may lead to data loss ((#update-primary-data-loss))
- 
-#### Affected versions
- 
-- All current versions of Vault
- 
-<Tip title="We are actively working on the underlying issue">
- 
-Look for **Fix a race condition with update-primary that could result in data
-loss after a DR failover.** in a future changelog for the resolution.
- 
-</Tip>
- 
-#### Issue
- 
-The [update-primary](/vault/api-docs/system/replication/replication-performance#update-performance-secondary-s-primary)
-endpoint temporarily removes all mount entries except for those that are managed
-automatically by vault (e.g. identity mounts). In certain situations, a race
-condition between mount table truncation replication repairs may lead to data
-loss when updating secondary replication clusters. 
- 
-Situations where the race condition may occur:
- 
-- **When the cluster has local data (e.g., PKI certificates, app role secret IDs)
-  in shared mounts**.
-  Calling `update-primary` on a performance secondary with local data in shared
-  mounts may corrupt the merkle tree on the secondary. The secondary still
-  contains all the previously stored data, but the corruption means that
-  downstream secondaries will not receive the shared data and will interpret the
-  update as a request to delete the information. If the downstream secondary is
-  promoted before the merkle tree is repaired, the newly promoted secondary will
-  not contain the expected local data. The missing data may be unrecoverable if
-  the original secondary is is lost or destroyed.
-- **When the cluster has an `Allow` paths defined.**
-  As of Vault 1.0.3.1, startup, unseal, and calling `update-primary` all trigger a
-  background job that looks at the current mount data and removes invalid entries
-  based on path filters. When a secondary has `Allow` path filters, the cleanup
-  code may misfire in the windown of time after update-primary truncats the mount
-  tables but before the mount tables are rewritten by replication. The cleanup
-  code deletes data associated with the missing mount entries but does not modify
-  the merkle tree. Because the merkle tree remains unchanged, replication will not
-  know that the data is missing and needs to be repaired.
- 
-#### Workaround 1: PR secondary with local data in shared mounts
- 
-Watch for `cleaning key in merkle tree` in the TRACE log immediately after an
-update-primary call on a PR secondary to indicate the merkle tree may be
-corrupt. Repair the merkle tree by issuing a
-[replication reindex request](/vault/api-docs/system/replication#reindex-replication)
-to the PR secondary.
- 
-If TRACE logs are no longer available, we recommend pre-emptively reindexing the
-PR secondary as a precaution.
- 
-#### Workaround 2: PR secondary with "Allow" path filters
- 
-Watch for `deleted mistakenly stored mount entry from backend` in the INFO log.
-Reindex the performance secondary to update the merkle tree with the missing
-data and allow replication to disseminate the changes. **You will not be able to
-recover local data on shared mounts (e.g., PKI certificates)**.
- 
-If INFO logs are no longer available, query the shared mount in question to
-confirm whether your role and configuration data are present on the primary but
-missing from the secondary.
+### API calls to update-primary may lead to data loss ((#update-primary-data-loss))
+ 
+#### Affected versions
+ 
+- All current versions of Vault
+ 
+<Tip title="We are actively working on the underlying issue">
+ 
+Look for **Fix a race condition with update-primary that could result in data
+loss after a DR failover.** in a future changelog for the resolution.
+ 
+</Tip>
+ 
+#### Issue
+ 
+The [update-primary](/vault/api-docs/system/replication/replication-performance#update-performance-secondary-s-primary)
+endpoint temporarily removes all mount entries except for those that are managed
+automatically by vault (e.g. identity mounts). In certain situations, a race
+condition between mount table truncation replication repairs may lead to data
+loss when updating secondary replication clusters. 
+ 
+Situations where the race condition may occur:
+ 
+- **When the cluster has local data (e.g., PKI certificates, app role secret IDs)
+  in shared mounts**.
+  Calling `update-primary` on a performance secondary with local data in shared
+  mounts may corrupt the merkle tree on the secondary. The secondary still
+  contains all the previously stored data, but the corruption means that
+  downstream secondaries will not receive the shared data and will interpret the
+  update as a request to delete the information. If the downstream secondary is
+  promoted before the merkle tree is repaired, the newly promoted secondary will
+  not contain the expected local data. The missing data may be unrecoverable if
+  the original secondary is is lost or destroyed.
+- **When the cluster has an `Allow` paths defined.**
+  As of Vault 1.0.3.1, startup, unseal, and calling `update-primary` all trigger a
+  background job that looks at the current mount data and removes invalid entries
+  based on path filters. When a secondary has `Allow` path filters, the cleanup
+  code may misfire in the windown of time after update-primary truncats the mount
+  tables but before the mount tables are rewritten by replication. The cleanup
+  code deletes data associated with the missing mount entries but does not modify
+  the merkle tree. Because the merkle tree remains unchanged, replication will not
+  know that the data is missing and needs to be repaired.
+ 
+#### Workaround 1: PR secondary with local data in shared mounts
+ 
+Watch for `cleaning key in merkle tree` in the TRACE log immediately after an
+update-primary call on a PR secondary to indicate the merkle tree may be
+corrupt. Repair the merkle tree by issuing a
+[replication reindex request](/vault/api-docs/system/replication#reindex-replication)
+to the PR secondary.
+ 
+If TRACE logs are no longer available, we recommend pre-emptively reindexing the
+PR secondary as a precaution.
+ 
+#### Workaround 2: PR secondary with "Allow" path filters
+ 
+Watch for `deleted mistakenly stored mount entry from backend` in the INFO log.
+Reindex the performance secondary to update the merkle tree with the missing
+data and allow replication to disseminate the changes. **You will not be able to
+recover local data on shared mounts (e.g., PKI certificates)**.
+ 
+If INFO logs are no longer available, query the shared mount in question to
+confirm whether your role and configuration data are present on the primary but
+missing from the secondary.
 <82990506+hc-github-team-secure-vault-core@users.noreply.github.com>
 6157b5ffcb6e0fcd1d0823e4929bf50ef1b00bc8 (#20837)
 changelog/20825.txt                                      | 3 +++
 physical/aerospike/aerospike_test.go                     | 4 ++++
 website/content/docs/configuration/storage/aerospike.mdx | 3 +++
 3 files changed, 10 insertions(+)
 create mode 100644 changelog/20825.txt

@@ -1,6 +1,11 @@
-output "public_ips" {
-  description = "Vault cluster target host public_ips"
-  value       = [for host in var.target_hosts : host.public_ip]
+output "audit_device_file_path" {
+  description = "The file path for the audit device, if enabled"
+  value       = var.enable_file_audit_device ? local.audit_device_file_path : "file audit device not enabled"
+}
+ 
+output "cluster_name" {
+  description = "The Vault cluster name"
+  value       = var.cluster_name
 }
  
 output "private_ips" {
@@ -8,29 +13,9 @@ output "private_ips" {
   value       = [for host in var.target_hosts : host.private_ip]
 }
  
-output "target_hosts" {
-  description = "The vault cluster instances that were created"
- 
-  value = var.target_hosts
-}
-output "root_token" {
-  value = coalesce(var.root_token, try(enos_vault_init.leader[0].root_token, null), "none")
-}
- 
-output "unseal_keys_b64" {
-  value = try(enos_vault_init.leader[0].unseal_keys_b64, [])
-}
- 
-output "unseal_keys_hex" {
-  value = try(enos_vault_init.leader[0].unseal_keys_hex, null)
-}
- 
-output "unseal_shares" {
-  value = try(enos_vault_init.leader[0].unseal_keys_shares, -1)
-}
- 
-output "unseal_threshold" {
-  value = try(enos_vault_init.leader[0].unseal_keys_threshold, -1)
+output "public_ips" {
+  description = "Vault cluster target host public_ips"
+  value       = [for host in var.target_hosts : host.public_ip]
 }
  
 output "recovery_keys_b64" {
@@ -49,12 +34,28 @@ output "recovery_threshold" {
   value = try(enos_vault_init.leader[0].recovery_keys_threshold, -1)
 }
  
-output "cluster_name" {
-  description = "The Vault cluster name"
-  value       = var.cluster_name
+output "root_token" {
+  value = coalesce(var.root_token, try(enos_vault_init.leader[0].root_token, null), "none")
 }
  
-output "audit_device_file_path" {
-  description = "The file path for the audit device, if enabled"
-  value       = var.enable_file_audit_device ? local.audit_device_file_path : "file audit device not enabled"
+output "target_hosts" {
+  description = "The vault cluster instances that were created"
+ 
+  value = var.target_hosts
+}
+ 
+output "unseal_keys_b64" {
+  value = try(enos_vault_init.leader[0].unseal_keys_b64, [])
+}
+ 
+output "unseal_keys_hex" {
+  value = try(enos_vault_init.leader[0].unseal_keys_hex, null)
+}
+ 
+output "unseal_shares" {
+  value = try(enos_vault_init.leader[0].unseal_keys_shares, -1)
+}
+ 
+output "unseal_threshold" {
+  value = try(enos_vault_init.leader[0].unseal_keys_threshold, -1)
 }

@@ -0,0 +1,322 @@
+terraform {
+  required_providers {
+    # We need to specify the provider source in each module until we publish it
+    # to the public registry
+    enos = {
+      source  = "app.terraform.io/hashicorp-qti/enos"
+      version = ">= 0.3.24"
+    }
+  }
+}
+ 
+data "aws_vpc" "vpc" {
+  id = var.vpc_id
+}
+ 
+data "aws_subnets" "vpc" {
+  filter {
+    name   = "vpc-id"
+    values = [var.vpc_id]
+  }
+}
+ 
+data "aws_kms_key" "kms_key" {
+  key_id = var.awskms_unseal_key_arn
+}
+ 
+data "aws_iam_policy_document" "target" {
+  statement {
+    resources = ["*"]
+ 
+    actions = [
+      "ec2:DescribeInstances",
+      "secretsmanager:*"
+    ]
+  }
+ 
+  statement {
+    resources = [var.awskms_unseal_key_arn]
+ 
+    actions = [
+      "kms:DescribeKey",
+      "kms:ListKeys",
+      "kms:Encrypt",
+      "kms:Decrypt",
+      "kms:GenerateDataKey"
+    ]
+  }
+}
+ 
+data "aws_iam_policy_document" "target_role" {
+  statement {
+    actions = ["sts:AssumeRole"]
+ 
+    principals {
+      type        = "Service"
+      identifiers = ["ec2.amazonaws.com"]
+    }
+  }
+}
+ 
+data "enos_environment" "localhost" {}
+ 
+resource "random_string" "random_cluster_name" {
+  length  = 8
+  lower   = true
+  upper   = false
+  numeric = false
+  special = false
+}
+ 
+resource "random_string" "unique_id" {
+  length  = 4
+  lower   = true
+  upper   = false
+  numeric = false
+  special = false
+}
+ 
+locals {
+  spot_allocation_strategy      = "price-capacity-optimized"
+  on_demand_allocation_strategy = "lowestPrice"
+  instances                     = toset([for idx in range(var.instance_count) : tostring(idx)])
+  cluster_name                  = coalesce(var.cluster_name, random_string.random_cluster_name.result)
+  name_prefix                   = "${var.project_name}-${local.cluster_name}-${random_string.unique_id.result}"
+  fleet_tag                     = "${local.name_prefix}-spot-fleet-target"
+  fleet_tags = {
+    Name                     = "${local.name_prefix}-target"
+    "${var.cluster_tag_key}" = local.cluster_name
+    Fleet                    = local.fleet_tag
+  }
+}
+ 
+resource "aws_iam_role" "target" {
+  name               = "${local.name_prefix}-target-role"
+  assume_role_policy = data.aws_iam_policy_document.target_role.json
+}
+ 
+resource "aws_iam_instance_profile" "target" {
+  name = "${local.name_prefix}-target-profile"
+  role = aws_iam_role.target.name
+}
+ 
+resource "aws_iam_role_policy" "target" {
+  name   = "${local.name_prefix}-target-policy"
+  role   = aws_iam_role.target.id
+  policy = data.aws_iam_policy_document.target.json
+}
+ 
+resource "aws_security_group" "target" {
+  name        = "${local.name_prefix}-target"
+  description = "Target instance security group"
+  vpc_id      = var.vpc_id
+ 
+  # SSH traffic
+  ingress {
+    from_port = 22
+    to_port   = 22
+    protocol  = "tcp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  # Vault traffic
+  ingress {
+    from_port = 8200
+    to_port   = 8201
+    protocol  = "tcp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+      formatlist("%s/32", var.ssh_allow_ips)
+    ])
+  }
+ 
+  # Consul traffic
+  ingress {
+    from_port = 8300
+    to_port   = 8302
+    protocol  = "tcp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  ingress {
+    from_port = 8301
+    to_port   = 8302
+    protocol  = "udp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  ingress {
+    from_port = 8500
+    to_port   = 8503
+    protocol  = "tcp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  ingress {
+    from_port = 8600
+    to_port   = 8600
+    protocol  = "tcp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  ingress {
+    from_port = 8600
+    to_port   = 8600
+    protocol  = "udp"
+    cidr_blocks = flatten([
+      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
+      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
+    ])
+  }
+ 
+  # Internal traffic
+  ingress {
+    from_port = 0
+    to_port   = 0
+    protocol  = "-1"
+    self      = true
+  }
+ 
+  # External traffic
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+ 
+  tags = merge(
+    var.common_tags,
+    {
+      Name = "${local.name_prefix}-sg"
+    },
+  )
+}
+ 
+resource "aws_launch_template" "target" {
+  name     = "${local.name_prefix}-target"
+  image_id = var.ami_id
+  key_name = var.ssh_keypair
+ 
+  iam_instance_profile {
+    name = aws_iam_instance_profile.target.name
+  }
+ 
+  network_interfaces {
+    associate_public_ip_address = true
+    delete_on_termination       = true
+    security_groups             = [aws_security_group.target.id]
+  }
+ 
+  tag_specifications {
+    resource_type = "instance"
+ 
+    tags = merge(
+      var.common_tags,
+      local.fleet_tags,
+    )
+  }
+}
+ 
+# There are three primary knobs we can turn to try and optimize our costs by
+# using a spot fleet: our min and max instance requirements, our max bid
+# price, and the allocation strategy to use when fulfilling the spot request.
+# We've currently configured our instance requirements to allow for anywhere
+# from 2-4 vCPUs and 4-16GB of RAM. We intentionally have a wide range
+# to allow for a large instance size pool to be considered. Our next knob is our
+# max bid price. As we're using spot fleets to save on instance cost, we never
+# want to pay more for an instance than we were on-demand. We've set the max price
+# to equal what we pay for t3.medium instances on-demand, which are the smallest
+# reliable size for Vault scenarios. The final knob is the allocation strategy
+# that AWS will use when looking for instances that meet our resource and cost
+# requirements. We're using the "lowestPrice" strategy to get the absolute
+# cheapest machines that will fit the requirements, but it comes with a slightly
+# higher capacity risk than say, "capacityOptimized" or "priceCapacityOptimized".
+# Unless we see capacity issues or instances being shut down then we ought to
+# stick with that strategy.
+resource "aws_ec2_fleet" "targets" {
+  terminate_instances = true // termiante instances when we "delete" the fleet
+  tags = merge(
+    var.common_tags,
+    local.fleet_tags,
+  )
+  type = "instant" // make a synchronous request for the entire fleet
+ 
+  launch_template_config {
+    launch_template_specification {
+      launch_template_id = aws_launch_template.target.id
+      version            = aws_launch_template.target.latest_version
+    }
+ 
+    override {
+      max_price = var.max_price
+      subnet_id = data.aws_subnets.vpc.ids[0]
+ 
+      instance_requirements {
+        burstable_performance = "included"
+ 
+        memory_mib {
+          min = var.instance_mem_min
+          max = var.instance_mem_max
+        }
+ 
+        vcpu_count {
+          min = var.instance_cpu_min
+          max = var.instance_cpu_max
+        }
+      }
+    }
+  }
+ 
+  on_demand_options {
+    allocation_strategy = local.on_demand_allocation_strategy
+    max_total_price     = (var.max_price * var.instance_count)
+    min_target_capacity = var.capacity_type == "on-demand" ? var.instance_count : null
+    // One of these has to be set to enforce our on-demand target capacity minimum
+    single_availability_zone = false
+    single_instance_type     = true
+  }
+ 
+  spot_options {
+    allocation_strategy = local.spot_allocation_strategy
+    // The instance_pools_to_use_count is only valid for the allocation_strategy
+    // lowestPrice. When we are using that strategy we'll want to always set it
+    // to non-zero to avoid rebuilding the fleet on a re-run. For any other strategy
+    // set it to zero to avoid rebuilding the fleet on a re-run.
+    instance_pools_to_use_count = local.spot_allocation_strategy == "lowestPrice" ? 1 : null
+  }
+ 
+  // Try and provision only spot instances and fall back to on-demand.
+  target_capacity_specification {
+    default_target_capacity_type = var.capacity_type
+    spot_target_capacity         = var.capacity_type == "spot" ? var.instance_count : 0
+    on_demand_target_capacity    = var.capacity_type == "on-demand" ? var.instance_count : 0
+    target_capacity_unit_type    = "units" // units == instance count
+    total_target_capacity        = var.instance_count
+  }
+}
+ 
+data "aws_instance" "targets" {
+  depends_on = [
+    aws_ec2_fleet.targets,
+  ]
+  for_each = local.instances
+ 
+  instance_id = aws_ec2_fleet.targets.fleet_instance_set[0].instance_ids[each.key]
+}

@@ -0,0 +1,64 @@
+### API calls to update-primary may lead to data loss ((#update-primary-data-loss))
+ 
+#### Affected versions
+ 
+- All current versions of Vault
+ 
+<Tip title="We are actively working on the underlying issue">
+ 
+Look for **Fix a race condition with update-primary that could result in data
+loss after a DR failover.** in a future changelog for the resolution.
+ 
+</Tip>
+ 
+#### Issue
+ 
+The [update-primary](/vault/api-docs/system/replication/replication-performance#update-performance-secondary-s-primary)
+endpoint temporarily removes all mount entries except for those that are managed
+automatically by vault (e.g. identity mounts). In certain situations, a race
+condition between mount table truncation replication repairs may lead to data
+loss when updating secondary replication clusters. 
+ 
+Situations where the race condition may occur:
+ 
+- **When the cluster has local data (e.g., PKI certificates, app role secret IDs)
+  in shared mounts**.
+  Calling `update-primary` on a performance secondary with local data in shared
+  mounts may corrupt the merkle tree on the secondary. The secondary still
+  contains all the previously stored data, but the corruption means that
+  downstream secondaries will not receive the shared data and will interpret the
+  update as a request to delete the information. If the downstream secondary is
+  promoted before the merkle tree is repaired, the newly promoted secondary will
+  not contain the expected local data. The missing data may be unrecoverable if
+  the original secondary is is lost or destroyed.
+- **When the cluster has an `Allow` paths defined.**
+  As of Vault 1.0.3.1, startup, unseal, and calling `update-primary` all trigger a
+  background job that looks at the current mount data and removes invalid entries
+  based on path filters. When a secondary has `Allow` path filters, the cleanup
+  code may misfire in the windown of time after update-primary truncats the mount
+  tables but before the mount tables are rewritten by replication. The cleanup
+  code deletes data associated with the missing mount entries but does not modify
+  the merkle tree. Because the merkle tree remains unchanged, replication will not
+  know that the data is missing and needs to be repaired.
+ 
+#### Workaround 1: PR secondary with local data in shared mounts
+ 
+Watch for `cleaning key in merkle tree` in the TRACE log immediately after an
+update-primary call on a PR secondary to indicate the merkle tree may be
+corrupt. Repair the merkle tree by issuing a
+[replication reindex request](/vault/api-docs/system/replication#reindex-replication)
+to the PR secondary.
+ 
+If TRACE logs are no longer available, we recommend pre-emptively reindexing the
+PR secondary as a precaution.
+ 
+#### Workaround 2: PR secondary with "Allow" path filters
+ 
+Watch for `deleted mistakenly stored mount entry from backend` in the INFO log.
+Reindex the performance secondary to update the merkle tree with the missing
+data and allow replication to disseminate the changes. **You will not be able to
+recover local data on shared mounts (e.g., PKI certificates)**.
+ 
+If INFO logs are no longer available, query the shared mount in question to
+confirm whether your role and configuration data are present on the primary but
+missing from the secondary.

@@ -1,181 +0,0 @@
-terraform {
-  required_providers {
-    # We need to specify the provider source in each module until we publish it
-    # to the public registry
-    enos = {
-      source  = "app.terraform.io/hashicorp-qti/enos"
-      version = ">= 0.3.24"
-    }
-  }
-}
- 
-data "aws_vpc" "vpc" {
-  id = var.vpc_id
-}
- 
-data "aws_subnets" "vpc" {
-  filter {
-    name   = "vpc-id"
-    values = [var.vpc_id]
-  }
-}
- 
-data "aws_kms_key" "kms_key" {
-  key_id = var.awskms_unseal_key_arn
-}
- 
-data "aws_iam_policy_document" "target" {
-  statement {
-    resources = ["*"]
- 
-    actions = [
-      "ec2:DescribeInstances",
-      "secretsmanager:*"
-    ]
-  }
- 
-  statement {
-    resources = [var.awskms_unseal_key_arn]
- 
-    actions = [
-      "kms:DescribeKey",
-      "kms:ListKeys",
-      "kms:Encrypt",
-      "kms:Decrypt",
-      "kms:GenerateDataKey"
-    ]
-  }
-}
- 
-data "aws_iam_policy_document" "target_instance_role" {
-  statement {
-    actions = ["sts:AssumeRole"]
- 
-    principals {
-      type        = "Service"
-      identifiers = ["ec2.amazonaws.com"]
-    }
-  }
-}
- 
-data "enos_environment" "localhost" {}
- 
-resource "random_string" "cluster_name" {
-  length  = 8
-  lower   = true
-  upper   = false
-  numeric = false
-  special = false
-}
- 
-locals {
-  instances    = toset([for idx in range(var.instance_count) : tostring(idx)])
-  cluster_name = coalesce(var.cluster_name, random_string.cluster_name.result)
-  name_prefix  = "${var.project_name}-${local.cluster_name}"
-}
- 
-resource "aws_iam_role" "target_instance_role" {
-  name               = "target_instance_role-${random_string.cluster_name.result}"
-  assume_role_policy = data.aws_iam_policy_document.target_instance_role.json
-}
- 
-resource "aws_iam_instance_profile" "target" {
-  name = "${local.name_prefix}-target"
-  role = aws_iam_role.target_instance_role.name
-}
- 
-resource "aws_iam_role_policy" "target" {
-  name   = "${local.name_prefix}-target"
-  role   = aws_iam_role.target_instance_role.id
-  policy = data.aws_iam_policy_document.target.json
-}
- 
-resource "aws_security_group" "target" {
-  name        = "${local.name_prefix}-target"
-  description = "Target instance security group"
-  vpc_id      = var.vpc_id
- 
-  # SSH traffic
-  ingress {
-    from_port = 22
-    to_port   = 22
-    protocol  = "tcp"
-    cidr_blocks = flatten([
-      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
-      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
-    ])
-  }
- 
-  # Vault traffic
-  ingress {
-    from_port = 8200
-    to_port   = 8201
-    protocol  = "tcp"
-    cidr_blocks = flatten([
-      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
-      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
-      formatlist("%s/32", var.ssh_allow_ips)
-    ])
-  }
- 
-  # Consul traffic
-  ingress {
-    from_port = 8301
-    to_port   = 8301
-    protocol  = "tcp"
-    cidr_blocks = flatten([
-      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
-      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
-    ])
-  }
- 
-  ingress {
-    from_port = 8301
-    to_port   = 8301
-    protocol  = "udp"
-    cidr_blocks = flatten([
-      formatlist("%s/32", data.enos_environment.localhost.public_ip_addresses),
-      join(",", data.aws_vpc.vpc.cidr_block_associations.*.cidr_block),
-    ])
-  }
- 
-  # Internal traffic
-  ingress {
-    from_port = 0
-    to_port   = 0
-    protocol  = "-1"
-    self      = true
-  }
- 
-  # External traffic
-  egress {
-    from_port   = 0
-    to_port     = 0
-    protocol    = "-1"
-    cidr_blocks = ["0.0.0.0/0"]
-  }
- 
-  tags = merge(
-    var.common_tags,
-    {
-      Name = "${local.name_prefix}-sg"
-    },
-  )
-}
- 
-resource "aws_instance" "targets" {
-  for_each               = local.instances
-  ami                    = var.ami_id
-  instance_type          = var.instance_type
-  vpc_security_group_ids = [aws_security_group.target.id]
-  subnet_id              = tolist(data.aws_subnets.vpc.ids)[each.key % length(data.aws_subnets.vpc.ids)]
-  key_name               = var.ssh_keypair
-  iam_instance_profile   = aws_iam_instance_profile.target.name
-  tags = merge(
-    var.common_tags,
-    {
-      Name = "${local.name_prefix}-target-instance"
-      Type = local.cluster_name
-    },
-  )
-}

@@ -20,7 +20,8 @@ utilization as defined in our contracts. They consist of mostly computed metrics
 and will never contain Personal Identifiable Information (PII) or other
 sensitive information. Automated reporting shares the data with HashiCorp using
 a secure, unidirectional HTTPS API and makes an auditable record in the product
-logs each time it submits a report.
+logs each time it submits a report. The reports are submitted once every 24
+hours.
  
 ## Enable automated reporting
  
@@ -42,11 +43,13 @@ addresses:
  
 ### 2. Upgrade
  
-Upgrade to a release that supports entitlement utilization reporting. These
+Upgrade to a release that supports license utilization reporting. These
 releases include:
  
 - [Vault Enterprise 1.14.0](https://releases.hashicorp.com/vault/) and later
- 
+- [Vault Enterprise 1.13.4](https://releases.hashicorp.com/vault/) and later 1.13.x versions
+- [Vault Enterprise 1.12.8](https://releases.hashicorp.com/vault/) and later 1.12.x versions
+- [Vault Enterprise 1.11.12](https://releases.hashicorp.com/vault/)
  
 ### 3. Check logs
  
 website/content/docs/release-notes/1.14.0.mdx | 289 ++++++++++++++++--
 .../docs/upgrading/upgrade-to-1.14.x.mdx      |  26 +-
 website/content/partials/alerts/rc-alert.mdx  |   7 +
 .../known-issues/ui-pki-control-groups.mdx    |  17 ++
 .../known-issues/update-primary-data-loss.mdx |  64 ++++
 .../release-notes/deprecation-note.mdx        |   4 +
 .../content/partials/release-notes/intro.mdx  |   8 +
 .../ui-pki-control-groups-known-issue.mdx     |  21 +-
 .../partials/update-primary-known-issue.mdx   | 136 ++++-----
 website/data/docs-nav-data.json               |   6 +-
 10 files changed, 458 insertions(+), 120 deletions(-)
 create mode 100644 website/content/partials/alerts/rc-alert.mdx
 create mode 100644 website/content/partials/known-issues/ui-pki-control-groups.mdx
 create mode 100644 website/content/partials/known-issues/update-primary-data-loss.mdx
 create mode 100644 website/content/partials/release-notes/deprecation-note.mdx
 create mode 100644 website/content/partials/release-notes/intro.mdx

@@ -1,31 +1,284 @@
 ---
 layout: docs
-page_title: 1.14.0
+page_title: "1.14.0 release notes"
 description: |-
-  This page contains release notes for Vault 1.14.0
+  Key updates for  Vault 1.14.0
 ---
  
-# Vault 1.14.0 Release Notes
+# Vault 1.14.0 release notes
  
-**Software Release date:** TBD
+**GA date:** June 21, 2023
  
-**Summary:** Vault Release 1.14.0 offers features and enhancements that improve
-the user experience while solving critical  issues previously encountered by our
-customers. We are providing an overview  of improvements in this set of  release
-notes.
+@include 'release-notes/intro.mdx'
  
-~> Note: these are **draft** notes for a future version of Vault. They should not be considered
-official guidance until the release has been completed.
+## Known issues and breaking changes
  
-## Known issues
+Version | Issue 
+------- | ------------------------------------------------------------
+1.14.0+ | [Users limited by control groups can only access issuer detail from PKI overview page](/vault/docs/upgrading/upgrade-to-1.14.x#ui-pki-control-groups)
+All     | [API calls to update-primary may lead to data loss](/vault/docs/upgrading/upgrade-to-1.14.x#update-primary-data-loss)
  
-@include 'ui-pki-control-groups-known-issue.mdx'
+## Vault companion updates
  
-@include 'update-primary-known-issue.mdx'
+Companion updates are Vault updates that live outside the main Vault binary.
  
-## Feature Deprecations and EOL
+<table>
+  <thead>
+    <tr>
+      <th style={{verticalAlign: 'middle'}}>Release</th>
+      <th style={{verticalAlign: 'middle'}}>Update</th>
+      <th style={{verticalAlign: 'middle'}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
  
-Please refer to the [Deprecation Plans and Notice](/vault/docs/deprecation) page
-for up-to-date information on feature deprecations and plans. A [Feature
-Deprecation FAQ](/vault/docs/deprecation/faq) page addresses questions about
-decisions made about Vault feature deprecations.
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      Vault Secrets Operator for Kubernetes
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Directly connect Vault secrets into Pods as native Kubernetes Secrets
+      without modifying your application code.
+      <br /><br />
+      Learn more: <a href="/vault/docs/platform/k8s/vso">Vault Secrets Operator</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td rowspan={2} style={{verticalAlign: 'middle'}}>
+      Terraform
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use LDAP authentication from the unified LDAP engine to Terraform Vault
+      Provider.
+      <br /><br />
+      Learn more: <a href="/vault/docs/secrets/ldap">LDAP Secrets Engine</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Support for additional PKI issuers and keys endpoints.
+      <br /><br />
+      Learn more: <a href="/vault/docs/secrets/pki">PKI Secrets Engine</a>
+    </td>
+  </tr>
+  </tbody>
+</table>
+ 
+## Core updates
+ 
+Follow the learn more links for more information, or browse the list of
+[Vault tutorials updated to highlight changes for the most recent GA release](/vault/tutorials/new-release).
+ 
+<table>
+  <thead>
+    <tr>
+      <th style={{verticalAlign: 'middle'}}>Release</th>
+      <th style={{verticalAlign: 'middle'}}>Update</th>
+      <th style={{verticalAlign: 'middle'}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+  
+  <tr>
+    <td rowspan={2} style={{verticalAlign: 'middle'}}>
+      Public Key Infrastructure (PKI)
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use ACME to automate certificate lifecycle management for private PKI
+      needs with standard ACME clients like Certbot and k8s cert-manager.
+      Request certificates from a Vault server without needing to know Vault
+      APIs or authentication mechanisms.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/api-docs/secret/pki#acme-certificate-issuance">PKI Secrets Engine API: ACME</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use the improved PKI web UI to manage your PKI instance with intuitive
+      configuration and reasonable defaults for workflows, metadata, issuer
+      info, mount and tidy configuration, cross signing, multi-issuers etc.and
+      includes.   
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/api-docs/secret/pki#acme-certificate-issuance">PKI Secrets Engine</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      Security patches
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Various security improvements to remediate low severity and informational
+      findings from a 3rd party security audit.
+      <br /><br />
+      Learn more: <a href="/vault/docs/internals/security">Vault security model</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td rowspan={2} style={{verticalAlign: 'middle'}}>
+      Vault Agent
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>BETA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Fetch secrets directly into your application as environment variables.
+      <br /><br />
+      Learn more: <a href="/vault/docs/agent-and-proxy/agent/process-supervisor">Process Supervisor Mode</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>TEXT</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use a new subcommand and daemon, Vault Proxy, to access the proxy
+      functionality of Vault Agent. Vault Proxy will handle Vault Agent proxy
+      functionality going forward to simplify use case decisions for customers.
+      <br /><br />
+      Learn more: <a href="/vault/docs/agent-and-proxy/proxy">Vault Proxy</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td rowspan={3} style={{verticalAlign: 'middle'}}>
+      Plugin support
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Capture plugin metadata in the Vault audit log.
+      <br /><br />
+      Learn more: <a href="/vault/docs/audit/syslog">Syslog audit device</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use X509 Authentication and Terraform Vault Provider in the MongoDB Atlas
+      Database Secrets Engine.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/docs/secrets/databases/mongodbatlas">MongoDB Atlas Database Secrets Engine</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Dependency updates and more robust multiplexing for secrets and
+      authentication plugins.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/docs/plugins/plugin-development#serving-a-plugin-with-multiplexing">
+        Serving a plugin with multiplexing (Plugin Development)
+      </a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td rowspan={2} style={{verticalAlign: 'middle'}}>
+      AWS support
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Monitoring and performance enhancements for the Vault Lambda extension.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/docs/platform/aws/lambda-extension">Vault Lambda Extension guide</a>
+    </td>
+  </tr>
+  <tr>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Use static roles for IAM users in the AWS Secrets Engine.
+      <br /><br />
+      Learn more: <a href="/vault/docs/secrets/aws">AWS Secrets Engine</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      Vault GUI
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Streamlined and aligned navigation with HCP Vault UI.
+      <br /><br />
+      Learn more: <a href="/vault/docs/configuration/ui">Vault UI</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      Transit
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      <b>Contributed by the OSS community</b>. Support for public-key only Transit
+      keys and BYOK-secured export of key material.
+      <br /><br />
+      Learn more: <a href="/vault/api-docs/secret/transit">Transit Secrets Engine</a>
+    </td>
+  </tr>
+ 
+  </tbody>
+</table>
+ 
+## Enterprise updates
+ 
+<table>
+  <thead>
+    <tr>
+      <th style={{verticalAlign: 'middle'}}>Release</th>
+      <th style={{verticalAlign: 'middle'}}>Update</th>
+      <th style={{verticalAlign: 'middle'}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+  
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      Vault replication
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>ENHANCED</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Stability improvements based on customer feedback for Vault 1.13. See the 
+      <a href="https://raw.githubusercontent.com/hashicorp/vault/main/CHANGELOG.md">
+        Vault changelog
+      </a>
+      for a full list of bug fixes.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/docs/internals/replication">Replication overview</a>
+    </td>
+  </tr>
+ 
+  <tr>
+    <td style={{verticalAlign: 'middle'}}>
+      License utilization reporting
+    </td>
+    <td style={{verticalAlign: 'middle', textAlign: 'center'}}>GA</td>
+    <td style={{verticalAlign: 'middle'}}>
+      Enables automatic license utilization reporting for you and HashiCorp to
+      ensure transparent, accurate billing.
+      <br /><br />
+      Learn more:&nbsp;
+      <a href="/vault/docs/enterprise/license/utilization-reporting">Automated License utilization reporting</a>
+    </td>
+  </tr>
+ 
+  </tbody>
+</table>
+ 
+## Feature deprecations and EOL
+ 
+Deprecated in 1.14 | Retired in 1.14
+------------------ | ---------------
+Vault Agent API proxy support | [Duplicative Docker Images](https://hub.docker.com/_/vault)
+ 
+@include 'release-notes/deprecation-note.mdx'

@@ -82,5 +82,45 @@ tutorial for more details.
  
 If you're currently on a version of Vault before 1.11, or you've chosen to opt-out the Autopilot automated upgrade features when running Vault after 1.11 with Integrated Storage, or if you are running Vault with other storage backend such as Consul. Please refer to our [Vault HA upgrades Pre 1.11/Without Autopilot Upgrade Automation](/vault/docs/upgrading/vault-ha-upgrade) documentation for more details. Please note that this upgrade procedure also applies if you are upgrading Vault from pre 1.11 to post 1.11.
  
+## Enterprise Replication Installations
  
+<Note>
  
+Prior to any upgrade, be sure to also read and follow any instructions in the
+version-specific upgrade notes which are found in the navigation menu for this
+documentation.
+ 
+</Note>
+ 
+Upgrading Vault Enterprise clusters which participate in [Enterprise
+Replication](/vault/docs/enterprise/replication) requires the following basic
+order of operations:
+ 
+- **Upgrade the replication secondary instances first** using appropriate
+  guidance from the previous sections
+- Verify functionality of each secondary instance after upgrading
+- When satisfied with functionality of upgraded secondary instances, upgrade
+  the primary instance
+ 
+<Note>
+ 
+It is not safe to replicate from a newer version of Vault to an older version.
+When upgrading replicated clusters, ensure that upstream clusters are always on
+older versions of Vault than downstream clusters.
+ 
+</Note>
+ 
+Here is an example of upgrading four Vault replicated Vault clusters:
+ 
+![Upgrading multiple replicated clusters](/img/vault-replication-upgrade.png)
+ 
+In the above scenario, the ideal upgrade procedure would be as follows,
+verifying functionality after each cluster upgrade.
+ 
+1. Upgrade Clusters B and D. These clusters have no downstream clusters, so they
+   should be upgraded first, but the ordering of B vs D does not matter.
+2. Upgrade Cluster C, which now has an upgraded downstream cluster (Cluster D).
+   Because Cluster C is a cluster, it should also use the HA upgrade process.
+3. Finally, upgrade Cluster A. All clusters downstream of A will already be
+   upgraded. It should be upgraded last, as it is a Performance Primary and a DR
+   Primary.

@@ -64,14 +64,14 @@ Perform these steps on each standby:
  
 At this point all standby nodes are upgraded and ready to take over. The
 upgrade will not complete until one of the upgraded standby nodes takes over
-active duty. 
+active duty.
  
 To complete the cluster upgrade:
  
 1. Properly shut down the remaining (active) node
  
    <Note>
-   
+ 
    It is important that you shut the node down properly.
    This will perform a step-down and release the HA lock, allowing a standby
    node to take over with a very short delay.
@@ -97,45 +97,4 @@ upgrade notes.
  
 ## Enterprise Replication Installations
  
-<Note>
- 
-Prior to any upgrade, be sure to also read and follow any instructions in the
-version-specific upgrade notes which are found in the navigation menu for this
-documentation.
- 
-</Note>
- 
-Upgrading Vault Enterprise clusters which participate in [Enterprise
-Replication](/vault/docs/enterprise/replication) requires the following basic
-order of operations:
- 
-- **Upgrade the replication secondary instances first** using appropriate
-  guidance from the previous sections depending on whether each secondary
-  instance is non-HA or HA
-- Verify functionality of each secondary instance after upgrading
-- When satisfied with functionality of upgraded secondary instances, upgrade
-  the primary instance
- 
-<Note>
- 
-It is not safe to replicate from a newer version of Vault to an older version.
-When upgrading replicated clusters, ensure that upstream clusters are always on
-older versions of Vault than downstream clusters.
- 
-</Note>
- 
-Here is an example of upgrading four Vault replicated Vault clusters:
- 
-![Upgrading multiple replicated clusters](/img/vault-replication-upgrade.png)
- 
-In the above scenario, the ideal upgrade procedure would be as follows,
-verifying functionality after each cluster upgrade.
- 
-1. Upgrade Clusters B and D, using the HA upgrade process above. These clusters
-   have no downstream clusters, so they should be upgraded first, but the
-   ordering of B vs D does not matter.
-2. Upgrade Cluster C, which now has an upgraded downstream cluster (Cluster D).
-   Because Cluster C is a cluster, it should also use the HA upgrade process.
-3. Finally, upgrade Cluster A. All clusters downstream of A will already be
-   upgraded. It should be upgraded last, as it is a Performance Primary and a DR
-   Primary.
+See the main [upgrading](/vault/docs/upgrading#enterprise-replication-installations) page.
 <82990506+hc-github-team-secure-vault-core@users.noreply.github.com>
 5d13092b038397d5970dcaa411c43defe993ab3a (#21422)
 website/content/docs/auth/index.mdx | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

@@ -14,12 +14,12 @@ import (
 	"sync"
 	"time"
  
-	atomic2 "go.uber.org/atomic"
- 
+	"github.com/hashicorp/go-secure-stdlib/parseutil"
 	"github.com/hashicorp/vault/sdk/helper/certutil"
 	"github.com/hashicorp/vault/sdk/helper/consts"
 	"github.com/hashicorp/vault/sdk/helper/errutil"
 	"github.com/hashicorp/vault/sdk/logical"
+	atomic2 "go.uber.org/atomic"
 )
  
 const (
@@ -248,12 +248,12 @@ func (cb *crlBuilder) checkForAutoRebuild(sc *storageContext) error {
 	// the grace period and act accordingly.
 	now := time.Now()
  
-	period, err := time.ParseDuration(cfg.AutoRebuildGracePeriod)
+	period, err := parseutil.ParseDurationSecond(cfg.AutoRebuildGracePeriod)
 	if err != nil {
 		// This may occur if the duration is empty; in that case
 		// assume the default. The default should be valid and shouldn't
 		// error.
-		defaultPeriod, defaultErr := time.ParseDuration(defaultCrlConfig.AutoRebuildGracePeriod)
+		defaultPeriod, defaultErr := parseutil.ParseDurationSecond(defaultCrlConfig.AutoRebuildGracePeriod)
 		if defaultErr != nil {
 			return fmt.Errorf("error checking for auto-rebuild status: unable to parse duration from both config's grace period (%v) and default grace period (%v):\n- config: %v\n- default: %w\n", cfg.AutoRebuildGracePeriod, defaultCrlConfig.AutoRebuildGracePeriod, err, defaultErr)
 		}
@@ -436,7 +436,7 @@ func (cb *crlBuilder) rebuildDeltaCRLsIfForced(sc *storageContext, override bool
 		return nil, nil
 	}
  
-	deltaRebuildDuration, err := time.ParseDuration(cfg.DeltaRebuildInterval)
+	deltaRebuildDuration, err := parseutil.ParseDurationSecond(cfg.DeltaRebuildInterval)
 	if err != nil {
 		return nil, err
 	}
@@ -2118,7 +2118,7 @@ func augmentWithRevokedIssuers(issuerIDEntryMap map[issuerID]*issuerEntry, issue
 func buildCRL(sc *storageContext, crlInfo *crlConfig, forceNew bool, thisIssuerId issuerID, revoked []pkix.RevokedCertificate, identifier crlID, crlNumber int64, isUnified bool, isDelta bool, lastCompleteNumber int64) (*time.Time, error) {
 	var revokedCerts []pkix.RevokedCertificate
  
-	crlLifetime, err := time.ParseDuration(crlInfo.Expiry)
+	crlLifetime, err := parseutil.ParseDurationSecond(crlInfo.Expiry)
 	if err != nil {
 		return nil, errutil.InternalError{Err: fmt.Sprintf("error parsing CRL duration of %s", crlInfo.Expiry)}
 	}

@@ -46,11 +46,19 @@ The PKI Secrets Engine documentation is split into the following pieces:
  
 ## Tutorial
  
-Refer to the [Build Your Own Certificate Authority (CA)](/vault/tutorials/secrets-management/pki-engine)
-guide for a step-by-step tutorial.
+Refer to the following tutorials for PKI secrets engine usage examples:
+ 
+- [Build Your Own Certificate Authority (CA)](/vault/tutorials/secrets-management/pki-engine)
+- [Build Certificate Authority (CA) in Vault with an offline Root](/vault/tutorials/secrets-management/pki-engine-external-ca)
+- [Enable ACME with PKI secrets engine](/vault/tutorials/new-release/pki-acme-caddy)
+- [PKI Secrets Engine with Managed Keys](/vault/tutorials/enterprise/managed-key-pki)
+- [PKI Unified CRL and OCSP With Cross Cluster
+  Revocation](/vault/tutorials/secrets-management/pki-unified-crl-ocsp-cross-cluster)
+- [Configure Vault as a Certificate Manager in Kubernetes with
+  Helm](/vault/tutorials/kubernetes/kubernetes-cert-manager)
+- [Generate mTLS Certificates for Nomad using
+  Vault](/vault/tutorials/secrets-management/vault-pki-nomad)
  
-Have a look at the [PKI Secrets Engine with Managed Keys](/vault/tutorials/enterprise/managed-key-pki)
-for more about how to use externally managed keys with PKI.
  
 ## API
  
 <82990506+hc-github-team-secure-vault-core@users.noreply.github.com>
 8e40823fe1cf40cecb1e004be29fde5fdb875bdb (#21416)
 website/content/api-docs/secret/cubbyhole.mdx |  2 +-
 website/content/api-docs/secret/kv/kv-v1.mdx  | 16 +++++++++-------
 website/content/api-docs/secret/kv/kv-v2.mdx  |  7 ++++---
 3 files changed, 14 insertions(+), 11 deletions(-)

@@ -15,33 +15,35 @@ scenario "ui" {
   ]
  
   locals {
-    arch           = "amd64"
-    distro         = "ubuntu"
-    seal           = "awskms"
-    artifact_type  = "bundle"
-    consul_version = "1.14.2"
+    arch            = "amd64"
+    artifact_type   = "bundle"
+    backend_tag_key = "VaultStorage"
     build_tags = {
       "oss" = ["ui"]
       "ent" = ["ui", "enterprise", "ent"]
     }
-    bundle_path = abspath(var.vault_bundle_path)
+    bundle_path    = abspath(var.vault_bundle_path)
+    distro         = "ubuntu"
+    consul_version = "1.14.2"
+    max_price = {
+      // These prices are based on on-demand cost for t3.large in us-east
+      "rhel"   = "0.1432"
+      "ubuntu" = "0.0832"
+    }
+    seal = "awskms"
     tags = merge({
       "Project Name" : var.project_name
       "Project" : "Enos",
       "Environment" : "ci"
     }, var.tags)
-    vault_instance_types = {
-      amd64 = "t3a.small"
-      arm64 = "t4g.small"
-    }
-    vault_instance_type = coalesce(var.vault_instance_type, local.vault_instance_types[local.arch])
-    vault_license_path  = abspath(var.vault_license_path != null ? var.vault_license_path : joinpath(path.root, "./support/vault.hclic"))
     vault_install_dir_packages = {
       rhel   = "/bin"
       ubuntu = "/usr/bin"
     }
-    vault_install_dir = var.vault_install_dir
-    ui_test_filter    = var.ui_test_filter != null && try(trimspace(var.ui_test_filter), "") != "" ? var.ui_test_filter : (matrix.edition == "oss") ? "!enterprise" : null
+    vault_install_dir  = var.vault_install_dir
+    vault_license_path = abspath(var.vault_license_path != null ? var.vault_license_path : joinpath(path.root, "./support/vault.hclic"))
+    vault_tag_key      = "Type" // enos_vault_start expects Type as the tag key
+    ui_test_filter     = var.ui_test_filter != null && try(trimspace(var.ui_test_filter), "") != "" ? var.ui_test_filter : (matrix.edition == "oss") ? "!enterprise" : null
   }
  
   step "get_local_metadata" {
@@ -62,24 +64,15 @@ scenario "ui" {
     }
   }
  
-  step "find_azs" {
-    module = module.az_finder
- 
-    variables {
-      instance_type = [
-        var.backend_instance_type,
-        local.vault_instance_type
-      ]
-    }
+  step "ec2_info" {
+    module = module.ec2_info
   }
  
   step "create_vpc" {
     module = module.create_vpc
  
     variables {
-      ami_architectures  = [local.arch]
-      availability_zones = step.find_azs.availability_zones
-      common_tags        = local.tags
+      common_tags = local.tags
     }
   }
  
@@ -92,8 +85,8 @@ scenario "ui" {
     }
   }
  
-  step "create_backend_cluster" {
-    module     = "backend_${matrix.backend}"
+  step "create_vault_cluster_targets" {
+    module     = module.target_ec2_spot_fleet
     depends_on = [step.create_vpc]
  
     providers = {
@@ -101,20 +94,17 @@ scenario "ui" {
     }
  
     variables {
-      ami_id      = step.create_vpc.ami_ids["ubuntu"]["amd64"]
-      common_tags = local.tags
-      consul_release = {
-        edition = var.backend_edition
-        version = local.consul_version
-      }
-      instance_type = var.backend_instance_type
-      kms_key_arn   = step.create_vpc.kms_key_arn
-      vpc_id        = step.create_vpc.vpc_id
+      ami_id                = step.ec2_info.ami_ids[local.arch][local.distro][var.ubuntu_distro_version]
+      awskms_unseal_key_arn = step.create_vpc.kms_key_arn
+      cluster_tag_key       = local.vault_tag_key
+      common_tags           = local.tags
+      max_price             = local.max_price[local.distro]
+      vpc_id                = step.create_vpc.vpc_id
     }
   }
  
-  step "create_vault_cluster_targets" {
-    module     = module.target_ec2_spot_fleet // "target_ec2_instances" can be used for on-demand instances
+  step "create_vault_cluster_backend_targets" {
+    module     = module.target_ec2_spot_fleet
     depends_on = [step.create_vpc]
  
     providers = {
@@ -122,14 +112,36 @@ scenario "ui" {
     }
  
     variables {
-      ami_id                = step.create_vpc.ami_ids[local.distro][local.arch]
+      ami_id                = step.ec2_info.ami_ids["arm64"]["ubuntu"]["22.04"]
       awskms_unseal_key_arn = step.create_vpc.kms_key_arn
+      cluster_tag_key       = local.backend_tag_key
       common_tags           = local.tags
-      instance_type         = local.vault_instance_type // only used for on-demand instances
+      max_price             = local.max_price["ubuntu"]
       vpc_id                = step.create_vpc.vpc_id
     }
   }
  
+  step "create_backend_cluster" {
+    module = "backend_${matrix.backend}"
+    depends_on = [
+      step.create_vault_cluster_backend_targets,
+    ]
+ 
+    providers = {
+      enos = provider.enos.ubuntu
+    }
+ 
+    variables {
+      cluster_name    = step.create_vault_cluster_backend_targets.cluster_name
+      cluster_tag_key = local.backend_tag_key
+      release = {
+        edition = var.backend_edition
+        version = local.consul_version
+      }
+      target_hosts = step.create_vault_cluster_backend_targets.hosts
+    }
+  }
+ 
   step "create_vault_cluster" {
     module = module.vault_cluster
     depends_on = [
@@ -143,23 +155,21 @@ scenario "ui" {
     }
  
     variables {
-      awskms_unseal_key_arn = step.create_vpc.kms_key_arn
-      cluster_name          = step.create_vault_cluster_targets.cluster_name
-      config_env_vars = {
-        VAULT_LOG_LEVEL = var.vault_log_level
-      }
-      consul_cluster_tag = step.create_backend_cluster.consul_cluster_tag
+      awskms_unseal_key_arn   = step.create_vpc.kms_key_arn
+      backend_cluster_name    = step.create_vault_cluster_backend_targets.cluster_name
+      backend_cluster_tag_key = local.backend_tag_key
+      cluster_name            = step.create_vault_cluster_targets.cluster_name
       consul_release = matrix.backend == "consul" ? {
         edition = var.backend_edition
         version = local.consul_version
       } : null
+      enable_file_audit_device = var.vault_enable_file_audit_device
       install_dir              = local.vault_install_dir
       license                  = matrix.edition != "oss" ? step.read_license.license : null
       local_artifact_path      = local.bundle_path
       storage_backend          = matrix.backend
       target_hosts             = step.create_vault_cluster_targets.hosts
       unseal_method            = local.seal
-      enable_file_audit_device = var.vault_enable_file_audit_device
     }
   }
  
@@ -175,6 +185,11 @@ scenario "ui" {
     }
   }
  
+  output "audit_device_file_path" {
+    description = "The file path for the file audit device, if enabled"
+    value       = step.create_vault_cluster.audit_device_file_path
+  }
+ 
   output "awskms_unseal_key_arn" {
     description = "The Vault cluster KMS key arn"
     value       = step.create_vpc.kms_key_arn
@@ -200,11 +215,6 @@ scenario "ui" {
     value       = step.create_vault_cluster.public_ips
   }
  
-  output "root_token" {
-    description = "The Vault cluster root token"
-    value       = step.create_vault_cluster.root_token
-  }
- 
   output "recovery_key_shares" {
     description = "The Vault cluster recovery key shares"
     value       = step.create_vault_cluster.recovery_key_shares
@@ -220,14 +230,9 @@ scenario "ui" {
     value       = step.create_vault_cluster.recovery_keys_hex
   }
  
-  output "unseal_keys_b64" {
-    description = "The Vault cluster unseal keys"
-    value       = step.create_vault_cluster.unseal_keys_b64
-  }
- 
-  output "unseal_keys_hex" {
-    description = "The Vault cluster unseal keys hex"
-    value       = step.create_vault_cluster.unseal_keys_hex
+  output "root_token" {
+    description = "The Vault cluster root token"
+    value       = step.create_vault_cluster.root_token
   }
  
   output "ui_test_environment" {
@@ -245,8 +250,13 @@ scenario "ui" {
     value       = step.test_ui.ui_test_stdout
   }
  
-  output "vault_audit_device_file_path" {
-    description = "The file path for the file audit device, if enabled"
-    value       = step.create_vault_cluster.audit_device_file_path
+  output "unseal_keys_b64" {
+    description = "The Vault cluster unseal keys"
+    value       = step.create_vault_cluster.unseal_keys_b64
+  }
+ 
+  output "unseal_keys_hex" {
+    description = "The Vault cluster unseal keys hex"
+    value       = step.create_vault_cluster.unseal_keys_hex
   }
 }

@@ -0,0 +1,98 @@
+variable "ami_id" {
+  description = "The machine image identifier"
+  type        = string
+}
+ 
+variable "awskms_unseal_key_arn" {
+  type        = string
+  description = "The AWSKMS key ARN if using the awskms unseal method. If specified the instances will be granted kms permissions to the key"
+  default     = null
+}
+ 
+variable "cluster_name" {
+  type        = string
+  description = "A unique cluster identifier"
+  default     = null
+}
+ 
+variable "cluster_tag_key" {
+  type        = string
+  description = "The key name for the cluster tag"
+  default     = "TargetCluster"
+}
+ 
+variable "common_tags" {
+  description = "Common tags for cloud resources"
+  type        = map(string)
+  default = {
+    Project = "Vault"
+  }
+}
+ 
+variable "instance_mem_min" {
+  description = "The minimum amount of memory in mebibytes for each instance in the fleet. (1 MiB = 1024 bytes)"
+  type        = number
+  default     = 4096 // ~4 GB
+}
+ 
+variable "instance_mem_max" {
+  description = "The maximum amount of memory in mebibytes for each instance in the fleet. (1 MiB = 1024 bytes)"
+  type        = number
+  default     = 16385 // ~16 GB
+}
+ 
+variable "instance_cpu_min" {
+  description = "The minimum number of vCPU's for each instance in the fleet"
+  type        = number
+  default     = 2
+}
+ 
+variable "instance_cpu_max" {
+  description = "The maximum number of vCPU's for each instance in the fleet"
+  type        = number
+  default     = 8 // Unlikely we'll ever get that high due to spot price bid protection
+}
+ 
+variable "instance_count" {
+  description = "The number of target instances to create"
+  type        = number
+  default     = 3
+}
+ 
+variable "max_price" {
+  description = "The maximum hourly price to pay for each target instance"
+  type        = string
+  default     = "0.0416"
+}
+ 
+variable "project_name" {
+  description = "A unique project name"
+  type        = string
+}
+ 
+variable "ssh_allow_ips" {
+  description = "Allowlisted IP addresses for SSH access to target nodes. The IP address of the machine running Enos will automatically allowlisted"
+  type        = list(string)
+  default     = []
+}
+ 
+variable "ssh_keypair" {
+  description = "SSH keypair used to connect to EC2 instances"
+  type        = string
+}
+ 
+variable "capacity_type" {
+  description = "What capacity type to use for EC2 instances"
+  type        = string
+  default     = "on-demand"
+ 
+  validation {
+    condition     = contains(["on-demand", "spot"], var.capacity_type)
+    error_message = "The capacity_type must be either 'on-demand' or 'spot'."
+  }
+}
+ 
+variable "vpc_id" {
+  description = "The identifier of the VPC where the target instances will be created"
+  type        = string
+}

@@ -19,124 +19,97 @@ variable "vault_root_token" {
   description = "The vault root token"
 }
  
-variable "node_public_ip" {
-  type        = string
-  description = "The primary node public ip"
-  default     = ""
+variable "vault_instance_count" {
+  type        = number
+  description = "The number of instances in the vault cluster"
 }
  
-variable "vault_instances" {
+variable "vault_hosts" {
   type = map(object({
     private_ip = string
     public_ip  = string
   }))
-  description = "The vault cluster instances that were created"
-}
- 
-variable "added_vault_instances" {
-  type = map(object({
-    private_ip = string
-    public_ip  = string
-  }))
-  description = "The vault cluster instances that were added"
-  default     = {}
+  description = "The vault cluster hosts. These are required to map private ip addresses to public addresses."
 }
  
 locals {
-  leftover_primary_instances = var.node_public_ip != "" ? {
-    for k, v in var.vault_instances : k => v if contains(values(v), trimspace(var.node_public_ip))
-  } : null
-  all_instances          = var.node_public_ip != "" ? merge(var.added_vault_instances, local.leftover_primary_instances) : var.vault_instances
-  updated_instance_count = length(local.all_instances)
-  updated_instances = {
-    for idx in range(local.updated_instance_count) : idx => {
-      public_ip  = values(local.all_instances)[idx].public_ip
-      private_ip = values(local.all_instances)[idx].private_ip
+  follower_hosts_list = [for idx in range(var.vault_instance_count - 1) : {
+    private_ip = local.follower_private_ips[idx]
+    public_ip  = local.follower_public_ips[idx]
     }
-  }
-  node_ip = var.node_public_ip != "" ? var.node_public_ip : local.updated_instances[0].public_ip
-  instance_private_ips = [
-    for k, v in values(tomap(local.updated_instances)) :
-    tostring(v["private_ip"])
-  ]
-  follower_public_ips = [
-    for k, v in values(tomap(local.updated_instances)) :
-    tostring(v["public_ip"]) if v["private_ip"] != trimspace(enos_remote_exec.get_leader_private_ip.stdout)
   ]
-  follower_private_ips = [
-    for k, v in values(tomap(local.updated_instances)) :
-    tostring(v["private_ip"]) if v["private_ip"] != trimspace(enos_remote_exec.get_leader_private_ip.stdout)
+  follower_hosts = {
+    for idx in range(var.vault_instance_count - 1) : idx => try(local.follower_hosts_list[idx], null)
+  }
+  follower_private_ips = jsondecode(enos_remote_exec.get_follower_private_ips.stdout)
+  follower_public_ips = [for idx in range(var.vault_instance_count) : var.vault_hosts[idx].public_ip if contains(
+    local.follower_private_ips, var.vault_hosts[idx].private_ip)
   ]
+  leader_host = {
+    private_ip = local.leader_private_ip
+    public_ip  = local.leader_public_ip
+  }
+  leader_private_ip = trimspace(enos_remote_exec.get_leader_private_ip.stdout)
+  leader_public_ip = element([
+    for idx in range(var.vault_instance_count) : var.vault_hosts[idx].public_ip if var.vault_hosts[idx].private_ip == local.leader_private_ip
+  ], 0)
+  private_ips = [for k, v in values(tomap(var.vault_hosts)) : tostring(v["private_ip"])]
 }
  
 resource "enos_remote_exec" "get_leader_private_ip" {
   environment = {
-    VAULT_ADDR                 = "http://127.0.0.1:8200"
-    VAULT_TOKEN                = var.vault_root_token
-    VAULT_INSTALL_DIR          = var.vault_install_dir
-    VAULT_INSTANCE_PRIVATE_IPS = jsonencode(local.instance_private_ips)
+    VAULT_ADDR        = "http://127.0.0.1:8200"
+    VAULT_TOKEN       = var.vault_root_token
+    VAULT_INSTALL_DIR = var.vault_install_dir
   }
  
   scripts = [abspath("${path.module}/scripts/get-leader-private-ip.sh")]
  
   transport = {
     ssh = {
-      host = local.node_ip
+      host = var.vault_hosts[0].public_ip
     }
   }
 }
  
-output "leftover_primary_instances" {
-  value = local.leftover_primary_instances
-}
- 
-output "all_instances" {
-  value = local.all_instances
-}
- 
-output "updated_instance_count" {
-  value = local.updated_instance_count
-}
+resource "enos_remote_exec" "get_follower_private_ips" {
+  environment = {
+    VAULT_ADDR                 = "http://127.0.0.1:8200"
+    VAULT_TOKEN                = var.vault_root_token
+    VAULT_LEADER_PRIVATE_IP    = local.leader_private_ip
+    VAULT_INSTANCE_PRIVATE_IPS = jsonencode(local.private_ips)
+    VAULT_INSTALL_DIR          = var.vault_install_dir
+  }
  
-output "updated_instances" {
-  value = local.updated_instances
-}
+  scripts = [abspath("${path.module}/scripts/get-follower-private-ips.sh")]
  
-output "leader_private_ip" {
-  value = trimspace(enos_remote_exec.get_leader_private_ip.stdout)
+  transport = {
+    ssh = {
+      host = var.vault_hosts[0].public_ip
+    }
+  }
 }
  
-output "leader_public_ip" {
-  value = element([
-    for k, v in values(tomap(local.all_instances)) :
-    tostring(v["public_ip"]) if v["private_ip"] == trimspace(enos_remote_exec.get_leader_private_ip.stdout)
-  ], 0)
+output "follower_hosts" {
+  value = local.follower_hosts
 }
  
-output "vault_instance_private_ips" {
-  value = jsonencode(local.instance_private_ips)
+output "follower_private_ips" {
+  value = local.follower_private_ips
 }
  
 output "follower_public_ips" {
   value = local.follower_public_ips
 }
  
-output "follower_public_ip_1" {
-  value = element(local.follower_public_ips, 0)
-}
- 
-output "follower_public_ip_2" {
-  value = element(local.follower_public_ips, 1)
-}
- 
-output "follower_private_ips" {
-  value = local.follower_private_ips
+output "leader_host" {
+  value = local.leader_host
 }
  
-output "follower_private_ip_1" {
-  value = element(local.follower_private_ips, 0)
+output "leader_private_ip" {
+  value = local.leader_private_ip
 }
  
-output "follower_private_ip_2" {
-  value = element(local.follower_private_ips, 1)
+output "leader_public_ip" {
+  value = local.leader_public_ip
 }

@@ -9,66 +9,76 @@
  
 set -e
  
-binpath=${VAULT_INSTALL_DIR}/vault
- 
-function fail() {
-	echo "$1" 1>&2
-	exit 1
+fail() {
+  echo "$1" 1>&2
+  exit 1
 }
  
+[[ -z "$PRIMARY_LEADER_PRIV_IP" ]] && fail "PRIMARY_LEADER_PRIV_IP env variable has not been set"
+[[ -z "$SECONDARY_LEADER_PRIV_IP" ]] && fail "SECONDARY_LEADER_PRIV_IP env variable has not been set"
+[[ -z "$VAULT_ADDR" ]] && fail "VAULT_ADDR env variable has not been set"
+[[ -z "$VAULT_INSTALL_DIR" ]] && fail "VAULT_INSTALL_DIR env variable has not been set"
+ 
+binpath=${VAULT_INSTALL_DIR}/vault
+test -x "$binpath" || fail "unable to locate vault binary at $binpath"
+ 
 retry() {
   local retries=$1
   shift
   local count=0
  
   until "$@"; do
-    exit=$?
     wait=$((2 ** count))
     count=$((count + 1))
     if [ "$count" -lt "$retries" ]; then
       sleep "$wait"
     else
-      return "$exit"
+      fail "$($binpath read -format=json sys/replication/performance/status)"
     fi
   done
 }
  
-test -x "$binpath" || exit 1
- 
 check_pr_status() {
   pr_status=$($binpath read -format=json sys/replication/performance/status)
-  cluster_state=$(echo $pr_status | jq -r '.data.state')
-  connection_mode=$(echo $pr_status | jq -r '.data.mode')
+  cluster_state=$(echo "$pr_status" | jq -r '.data.state')
+  connection_mode=$(echo "$pr_status" | jq -r '.data.mode')
  
   if [[ "$cluster_state" == 'idle' ]]; then
-    fail "replication cluster state is $cluster_state"
+    echo "replication cluster state is idle" 1>&2
+    return 1
   fi
  
   if [[ "$connection_mode" == "primary" ]]; then
-    connection_status=$(echo $pr_status | jq -r '.data.secondaries[0].connection_status')
+    connection_status=$(echo "$pr_status" | jq -r '.data.secondaries[0].connection_status')
     if [[ "$connection_status" == 'disconnected' ]]; then
-      fail "replication connection status of secondaries is $connection_status"
+      echo ".data.secondaries[0].connection_status from primary node is 'disconnected'" 1>&2
+      return 1
     fi
-    secondary_cluster_addr=$(echo $pr_status | jq -r '.data.secondaries[0].cluster_address')
-    if [[ "$secondary_cluster_addr" != "https://"${SECONDARY_LEADER_PRIV_IP}":8201" ]]; then
-      fail "Expected secondary cluster address $SECONDARY_LEADER_PRIV_IP got  $secondary_cluster_addr "
+    secondary_cluster_addr=$(echo "$pr_status" | jq -r '.data.secondaries[0].cluster_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")')
+    if [[ "$secondary_cluster_addr" != "$SECONDARY_LEADER_PRIV_IP" ]]; then
+      echo ".data.secondaries[0].cluster_address should have an IP address of $SECONDARY_LEADER_PRIV_IP, got: $secondary_cluster_addr" 1>&2
+      return 1
     fi
   else
-    connection_status=$(echo $pr_status | jq -r '.data.primaries[0].connection_status')
+    connection_status=$(echo "$pr_status" | jq -r '.data.primaries[0].connection_status')
     if [[ "$connection_status" == 'disconnected' ]]; then
-      fail "replication connection status of secondaries is $connection_status"
+      echo ".data.primaries[0].connection_status from secondary node is 'disconnected'" 1>&2
+      return 1
     fi
-    primary_cluster_addr=$(echo $pr_status | jq -r '.data.primaries[0].cluster_address')
-    if [[ "$primary_cluster_addr" != "https://"${PRIMARY_LEADER_PRIV_IP}":8201" ]]; then
-      fail "Expected primary cluster address $PRIMARY_LEADER_PRIV_IP got  $primary_cluster_addr"
+    primary_cluster_addr=$(echo "$pr_status" | jq -r '.data.primaries[0].cluster_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")')
+    if [[ "$primary_cluster_addr" != "$PRIMARY_LEADER_PRIV_IP" ]]; then
+      echo ".data.primaries[0].cluster_address should have an IP address of $PRIMARY_LEADER_PRIV_IP, got: $primary_cluster_addr" 1>&2
+      return 1
     fi
-    known_primary_cluster_addrs=$(echo $pr_status | jq -r '.data.known_primary_cluster_addrs')
-    # IFS="," read -a cluster_addr <<< ${known_primary_cluster_addrs}
-    if ! $(echo $known_primary_cluster_addrs |grep -q $PRIMARY_LEADER_PRIV_IP); then
-      fail "Primary leader address $PRIMARY_LEADER_PRIV_IP not found in Known primary cluster addresses $known_primary_cluster_addrs"
+    known_primary_cluster_addrs=$(echo "$pr_status" | jq -r '.data.known_primary_cluster_addrs')
+    if ! echo "$known_primary_cluster_addrs" | grep -q "$PRIMARY_LEADER_PRIV_IP"; then
+      echo "$PRIMARY_LEADER_PRIV_IP is not in .data.known_primary_cluster_addrs: $known_primary_cluster_addrs" 1>&2
+      return 1
     fi
   fi
-  echo $pr_status
+ 
+  echo "$pr_status"
+  return 0
 }
  
 # Retry a few times because it can take some time for replication to sync

@@ -5,31 +5,42 @@
  
 set -e
  
-binpath=${VAULT_INSTALL_DIR}/vault
-instance_ips=${VAULT_INSTANCE_PRIVATE_IPS}
- 
 function fail() {
-	echo "$1" 1>&2
-	exit 1
+  echo "$1" 1>&2
+  exit 1
 }
  
+[[ -z "$VAULT_ADDR" ]] && fail "VAULT_ADDR env variable has not been set"
+[[ -z "$VAULT_INSTALL_DIR" ]] && fail "VAULT_INSTALL_DIR env variable has not been set"
+[[ -z "$VAULT_TOKEN" ]] && fail "VAULT_TOKEN env variable has not been set"
+ 
+binpath=${VAULT_INSTALL_DIR}/vault
+test -x "$binpath" || fail "Unable to locate vault binary at $binpath"
+ 
 count=0
 retries=5
 while :; do
-    # Find the leader private IP address
-    leader_private_ip=$($binpath status -format json | jq '.leader_address | rtrimstr(":8200") | ltrimstr("http://")')
-    match_ip=$(echo "$instance_ips" |jq -r --argjson ip "$leader_private_ip" 'map(select(. == $ip))')
- 
-    if [[ "$leader_private_ip" != 'null' ]] && [[ "$match_ip" != '[]' ]]; then
-        echo "$leader_private_ip" | sed 's/\"//g'
-        exit 0
+  # Find the leader private IP address
+  if ip=$($binpath read sys/leader -format=json | jq -r '.data.leader_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")'); then
+    if [[ -n "$ip" ]]; then
+      echo "$ip"
+      exit 0
     fi
+  fi
  
-    wait=$((5 ** count))
-    count=$((count + 1))
-    if [ "$count" -lt "$retries" ]; then
-        sleep "$wait"
-    else
-        fail "leader IP address $leader_private_ip was not found in $instance_ips"
+  # Some older versions of vault don't support reading sys/leader. Try falling back to the cli status.
+  if ip=$($binpath status -format json | jq -r '.leader_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")'); then
+    if [[ -n "$ip" ]]; then
+      echo "$ip"
+      exit 0
     fi
+  fi
+ 
+  wait=$((2 ** count))
+  count=$((count + 1))
+  if [ "$count" -lt "$retries" ]; then
+    sleep "$wait"
+  else
+    fail "Timed out trying to obtain the cluster leader"
+  fi
 done

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: BUSL-1.1
+ 
+ 
+set -e
+ 
+fail() {
+  echo "$1" 1>&2
+  exit 1
+}
+ 
+[[ -z "$RETRY_INTERVAL" ]] && fail "RETRY_INTERVAL env variable has not been set"
+[[ -z "$TIMEOUT_SECONDS" ]] && fail "TIMEOUT_SECONDS env variable has not been set"
+[[ -z "$VAULT_ADDR" ]] && fail "VAULT_ADDR env variable has not been set"
+[[ -z "$VAULT_INSTALL_DIR" ]] && fail "VAULT_INSTALL_DIR env variable has not been set"
+[[ -z "$VAULT_INSTANCE_PRIVATE_IPS" ]] && fail "VAULT_INSTANCE_PRIVATE_IPS env variable has not been set"
+[[ -z "$VAULT_TOKEN" ]] && fail "VAULT_TOKEN env variable has not been set"
+ 
+binpath=${VAULT_INSTALL_DIR}/vault
+test -x "$binpath" || fail "unable to locate vault binary at $binpath"
+ 
+findLeaderInPrivateIPs() {
+  # Find the leader private IP address
+  local leader_private_ip
+  if ! leader_private_ip=$($binpath read sys/leader -format=json | jq -r '.data.leader_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")') ; then
+    # Some older versions of vault don't support reading sys/leader. Fallback to the cli status.
+    if leader_private_ip=$($binpath status -format json | jq '.leader_address | scan("[0-9]+.[0-9]+.[0-9]+.[0-9]+")'); then
+      return 1
+    fi
+  fi
+ 
+  if isIn=$(jq -r --arg ip "$leader_private_ip" 'map(select(. == $ip)) | length == 1' <<< "$VAULT_INSTANCE_PRIVATE_IPS"); then
+    if [[ "$isIn" == "true" ]]; then
+      echo "$leader_private_ip"
+      return 0
+    fi
+  fi
+ 
+  return 1
+}
+ 
+begin_time=$(date +%s)
+end_time=$((begin_time + TIMEOUT_SECONDS))
+while [ "$(date +%s)" -lt "$end_time" ]; do
+  if findLeaderInPrivateIPs; then
+    exit 0
+  fi
+ 
+  sleep "$RETRY_INTERVAL"
+done
+ 
+fail "Timed out waiting for one of $VAULT_INSTANCE_PRIVATE_IPS to be leader."
 <82990506+hc-github-team-secure-vault-core@users.noreply.github.com>
 d7e4447ec054ad9b8ce2dbdad6111f1587b325f8 (#23319)
 .../docs/commands/pki/health-check.mdx        | 64 ++++++++++---------
 website/data/docs-nav-data.json               | 58 ++++++++---------
 2 files changed, 64 insertions(+), 58 deletions(-)

@@ -0,0 +1,440 @@
+---
+layout: api
+page_title: SAML - Auth Methods - HTTP API
+description: |-
+  This is the API documentation for the Vault SAML auth method.
+---
+ 
+# SAML auth method (API)
+ 
+<EnterpriseAlert />
+ 
+This is the API documentation for the Vault SAML auth method. To learn more about the
+usage and operation, see the [Vault SAML auth method documentation](/vault/docs/auth/saml).
+ 
+This documentation assumes the SAML auth method is mounted at the `/auth/saml` path in
+Vault. Since it is possible to enable auth methods at any location, please update your
+API calls accordingly.
+ 
+## Create or update configuration
+ 
+Configures the auth method with a SAML identity provider.
+ 
+| Method     | Path                |
+|:-----------| :------------------ |
+| `POST/PUT` | `/auth/saml/config` |
+ 
+### Parameters
+ 
+- `idp_metadata_url` `(string, <required>)` - The metadata URL of the identity provider.
+  Mutually exclusive with `idp_sso_url`, `idp_issuer` and `idp_cert`. Must be a
+  well-formatted URL.
+- `idp_sso_url` `(string, <required if idp_metadata_url is not set>)` - The SSO URL of the
+  identity provider. Mutually exclusive with `idp_metadata_url`. Must be a
+  well-formatted URL.
+- `idp_entity_id` `(string, <required if idp_metadata_url is not set>)` - The entity ID of
+  the identity provider. Mutually exclusive with `idp_metadata_url`.
+- `idp_cert` `(string, <required if idp_metadata_url is not set>)` - The PEM-encoded
+  certificate of the identity provider used to verify response and assertion signatures.
+  Mutually exclusive with `idp_metadata_url`.
+- `entity_id` `(string, <required>)` - The entity ID of the SAML authentication
+  service provider. Must match entity ID configured for the application in the
+  SAML identity provider.
+- `acs_urls` `(list, <required>)` - The well-formated URLs of your Assertion
+  Consumer Service (ACS) that should receive a response from the identity
+  provider. Vault returns a security warning if any of the given URLs lack TLS
+  protection.
+- `default_role` `(string, <optional>)` - The role to use if no role is provided during login.
+  If not set, a role is required during login.
+ 
+### Sample payload
+ 
+```json
+{
+  "acs_urls": "https://my.vault/v1/auth/saml/callback",
+  "default_role": "admin",
+  "entity_id": "https://my.vault/v1/auth/saml",
+  "idp_metadata_url": "https://company.okta.com/app/abc123eb9xnIfzlaf697/sso/saml/metadata"
+}
+```
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request PUT \
+    --data @payload.json \
+    http://127.0.0.1:8200/v1/auth/saml/config
+```
+ 
+## Read configuration
+ 
+Reads the auth method configuration.
+ 
+| Method  | Path                |
+| :------ | :------------------ |
+| `GET`   | `/auth/saml/config` |
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request GET \
+    --data @payload.json \
+    http://127.0.0.1:8200/v1/auth/saml/config
+```
+ 
+### Sample response
+ 
+```json
+{
+  "request_id": "09c907d2-2dbe-8a5c-ca97-fad83195738b",
+  "lease_id": "",
+  "lease_duration": 0,
+  "renewable": false,
+  "data": {
+    "acs_urls": [
+      "https://my.vault/v1/auth/saml/callback"
+    ],
+    "default_role": "admin",
+    "entity_id": "https://my.vault/v1/auth/saml",
+    "idp_metadata_url": "https://company.okta.com/app/abc123eb9xnIfzlaf697/sso/saml/metadata"
+  },
+  "warnings": null
+}
+```
+ 
+## Create or update role
+ 
+Configures a role in the auth method. Roles define specific constraints required for
+authentication and properties of resulting Vault tokens.
+ 
+| Method     | Path                     |
+|:-----------|:-------------------------|
+| `POST/PUT` | `/auth/saml/role/:name`  |
+ 
+### Parameters
+ 
+- `name` `(string: <required>)` - URL parameter that provides the name of the role to create.
+- `bound_subjects` `(string: <optional>)` - The subject being asserted for SAML
+  authentication. One of the provided values must match the subject returned in
+  the SAML assertion from the identity provider.
+- `bound_subjects_type` `(string: <optional>)` - The type of matching assertion to perform
+  on `bound_subjects`. If `string`, requires a direct string match. If `glob`, allows for
+  wildcard matching using the `*` character.
+- `bound_attributes` `(map: <optional>)` - Mapping of attribute names to values that are
+  expected to exist in the SAML assertion. The expected value may be a single string or a
+  comma-separated list of strings. The user will be authenticated if the SAML attributes
+  match at least one of the expected values.
+- `bound_attributes_type` `(string: "string")` - The type of matching assertion to perform
+  on the key-value pairs provided by `bound_attributes`. If set to `string`, a direct string
+  match is required. If set to `glob`, allows for wildcard matching using the `*` character.
+- `groups_attribute` `(string: <optional>)` - The attribute to use to identify the set of
+  groups to which the user belongs. This will be used as the names for the Identity group
+  aliases created due to a successful login.
+ 
+@include 'tokenfields.mdx'
+ 
+### Sample payload
+ 
+```json
+{
+  "bound_attributes": "group=admin",
+  "bound_subjects": "*@hashicorp.com",
+  "bound_subjects_type": "glob",
+  "token_policies": "writer",
+  "ttl": "1h"
+}
+```
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request PUT \
+    --data @payload.json \
+    http://127.0.0.1:8200/v1/auth/saml/role/admin
+```
+ 
+## Read role
+ 
+Reads a configured role.
+ 
+| Method | Path                    |
+| :----- |:------------------------|
+| `GET`  | `/auth/saml/role/:name` |
+ 
+### Parameters
+ 
+- `name` `(string: <required>)` - URL parameter that provides the name of the
+  role to read.
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request GET \
+    http://127.0.0.1:8200/v1/auth/saml/role/admin
+```
+ 
+### Sample response
+ 
+```json
+{
+  "request_id": "3148ca9a-286e-a0a4-5a4b-31b6abb63d37",
+  "lease_id": "",
+  "lease_duration": 0,
+  "renewable": false,
+  "data": {
+    "bound_attributes": {
+      "group": [
+        "admin"
+      ]
+    },
+    "bound_attributes_type": "string",
+    "bound_subjects": [
+      "*@hashicorp.com"
+    ],
+    "bound_subjects_type": "glob",
+    "groups_attribute": "",
+    "token_bound_cidrs": [],
+    "token_explicit_max_ttl": 0,
+    "token_max_ttl": 0,
+    "token_no_default_policy": false,
+    "token_num_uses": 0,
+    "token_period": 0,
+    "token_policies": [
+      "writer"
+    ],
+    "token_ttl": 0,
+    "token_type": "default"
+  },
+  "warnings": null
+}
+```
+ 
+## List roles
+ 
+Lists all the configured roles.
+ 
+| Method | Path              |
+| :----- | :---------------- |
+| `LIST` | `/auth/saml/role` |
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request GET \
+    http://127.0.0.1:8200/v1/auth/saml/role?list=true
+```
+ 
+### Sample response
+ 
+```json
+[
+  "admin",
+  "operations"
+]
+```
+ 
+## Delete Role
+ 
+Deletes a configured role.
+ 
+| Method   | Path                    |
+| :------- | :---------------------- |
+| `DELETE` | `/auth/saml/role/:name` |
+ 
+### Parameters
+ 
+- `name` `(string: <required>)` - URL parameter that provides the name of the
+  role to delete.
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request DELETE \
+    http://127.0.0.1:8200/v1/auth/saml/role/admin
+```
+ 
+## Obtain SSO service URL
+ 
+Starts a login flow by providing a SAML Single Sign-On (SSO) Service URL for the
+configured identity provider. The returned `token_poll_id` can be used to obtain
+the Vault token after the user is authenticated with the identity provider and the
+SAML response has passed validation.
+ 
+<Note title="Unauthenticated">
+A Vault token is not required to interact with this API.
+</Note>
+ 
+| Method  | Path                         |
+|:--------|:-----------------------------|
+| `POST`  | `/auth/saml/sso_service_url` |
+ 
+### Parameters
+ 
+- `role` `(string, <optional>)` - The role name to use for the login flow.
+  Defaults to the role configured with `default_role`.
+- `client_challenge` `(string, <required>)` - The client challenge value. Must be the
+  output of a base64-encoded, sha256 digest of the `client_verifier` eventually provided
+  to the [Token API](/vault/api-docs/auth/saml#obtain-vault-token). Must be at least 44
+  bytes in length.
+- `client_type` `(string, <required>)` - The type of the requesting client. The response
+  from the Assertion Consumer Service [Callback API](/vault/api-docs/auth/saml#assertion-consumer-service-callback)
+  will differ based on the provided type. If `cli`, an HTML success page will be returned
+  in the response. If `browser`, a blank HTML page will be returned in the response.
+- `acs_url` `(string, <required>)` - The URL where the identity provider will send its
+  SAML response. Must be in the set of configured [`acs_urls`](/vault/api-docs/auth/saml#acs_urls).
+ 
+### Sample payload
+ 
+```json
+{
+  "acs_url": "https://my.vault/v1/auth/saml/callback",
+  "client_challenge": "Z6+7owP80d1aHTha1kdixtT99JkvmG4TPSgbvDwZ70A=",
+  "client_type": "cli",
+  "role": "admin"
+}
+```
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request POST \
+    --data @payload.json \
+    http://127.0.0.1:8200/v1/auth/saml/sso_service_url
+```
+ 
+### Sample response
+ 
+```json
+{
+  "sso_service_url": "https://example.okta.com/app/abc123eb9xnIfzlaf697/id/sso/saml?RelayState=...&SAMLRequest=...",
+  "token_poll_id": "ee442348-159b-df10-4c59-63050069df4d"
+}
+```
+ 
+## Assertion consumer service callback
+ 
+The assertion consumer service URL of the auth method. Completes the round trip from
+the identity provider and performs validations on the SAML response.
+ 
+<Note title="Unauthenticated">
+A Vault token is not required to interact with this API.
+</Note>
+ 
+| Method  | Path                  |
+|:--------|:----------------------|
+| `POST`  | `/auth/saml/callback` |
+ 
+### Parameters
+ 
+- `RelayState` `(string, <required>)` - The relay state from the original SAML
+  authentication request that was returned by the identity provider.
+- `SAMLResponse` `(string, <required>)` - The signed SAML response from the identity
+  provider.
+ 
+### Sample payload
+ 
+```json
+{
+  "RelayState": "0afe62a9-7b83-a182-0650-c749badfb900",
+  "SAMLResponse": "..."
+}
+```
+ 
+### Sample request
+ 
+```shell-session
+$ curl \
+    --header "X-Vault-Token: ..." \
+    --request POST \
+    --data @payload.json \
+    http://127.0.0.1:8200/v1/auth/saml/callback
+```
+ 
+## Obtain vault token
+ 
+The token endpoint completes the login flow by returning a Vault token.
+ 
+<Note title="Unauthenticated">
+A Vault token is not required to interact with this API.
+</Note>
+ 
+| Method  | Path               |
+|:--------|:-------------------|
+| `POST`  | `/auth/saml/token` |
+ 
+### Parameters
+ 
+- `client_verifier` `(string, <required>)` - The value which produced the `client_challenge`
+  provided to the [SSO Service URL API](/vault/api-docs/auth/saml#obtain-sso-service-url)
+  at the start of the authentication flow. Its base64-encoded, sha256 digest must match the
+  `client_challenge` value.
+- `token_poll_id` `(string, <required>)` - The `token_poll_id` value returned from the
+  [SSO Service URL API](/vault/api-docs/auth/saml#obtain-sso-service-url) at the start of
+  the authentication flow.
+ 
+### Sample payload
+ 
+```json
+{
+  "client_verifier": "59634224-5869-6002-e0b1-35370b8f6b82",
+  "token_poll_id": "ee442348-159b-df10-4c59-63050069df4d"
+}
+```
+ 
+### Sample request
+ 
… diff truncated