1 - Zero Trust Security

Zero Trust Security with Service Mesh Manager

Service Mesh Manager brings the zero trust security to modern apps. The zero trust security model is based on the following points:

  • don’t trust any system, not even the ones within your network
  • verify the identity, authorization, and tool that the user or service uses before establishing trust,
  • grant only the minimal access needed for the task or functionality.

Service Mesh Manager gives you the following security controls to your service mesh to achieve zero trust security:

Control ingress and egress traffic at the edge

To protect your applications and your sensitive data, you need to have a control over the incoming traffic from external sources. Service Mesh Manager allows you to:

  • create and manage ingress and egress gateways,
  • apply rate-limiting to mitigate denial-of-service attacks,
  • encrypt and mutually authenticate traffic on the gateway,
  • permit egress traffic only to specific endpoints to prevent data exfiltration.

Authenticate, authorize, and encrypt all connections

To make zero trust security work, every connection should be authenticated, authorizes, and encrypted. Service Mesh Manager uses mTLS encryption within the mesh to protect data-in-motion on all service-to-service connections, and can do the same for all ingress and egress traffic. If you need to create a FIPS-compliant service mesh, you can use the FIPS-compliant builds of Service Mesh Manager and Istio that use only FIPS 140-2 compliant cipher suites.

For fine-grained authorization control, you can use Istio Authorization Policies to set access control rules on workloads in the mesh. You can also restrict outbound traffic for workloads and namespaces, manually or automatically.

Health monitoring, logging, and tracing

For maintenance and security monitoring, Service Mesh Manager allows you to configure health monitoring and alerts for your services and workloads to quickly and easily notice issues that might cause a service outage. In addition, Service Mesh Manager gives you quick access to pod logs, traces, and Prometheus metrics.

2 - Authentication

Service Mesh Manager leverages the Kubeconfig, the official client libraries and the Kubernetes API to perform authentication and authorization for its users.

If you’re allowed to add, edit or delete specific Istio custom resources, you’ll have the same permissions from Service Mesh Manager as well.

Authentication overview

The authentication flow consists of the following steps:

  • The CLI extracts authentication credentials from the user’s Kubeconfig the same way kubectl would do
  • The CLI sends these credentials (client certificate or bearer token) to Service Mesh Manager during the login process
  • Service Mesh Manager validates these credentials against the Kubernetes API Server Service Mesh Manager doesn’t store these credentials afterwards)
  • Once the credentials are proved to be valid Service Mesh Manager generates it’s own ID token (JWT) and encodes relevant user information in it
  • The user - in possession of the ID token - can then use the token to authenticate against Service Mesh Manager until it expires
  • Service Mesh Manager will send subsequent requests to the API server with impersonation headers set to the user’s name and groups to delegate Authorization entirely to Kubernetes

Test authentication

Dashboard access

smm dashboard

When you open the dashboard through the recommended way of typing smm dashboard, you’re seamlessly authenticated with your Kubeconfig, logged in automatically and redirected to a browser tab with the Service Mesh Manager Dashboard open.

Login

smm login

You can explicitly log in any time using the smm login command, which gives you a short lifetime (10s), encrypted token to use over the UI login window.

Troubleshooting

The ID token will be saved to the current context’s config to reuse for subsequent CLI commands for efficiency. You can check or edit this config any time using the smm config get, smm config edit commands respectively.

Once the token expires (10h) the CLI performs a new login automatically within the next command.

If the token seems to be invalid for any reason you can always reauthenticate with the smm login command.

Anonymous mode

Service Mesh Manager provides a way to disable user authentication and use its own service account token for all communication with the Kubernetes API server.

Use the --anonymous-auth flag of the install command to disable authentication.

smm install --anonymous-auth

3 - Mutual TLS

TLS authentication overview

Istio offers mutual TLS as a solution for service-to-service authentication.

Note: For FIPS-compliant TLS settings, see FIPS-compliant service mesh.

Istio uses the sidecar pattern, meaning that each application container has a sidecar Envoy proxy container running beside it in the same pod.

  1. When a service receives or sends network traffic, the traffic always goes through the Envoy proxies first.
  2. When mTLS is enabled between two services, the client side and server side Envoy proxies verify each other’s identities before sending requests.
  3. If the verification is successful, then the client-side proxy encrypts the traffic, and sends it to the server-side proxy.
  4. The server-side proxy decrypts the traffic and forwards it locally to the actual destination service.

isito-mtls isito-mtls

In Service Mesh Manager, you can manage the mTLS settings:

Change service-specific mTLS settings using the UI

To configure service-specific mTLS settings using the UI, complete the following steps. You can change the mTLS settings of a namespace or the entire service mesh from the command line.

  1. Select the service on the MENU > TOPOLOGY or MENU > SERVICES page.

  2. Select MTLS POLICIES. You can configure the MTLS policy you want to use independently for each port of the service. The following policies are available:

    • STRICT: The service can accept only mutual TLS traffic.
    • PERMISSIVE: The service can accept both plaintext/unencrypted traffic and mutual TLS traffic at the same time.
    • DISABLED: The service can accept plaintext/unencrypted traffic only.
    • DEFAULT: Use the global MTLS policy.

    mtls-set mtls-set

  3. Select APPLY CHANGES.

  4. When a load is sent to the service, you can verify whether the traffic between your services is actually encrypted or not on the MENU > TOPOLOGY page by selecting EDGE LABELS > security.

    Either red open locks or green closed ones are displayed between the services in the UI, indicating non-encrypted or encrypted traffic between the services.

Change mTLS settings using PeerAuthentication

You can change the mTLS settings of a workload, namespace, or the entire service mesh from the command line using the PeerAuthentication custom resource. For example, the following CR disables mTLS for the “catalog” service.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: catalog
  namespace: smm-demo
spec:
  mtls:
    mode: DISABLE

For other examples, see the PeerAuthentication documentation.

4 - Authorization

5 - FIPS-compliant service mesh

Note: The default version of Service Mesh Manager is built with the standard SSL libraries. To use a FIPS-compliant version of Istio, see Install FIPS images.

FIPS compliance overview

The Istio distribution of Service Mesh Manager provides compliance with the rules for cryptographic modules of FIPS 140-2 Security Level 1. To achieve this, Service Mesh Manager provides the following measures:

  • Service Mesh Manager is built using a FIPS-compliant library (BoringCrypto).
  • Envoy is built with the same FIPS-compliant library (BoringCrypto).
  • Service Mesh Manager delivers a custom Istio build, using the same FIPS-compliant library (BoringCrypto).
  • For certificate management, Service Mesh Manager uses a version of cert-manager built with the same FIPS-compliant library (BoringCrypto).
  • Service Mesh Manager is tested with FIPS 140-2 compliant cipher suites (and rejects anything else).
  • Although FIPS 140 allows other ciphers, Service Mesh Manager only GCM ciphers are enabled, because only those can prevent the SSL LUCKY13 timing attack.
  • BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs. BoringSSL as a whole is not FIPS validated. However, there is a core library (called BoringCrypto) that has been FIPS validated.

FIPS 140-2 compliant Service Mesh Manager TLS settings

Allowed TLS versions

  • TLS v1.2
  • TLS v1.3

Although FIPS 140-2 would allow lower TLS versions under some circumstances, we disabled them for security reasons. TLS 1.0 and 1.1 are out-of-date protocols that do not support modern cryptographic algorithms, and they contain security vulnerabilities that may be exploited by attackers. The IETF is also planning to officially deprecate both protocols. In addition, the vast majority of encrypted Internet traffic is now over TLS 1.2, which was introduced over a decade ago.

Allowed FIPS compatible ciphers

  • ECDHE-RSA-AES128-GCM-SHA256
  • ECDHE-RSA-AES256-GCM-SHA384
  • ECDHE-ECDSA-AES128-GCM-SHA256
  • ECDHE-ECDSA-AES256-GCM-SHA384
  • AES128-GCM-SHA256
  • AES256-GCM-SHA384

There are more ciphers allowed by FIPS 140-2. We only enable GCM ciphers, because only those ciphers can prevent a LUCKY13 timing attack

Allowed Elliptic-curve algorithm

  • P-256

Installation

To install a FIPS-compliant version of Istio, see Install FIPS images.

6 - Open Port Inventory

A Service Mesh Manager installation requires the following open service ports. Opening only the required ports helps to keep deployment’s attack surface is the as small as possible. Each service is described in YAML format, with the list of all the ports how the service uses them. This helps to understand the risks associated with all the open ports.

Every service is described in a YAML file using the following format:

namespace: smm-system
name: mesh-manager
description: is used by our mesh-manager instance which manages istio operators on a kubernetes cluster
ports:
  - name: https
    number: 443
    use: handle https traffic and queries with tls/ssl

Useful commands

The following commands help you examine the services of your Service Mesh Manager deployment.

List services under smm-system namespace:

kubectl get services -n smm-system

Inspect a particular service, for example, smm-leo:

kubectl describe service smm-leo -n smm-system

Services (namespace-scoped)

cert-manager

The cert-manager namespace contains the following services.

cert-manager

The cert-manager instance of Service Mesh Manager provides Kubernetes certificate management as part of the controller-component. The service uses these ports:

  • tcp-prometheus-servicemonitor (9402): The default port for the cert-manager service.

cert-manager-webhook

The cert-manager instance of Service Mesh Manager uses the service as part of the webhook-component. The service uses these ports:

  • https (443): Incoming http traffic for the webhooks.

istio-system

The istio-system namespace contains the following services.

istio-meshexpansion-cp-v115x

Service Mesh Manager provides multicluster expansion capabilities using this service, which is used for version 1.10 istio mesh-gateway expansion. The service uses these ports:

  • tcp-status-port (15021): Health check (readiness-probe) port for Mesh Overview functionality and troubleshooting istio related issues.

  • tls-istiod (15012): Handles incoming gRPC traffic for accessing to istiod from passive clusters.

  • tls-istiodwebhook (15017): Handles incoming gRPC traffic for the istiod webhooks.

  • tls (15443): Handles incoming TLS traffic from other clusters.

  • tcp-smm-als-tls (50600): Handles incoming TLS traffic to access log services.

  • tcp-smm-zipkin-tls (59411): Default Zipkin port that handles incoming HTTPS traffic for distributed tracing services.

istio-meshexpansion-cp-v115x-external

Service Mesh Manager provides multicluster expansion capabilities using this external service, which is used for version 1.10 istio mesh-gateway expansion. The service uses these ports:

  • tcp-status-port (15021): Health check (readiness-probe) port for Mesh Overview functionality and troubleshooting istio related issues.

  • tls-istiod (15012): Handles incoming gRPC traffic for accessing to istiod from passive clusters.

  • tls-istiodwebhook (15017): Handles incoming gRPC traffic for the istiod webhooks.

  • tls (15443): Handles incoming TLS traffic from other clusters.

  • tcp-smm-als-tls (50600): Handles incoming TLS traffic to access log services.

  • tcp-smm-zipkin-tls (59411): Default Zipkin port that handles incoming HTTPS traffic for distributed tracing services.

istiod-cp-v115x

Used by the Istio 1.10 control plane. The service uses these ports:

  • grpc-xds (15010): Handles gRPC traffic for xds transport protocol, which is used for Envoy discovery services and Istio proxies.

  • https-dns (15012): Handles DNS requests (with TLS) for the Istio service-mesh.

  • https-webhook (443): Handles incoming HTTPS traffic (with TLS) for Istio webhook management.

  • http-monitoring (15014): Handles HTTP requests or queries for monitoring of the traffic management between microservices.

smm-system

The smm-system namespace contains the following services.

istio-operator-v113x

Istio-operator version 1.13, which is the controller of Istio resources, uses this service to manage Istio 1.13 deployments. The service uses these ports:

  • https (443): Handles the incoming HTTPS traffic for the Istio webhook.

istio-operator-v113x-authproxy

RBAC-authenticated endpoints use this service for istio-operator version 1.13. The service uses these ports:

  • https (8443): Authenticated (RBAC or TLS) endpoint for providing access to Prometheus metrics.

istio-operator-v115x

Istio-operator version 1.15, which is the controller of Istio resources, uses this service to manage Istio 1.15 deployments. The service uses these ports:

  • https (443): Handles the incoming HTTPS traffic for the Istio webhook.

istio-operator-v115x-authproxy

RBAC-authenticated endpoints use this service for istio-operator version 1.15. The service uses these ports:

  • https (8443): Authenticated (RBAC or TLS) endpoint for providing access to Prometheus metrics.

mesh-manager

The mesh-manager instance of Service Mesh Manager that manages Istio operators on a Kubernetes cluster uses this service. The service uses these ports:

  • https (443): Handles HTTPS traffic and queries with TLS/SSL.

mesh-manager-authproxy

The mesh-manager instance of Service Mesh Manager acts as an authentication proxy to the mesh-manager service. The service uses these ports:

  • https (8443): Authenticated (RBAC or TLS) endpoint for providing access to Prometheus metrics.

prometheus-node-exporter

The Prometheus instances use this service as an exporter for Kubernetes nodes. The service uses these ports:

  • metrics (19101): Exposes node-level metrics to Prometheus.

prometheus-operated

Used by the Prometheus instances of Service Mesh Manager. The service uses these ports:

  • http (9090): Handles normal HTTP traffic and Prometheus queries.

  • grpc (10901): Default port to handle incoming gRPC traffic.

smm

The Service Mesh Manager uses this service as part of the application-component. The service uses these ports:

  • http (80): Handles GraphQL API traffic and queries.

  • http-metrics (10000): Exports metrics to the Prometheus service.

smm-als

The Service Mesh Manager uses the smm-als service as part of the als-component. The service uses these ports:

  • grpc-als (50600): The container port of the grpc-als container, used for accessing log services.

smm-authentication

The smm-authentication instance verifies that a user has a valid token or certificate to make API calls to the backend service. The service uses these ports:

  • http (80): Handles HTTP traffic and GraphQL queries.

smm-expansion-gw

This is an external service of the mesh expansion gw of the local cluster. It is synced with the cluster registry to provide reachability of smm from the peer clusters even in case of multiple active istio controlplanes. The service uses these ports:

smm-federation-gateway

The smm-federation-gateway instance provides federation to GraphQL services via this service. The service uses these ports:

  • http (80): Handles HTTP traffic and GraphQL queries.

smm-grafana

The Service Mesh Manager uses the Grafana dashboard monitoring service as part of the grafana-component. The service uses these ports:

  • http (3000): Exposes GraphQL web interface and API endpoints over HTTP.

smm-health

is used by our for smm-health instance which includes health controller and exporter and is part of health-component The service uses these ports:

  • http-metrics (8080): export metrics to prometheus service

smm-health-api

The smm-health-api instance uses the service for the GraphQL API. The service uses these ports:

  • http-graphql (80): Handles HTTP traffic and GraphQL queries.

smm-ingressgateway

The service is part of the ingressgateway-component which acts as a loadbalancer, receiving incoming HTTP or TCP connections. The service uses these ports:

  • http2 (80): Handles traffic to the Service Mesh Manager dashboard.

smm-jaeger-agent

Jaeger agents can send traces to the Jaeger collector. The service is part of the tracing-component. The service uses these ports:

  • udp-agent-zipkin (5775): Default UDP port for zipkin-thrift tracing services.

  • udp-agent-compact (6831): Default UDP port for the Jaeger agent endpoint.

  • udp-agent-binary (6832): Default UDP port for the Jaeger agent binary protocol.

smm-jaeger-collector

The Jaeger collector receives traces from Jaeger agents via this service. It is part of the tracing-component. The service uses these ports:

  • tcp-smm-jaeger-collector-tchannel (14267): Default TCP port of tchannel for Jaeger collector.

  • http-jaeger-collector (14268): Default port of Jaeger collector to handle HTTP traffic.

smm-jaeger-query

Service Mesh Manager uses this service to access Jaeger. It is part of the tracing-component. The service uses these ports:

  • http-query (16686): Handles normal HTTP traffic and queries for the tracing services.

smm-kubelet-node-discovery

kubelet uses this service to export metrics. The service uses these ports:

  • https-metrics (10250): kubelet-service default port, used for exporting metrics to Prometheus services over TLS/SSL.

  • http-metrics (10255): kubelet-service default port, used for exporting metrics to Prometheus services without encryption.

  • cadvisor (4194): Port of container advisor that exposes Prometheus out of the box.

smm-kubestatemetrics

Part of the kubestatemetrics-component. The service uses these ports:

  • http-monitoring (42422): Monitoring port for the kube-state-metrics application (HTTP).

  • http-telemetry (15014): Telemetry port for the kube-state-metrics application (HTTP).

smm-leo

Makes cert-manager Istio-aware. It is part of the leo-component. The service uses these ports:

  • http-metrics (8080): Export metrics to Prometheus services.

smm-prometheus

Used for event monitoring and alerting as part of the prometheus-component. The service uses these ports:

  • http (59090): Default port of the Prometheus service for handing HTTP traffic and queries.

smm-prometheus-operator

The smm-prometheus-operator instance which is the controller of the Prometheus application uses this service. The service uses these ports:

  • http (8080): Incoming webhook traffic and Prometheus exporter for operator metrics.

smm-sre-alert-exporter

Used by the smm-sre-alert-exporter instance. The service uses these ports:

  • http-metrics (8080): Used for exporting Prometheus alerting status as metrics.

smm-sre-api

The smm-sre-api instance uses this service for the GraphQL API. The service uses these ports:

  • http-graphql (80): Handles HTTP traffics and GraphQL queries.

smm-sre-controller

Used by the smm-sre instance, as part of the sre-controller-component. The service uses these ports:

  • http-metrics (8080): Exports metrics to the Prometheus service.

smm-tracing

The Service Mesh Manager uses this service as part of the tracing-component. The service uses these ports:

  • http-query (80): Handles the Jaeger user interface and HTTP API traffic.

smm-vm-integration

The vm-integration instance uses the service for the GraphQL API. The service uses these ports:

  • http (8080): Handles GraphQL API traffic and queries.

  • http-metrics (8081): HTTP metrics from the controller.

  • http-istiostate (8082): Metrics mapping istio resources to together.

smm-web

The Service Mesh Manager uses this service as part of the web-component. The service uses these ports:

  • http (80): Serves static web content used by the Service Mesh Manager dashboard.

  • http-downloads (81): Serves the functionaliy for downloading the SMM binary file and is used by Service Mesh Manager dashboard.

smm-zipkin

The Service Mesh Manager uses this service as part of the tracing-component. The service uses these ports:

  • http (59411): Default Zipkin port for handling HTTP traffic and distributed tracing mechanisms.

7 - Recovery procedure

This document details how to change and revoke all keys related to a Service Mesh Manager installation when the cluster where it is running is compromised. We’ll assume that an attacker could have had access to various Service Mesh Manager related service account tokens or keys.

Disclaimer: This is not a full guide for Kubernetes security recovery, only covers parts related to Service Mesh Manager.

Service Account Tokens

Service account tokens provide access to specific actions related to the Kubernetes API. Service Mesh Manager is using multiple service account tokens to access the Kubernetes API service. If you think some, or all of these are compromised, and an attacker could have gained access, delete these tokens to revoke them, and restart the pods in the Service Mesh Manager related namespaces. Complete the following steps:

  1. Delete all secrets holding service account tokens.

    for namespace in smm-system smm-demo cert-manager cluster-registry istio-system smm-canary smm-registry-access
    do
      kubectl delete secret -n $namespace $(kubectl get secret -n smm-system --field-selector type=kubernetes.io/service-account-token -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')
    done
    
  2. Restart all pods.

    for namespace in smm-system smm-demo cert-manager cluster-registry istio-system smm-canary smm-registry-access
    do
      kubectl delete pods --all
    done
    

Istio root CAs

Istio root CAs are used to sign certificates provided to workloads in the service mesh. If an attacker gets access to your CA keys, they may be able to set up a man-in-the-middle attack and eavesdrop encrypted network traffic.

Using the default, generated CA

If you are using the default, generated CA, delete the secret that holds the generated root CA and restart all pods in the istio-system namespace.

k delete secret -n istio-system istio-ca-secret
k delete pods -n istio-system --all

After Istiod is restarted, and the CA is regenerated, you restart all pods with a sidecar in the system.

Using your own certificates

If using your own certificates with Istio, revoke them, create new ones, and configure those in the cluster.

This guide doesn’t cover how exactly you’ll need to revoke the certificates, or create new ones. It depends on how the original certificates were created, what is the signing root CA for them, and where else are they used. You’ll probably need to add them to the Certificate Revocation List (CRL) of the issuing Certificate Authority, and follow the issuing Certificate Authority’s guidelines to create a new one.

Once you’ve revoked the CA and generated a new one, set the new CA in Service Mesh Manager.

  1. If you’re storing the CAs in secrets on the cluster, update the contents of the configured secret specified in spec.citadel.caSecretName. To properly set up the secret with the certificates, follow the Istio documentation
  2. If you’ve configured Vault using istiod.ca.vault in the IstioControlPlane CR, update the CAs in Vault.

Once your CA is changed, restart all pods in the istio-system namespace:

kubectl delete pods -n istio-system --all

After Istiod is restarted, restart all pods with a sidecar in the system.

JWT backend key

A JWT backend key is generated during Service Mesh Manager installation to sign and validate JWT tokens for the GraphQL API authentication service. If a malicious attacker acquires this key, they could use it to forge a JWT token and gain unauthorized access to the SMM API using that key to retrieve or even change Kubernetes data.

The JWT backend key is stored in the smm-authentication Kubernetes secret in the smm-system namespace. If you suspect that this key is compromised, delete the Kubernetes secret:

kubectl delete secret -n smm-system smm-authentication

To regenerate a new key:

  • If you’re using the Service Mesh Manager operator, it will automatically regenerate the key and the secret.
  • Otherwise, rerun the smm install command.

IMPS

Service Mesh Manager can hold access key pairs in Kubernetes secrets to be able to access images in ECR (default) or other image registries. It’s always a good idea to restrict the permissions of these keys to image access, but in case a malicious attacker gets access to these secrets, revoke these keypairs, and create new ones instead. To be able to revoke the old keys and create new ones, you should follow the process of the image registry you’re using.

For ECR, you should be able to revoke and recreate credentials on the AWS console, or by contacting your AWS administrator.

To check the credentials configured in the environment, list secrets in the smm-registry-access namespace. Credentials are base64 encoded within the secrets:

k get secret -n smm-registry-access

Once you have new image registry credentials, use the smm registry commands of the Service Mesh Manager CLI to configure them in the cluster.

  • smm registry list lists the currently configured image registry settings.
  • smm registry remove remove the configured registry entry.
  • smm registry add configures your image registry with new access credentials.
  • smm registry reconcile updates the remote cluster’s image pull secrets from the local settings.

8 - Fetch CA certificate from Vault

Fetching CA certificate overview

Istio uses X.509 certificates to create strong identities to every workload. These certificates are used to authenticate the service-to-service communication in the mesh. The Istio agent of the workload obtains these certificates from istiod and provides them to the Envoy proxy of the workload. istiod acts as a Certificate Authority, and signs the certificates with the signing certificate of the Certificate Authority (CA) certificate.

How Istio provides certificates for the workloads How Istio provides certificates for the workloads

By default, the Certificate Authority has a self-signed root certificate and key and uses them to sign the workload certificates. The private key of the CA is stored in Kubernetes secrets. In production environments these are not recommended, and you should instead:

  • Manage the CA independently from Istio, in a solution that offers strong security protection.
  • Use a production-ready CA, such as Hashicorp Vault.

You can configure Service Mesh Manager to use Vault as a secret store and fetch the signing certificate of the Certificate Authority from a Vault instance. That way you can securely store the CA certificate and its private key outside Istio.

The following high-level procedure shows the main steps of configuring Service Mesh Manager to use Vault.

Note: Service Mesh Manager works with any Vault solution. For the following procedures and examples we used Bank-Vaults, which is an open-source solution to run and manage Vault in Kubernetes.

  1. Configure Vault so Istio (the istiod service account) can access the CA certificate.
  2. Configure Istio to use the CA from Vault. If you are using Bank-Vaults, then the following examples show how to configure Service Mesh Manager to inject the bank-vaults sidecar into Istio, so the istiod can access the CA certificate.
  3. (Optional) Test your setup to verify that it works properly.

Prerequisites

Before configuring Istio and Vault, make sure these prerequisites are met:

  • Make sure you have a Vault instance with CA certificate and private key ready to pull.

  • If you want to use Bank-Vaults to create and manage Vault, install Vault, the Vault operator, and the Vault Secrets Webhook using the Helm charts of the Bank-Vaults project. For details, see the Bank-Vaults documentation).

Configure Vault

Create a policy so that istiod can access the Vault secrets.

To give istiod the access right to Vault secrets, a policy needs to be created for authentication purpose. The Vault is not limited to Bank-Vaults, it can be any other vault solutions. But we will use bank-vaults as an example on how to set the authentication policy:

Edit Vault CR by running kubectl edit vault.

spec:
  caNamespaces:
  - istio-system
  externalConfig:
    auth:
    - roles:
      # Make sure the role name 'istiod' corresponds to the role name we set in Istio
      - bound_service_account_names:
        - istiod-service-account-cp-v115x
        bound_service_account_namespaces:
        - istio-system
        name: istiod
        policies:
        - allow_istiod_secrets
      - bound_service_account_names:
        - default
        - vault-secrets-webhook
        - vault
        bound_service_account_namespaces:
        - default
        name: default
        policies:
        - allow_secrets
        - allow_pki
        ttl: 1h
      type: kubernetes
    policies:
    # Policy created for istiod to read secrets under this specific path
    - name: allow_istiod_secrets
      rules: path "secret/data/pki/istiod" { capabilities = ["read", "list"] }
    - name: allow_secrets
      rules: path "secret/*" { capabilities = ["create", "read", "update", "delete",
        "list"] }
    - name: allow_pki
      rules: path "pki/*" { capabilities = ["create", "read", "update", "delete",
        "list"] }

Note: Remember to add the istio-system namespace under .spec.caNamespaces so that istiod has the required TLS certificate to access Vault.

Configure SMM

In order for istiod pod to access the Vault instance, you need to set values of .istiod.ca.vault.

  1. Collect the parameters you need to configure the Istio custom resource:

    • The address of your Vault instance (For example, https://vault.default:8200).
    • The path to the CA certificate in Vault. Istio will use this certificate to sign the workload certificates with. (For example, vault:secret/data/pki/istiod#certificate).
    • The path to the private key of the certificate in Vault. (For example, vault:secret/data/pki/istiod#privateKey).
    • The name of the role that Istio can use to access the CA certificate and its private key in Vault. (For example, istiod).
  2. Edit the Istio custom resource. You can use the following command to edit the Istio custom resource (adjust the name of the controlplane to your Istio version: kubectl edit istio cp-v115x -n istio-system

  3. Add the parameters of Vault under the .istiod.ca.vault key, for example:

      istiod:
        ca:
          vault:
            address: https://vault.default:8200
            certPath: vault:secret/data/pki/istiod#certificate
            enabled: true
            keyPath: vault:secret/data/pki/istiod#privateKey
            role: istiod
            vaultEnvImage: ghcr.io/banzaicloud/vault-env:1.13.1
    
  4. Set the .istiod.ca.vault.enabled key to true, so that vault-env sidecar will be injected into istiod

Test your setup

If you are using Bank-Vaults and followed the steps on this page, you should see that the vault-env sidecar has been injected into the istiod pod which pulls the certificate from the Vault instance.

Sidecar container injected in istiod Sidecar container injected in istiod

If not, delete the istiod pod and wait for it to restart with the latest Istio configuration.

Additional Info

Add secrets in Vault

For testing purposes, you may need this step-by-step guide to put some secrets in the specified path defiend in Istio CR.

  1. Port-forward into the pod:
    kubectl port-forward vault-0 8200 &
    
  2. Set the address of the Vault instance:
    export VAULT_ADDR=https://127.0.0.1:8200
    
  3. Import the CA certificate and private key of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):
    kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
    kubectl get secret vault-tls -o jsonpath="{.data.ca\.key}" | base64 --decode > $PWD/vault-ca.key
    export VAULT_CACERT=$PWD/vault-ca.crt
    
  4. To authenticate to Vault, you can access its root token by running:
    export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
    
  5. Now you can interact with Vault and add a secret by running:
    vault kv put secret/pki/istiod privateKey=@vault-ca.key certificate=@vault-ca.crt 
    
  6. To verify the secret has been added to Vault:
    vault kv get secret/pki/istiod