This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Dashboard

Service Mesh Manager provides a dashboard interface that can be used to diagnose any issues with the underlying deployment. This section provides an introduction to the list of available features on this user interface.

Accessing the dashboard

To access the dashboard, set your KUBECONFIG file to the cluster where the Service Mesh Manager control plane is running, then run the following command to open a browser window to the dashboard.

smm dashboard --kubeconfig <path/to/kubeconfig>

In case you are executing this command on a remote machine, complete the following additional steps.

  1. Check the output of the command and forward the indicated port to your local machine.

  2. Open a browser window and navigate to http://127.0.0.1:50500/.

  3. Service Mesh Manager asks for login credentials. To acquire the credentials, follow the instructions on the user interface.

    Alternatively, you can complete the following steps.

    1. Set your KUBECONFIG file to the cluster where the Service Mesh Manager control plane is running, then run the following command.

      smm login --kubeconfig <path/to/kubeconfig>
      

      A temporary login token is displayed. Now you can perform other actions from the command line.

1 - Dashboard overview

The MENU > OVERVIEW page on the Service Mesh Manager web interface shows information about your the traffic in your mesh and the health of your services and workloads.

overview overview

If your application hasn’t received any traffic yet, there will be no metrics in the system so you won’t see any visualization yet. To send some traffic to your services as a test, see Generate test load.

The page shows the following information and controls:

Metrics

  • Requests per second
  • Average latency (95th percentile)
  • Error rate (for 5XX errors). Client-side errors (with 4XX status code) are not included.
  • Clusters (number of clusters in the mesh)
  • Services (number of services in the mesh / total number of services)
  • Workloads (number of workloads in the mesh / total number of workloads)
  • Pods (number of pods in the mesh / total number of pods)
  • Validation issues

Dashboards

The OVERVIEW page shows charts about the health of the services and workloads, as well as the aggregated status of your service level objectives (SLOs). Click on the chart to get more information, for example, about the SLO burn rates that display warnings.

The OVERVIEW page also shows the following live Grafana dashboards:

  • Requests per second
  • Average latency (95th percentile)
  • Error rate (for 5XX errors). Client-side errors (with 4XX status code) are not included.

Validations

To check the validation status of your YAML configuration files, select OVERVIEW > VALIDATION ISSUES. For details, see Validation.

2 - Mesh

The MENU > MESH page on the Service Mesh Manager web interface shows information about your service mesh and the control planes.

Mesh overview Mesh overview

The page shows the following real-time information:

The mesh in numbers

  • CONTROL PLANES: The number of Istio control planes in the mesh.
  • CLUSTERS: The number of clusters in the mesh.
  • ISTIO PROXIES MEMORY USAGE: Current memory usage of the Istio proxies (sidecars).
  • ISTIO PROXIES CPU USAGE: Current CPU usage of the Istio proxies (sidecars).
  • ISTIO PROXIES NOT RUNNING: The Istio proxies (sidecars) that are not running for some reason.

Clusters

Displays basic status information about the clusters in the mesh.

Clusters Clusters

This is mostly useful in the multi-cluster setup when multiple clusters are in the mesh.

Control planes

This section displays information and metrics about the Istio control planes in the mesh, including version and revision information, and validation errors.

Istio control planes in the mesh Istio control planes in the mesh

Click on a specific control plane to display information about:

In addition, selecting a control plane also shows the following basic information:

  • CLUSTER NAME: The name of the cluster the control plane is running on.
  • VERSION: The Istio version of the service mesh.
  • ROOT NAMESPACE: The administrative root namespace for Istio configuration of the service mesh.
  • TRUST DOMAIN: The list of trust domains.
  • AUTOMATIC MTLS: Shows whether automatic mutual TLS is enabled for the service mesh.
  • OUTBOUND TRAFFIC POLICY: The default outbound traffic policy for accessing external services set for the mesh. For details, see External Services.
  • PROXIES: The number of sidecar proxies in the mesh.
  • CONFIG: Click the Show YAML configuration icon to display the configuration of the control plane.

Pods

Shows information and status about the pods of the control plane.

Control plane pods Control plane pods

Proxies

Lists the proxies managed by the control plane, and the synchronization status of the cluster discovery service (CDS), listener discovery service (LDS), endpoint discovery service(EDS), and route discovery service (RDS) for the proxy.

Control plane proxies Control plane proxies

Trust bundles

Shows the trust bundles defined for the control plane.

Validation issues

Lists the validation issues for the entire control plane.

Control plane validation Control plane validation

Metrics

The timeline charts show the version and revision of the Istio proxies used in the mesh, as well as error metrics from the Istio Pilot agent, for example, rejected CDS and EDS configurations. (Istio Pilot agent runs in the sidecar or gateway container and bootstraps Envoy.)

To display more detailed metrics about the resource usage of Istiod and the proxies, click on a control plane in the Control planes section.

Control plane metrics Control plane metrics

2.1 - Validation

The Service Mesh Manager product:

  • simplifies service mesh configuration and management,
  • guides you through setting up complex traffic routing rules
  • takes care of creating, merging and validating the YAML configuration.

And unlike some other similar products, it’s working in both directions: you can edit the YAML files manually, and you can still view and manipulate the configuration from Service Mesh Manager. That’s possible because there’s no intermediate configuration layer in Service Mesh Manager.

To support the bi-directional mesh configuration, Service Mesh Manager provides a validation subsystem for the entire mesh. Istio itself provides some syntactic and semantic validation for the individual Istio resources, but Service Mesh Manager goes even further. Service Mesh Manager performs complex validations which take the whole cluster state and related resources into account to check whether everything is configured correctly within the whole mesh.

Service Mesh Manager performs a lot of syntactical and semantical validation checks for various aspects of the configuration. The validation checks are constantly curated and new checks added with every release. For example:

  • Sidecar injection template validation: Validates whether there are any pods in the environment that run with outdated sidecar proxy image or configuration.
  • Gateway port protocol configuration conflict validation: Detects conflicting port configuration in different Gateway resources.
  • Multiple gateways with the same TLS certificate validation: Configuring multiple gateways to use the same TLS certificate causes most browsers to produce 404 errors when accessing a second host after a connection to another host has already been established.

Check validation results on the Service Mesh Manager UI

The validations are constantly running in the background. To display the actual results, navigate to OVERVIEW > VALIDATION ISSUES. You can use the NAMESPACES field to select the namespaces you want to observe.

Show validation results Show validation results

To display the invalid part of the configuration in the invalid resource, click the Show YAML configuration icon.

Show validation details Show validation details

To display every validation error of a control plane as a list, navigate to MENU > MESH, and click on the control plane in the Control planes section, then select VALIDATIONS. For details, see Validation issues.

Check validation results from the CLI

To check the results of the validation from the CLI, run the smm analyze command. To show only results affecting a specific namespace, use the –namespace option, for example: smm analyze --namespace smm-demo, or smm analyze --namespace istio-system

The smm analyze command can also produce JSON output, for example:

smm analyze --namespace istio-system -o json

Example output:

{
  "gateway.networking.istio.io:master:istio-system:demo-gw-demo1": [
    {
      "checkID": "gateway/reused-cert",
      "istioRevision": "cp-v115x.istio-system",
      "subjectContextKey": "gateway.networking.istio.io:master:istio-system:demo-gw-demo1",
      "passed": false,
      "error": {},
      "errorMessage": "multiple gateways configured with same TLS certificate"
    }
  ],
  "gateway.networking.istio.io:master:istio-system:demo-gw-demo2": [
    {
      "checkID": "gateway/reused-cert",
      "istioRevision": "cp-v115x.istio-system",
      "subjectContextKey": "gateway.networking.istio.io:master:istio-system:demo-gw-demo2",
      "passed": false,
      "error": {},
      "errorMessage": "multiple gateways configured with same TLS certificate"
    }
  ]
}

3 - Topology view

The MENU > TOPOLOGY page of the Service Mesh Manager web interface displays the topology of services and workloads inside the mesh, and annotates it with real-time and information about latency, throughput, or HTTP request failures. You can also display historical data by adjusting the timeline.

The topology view is almost entirely based on metrics: metrics received from Prometheus and enhanced with information from Kubernetes.

The topology page serves as a starting point of diagnosing problems within the mesh. Service Mesh Manager is integrated with Grafana and Jaeger for easy access to in-depth monitoring and distributed traces) of various services.

Topology view of the demo cluster Topology view of the demo cluster

The nodes in the graph are services or workloads, while the arrows represent network connections between different services. This is based on Istio metrics retrieved from Prometheus.

For certain services like MySQL and PostgreSQL, protocol-specific metrics normally not available in Istio are also shown, for example, sessions or transactions per second.

Note: Protocol-specific metrics for MySQL and PostgreSQL are available only for certain versions of MySQL and PostgreSQL. For details, see the documentation of the MySQL and the PostgreSQL Envoy filters.

For example, the following image shows SQL sessions and transactions per second.

Protocol-specific data Protocol-specific data

The graph serves as a visual monitoring tool, as it displays various errors and metrics in the system. Click the ? icon on the left to show a legend of the graph to better understand what the icons mean in the graph. Virtual machines integrated into the mesh are displayed as workloads with the Virtual machine workload icon in their corner.

Topology view legend Topology view legend

If your application hasn’t received any traffic yet, there will be no metrics in the system so you won’t see any visualization yet. To send some traffic to your services as a test, see Generate test load.

Select the data displayed

You can select and configure what is displayed on the graph in the top bar of the screen. You can also display historical data using the TIMELINE.

Configure the topology view Configure the topology view

Namespaces

Display data only for the selected namespaces.

Resources

You can select the type of resources you want to display in the graph. The following resources can be displayed in a cluster: clusters, namespaces, apps, services, and workloads.

Workloads are always shown, they cannot be disabled.

Here’s an example when only apps, services and workloads are shown:

Show selected resources of the mesh Show selected resources of the mesh

Showing clusters is important in multi-cloud and hybrid-cloud environments. For details, see Multi-cluster.

Edge labels

The labels on the edges of the graph can display various real-time information about the traffic between services. You can display the following information:

  • the protocol used in the communication (HTTP, gRPC, TCP) and the request rate (or throughput for TCP connections),
  • actual P95 latency, or
  • whether the connection is using mTLS or not.

For certain services like MySQL and PostgreSQL, protocol-specific metrics normally not available in Istio are also shown, for example, sessions or transactions per second.

Note: Protocol-specific metrics for MySQL and PostgreSQL are available only for certain versions of MySQL and PostgreSQL. For details, see the documentation of the MySQL and the PostgreSQL Envoy filters.

Show information on the graph edges Show information on the graph edges

Timeline

By default, the graph displays the current data. The timeline view allows you to select a specific point in time, and then move minutes back and forth to see how your most important metrics have changed. For example, you can use it to check how things changed for a specific service, when did the error rates go up, or how your latency values have changed over time when RPS values increased. This can be a good indicator to know where to look for errors in the logs, or to notice if something else has changed in the cluster that can be related to a specific failure.

  • To display the timeline, select TIMELINE on the left, then use the timeline bar to adjust the date and the time. The date corresponding to the displayed data is shown below the topology graph.
  • To return to the current real-time data, select LIVE.

Show historical topology and metrics Show historical topology and metrics

Drill-down to the pods and nodes

You can drill-down from the MENU > TOPOLOGY page by selecting a service or a workload in the Istio service mesh. You can trace back an issue from the top-level service mesh layer by navigating deeper in the stack, and see the status and most important metrics of your Kubernetes controllers, pods, and nodes.

See Drill-down for details.

3.1 - Drill-down

You can drill-down from the MENU > TOPOLOGY page by selecting a service or a workload in the Istio service mesh. You can trace back an issue from the top-level service mesh layer by navigating deeper in the stack, and see the status and most important metrics of your Kubernetes controllers, pods, and nodes.

For an example on how you can diagnose and solve a real-life problem using the drill-down view, see the Service Mesh Manager drill-down blog post.

Drill-down from the Topology view

The highest level of the Topology view is the service mesh layer. This level contains the most important network-level metrics, and an overview of the corresponding Kubernetes controllers.

Click on a workload ( Workload Workload ) or service ( Service Service ) to display its details on the right. Workloads running on virtual machines have a blue icon in the corner (Workload on a virtual machine ).

From the details overview, you can drill down through the following levels to the underlying resources of the infrastructure:

Details of a workload Details of a workload

To display the metrics-based health of the workload or the service, select the HEALTH tab. You can scroll down to display the charts of the selected metric (for example, saturation, latency, or success rate). Note that for workloads running on virtual machines, the total saturation of the virtual machine is shown. Application health metrics Application health metrics

Service overview

Details of a service Details of a service

The following details of the service are displayed:

  • Namespace: The namespace the service belongs to.

  • APP: The application exposed using the service.

  • PORTS: The ports where the service is accessible, for example:

    http         8080/TCP → 8080
    grpc         8082/TCP → 8082
    tcp          8083/TCP → 8083
    
  • Services: The services exposed in this resource. Click on the name of the service to display the details of the service.

  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

  • Traces: Click Open Jaeger tracing to run tracing with Jaeger.

CAUTION:

If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

Details of the workload

Details of a workload Details of a workload

The following details of the workload are displayed:

  • Namespace: The namespace the workload belongs to.
  • APP: The application running in the workload.
  • VERSION: The version number of the workload, for example, v2.
  • REPLICAS: The number of replicas for the workload.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Controllers: The controllers related to the workload. Click on a controller to display its details.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

Service details

Select a Service in the SERVICE section of a service overview to display the details of the service.

Details of a service Details of a service

The following details of the service are displayed:

  • Namespace: The namespace the service belongs to.

  • CLUSTER: The name of the Kubernetes cluster the service belongs to.

  • SELECTOR: The label selector used to select the set of Pods targeted by the Service.

  • PORTS: The ports where the service is accessible, for example:

    http         8080/TCP → 8080
    grpc         8082/TCP → 8082
    tcp          8083/TCP → 8083
    
  • TYPE: The ServiceType indicates how your service is exposed, for example, ClusterIP or LoadBalancer.

  • CLUSTER IP: The IP address corresponding to the ServiceType.

  • CREATED: The date when the service was started.

  • LABELS: The list of Kubernetes labels assigned to the resource.

  • Pods: The list of pods running this service, including their name, number of containers in the pod, and their status. Click on the name of the pod to display the details of the pod.

  • Events: Recent events related to the service resource.

Workload controller details

Select a deployment in the CONTROLLER section of a workload to display the details of the deployment.This view contains detailed information about the Kubernetes controller.

While the service mesh layer displays network level metrics and an aggregated view of the corresponding controllers, this view focuses on CPU and memory metrics, and the Kubernetes resources, like related pods or events. It’s also possible that multiple controllers belong to the same service mesh entity, for example, in a shared control plane multi-cluster scenario, when multiple clusters are running controllers that belong to the same logical workload.

Details of a workload controller Details of a workload controller

The following details of the workload controller are displayed:

  • Namespace: The namespace the workload belongs to.
  • CLUSTER: The name of the Kubernetes cluster the workload belongs to.
  • Kind: The type of the controller, for example, Deployment. If the workload is running on a virtual machine, the kind of the controller is WorkloadGroup.
  • APP: The application running in the workload.
  • VERSION: The version number of the workload, for example, v2.
  • REPLICAS: The number of replicas for the workload.
  • CREATED: The date when the workload was started.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Pods: The list of pods running this workload. Click on the name of the pod to display the details of the pod.
  • WorkloadEntries: The list of virtual machines running this workload. Click on the name of the WorkloadEntry to display the details of the WorkloadEntry.
  • Events: Recent events related to the resource.

Details of the pod

To check the details of a pod, select a pod in the CONTROLLER > POD or the SERVICE > POD section.

Details of a pod

The following details of the pod are displayed:

  • Namespace: The namespace the pod belongs to.
  • CLUSTER: The name of the Kubernetes cluster the pod belongs to.
  • NODE: The hostname of the node the pod is running on, for example, ip-192-168-1-1.us-east-2.compute.internal. Click on the name of the node to display the details of the node.
  • IP: The IP address of the pod.
  • STARTED: The date when the pod was started.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Containers: The list of containers in the pod. Also includes the Name, Image, and Status of the container.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

To display the logs of the pod, click the Show pod logs icon icon. The pod logs are displayed at the bottom of the screen.

Note: In multi-cluster scenarios, live log streaming is available only for pods running on the primary cluster.

Show pod logs

Details of the node

To check the health of a node, select a node in the pod details view. The node view is the deepest layer of the drill-down view and shows information about a Kubernetes node.

Details of a node

The following details of the node that the pod is running on are displayed:

  • CLUSTER: The name of the Kubernetes cluster the node belongs to.
  • OS: The operating system running on the node, for example: linux amd64 (Ubuntu 18.04.4 LTS)
  • STARTED: The date when the node was started.
  • TAINTS: The list of Kubernetes taints assigned to the node.
  • LABELS: The list of Kubernetes labels assigned to the node.
  • Conditions: The status of the node, for example, disk and memory pressure, or network and kubelet status.
  • Pods: The list of pods currently running on the node.
  • Events: Recent events related to the node.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

Details of the WorkloadEntry

Details of a WorkloadEntry

The following details of the pod are displayed:

  • NAMESPACE: The namespace the pod belongs to.
  • CLUSTER: The name of the Kubernetes cluster the pod belongs to.
  • NETWORK: The name of the network the virtual machine running the workload belongs to.
  • ADDRESS: The IP address of the virtual machine.
  • PORTS: The open ports and related protocols of the WorkloadGroup.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

3.2 - Generate test load

There are several places in the Service Mesh Manager interface where you can’t see anything if your application hasn’t received any traffic yet. For example, if there are no metrics in the system you won’t see any visualization on the MENU > OVERVIEW page.

Generate load on the UI

To send generated traffic to an endpoint, complete the following steps.

  1. On the Service Mesh Manager web interface, select MENU > TOPOLOGY.
  2. Click HTTP on the left side of the screen. Generate test traffic Generate test traffic
  3. Complete the form with the endpoint details and click SEND to generate test traffic to your services. Set traffic parameters Set traffic parameters
  4. In a few seconds a graph of your services appears.

Generate load in the CLI

If needed, you can generate constant traffic in the demo application by running: smm demoapp load start

3.3 - Pod logs

You can display the logs of Kubernetes pods on the Service Mesh Manager web interface to help troubleshooting your infrastructure.

Note: In multi-cluster scenarios, live log streaming is available only for pods running on the primary cluster.

To display the logs of a pod, complete the following steps.

  1. Open the Service Mesh Manager dashboard.
  2. Drill down to the pod you want to inspect. Drill down to the pod level Drill down to the pod level
  3. Click the Show pod logs icon icon. The pod logs are displayed at the bottom of the screen. Show pod logs Show pod logs
  4. You can also filter the logs to a specific container. Filter pod logs Filter pod logs

From the pod level, you can also go up or down in your infrastructure to inspect other components.

4 - Workloads

You can drill-down from the MENU > TOPOLOGY page by selecting a service or a workload in the Istio service mesh. You can trace back an issue from the top-level service mesh layer by navigating deeper in the stack, and see the status and most important metrics of your Kubernetes controllers, pods, and nodes.

List of workloads

The MENU > WORKLOADS page contains information about the workloads in your service mesh. Above the list of workloads, there is a summary dashboard about the state of your workloads, showing the following information:

  • Requests per second: Requests per second for the workloads.
  • Average latency: Average latency for the workloads (95th percentile latency in milliseconds).
  • Error rate: The percentage of requests returning a 5xx status code. Client-side errors (with 4XX status code) are not included.
  • Clusters: The number of clusters in the service mesh.
  • Workloads: The number of workloads in the mesh and the total number of workloads.
  • Pods: The number of running pods and the desired number of pods.

The list displays the workloads (grouped by namespaces), and a timeline of the metrics-based health score of each workload. The Kubernetes workload icon indicates Kubernetes workloads, while the Virtual machine workload icon indicates workloads running on virtual machines. You can filter the list to show only the selected namespaces, and display historical data by adjusting the timeline.

  • To display the details of a workload, click the name of the workload.
  • To open the Grafana dashboards related to the workload, click Open metrics in Grafana .
  • To display the detailed health metrics of a workload, click the health indicator of the workload for the selected period. Health metrics of a workload Health metrics of a workload

From the mesh workload overview, you can drill down through the following levels to the underlying resources of the infrastructure:

Details of a workload Details of a workload

Workload details

Select a Workload from the list to display its details.

Details of a workload Details of a workload

The following details of the workload are displayed:

  • NAMESPACE: The namespace the workload belongs to.
  • APP: The application running in the workload.
  • VERSION: The version number of the workload, for example, v2.
  • REPLICAS: The number of replicas for the workload.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • HEALTH: Indicates the health score of the workload. Click the chart to display more details.
  • Controller: The controllers related to the workload. Click on a controller to display its details.
  • CLUSTER: The name of the Kubernetes cluster the workload belongs to.
  • KIND: The kind of the workload, for example, DaemonSet, Deployment, ReplicaSet, or StatefulSet.
  • Pods: The list of pods running this workload. Click on the name of the pod to display the details of the pod. You can also display and filter logs of the pod.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

Details of a controller Details of a controller

Pod details

To check the details of a pod, click the name of a pod in the Pods section.

Details of a pod

The following details of the pod are displayed:

  • Namespace: The namespace the pod belongs to.
  • CLUSTER: The name of the Kubernetes cluster the pod belongs to.
  • NODE: The hostname of the node the pod is running on, for example, ip-192-168-1-1.us-east-2.compute.internal. Click on the name of the node to display the details of the node.
  • IP: The IP address of the pod.
  • STARTED: The date when the pod was started.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Containers: The list of containers in the pod. Also includes the Name, Image, and Status of the container.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

To display the logs of the pod, click the Show pod logs icon icon. The pod logs are displayed at the bottom of the screen.

Note: In multi-cluster scenarios, live log streaming is available only for pods running on the primary cluster.

Show pod logs

Node details

To check the health of a node, select a node in the pod details view. The node view is the deepest layer of the drill-down view and shows information about a Kubernetes node.

Details of a node

The following details of the node that the pod is running on are displayed:

  • CLUSTER: The name of the Kubernetes cluster the node belongs to.
  • OS: The operating system running on the node, for example: linux amd64 (Ubuntu 18.04.4 LTS)
  • STARTED: The date when the node was started.
  • TAINTS: The list of Kubernetes taints assigned to the node.
  • LABELS: The list of Kubernetes labels assigned to the node.
  • Conditions: The status of the node, for example, disk and memory pressure, or network and kubelet status.
  • Pods: The list of pods currently running on the node.
  • Events: Recent events related to the node.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

WorkloadEntry details

Details of a WorkloadEntry

The following details of the pod are displayed:

  • NAMESPACE: The namespace the pod belongs to.
  • CLUSTER: The name of the Kubernetes cluster the pod belongs to.
  • NETWORK: The name of the network the virtual machine running the workload belongs to.
  • ADDRESS: The IP address of the virtual machine.
  • PORTS: The open ports and related protocols of the WorkloadGroup.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

5 - Services

You can drill-down from the MENU > TOPOLOGY page by selecting a service or a workload in the Istio service mesh. You can trace back an issue from the top-level service mesh layer by navigating deeper in the stack, and see the status and most important metrics of your Kubernetes controllers, pods, and nodes.

List of services

The MENU > SERVICES page contains information about the services in your service mesh. Above the list of services, there is a summary dashboard about the state of your services, showing the following information:

List of services List of services

The list displays the services (grouped by namespaces), and a timeline of the metrics-based health score of each service. You can filter the list to show only the selected namespaces, and display historical data by adjusting the timeline.

  • To display the details of a service, click the name of the service.
  • To open the Grafana dashboards related to the service, click Open metrics in Grafana .
  • To run tracing with Jaeger click Open Jaeger tracing .

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.
  • To display the detailed health metrics of a service, click the health indicator of the service for the selected period. Health metrics of a service Health metrics of a service

From the services list, you can drill down through the following levels to the underlying resources of the infrastructure: List of services > Mesh Service > Service > Pod > Node.

Mesh Service details

Details of a mesh service Details of a mesh service

  • Mesh Service: The multi-cluster mesh service available in the current mesh. Select a service from the SERVICE field to display the details of the service in any of the attached clusters.

  • Namespace: The namespace the service belongs to.

  • APP: The application exposed using the service.

  • PORTS: The ports where the service is accessible, for example:

    http         8080/TCP → 8080
    grpc         8082/TCP → 8082
    tcp          8083/TCP → 8083
    
  • HEALTH: Indicates the health score of the workload. Click the chart to display more details.

  • Service Level Objectives: The details of the Service Level Objectives (SLOs) defined for the service. Click on an SLO to display its details.

  • Services: The list of services belonging to the mesh service.

  • Metrics: Dashboards of the most important metrics. Click on a service to display its details.

    • To open the related Grafana dashboards, click Open metrics in Grafana .
    • To run tracing with Jaeger click Open Jaeger tracing .

Display service details

Select a Service from the list to display its details.

  • To run tracing with Jaeger click Open Jaeger tracing . In case of a multi-cluster setup, you can select which cluster’s data to display.

Details of a service Details of a service

The following details of the service are displayed:

  • Service: The services exposed in this resource. To display the details of the service, click the name of the service.

  • Namespace: The namespace the service belongs to.

  • CLUSTER: The name of the Kubernetes cluster the service belongs to.

  • SELECTOR: The label selector used to select the set of Pods targeted by the Service.

  • PORTS: The ports where the service is accessible, for example:

    http         8080/TCP → 8080
    grpc         8082/TCP → 8082
    tcp          8083/TCP → 8083
    
  • TYPE: The ServiceType indicates how your service is exposed, for example, ClusterIP or LoadBalancer.

  • CLUSTER IP: The IP address corresponding to the ServiceType.

  • CREATED: The date when the service was started.

  • LABELS: The list of Kubernetes labels assigned to the resource.

  • Pods: The list of pods running this service, including their name, number of containers in the pod, and their status. Click on the name of the pod to display the details of the pod.

  • Events: Recent events related to the service resource.

Display pod details

To check the details of a pod, click the name of the pod in the PODS section.

Details of a pod

The following details of the pod are displayed:

  • Namespace: The namespace the pod belongs to.
  • CLUSTER: The name of the Kubernetes cluster the pod belongs to.
  • NODE: The hostname of the node the pod is running on, for example, ip-192-168-1-1.us-east-2.compute.internal. Click on the name of the node to display the details of the node.
  • IP: The IP address of the pod.
  • STARTED: The date when the pod was started.
  • LABELS: The list of Kubernetes labels assigned to the resource.
  • Containers: The list of containers in the pod. Also includes the Name, Image, and Status of the container.
  • Events: Recent events related to the resource.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

To display the logs of the pod, click the Show pod logs icon icon. The pod logs are displayed at the bottom of the screen.

Note: In multi-cluster scenarios, live log streaming is available only for pods running on the primary cluster.

Show pod logs

Display node details

To check the health of a node, select a node in the pod details view. The node view is the deepest layer of the drill-down view and shows information about a Kubernetes node.

Details of a node

The following details of the node that the pod is running on are displayed:

  • CLUSTER: The name of the Kubernetes cluster the node belongs to.
  • OS: The operating system running on the node, for example: linux amd64 (Ubuntu 18.04.4 LTS)
  • STARTED: The date when the node was started.
  • TAINTS: The list of Kubernetes taints assigned to the node.
  • LABELS: The list of Kubernetes labels assigned to the node.
  • Conditions: The status of the node, for example, disk and memory pressure, or network and kubelet status.
  • Pods: The list of pods currently running on the node.
  • Events: Recent events related to the node.
  • Metrics: Dashboards of the most important metrics. Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

6 - Gateways

The MENU > GATEWAYS page of the Service Mesh Manager web interface allows you to:

Note: Service Mesh Manager uses the concept of IstioMeshGateways, a declarative representation of Istio ingress and egress gateway services and deployments. With the help of IstioMeshGateways, you can set up multiple gateways in a cluster, and use them for different purposes.

To create a new ingress gateway, see Create ingress gateway.

List gateways

To list the gateways of your service mesh, navigate to MENU > GATEWAYS.

List of Istio ingress gateways List of Istio ingress gateways

For each gateway, the following information is shown:

  • Name: The name of the gateway.
  • Namespace: The namespace the gateway belongs to.
  • Cluster: The cluster the gateway belongs to. Mainly useful in multi-cluster scenarios.
  • Type: Type of the gateway. Ingress gateways define an entry point into your Istio mesh for incoming traffic, while egress gateways define an exit point from your Istio mesh for outgoing traffic.
  • Open ports: The ports the gateway accepts connections on.
  • Hosts: Number of hosts accessible using the gateway.
  • Routes: Number of routing rules configured for the ingress traffic.
  • Error rate: The number of errors during the last polling interval, for 5XX errors. Client-side errors (with 4XX status code) are not included.
  • Requests per second: The number of requests per second during the last polling interval.
  • Status: Status of the gateway.

Click the name of a gateway to display the details of the gateway (grouped into several tabs: Overview and host configuration, Routes, Deployment and Service).

To display the YAML configuration of MeshGateways, Gateways, or VirtualServices, click the name of the gateway in the list, then click the Show YAML configuration icon next to their name.

YAML configuration of a gateways YAML configuration of a gateways

YAML configuration of a gateways YAML configuration of a gateways

Monitor upstream traffic

Service Mesh Manager collects upstream metrics like latencies, throughput, RPS, or error rate from Prometheus, and provides a summary for each gateway. It also sets up a Grafana dashboard and displays appropriate charts in-place.

To monitor the upstream traffic of your Istio gateways, complete the following steps.

  1. Open the Service Mesh Manager web interface, and navigate to MENU > GATEWAYS.

  2. From the list of gateways, click the gateway you want to monitor.

    List of Istio ingress gateways List of Istio ingress gateways

  3. On the OVERVIEW tab, scroll down to the METRICS section. The most important metrics of the gateway are displayed on the Service Mesh Manager web interface (for example, upstream requests per second and error rate).

    Note: You can also view the details of the service or the deployment related to the gateway.

    Istio ingress gateway metrics Istio ingress gateway metrics

    Click Open metrics in Grafana to open the related dashboards in Grafana.

    CAUTION:

    If you have installed Service Mesh Manager in Anonymous mode, you won’t be able to access the Metrics and Traces dashboards from the UI. Clicking the Open metrics in Grafana or Open Jaeger tracing icon in anonymous mode causes the RBAC: access denied error message.

Gateway deployment and service details

To display the details, events, and most important metrics of the deployment and service related to a gateway, navigate to MENU > GATEWAYS > <Gateway-to-inspect>, then click SERVICE or DEPLOYMENT.

Details of the gateway service Details of the gateway service

Details of the gateway deployment Details of the gateway deployment

6.1 - Create ingress gateway

Overview

Ingress gateways define an entry point into your Istio mesh for incoming traffic.

Multiple ingress gateways in Istio

You can configure gateways using the Gateway and VirtualService custom resources of Istio, and the IstioMeshGateway CR of Service Mesh Manager.

  • The Gateway resource describes the port configuration of the gateway deployment that operates at the edge of the mesh and receives incoming or outgoing HTTP/TCP connections. The specification describes a set of ports that should be exposed, the type of protocol to use, TLS configuration – if any – of the exposed ports, and so on. For more information about the gateway resource, see the Istio documentation.
  • The VirtualService resource defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for the traffic of a specific protocol. If the traffic matches a routing rule, then it is sent to a named destination service defined in the registry. For example, it can route requests to different versions of a service or to a completely different service than was requested. Requests can be routed based on the request source and destination, HTTP paths and header fields, and weights associated with individual service versions. For more information about VirtualServices, see the Istio documentation.
  • Service Mesh Manager provides a custom resource called IstioMeshGateway and uses a separate controller to reconcile gateways, allowing you to use multiple gateways in multiple namespaces. That way you can also control who can manage gateways, without having permissions to modify other parts of the Istio mesh configuration.

Using IstioMeshGateway, you can add Istio ingress or egress gateways in the mesh and configure them. When you create a new IstioMeshGateway CR, Service Mesh Manager takes care of configuring and reconciling the necessary resources, including the Envoy deployment and its related Kubernetes service.

Note: Service Mesh Manager automatically creates an ingress gateway called smm-ingressgateway and an istio-meshexpansion-cp-v115x. The smm-ingressgateway serves as the main entry point for the services of Service Mesh Manager, for example, the dashboard and the API, while the meshexpansion gateway is used in multi-cluster setups to ensure communication between clusters for the Istio control plane and the user services.

Do not use this gateway for user workloads, because it is managed by Service Mesh Manager, and any change to its port configuration will be overwritten. Instead, create a new mesh gateway using the IstioMeshGateway custom resource.

Prerequisites

Auto sidecar injection must be enabled for the namespace of the service you want to make accessible.

Steps

To create a new ingress gateway and expose a service, complete the following steps.

  1. If you haven’t already done so, create and expose the service you want to make accessible through the gateway.

    For testing, you can download and apply the following echo service:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echo
      labels:
        k8s-app: echo
      namespace: default
    spec:
      replicas: 1
      selector:
        matchLabels:
          k8s-app: echo
      template:
        metadata:
          labels:
            k8s-app: echo
        spec:
          terminationGracePeriodSeconds: 2
          containers:
          - name: echo-service
            image: k8s.gcr.io/echoserver:1.10
            ports:
            - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: echo
      labels:
        k8s-app: echo
      namespace: default
    spec:
      ports:
      - name: http
        port: 80
        targetPort: 8080
      selector:
        k8s-app: echo
    
    kubectl apply -f echo.yaml
    

    Expected output:

    deployment.apps/echo created
    service/echo created
    
  2. Create a new ingress gateway using the IstioMeshGateway resource.

    1. Download the following resource and adjust it as needed for your environment:

      CAUTION:

      By the default, the IstioMeshGateway pod is running without root privileges, therefore it cannot use ports under 1024. Either use ports above 1024 as targetports (for example, 8080 instead of 80) or run the gateway pod with root privileges by setting spec.runAsRoot: true in the IstioMeshGateway custom resource.
      apiVersion: servicemesh.cisco.com/v1alpha1
      kind: IstioMeshGateway
      metadata:
        name: demo-gw
      spec:
        istioControlPlane:
          name: cp-v115x
          namespace: istio-system
        runAsRoot: false
        service:
          ports:
            - name: tcp-status-port
              port: 15021
              protocol: TCP
              targetPort: 15021
            - name: http
              port: 80
              protocol: TCP
              targetPort: 8080
          type: LoadBalancer
        type: ingress 
      
    2. Apply the IstioMeshGateway resource. Service Mesh Manager creates a new ingress gateway deployment and a corresponding service, and automatically labels them with the gateway-name and gateway-type labels and their corresponding values.

      kubectl apply -f meshgw.yaml
      

      Expected output:

      istiomeshgateway.servicemesh.cisco.com/demo-gw created
      
    3. Get the IP address of the gateway. (Adjust the name and namespace of the IstioMeshGateway as needed for your environment.)

      kubectl -n default get istiomeshgateways demo-gw
      

      The output should be similar to:

      NAME      TYPE      SERVICE TYPE   STATUS      INGRESS IPS       ERROR   AGE    CONTROL PLANE
      demo-gw   ingress   LoadBalancer   Available   ["3.10.16.232"]           107s   {"name":"cp-v115x","namespace":"istio-system"}
      
    4. Create the Gateway and VirtualService resources to configure listening ports on the matching gateway deployment. Make sure to adjust the hosts fields to the external hostname of the service. (You should manually set an external hostname that points to these addresses, but for testing purposes you can use for example nip.io, which is a domain name that provides wildcard DNS for any IP address.)

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: echo
        namespace: default
      spec:
        selector:
          gateway-name: demo-gw
          gateway-type: ingress
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "echo.3.10.16.232.nip.io"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: echo
        namespace: default
      spec:
        hosts:
        - "echo.3.10.16.232.nip.io"
        gateways:
        - echo 
        http:
        - route:
          - destination:
              port:
                number: 80
              host: echo.default.svc.cluster.local
      
      kubectl apply -f gwvs.yaml
      

      Expected output:

      gateway.networking.istio.io/echo created
      virtualservice.networking.istio.io/echo created
      
  3. Access the service on the external address.

    curl -i echo.3.10.16.232.nip.io
    

    The output should be similar to:

    HTTP/1.1 200 OK
    date: Mon, 07 Mar 2022 19:22:15 GMT
    content-type: text/plain
    server: istio-envoy
    x-envoy-upstream-service-time: 1
    
    Hostname: echo-68578cf9d9-874rz
    ...
    

IstioMeshGateway CR reference

This section describes the fields of the IstioMeshGateway custom resource.

apiVersion (string)

Must be servicemesh.cisco.com/v1alpha1

kind (string)

Must be IstioMeshGateway

spec (object)

The configuration and parameters of the IstioMeshGateway.

spec.type (string, required)

Type of the mesh gateway. Ingress gateways define an entry point into your Istio mesh for incoming traffic, while egress gateways define an exit point from your Istio mesh for outgoing traffic. Possible values:

  • ingress
  • egress

spec.istioControlPlane (object, required)

Specifies the istiocontrolplane cr the istio-proxy connects to by a namespaced name. When upgrading to a new Istio version (thus to a new control plane), this should be upgraded.

For example:

spec:
  istioControlPlane:
    name: cp-v115x
    namespace: istio-system

spec.deployment (object)

Configuration options for the Kubernetes istio-proxy deployment. Metadata like labels and annotations can be set here for the deployment or pods as well, in spec.deployment.metadata.annotations or spec.deployment.podMetadata.annotations.

spec.service (object, required)

Configuration options for the Kubernetes service. Annotations can be set here as well as in spec.service.metadata.annotations, they are often useful in cloud loadbalancer cases, for example to specify some configuration for AWS.

For example:

  service:
    ports:
      - name: tcp-status-port
        port: 15021
        protocol: TCP
        targetPort: 15021
      - name: http
        port: 80
        protocol: TCP
        targetPort: 8080
    type: LoadBalancer

spec.runAsRoot (true | false)

Whether to run the gateway in a privileged container. If not running as root, only ports higher than 1024 can be opened on the container. Default value: false

6.2 - Create egress gateway

Egress traffic

Traffic that’s outbound from a pod that has an Istio sidecar also passes through that sidecar’s container, (more precisely, through Envoy). Therefore, the accessibility of external services depends on the configuration of that Envoy proxy.

By default, Istio configures the Envoy proxy to enable requests for unknown services. Although this provides a convenient way of getting started with Istio, it’s generally a good idea to put stricter controls in place.

Egress traffic without egress gateway Egress traffic without egress gateway

Allow only registered access

You can configure Service Mesh Manager to permit access only to specific external services. For details, see External Services.

Egress gateway

Egress gateways define an exit point from your Istio mesh for outgoing traffic. Egress gateways also allow you to apply Istio features on the traffic that exits the mesh, for example monitoring, routing rules, or retries.

Egress traffic using egress gateway Egress traffic using egress gateway

Why do you need egress gateways? For example:

  • Your organization requires some, or all, outbound traffic to go through dedicated nodes. These nodes could be separated from the rest of the nodes for the purposes of monitoring and policy enforcement.
  • The application nodes of a cluster don’t have public IPs, so the in-mesh services that run on them cannot access the internet directly. Allocating public IPs to the egress gateway nodes and routing egress traffic through the gateway allows for controlled access to external services.

CAUTION:

Using an egress gateway doesn’t restrict outgoing traffic, it only routes it through the egress gateway. To limit access only to selected external services, see External Services.

Create egress gateway

To create an egress gateway and route egress traffic through it, complete the following steps.

Note: The YAML samples work with the Service Mesh Manager demo application. Adjust their parameters (for example, namespace, service name, and so on) as needed for your environment.

CAUTION:

Using an egress gateway doesn’t restrict outgoing traffic, it only routes it through the egress gateway. To limit access only to selected external services, see External Services.
  1. Create an egress gateway using the IstioMeshGateway resource.

    CAUTION:

    By the default, the IstioMeshGateway pod is running without root privileges, therefore it cannot use ports under 1024. Either use ports above 1024 as targetports (for example, 8080 instead of 80) or run the gateway pod with root privileges by setting spec.runAsRoot: true in the IstioMeshGateway custom resource.

    For testing, you can download and apply the following resource to create a new egress gateway deployment and a corresponding service in the smm-demo namespace.

    apiVersion: servicemesh.cisco.com/v1alpha1
    kind: IstioMeshGateway
    metadata:
      name: egress-demo
      namespace: smm-demo
    spec:
      istioControlPlane:
        name: cp-v115x
        namespace: istio-system
      deployment:
        replicas:
          max: 1
          min: 1
      service:
        ports:
        - name: http
          port: 80
          protocol: TCP
          targetPort: 8080
        type: ClusterIP
      type: egress
    
    kubectl apply -f egress-meshgateway.yaml
    

    Expected output:

    istiomeshgateway.servicemesh.cisco.com/egress-demo created
    
  2. Create a Gateway resource for the egress gateway. The Gateway resource connects the Istio configuration resources and the deployment of a matching gateway. Apply the following Gateway resource to configure the outbound port (80 in the previous example) on the egress gateway that you have defined in the previous step.

    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: egress-demo
      namespace: smm-demo
    spec:
      selector:
        gateway-name: egress-demo
        gateway-type: egress
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:
        - "*"
    kubectl apply -f egress-gateway.yaml
    

    Expected output:

    gateway.networking.istio.io/egress-demo created
    
  3. Define a VirtualService resource to direct traffic from the sidecars to the egress gateway.

    Apply the following VirtualService to direct traffic from the sidecars to the egress gateway, and also from the egress gateway to the external service. Edit the VirtualService to match the external service you want to permit access to.

    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: httpbin-egress
      namespace: smm-demo
    spec:
      hosts:
      - httpbin.org
      gateways:
      - egress-demo
      - mesh
      http:
      - match:
        - gateways:
          - mesh
          port: 80
        route:
        - destination:
            host: egress-demo.smm-demo.svc.cluster.local
            port:
              number: 80
      - match:
        - gateways:
          - egress-demo
          port: 80
        route:
        - destination:
            host: httpbin.org
            port:
              number: 80
    
    kubectl apply -f egress-virtualservice.yaml
    

    Expected output:

    virtualservice.networking.istio.io/httpbin-egress created
    
  4. Test access to the external service.

    If you have installed the Service Mesh Manager demo application and used the examples in the previous steps, you can run the following command to start requests from the notifications-v1 workload to the external http-bin service:

    kubectl -n smm-demo set env deployment/notifications-v1 'REQUESTS=http://httpbin.org/get#1'
    
    • If everything is set up correctly, the new egress gateway appears on the MENU > GATEWAYS page. The new egress gateway on the <strong>MENU &gt; GATEWAYS</strong> page The new egress gateway on the <strong>MENU &gt; GATEWAYS</strong> page
    • If there is egress traffic, the gateway appears on theMENU > GATEWAYS page (make sure to select the relevant namespace). Note that the traffic from the gateway to the external service is visible only if you create a ServiceEntry resource for the external service. The new egress gateway on the <strong>MENU &gt; TOPOLOGY</strong> page The new egress gateway on the <strong>MENU &gt; TOPOLOGY</strong> page
  5. If needed, permit access only to specific external services. For details, see External Services.

IstioMeshGateway CR reference

This section describes the fields of the IstioMeshGateway custom resource.

apiVersion (string)

Must be servicemesh.cisco.com/v1alpha1

kind (string)

Must be IstioMeshGateway

spec (object)

The configuration and parameters of the IstioMeshGateway.

spec.type (string, required)

Type of the mesh gateway. Ingress gateways define an entry point into your Istio mesh for incoming traffic, while egress gateways define an exit point from your Istio mesh for outgoing traffic. Possible values:

  • ingress
  • egress

spec.istioControlPlane (object, required)

Specifies the istiocontrolplane cr the istio-proxy connects to by a namespaced name. When upgrading to a new Istio version (thus to a new control plane), this should be upgraded.

For example:

spec:
  istioControlPlane:
    name: cp-v115x
    namespace: istio-system

spec.deployment (object)

Configuration options for the Kubernetes istio-proxy deployment. Metadata like labels and annotations can be set here for the deployment or pods as well, in spec.deployment.metadata.annotations or spec.deployment.podMetadata.annotations.

spec.service (object, required)

Configuration options for the Kubernetes service. Annotations can be set here as well as in spec.service.metadata.annotations, they are often useful in cloud loadbalancer cases, for example to specify some configuration for AWS.

For example:

  service:
    ports:
      - name: tcp-status-port
        port: 15021
        protocol: TCP
        targetPort: 15021
      - name: http
        port: 80
        protocol: TCP
        targetPort: 8080
    type: LoadBalancer

spec.runAsRoot (true | false)

Whether to run the gateway in a privileged container. If not running as root, only ports higher than 1024 can be opened on the container. Default value: false

6.3 - Manage host and port configuration

Service Mesh Manager understands the Gateway CRs of Istio and the gateway’s service configuration in Kubernetes (with the help of the MeshGateway CR), so it can display information about ports, hosts, and protocols that are configured on a specific gateway.

  1. Open the Service Mesh Manager web interface, and navigate to MENU > GATEWAYS.

  2. From the list of gateways, click the gateway you want to monitor.

  3. You can see the host and port configurations on the OVERVIEW tab, in the Ports & Hosts section.

    Istio ingress gateway hosts Istio ingress gateway hosts

    The following information is shown about each entry point.

    • GATEWAY NAME: The name of the gateway.
    • GATEWAY NAMESPACE: The namespace the gateway belongs to.
    • PORT: The list of open ports on the gateway.
    • PROTOCOL: The protocols permitted on the gateway.
    • HOSTS: The host selector that determines which hosts are accessible using the gateway.
    • TLS: The TLS settings applying to the gateway.
  • To modify an existing route, click Edit , change the settings as needed, then click APPLY.
  • To delete a route, click Delete .
  • To create a new entry point, click CREATE NEW.

Create new ingress entry point

You can set up a new entry point for your Istio ingress gateways, and Service Mesh Manager translates your configuration to declarative custom resources.

  1. Navigate to MENU > GATEWAYS > <Gateway-to-modify> > OVERVIEW.

  2. In the Ports & Hosts section, click CREATE NEW.

    Istio ingress gateway create new entry point

  3. Set the parameters of the entry point. As a minimum, you must set the port number for the entry point, and the protocol (for example, HTTP, HTTPS, or GRPC) that is accepted at the entry point.

    Note: DNS resolution is not managed by Service Mesh Manager. Once you’ve configured ingress for a particular service, Service Mesh Manager will display the IP of the ingress gateway service. Do not forget to create the corresponding DNS records that point to this IP.

  4. Click CREATE.

Gateway TLS settings

When setting up a service on a gateway with TLS, you need to configure a certificate for the host(s). You can do that by bringing your own certificate, putting it down in a Kubernetes secret, and configuring it for a gateway server. This works for simple use cases, but involves lots of manual steps when obtaining or renewing a certificate. Automated Certificate Management Environments (ACME) automates these kinds of interactions with the certificate provider.

ACME is most widely used with Let’s Encrypt and - when in a Kubernetes environment - cert-manager. Service Mesh Manager helps you set up cert-manager, and you can quickly obtain a valid Let’s Encrypt certificate through the dashboard with a few clicks.

Note: For details on using your own domain name with Let’s Encrypt, see Using Let’s Encrypt with your own domain name.

To set TLS encryption for a gateway, complete the following steps.

  1. Navigate to MENU > GATEWAYS > <Gateway-to-modify> > OVERVIEW.

  2. In the Ports & Hosts section, click Edit in the row of the gateway you want to modify.

  3. Set PORT PROTOCOL to HTTPS.

    TLS on the gateway with Let&rsquo;s Encrypt TLS on the gateway with Let&rsquo;s Encrypt

  4. Decide how you want to provide the certificate for the gateway.

    • To use Let’s Encrypt, select USE LET’S ENCRYPT FOR TLS, then enter a valid email address into the CONTACT EMAIL field. The provided email address will be used to notify about expirations and to communicate about any issues specific to your account.
    • To use a custom certificate, upload a certificate as a Kubernetes secret, then set the name of the secret in the TLS SECRET NAME field. Note that currently you cannot upload the certificate from the Service Mesh Manager UI, use regular Kubernetes tools instead.
  5. Click CREATE.

6.4 - Routes and traffic management

Note: This section describes the routing rules of ingress gateways. To configure routing rules for in-mesh services, see Routing.

One of the main reasons to use Istio gateways instead of native Kubernetes ingress is that you can use VirtualServices to configure the routing of incoming traffic, just like for in-mesh routes. You can apply Istio concepts to incoming requests, like redirects, rewrites, timeouts, retries, or fault injection.

Service Mesh Manager displays routes and their related configuration on the Gateways page, and gives you the ability to configure routing. Service Mesh Manager translates the inputs to Istio CRs (mainly VirtualServices), then validates and applies them to the cluster.

The MENU > GATEWAYS > <Gateway-to-inspect> > ROUTES page displays the following information about the routes of the gateway.

  • VIRTUAL SERVICE: The name of the VirtualService resource for the gateway. To display the YAML configuration of the VirtualService, click the Show YAML configuration icon next to its name.
  • GATEWAYS: The names of gateways and sidecars that apply these routes.
  • HOSTS: The host selector that determines which hosts are accessible using the route.
  • MATCH: The route applies only to requests matching this expression.
  • DESTINATION: The destinations of the routing rule.
  • ACTIONS: Any special action related to the route (for example, rewrite).
  • PROTOCOL: The protocol permitted in the route.

Istio ingress gateway routes Istio ingress gateway routes

  • To modify an existing route, click Edit .
  • To delete a route, click Delete .
  • To create a new route, click CREATE NEW.

Routing rule precedence

Note the following points about how Service Mesh Manager evaluates the routing rules:

  • Rules are evaluated in top-down order.
  • Rules that match on any traffic are always the last to help avoid rule shadowing.
  • Changing the order of rules is not supported in Service Mesh Manager.

When you specify multiple MATCH arguments, they have a logical OR relationship: the rule matches any request that fits one of the match rules. Within a match rule, you can specify multiple rules that have an AND relation. That way you can match requests against a specific URL and an HTTP method, for example.

Create a new route

To create a new route on the gateway, or to apply redirects, rewrites, or other changes to the incoming requests, complete the following steps.

Note: Rules are evaluated in top-down order. For more details, see Rule precedence.

  1. Navigate to MENU > GATEWAYS > <Gateway-to-modify> > ROUTES.

  2. Click CREATE NEW.

    Istio ingress gateway create new entry point Istio ingress gateway create new entry point

  3. Select the gateways that should apply this rule in the GATEWAYS field.

  4. By default, the new rule matches every incoming request. Click ADD CUSTOM MATCH to select only specific traffic for the rule based on scheme, method, URI, host, port, or authority.

    When you specify multiple MATCH arguments, they have a logical OR relationship: the rule matches any request that fits one of the match rules. Within a match rule, you can specify multiple rules that have an AND relation. That way you can match requests against a specific URL and an HTTP method, for example.

    For example, the following rule matches GET requests, and PUT requests received on port 8080.

    Match requests Match requests

  5. Set the action you want to execute on the matching requests.

    • You can Route the requests to a specific service. To route a portion of the traffic to a different destination, select ADD DESTINATION and use the WEIGHT parameter to split the traffic between multiple destination services.

      Multiple destinations Multiple destinations

      Note: If you want to mirror the traffic (that is, send the same requests to multiple destinations), see Mirroring.

    • Alternatively, you can Redirect the traffic to a specific URI. Redirect rules overwrite the Path portion of the URL with this value. Note that the entire path is replaced, irrespective of the request URI being matched as an exact path or prefix. To overwrite the Authority/Host portion of the URL, set the AUTHORITY FIELD as well.

      Redirect traffic Redirect traffic

  6. Set TIMEOUT and RETRY options as needed for your environment.

    Set timeout and retry parameters Set timeout and retry parameters

  7. Click Apply. The new rule appears on the ROUTES tab.

    You can later edit or delete the routing rule by clicking the Edit or Delete icons, respectively.

6.5 - Using Let's Encrypt with your own domain name

The following procedure shows you how to set up an encrypted HTTPS port under your own domain name for your services, and obtain a matching certificate from Let’s Encrypt.

This requires solving the ACME HTTP-01 challenge, and this involves routing an HTTP request from the ACME server (the Certificate Authority) to the cert-manager challenge-solver pod.

Complete the following steps.

  1. Open the Service Mesh Manager web interface, and navigate to MENU > GATEWAYS > OVERVIEW.

  2. Select the gateway you want secured. Note that the SERVICE TYPE of the gateway must be LoadBalancer. The load balancer determines the IP address(es) to be used for the ACME HTTP-01 challenge. In the following example, it’s istio-meshexpansion-gateway-cp-v115x.

    gateways gateways

  3. Point your domain name to the IP address or DNS name found in the ADDRESS field.

  4. Configure the ingress gateway.

    1. In the Ports & Hosts section, click CREATE NEW in the upper right corner.

    2. Select the HTTPS protocol and the port you want to accept incoming connections on (probably 443).

    3. Enter your domain name into the HOSTS field. To enter multiple domain names, use Enter.

    4. Select Use Let’s Encrypt for TLS to get a certificate for your domain from Let’s Encrypt.

    5. Enter your email address. This address is forwarded to Let’s Encrypt and is used for ACME account management.

    6. Click CREATE.

    7. Two more items appear in the Ports & Hosts list for your domain name:

      • One on the HTTPS port (for example, 443) for the incoming connection requests, and
      • the other on port 80 for solving the ACME HTTP-01 challenge.

      A warning icon shows if the HTTPS port is not valid yet.

      gateways gateways

  5. Wait while the certificate arrives. After a short while the item with port 80 and protocol HTTP disappears, and a green check mark appears next to HTTPS. This shows that the certificate has been issued and is used to secure your domain:

    gateways gateways

  6. Set up routing for your service. Use the gateway, host, and port number you provided in this procedure. For details, see Routes and traffic management.

    gateways gateways

  7. Test that your service can be accessed, and that it shows the proper certificate.

7 - Traffic tap

The traffic tap feature of Service Mesh Manager enables you to monitor live access logs of the Istio sidecar proxies. Each sidecar proxy outputs access information for the individual HTTP requests or HTTP/gRPC streams.

The access logs contain information about the:

  • reporter proxy,
  • source and destination workloads,
  • request,
  • response, as well as the
  • timings.

Note: For workloads that are running on virtual machines, the name of the pod is the hostname of the virtual machine.

Traffic tap using the UI

Traffic tap is also available from the dashboard. To use it, complete the following steps.

  1. Select MENU > TRAFFIC TAP.

  2. Select the reporter (namespace or workload) from the REPORTING SOURCE field.

    Service Mesh Manager tap Service Mesh Manager tap

  3. Click FILTERS to set additional filters, for example, HTTP method, destination, status code, or HTTP headers.

  4. Click START STREAMING.

  5. Select an individual log to see its details:

    Service Mesh Manager tap Service Mesh Manager tap

  6. After you are done, click PAUSE STREAMING.

Traffic tap using the CLI

These examples work out of the box with the demo application packaged with Service Mesh Manager. Change the service name and namespace to match your service.

To watch the access logs for an individual namespace, workload or pod, use the smm tap command. For example, to tap the smm-demo namespace, run:

smm tap ns/smm-demo

The output should be similar to:

✓ start tapping max-rps=0
2022-04-25T10:56:47Z outbound frontpage-v1-776d76965-b7w76 catalog-v1-5864c4b7d7-j5cmf "http GET / HTTP11" 200 121.499879ms "tcp://10.10.48.169:8080"
2022-04-25T10:56:47Z outbound frontpage-v1-776d76965-b7w76 catalog-v1-5864c4b7d7-j5cmf "http GET / HTTP11" 200 123.066985ms "tcp://10.10.48.169:8080"
2022-04-25T10:56:47Z inbound bombardier-66786577f7-sgv8z frontpage-v1-776d76965-b7w76 "http GET / HTTP11" 200 145.422013ms "tcp://10.20.2.98:8080"
2022-04-25T10:56:47Z outbound frontpage-v1-776d76965-b7w76 catalog-v1-5864c4b7d7-j5cmf "http GET / HTTP11" 200 129.024302ms "tcp://10.10.48.169:8080"
2022-04-25T10:56:47Z outbound frontpage-v1-776d76965-b7w76 catalog-v1-5864c4b7d7-j5cmf "http GET / HTTP11" 200 125.462172ms "tcp://10.10.48.169:8080"
2022-04-25T10:56:47Z inbound bombardier-66786577f7-sgv8z frontpage-v1-776d76965-b7w76 "http GET / HTTP11" 200 143.590923ms "tcp://10.20.2.98:8080"
2022-04-25T10:56:47Z outbound frontpage-v1-776d76965-b7w76 catalog-v1-5864c4b7d7-j5cmf "http GET / HTTP11" 200 121.868301ms "tcp://10.10.48.169:8080"
2022-04-25T10:56:47Z inbound bombardier-66786577f7-sgv8z frontpage-v1-776d76965-b7w76 "http GET / HTTP11" 200 145.090036ms "tcp://10.20.2.98:8080"
...

Filter on workload or pod

You can tap into specific workloads and pods, for example:

  • Tap the bookings-v1 workload in the smm-demo namespace:

    smm tap --ns smm-demo workload/bookings-v1
    
  • Tap a pod of the bookings app in the smm-demo namespace:

    POD_NAME=$(kubectl get pod -n smm-demo -l app=bookings -o jsonpath="{.items[0]..metadata.name}")
    smm tap --ns smm-demo pod/$POD_NAME
    

At large volume it’s difficult to find the relevant or problematic logs, but you can use filter flags to display only the relevant lines, for example:

# Show only server errors
smm tap ns/smm-demo --method GET --response-code 500,599

The output can be similar to:

2020-02-06T14:00:13Z outbound frontpage-v1-57468c558c-8c9cb bookings:8080 " GET / HTTP11" 503 173.099µs "tcp://10.10.111.111:8080"                               2020-02-06T14:00:18Z outbound frontpage-v1-57468c558c-8c9cb bookings:8080 " GET / HTTP11" 503 157.164µs "tcp://10.10.111.111:8080"                               2020-02-06T14:00:19Z outbound frontpage-v1-57468c558c-4w26k bookings:8080 " GET / HTTP11" 503 172.541µs "tcp://10.10.111.111:8080"                               2020-02-06T14:00:15Z outbound frontpage-v1-57468c558c-8c9cb bookings:8080 " GET / HTTP11" 503 165.05µs "tcp://10.10.111.111:8080"                                2020-02-06T14:00:15Z outbound frontpage-v1-57468c558c-8c9cb bookings:8080 " GET / HTTP11" 503 125.671µs "tcp://10.10.111.111:8080"                               2020-02-06T14:00:19Z outbound frontpage-v1-57468c558c-8c9cb bookings:8080 " GET / HTTP11" 503 101.701µs "tcp://10.10.111.111:8080"

You can also change the output format to JSON, and use the jq command line tool to further filter or map the log entries, for example:

# Show pods with a specific user-agent
smm tap ns/smm-demo -o json | jq 'select(.request.userAgent=="fasthttp") | .source.name'

The output can be similar to:

"payments-v1-7c955bccdd-vt2pg"
"bookings-v1-7d8d76cd6b-f96tm"
"bookings-v1-7d8d76cd6b-f96tm"
"payments-v1-7c955bccdd-vt2pg"
"bookings-v1-7d8d76cd6b-f96tm"

8 - Managing Kafka clusters

The Streaming Data Manager dashboard is integrated into the Service Mesh Manager dashboard, allowing you to manage and overview your Apache Kafka deployments, including brokers, topics, ACLs, and more. For details on using the dashboard features related to Apache Kafka, see Dashboard.

9 - Configuring the Dashboard

9.1 - Exposing the Dashboard

By default, Service Mesh Manager relies on Kubernetes' built-in authentication and proxying capabilities to allow our users to access the Dashboard. In some cases, it makes sense to allow developers to access the Dashboard via a public URL, to make distributing Service Mesh Manager client binaries easier.

You can download the Service Mesh Manager client binaries from the login page:

Download the CLI Download the CLI

Or alternatively, the deployment can use an OIDC-compliant External Provider for authentication so that there’s no need for downloading and installing the CLI binary.

Expose the dashboard

While planning to expose the dashboard, consider the following:

  1. Does the Kubernetes cluster running Service Mesh Manager support LoadBalancer typed services natively? If not, see exposing via NodePort.
  2. Where to terminate the TLS connections? (Should it be terminated by Istio inside the cluster, or should it be terminated by an external LoadBalancer?)
  3. How to manage the TLS certificate for the dashboard? (Do you want to use Let’s Encrypt for certificates, or does your organization have its own certificate authority?)

For some of the examples, we assume that the external dns controller is installed and functional on the cluster. If not, make sure to manually set up the required DNS record based on your deployment.

This document covers a few scenarios to address the setups based on the answers to the previous questions.

In this scenario, we are assuming that:

  1. Your Kubernetes cluster supports LoadBalancer typed services to expose services externally.
  2. You use Istio to terminate the TLS connections inside the cluster.
  3. You want to use Let’s Encrypt to manage the certificates.
  4. External dns is operational on the cluster.

The dashboard will be exposed on the domain name smm.example.org. To expose Service Mesh Manager on that URL, add the following to the smm ControlPlane resource:

cat > enable-dashboard-expose.yaml <<EOF
spec:
  smm:
   exposeDashboard:
      meshGateway:
        enabled: true
        service:
          annotations:
            external-dns.alpha.kubernetes.io/hostname: smm.example.org.
        tls:
          enabled: true
          letsEncrypt:
            dnsNames:
            - smm.example.org
            enabled: true
            # server: https://acme-staging-v02.api.letsencrypt.org/directory
EOF
kubectl patch controlplane --type=merge --patch "$(cat enable-dashboard-expose.yaml )" smm
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

The dashboard is now available on the https://smm.example.org/ URL.

Note: When external dns is not present on the cluster, make sure that the external name of the MeshGateway service is assigned to the right DNS name. Otherwise, Certificate requests will fail. To check the IP address/name of the service, run the kubectl get service smm-ingressgateway-external --namespace smm-system command. The output should be similar to:

NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)                                      AGE
smm-ingressgateway-external   LoadBalancer   10.10.157.144   afd8bac546b1e46faab0e284fa0dc5da-580525876.eu-north-1.elb.amazonaws.com   15021:30566/TCP,80:32436/TCP,443:30434/TCP   20h

Terminate TLS on the LoadBalancer

To terminate TLS on the LoadBalancer, in the smm ControlPlane resource you must set the .spec.smm.exposeDashboard.meshGateway.tls.enabled value to false.

If the Kubernetes Service requires additional annotations to enable TLS, add these annotations to the ControlPlane resource. For example, for AWS/EKS you can use the following settings to terminate TLS with AWS Certificate Manager:

cat > enable-dashboard-expose.yaml <<EOF
spec:
  smm:
   exposeDashboard:
      meshGateway:
        enabled: true
        service:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:{region}:{user id}:certificate/{id}
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            external-dns.alpha.kubernetes.io/hostname: smm.example.org.
        tls:
          enabled: true
          externalTermination: true
EOF
kubectl patch controlplane --type=merge --patch "$(cat enable-dashboard-expose.yaml )" smm
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

Note: In the previous example, the externalTermination: true instructs Service Mesh Manager to expose a plain http endpoint on port 443 so that the external LoadBalancer can terminate TLS for that port too.

Using NodePort

In this setup the LoadBalancer is managed externally. Each worker node will expose the set ports and you can create a LoadBalancer by pointing it to all the worker node’s relevant port.

To enable NodePort-based exposing of the SMM service, run the following command. This example exposes the HTTP on all worker node’s 40080 port, and https on port 40443.

Note: The https port is only available if the tls settings are explicitly enabled, this example omits that part. Either use the TLS settings from the LoadBalancer example, or check the section on user-provided TLS settings.

cat > enable-dashboard-expose.yaml <<EOF
spec:
  smm:
   exposeDashboard:
      meshGateway:
        enabled: true
        service:
          type: NodePort
          nodePorts:
            http: 40080
            https: 40443
EOF
kubectl patch controlplane --type=merge --patch "$(cat enable-dashboard-expose.yaml )" smm

After that, set up the LoadBalancer and the DNS names manually.

  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

Expose using custom TLS credentials

You can provide a custom TLS secret in the secret called my-own-secret in the smm-system namespace. The following command configures the system to use that for in-cluster TLS termination:

cat > enable-dashboard-expose.yaml <<EOF
spec:
  smm:
   exposeDashboard:
      meshGateway:
        enabled: true
        tls:
          enabled: true
          credentialName: "my-own-secret"
EOF
kubectl patch controlplane --type=merge --patch "$(cat enable-dashboard-expose.yaml )" smm
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

Known limitations in HTTP access

As a security measure, Service Mesh Manager operates only over HTTPS when exposed via an external URL. Make sure that somewhere in the traffic chain some component (Istio or LoadBalancer) terminates the TLS connections, otherwise every login attempt to the dashboard will fail.

9.2 - OIDC authentication

Service Mesh Manager allows for authenticating towards an OIDC External Provider instead of relying on the kubeconfig based authentication. This is useful when your organization already has an existing OIDC Provider that is used for user authentication on the Kubernetes clusters.

Since Service Mesh Manager does not require the Kubernetes cluster to be relying on OIDC authentication, you (or the operator of the cluster) might need to set up additional Groups in the cluster (for details, see the Setting up user permissions).

If your organization uses a central authentication database which is not OIDC compliant, check out Dex. Dex can act as an OIDC provider and supports LDAP, GitHub, or any OAuth2 identity provider as a backend. For an example on setting up Service Mesh Manager to use GitHub authentication using Dex, see Using Dex for authentication.

Note: Even if OIDC is enabled in Service Mesh Manager, you can access Service Mesh Manager from the command line by running smm dashboard. This is a fallback authentication/access method in case the OIDC provider is down.

Prerequisites

Before starting to set up OIDC authentication, make sure that you have already:

Enable OIDC authentication

To enable the OIDC authentication, patch the ControlPlane resource with the following settings:

cat > oidc-enable.yaml <<EOF
spec:
  smm:
    auth:
      oidc:
        enabled: true
        client:
          id: ${OIDC_CLIENT_ID}
          issuerURL: https://${IDENTITY_PROVIDER_EXTERNAL_URL}
          secret: ${OIDC_CLIENT_SECRET}
EOF

Where:

  • ${OIDC_CLIENT_ID} is the client id obtained from the External OIDC Provider of your organization.
  • ${OIDC_CLIENT_SECRET} is the client secret obtained from the External OIDC Provider of your organization.
  • ${IDENTITY_PROVIDER_EXTERNAL_URL} is the URL of the External OIDC Provider of your organization.
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

After this change, the dashboard will allow logging in using an External OIDC Provider:

Login using OIDC

Set up user and group mappings

After completing the previous step, the users will be able to authenticate via OIDC. However, Service Mesh Manager needs to map them to Kubernetes users. As Service Mesh Manager uses Kubernetes RBAC for access control, it relies on the same mapping as the Kubernetes API Server’s OIDC authentication backend.

You can use the following settings in the ControlPlane resource :

spec:
  smm:
    auth:
      oidc:
        username:
            claim:  # Claim to take the username from
            prefix: # Append this prefix to all usernames
        groups:
            claim:  # Claim to take the user's groups from
            prefix: # Append this prefix to all group names the user has
        requiredClaims:
            <CLAIM>: "<VALUE>"  # Only allow authentication if the given claim has the specified value

If the target cluster has OIDC enabled, the following table helps mapping the OIDC options of the API server to the settings of Service Mesh Manager:

API Server Setting Description ControlPlane setting
--oidc-issuer-url URL of the provider which allows the API server to discover public signing keys. Only URLs which use the https:// scheme are accepted. This URL should point to the level below .well-known/openid-configuration .spec.smm.auth.client.issuerURL
--oidc-client-id A client id that all tokens must be issued for. .spec.smm.auth.client.id
A client secret that all tokens must be issued for. .spec.smm.auth.client.secret
--oidc-username-claim JWT claim to use as the user name. By default sub, which is expected to be a unique identifier of the end user. .spec.smm.auth.username.claim
--oidc-username-prefix Prefix prepended to username claims to prevent clashes with existing names (such as system:users). For example, the value oidc: will create usernames like oidc:jane.doe. If this flag isn’t provided and –oidc-username-claim is a value other than email, the prefix defaults to the value of –oidc-issuer-url. Use the - value to disable all prefixing. .spec.smm.auth.username.prefix
--oidc-groups-claim JWT claim to use as the user’s group. If the claim is present, it must be an array of strings. .spec.smm.auth.groups.claim
--oidc-groups-prefix Prefix prepended to group claims to prevent clashes with existing names (such as system:groups). For example, the value oidc: will create group names like oidc:engineering and oidc:infra. .spec.smm.auth.groups.prefix
--oidc-required-claim A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. .spec.smm.auth.requiredClaims
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

Set up user permissions

Note: This step is only required if the target cluster does not already have OIDC authentication set up. If the Kubernetes cluster’s OIDC authentication settings are matching the ones set in the ControlPlane resource, no further action is needed.

By default, when using OIDC authentication, users and groups cannot modify the resources in the target cluster, so you need to create the ClusterRoleBinding right for these Groups or Users.

The groups a given user belongs to is shown in the right hand menu on the user interface:

Menu with group information

In this example, the username is oidc:test@example.org and the user belongs to only one group, called oidc:example-org:test.

If the Kubernetes Cluster is not using OIDC for authentication, create the relevant ClusterRoleBindings against these Groups and Users.

9.2.1 - Using Dex for authentication

Dex is an identity service that uses OpenID Connect to drive authentication for other apps.

Dex acts as a portal to other identity providers through “connectors.” This lets Dex defer authentication to LDAP servers, SAML providers, or established identity providers like GitHub, Google, and Active Directory. Clients write their authentication logic once to talk to Dex, then Dex handles the protocols for a given backend.

This section shows you how to set up GitHub authentication using Service Mesh Manager. To set up other authentication backends such as Active Directory or LDAP, see the DEX Connectors documentation.

Enable GitHub authentication

As GitHub is an OAuth 2 provider, Service Mesh Manager requires a bridge between OAuth 2 (or any other authentication backend) and OIDC.

Prerequisites

Before starting to set up GitHub authentication to Service Mesh Manager, make sure that you have already:

You need the following information to follow this guide:

  • GITHUB_CLIENT_ID: The Client ID from the GitHub OAuth 2 registration.
  • GITHUB_CLIENT_SECRET: The Client Secret from the GitHub OAuth 2 registration.
  • GITHUB_ORG_NAME: The name of the GitHub organization to authenticate against. If you want to support multiple organizations, consult the Dex manual.
  • GITHUB_ADMIN_TEAM_NAME: The name of the GitHub team that contains the users who receive administrative privileges.
  • DEX_EXTERNAL_URL: The URL where Dex will be exposed. This must be separate from the dashboard URL.
  • SMM_DASHBOARD_URL: The URL where the Service Mesh Manager dashboard is exposed.
  • OIDC_CLIENT_SECRET: The secret to be used between Dex and the Service Mesh Manager authentication backend (can be any random string).

To follow the examples, export these values as environment variables from your terminal, as these will be needed in multiple steps:

export GITHUB_CLIENT_ID=<***>
export GITHUB_CLIENT_SECRET=<***>
export GITHUB_ORG_NAME=my-github-org
export GITHUB_ADMIN_TEAM_NAME=admin
export DEX_EXTERNAL_URL=dex.example.org
export SMM_DASHBOARD_URL=smm.example.org
export OIDC_CLIENT_SECRET=$(openssl rand -base64 32) # or any random string

Create namespace for Dex

Dex will be installed into its own namespace for isolation. Create the namespace for it:

kubectl create ns dex

Dex will be exposed externally using Istio, so enable Istio sidecar injection on the namespace:

kubectl label ns dex istio.io/rev=cp-v115x.istio-system

Create MeshGateway for Dex

GitHub needs to access Dex to invoke the OAuth 2 callback, so that Dex can understand what was the result of the authentication on the GitHub side.

Create an externally available MeshGateway:

cat > dex-meshgateway.yaml <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioMeshGateway
metadata:
    labels:
        app.kubernetes.io/instance: dex
        app.kubernetes.io/name: dex-ingress
    name: dex-ingress
    namespace: dex
spec:
    istioControlPlane:
        name: cp-v115x
        namespace: istio-system
    deployment:
      metadata:
        labels:
          app.kubernetes.io/instance: dex
          app.kubernetes.io/name: dex-ingress
          gateway-name: dex-ingress
          gateway-type: ingress
      replicas:
        max: 1
        min: 1
        count: 1
    service:
      metadata:
        annotations:
          external-dns.alpha.kubernetes.io/hostname: ${DEX_EXTERNAL_URL}.
      ports:
      - name: http2
        port: 80
        protocol: TCP
        targetPort: 8080
      - name: https
        port: 443
        protocol: TCP
        targetPort: 8443
      type: LoadBalancer
    type: ingress
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app.kubernetes.io/instance: dex
    app.kubernetes.io/name: dex-ingress
  name: dex-ingress
  namespace: dex
spec:
  selector:
    app.kubernetes.io/instance: dex
    app.kubernetes.io/name: dex-ingress
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: dex-ingress-tls
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app.kubernetes.io/instance: dex
    app.kubernetes.io/name: dex-ingress
  name: dex-ingress
  namespace: dex
spec:
  gateways:
    - dex-ingress
  hosts:
    - '*'
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: dex
        port:
          number: 80
EOF
kubectl apply -f dex-meshgateway.yaml

Get certificates for Dex

The secret referenced in the MeshGateway resource is not yet available. To secure the communication between the end-user’s browser and your Dex installation, enable the Let’s Encrypt support for gateways in Service Mesh Manager:

cat > certs.yaml <<EOF
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: dex-issuer
  namespace: dex
spec:
  acme:
    email: noreply@cisco.com
    preferredChain: ""
    privateKeySecretRef:
      name: smm-letsencrypt-issuer
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dex-tls
  namespace: dex
  annotations:
    acme.smm.cisco.com/gateway-selector: |
      {
        "app.kubernetes.io/instance": "dex",
        "app.kubernetes.io/name": "dex-ingress"
      }      
spec:
  dnsNames:
  - ${DEX_EXTERNAL_URL}
  duration: 2160h0m0s
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: dex-issuer
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  renewBefore: 360h0m0s
  secretName: dex-ingress-tls
  usages:
  - server auth
  - client auth
EOF
kubectl apply -f certs.yaml

After executing the previous commands, check that the Certificate has been successfully issued by running the kubectl get certificate command. The output should be similar to:

NAME      READY   SECRET            AGE
dex-tls   True    dex-ingress-tls   24h

If the READY column shows True, then the Certificate has been issued. If not, refer to the Cert Manager documentation for troubleshooting the issue.

Provision Dex

Now you can install Dex onto the namespace using helm. First create a file called dex-values.yaml for the Dex installation:

cat > dex-values.yaml <<EOF
---
config:
  issuer: https://${DEX_EXTERNAL_URL}
  storage:
    type: kubernetes
    config:
      inCluster: true

  connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $GITHUB_CLIENT_ID
        clientSecret: "$GITHUB_CLIENT_SECRET"
        redirectURI: https://${DEX_EXTERNAL_URL}/callback
        orgs:
        - name: $GITHUB_ORG_NAME
        loadAllGroups: true

  oauth2:
    skipApprovalScreen: true

  staticClients:
    - id: smm-app
      redirectURIs:
        - "https://${SMM_DASHBOARD_URL}/auth/callback"
      name: 'Cisco Service Mesh Manager'
      secret: ${OIDC_CLIENT_SECRET}

service:
  enabled: true
  ports:
    http:
      port: 80

    https:
      port: 443
EOF

Run the following commands to install Dex using these values:

helm repo add dex https://charts.dexidp.io
helm install -n dex dex -f dex-values.yaml dex/dex

Verify that Dex has started successfully by running the kubectl get pods -n dex command. The output should be similar to:

NAME                           READY   STATUS    RESTARTS   AGE
dex-6d879bb86d-pxtvm           2/2     Running   1          20m
dex-ingress-6885b4f747-c5l96   1/1     Running   0          24m

Configure SMM to use OIDC provider

Enable Dex as an OIDC provider to Service Mesh Manager by patching the ControlPlane resource:

cat > smm-oidc-enable.yaml <<EOF
spec:
  smm:
    auth:
      oidc:
        enabled: true
        client:
          id: smm-app
          issuerURL: https://${DEX_EXTERNAL_URL}
          secret: ${OIDC_CLIENT_SECRET}
        groups:
          claim: groups
          prefix: 'oidc:'
        username:
          claim: email
          prefix: 'oidc:'
EOF
kubectl patch --type=merge --patch "$(cat smm-oidc-enable.yaml )" controlplane smm
  • If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
  • If you are using the imperative mode, run the smm operator reconcile command to apply the changes.

Create user mapping

After logging in, the users will be mapped to have the:

  • Username of oidc:<email-of-the-github-user>, and the
  • groups of oidc:$$GITHUB_ORG_NAME:<team-name> for each of the GitHub Teams the user is a member of.

By default, these users and groups cannot modify the resources in the target cluster, so you need to create the ClusterRoleBinding right for these Groups or Users. For example, to grant administrative access to the users in the $GITHUB_ADMIN_TEAM_NAME GitHub Team, run the following command:

cat > allow-admin-access.yaml <<EOF
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: oidc-admin-access
subjects:
- kind: Group
  name: 'oidc:$GITHUB_ORG_NAME:$GITHUB_ADMIN_TEAM_NAME'
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f allow-admin-access.yaml

The groups a given user belongs to is shown in the right hand menu on the user interface:

Menu with group information Menu with group information

In this example, the username is oidc:test@example.org and the user belongs to only one group, called oidc:example-org:test.

Verify login

To test that the login works, navigate to the URL where the Service Mesh Manager dashboard is exposed ($SMM_DASHBOARD_URL), and select Sign in with OIDC.

Login using OIDC Login using OIDC