Leveraging Custos for Gateways

As shown in the following diagram, the Custos acts as a  security middleware for gateways. Basically, it interacts with the federated and local services to provide conglomerated security services. Moreover, it maintains a user store for each gateway and provides user and group related security functions. The Custos services are deployed in a clustered environment and expose a set of APIs for gateways. The Custos provides REST APIs, Python SDK build on top of gRPC stubs. Moreover, anyone can use the gRPC stubs of the Custos services to build clients in their preferred language. Since gRPC supports forward and backward compatibility, seamless version upgrades are supported while keeping existing clients uninterrupted. Furthermore, services can be upgraded or downgraded with zero downtimes using K8 rollout capabilities.

Custos APIs Overview

 The following diagram shows the APIs' functional groups supported by the Custos middleware.

• Tenant Management

          A Tenant represents a  Gateway within the Custos. At first,  gateways should register with the Custos. After successful registration, the tenants get Custos Credentials and those tenants are manually verified by the Custos Admin and will be approved. Approved clients can consume Custos APIs. Tenant Management API group mainly composed of tenant management methods such as register, update, delete, and other tenant-related functions. For further information please check the API Guide.

• User Management

Tenants can register users, enable and disable users, add/remove roles and attributes to users and many more user-related functions are available under user management APIs.

• Group Management

User and Agent Groups are used by Gateways for different perspectives such as access management, sharing management, etc. Custos is capable of maintaining a Group representation of  Users and  Agents and assign attributes, roles to them. Moreover, Custus supports a hierarchical grouping and provides the SCIM interface other than basic group management APIs for interoperability.

• Identity and Access Management

Authentication and Authorization related functions are available under this API Group. For e.g get user tokens, get agent tokens, revoke user sessions, etc.

• Resource and Secret Management

This provides secret management functions such as getting server certificates, get JWKS certificates, get SSH credentials for access to external resources, etc.

• Agent Management

Agents are lightweight distributed components managed under a Gateway. Agents are provided with specific credentials to access  Custos APIs. Agents obtain JWT access tokens from the Custos to access secured APIs, and resources of Gateways. Those access tokens are composed of relevant attributes and roles of agents for authorization purposes.

• Observability Management

The Custos is capable of providing security audit logs and it captures important events such as user logins, status changes of tenant data, and is configurable etc. 

Custos Component Segregation

We have used microservice architectural styles to develop our services. As shown in the below diagram we have developed a set of integration services that orchestrate between core services to fulfill a use case. The core services focus more on a single core function other than focusing on use cases. Some core services are brokering services that connect to external services such as CILogon, Keycloak,  HashCorp Vault, and others are local services that maintain local data in its own database.

Moreover, we use the gRPC protocol for inter-service communication and the Envoy sidecar for REST to gRPC conversion. The K8 Ingress-Controller is used to expose APIs to external traffic. Moreover, the  Linkerd service mesh is used to enable SSL communication between the services and for monitoring purposes.  The Keycloak is used as our IDP implementation, HarshiCorp Vault is used for secret storage, and the CILogon provides Institutional login through federated identity. 

Internals of Custos Microservices

The following sections describe the internal components of the Custos and their attributes. 

A Custos microservice is composed of the following components 

•   Preprocessing Interceptor Pipeline

 Mainly, preprocessing interceptors intercept inbound requests and do necessary validations before dispatching requests to Grpc service. Primarily, there are three main interceptors, Input validator does the request validation such as checking for required parameters and values, Auth Validator is responsible for the token authorization. Moreover, Grpc Tracer publishes important metadata to tracing servers. Interceptors are executed by the Evaluator. Interceptors are the main extensions for Custos services pre validations.

•     Evaluator

     The Evaluator is responsible for executing each interceptor sequentially against incoming requests. Upon successful execution of all the interceptors, the request is dispatched to Grpc service, otherwise, an error message is sent back to the client.  Moreover, evaluators can access external services to pull the required metadata for evaluation. 

• Grpc Service

This is the main service that implements business logic. It might call external services to get their jobs done. For the Custos integration services, this will be the orchestrator component and core services may have its own functions implemented. 

• Status Updater

The Status updater is the post-processing component, that updates operation status after completion of the operation.

• External Service Invoker

 This component contains external services client stubs and synchronous and asynchronous messaging implementation. 

• Data Persistence Handler

The data persistence handler is responsible for saving and retrieving data from DB. It is written on top of the Spring Data JPA framework and it utilizes the Hibernate and Java Persistence API to connect to databases.

Bootstrapping Services

The Custos services are implemented on top of the Spring Boot and Spring Data JPA frameworks. Those are responsible for bootstrap Grpc services and data services of the Custos. Moreover, all services configurations are dynamically loaded through separate configuration service. The data framework uses Hibernate and Java Persistence APIs for data storage. MySQL databases are used as storage.

 SCIM Adapter

There are some open-source SCIM implementations available. Among them, we have chosen WSO2 Charon implementation due to the decoupled nature of components and the ability to use the relevant set of features required. Moreover, it supports registering extended user schema. We have used Charon core for SCIM schema validation. In order to convert the core user/group model to the Custos user/group model, we have implemented a resource manager component that receives messages from core and transfers messages to the Custos services. The following diagram shows the architecture of SCIM Implementation.

Token management and fine-grained authorization

Token-based authentication and authorization are supported through the Custos.  Primarily, Custos itself uses tokens for authorizing its own APIs, and gateways use tokens issued by the Custos to authorize their APIs. The Custos supports OAuth2 protocol and  JWT tokens. Moreover, the Keycloak is used as the token server for the Custos. The following diagram provides insights into the implementation. 

• User authorization

   After successful authentication, the client receives the JWT based access token, refresh token, and Id token of the user.  Tokens contain attributes and roles of users. Based on those attributes and roles clients authorize their users. The user token issuer is configurable to issue relevant information in the token. The Custos Identity service supports authorization code, refresh token and password grant types.

• Agent authentication and authorization

  Agents are deployed by gateways to do some limited and specific set of tasks on behalf of gateways. These agents can be registered with the Custos and provided with an agent Id and agent Secret. 

Then agents can obtain their tokens through the Custos services. These tokens contain attributes and roles of agents. The agent token issuer can be configured separately to the User token issuer. Custos support client credentials, refresh token grant types for agents.

Server and Client Certificate Auto-Updater

We are using   Let's Encrypt certificate issuer for the Custos services. Since Let's Encrypt certificates are expired within 90 days interval, we have set up auto certificate renewal for Custos as shown in the below diagram.

• Cert-manager

Kubernetes Cert-manager talks to Let’s encrypt through ACME protocol and creates a new account for the domain. It stores the account's private key in a K8 secret. Moreover, it fetches a certificate for the domain and stores certificate and private key in Kubernetes secrets. Cert-manager is responsible for certificate auto-updates before expiring them.

•   Custos Cluster Manager

 This component obtains certificates on demand through K8 Secrets via Kubernetes APIs and provides them to clients. So, clients that consume Custos services should use this API to fetch the latest certificates.