Add hybrid multi-tenancy documentation (Capsule + Kamaji)
This commit is contained in:
govardhan
244
HYBRID-MULTI-TENANCY-GUIDE.md
Normal file
244
HYBRID-MULTI-TENANCY-GUIDE.md
Normal file
@ -0,0 +1,244 @@
|
||||
# Hybrid Multi-Tenancy Model: Capsule + Kamaji
|
||||
|
||||
## Overview
|
||||
|
||||
Your Kubernetes cluster now supports **TWO types of tenants**:
|
||||
|
||||
### 1. **Capsule Tenants** (Lightweight, Namespace-based)
|
||||
- Best for: Internal teams, dev/qa/staging environments
|
||||
- Isolation: Namespace-level
|
||||
- Overhead: Very low
|
||||
- User experience: Limited Kubernetes (namespaces only)
|
||||
|
||||
### 2. **Kamaji Tenants** (Virtual Clusters)
|
||||
- Best for: External customers, production workloads requiring full cluster experience
|
||||
- Isolation: Control plane-level
|
||||
- Overhead: Medium (dedicated API server per tenant)
|
||||
- User experience: Full Kubernetes cluster
|
||||
|
||||
---
|
||||
|
||||
## Current Tenants
|
||||
|
||||
### Capsule Tenants
|
||||
|
||||
#### 1. **dev-team**
|
||||
- **Owner**: dev user
|
||||
- **Quota**: 5 namespaces max
|
||||
- **Resources**:
|
||||
- Max 50 pods
|
||||
- Max 8 CPU cores (limits), 4 cores (requests)
|
||||
- Max 16 GiB memory (limits), 8 GiB (requests)
|
||||
- Max 10 PVCs, 10 services
|
||||
- **Network**: Isolated, can only talk to dev-team namespaces
|
||||
- **Storage**: standard, hostpath
|
||||
- **Access**: Login to Rancher with `dev` / `devuser123456`
|
||||
|
||||
#### 2. **prod-team**
|
||||
- **Quota**: 10 namespaces max
|
||||
- Similar resource quotas (check tenant spec for details)
|
||||
|
||||
#### 3. **qa-team**
|
||||
- **Quota**: 7 namespaces max
|
||||
- Similar resource quotas (check tenant spec for details)
|
||||
|
||||
### Kamaji Tenants
|
||||
|
||||
#### 1. **customer1** (Virtual Cluster)
|
||||
- **Version**: Kubernetes v1.28.0
|
||||
- **Control Plane**: Dedicated API server, controller-manager, scheduler
|
||||
- **Endpoint**: https://160.30.114.10:31443
|
||||
- **Kubeconfig**: `~/Documents/kuber/customer1-kubeconfig-external.yaml`
|
||||
- **Resources**:
|
||||
- API Server: 250m-500m CPU, 512Mi-1Gi memory
|
||||
- Controller Manager: 125m-250m CPU, 256Mi-512Mi memory
|
||||
- Scheduler: 125m-250m CPU, 256Mi-512Mi memory
|
||||
- **Pod CIDR**: 10.244.0.0/16
|
||||
- **Service CIDR**: 10.96.0.0/16
|
||||
- **Access**: Use kubeconfig file
|
||||
|
||||
---
|
||||
|
||||
## When to Use Which?
|
||||
|
||||
### Use **Capsule** when:
|
||||
✅ Internal teams (dev, qa, staging)
|
||||
✅ Simple app deployments
|
||||
✅ Resource-constrained environments
|
||||
✅ Need Rancher UI access
|
||||
✅ Don't need cluster-admin features
|
||||
✅ Want low overhead
|
||||
|
||||
### Use **Kamaji** when:
|
||||
✅ External customers paying for dedicated clusters
|
||||
✅ Need complete Kubernetes API experience
|
||||
✅ Want to install CRDs or cluster-level resources
|
||||
✅ Need different Kubernetes versions per tenant
|
||||
✅ Strong isolation requirements
|
||||
✅ Selling "Kubernetes-as-a-Service"
|
||||
|
||||
---
|
||||
|
||||
## Managing Capsule Tenants
|
||||
|
||||
### Add User to Tenant
|
||||
```bash
|
||||
kubectl patch tenant dev-team --type='json' \
|
||||
-p='[{"op": "add", "path": "/spec/owners/-", "value": {"kind": "User", "name": "newuser"}}]'
|
||||
```
|
||||
|
||||
### Update Resource Quotas
|
||||
```bash
|
||||
kubectl edit tenant dev-team
|
||||
# Modify spec.resourceQuotas.items[0].hard
|
||||
```
|
||||
|
||||
### Create Namespace as Tenant Owner
|
||||
```bash
|
||||
# Login as dev user in Rancher, create namespace in UI
|
||||
# Or use kubectl with dev user credentials
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Managing Kamaji Tenants
|
||||
|
||||
### Create New Tenant
|
||||
```bash
|
||||
kubectl apply -f - << 'YAML'
|
||||
apiVersion: kamaji.clastix.io/v1alpha1
|
||||
kind: TenantControlPlane
|
||||
metadata:
|
||||
name: customer2
|
||||
namespace: kamaji-system
|
||||
spec:
|
||||
controlPlane:
|
||||
deployment:
|
||||
replicas: 1
|
||||
service:
|
||||
serviceType: ClusterIP
|
||||
kubernetes:
|
||||
version: "v1.28.0"
|
||||
networkProfile:
|
||||
port: 6443
|
||||
podCidr: "10.245.0.0/16" # Different from customer1
|
||||
serviceCidr: "10.97.0.0/16" # Different from customer1
|
||||
addons:
|
||||
coreDNS: {}
|
||||
kubeProxy: {}
|
||||
YAML
|
||||
```
|
||||
|
||||
### Get Tenant Kubeconfig
|
||||
```bash
|
||||
kubectl get secret customer2-admin-kubeconfig -n kamaji-system \
|
||||
-o jsonpath='{.data.admin\.conf}' | base64 -d > customer2-kubeconfig.yaml
|
||||
```
|
||||
|
||||
### Create NodePort for External Access
|
||||
```bash
|
||||
kubectl apply -f - << 'YAML'
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: customer2-external
|
||||
namespace: kamaji-system
|
||||
spec:
|
||||
type: NodePort
|
||||
selector:
|
||||
kamaji.clastix.io/name: customer2
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
targetPort: 6443
|
||||
nodePort: 31444 # Different port for each tenant
|
||||
YAML
|
||||
```
|
||||
|
||||
### Update Kubeconfig for External Access
|
||||
```bash
|
||||
sed 's|server: https://.*:6443|server: https://160.30.114.10:31444|g' \
|
||||
customer2-kubeconfig.yaml > customer2-kubeconfig-external.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resource Usage
|
||||
|
||||
### Capsule
|
||||
- **dev-team**: ~0 overhead (just RBAC policies)
|
||||
- **prod-team**: ~0 overhead
|
||||
- **qa-team**: ~0 overhead
|
||||
|
||||
### Kamaji
|
||||
- **Etcd cluster**: ~3 GB RAM (3 replicas)
|
||||
- **Kamaji controller**: ~256 MB RAM
|
||||
- **customer1 control plane**: ~1.5 GB RAM
|
||||
- **Per additional tenant**: ~1.5 GB RAM
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Physical Kubernetes Cluster │
|
||||
│ ┌────────────────────────────────────────────────────────┐ │
|
||||
│ │ Rancher (Cluster Management) │ │
|
||||
│ └────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────┐ ┌──────────────────────────────┐ │
|
||||
│ │ Capsule Tenants │ │ Kamaji Tenants │ │
|
||||
│ │ ──────────────── │ │ ───────────────── │ │
|
||||
│ │ • dev-team │ │ ┌────────────────────────┐ │ │
|
||||
│ │ - 5 namespaces │ │ │ customer1 │ │ │
|
||||
│ │ - 50 pods max │ │ │ ├─ API Server │ │ │
|
||||
│ │ - 8 CPU max │ │ │ ├─ Controller Manager │ │ │
|
||||
│ │ │ │ │ ├─ Scheduler │ │ │
|
||||
│ │ • prod-team │ │ │ └─ Etcd (shared) │ │ │
|
||||
│ │ • qa-team │ │ └────────────────────────┘ │ │
|
||||
│ └─────────────────────┘ └──────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────┐ │
|
||||
│ │ Shared Worker Nodes (4 nodes, 16 cores) │ │
|
||||
│ └────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Analysis
|
||||
|
||||
### Capsule (3 tenants)
|
||||
- Infrastructure: $0 (pure RBAC)
|
||||
- Management: Minimal
|
||||
|
||||
### Kamaji (1 tenant)
|
||||
- Etcd cluster: 3 GB RAM
|
||||
- Control plane: 1.5 GB RAM per tenant
|
||||
- Total: ~4.5 GB RAM for first tenant, +1.5 GB per additional
|
||||
|
||||
**Recommendation**: Use Capsule for internal teams, Kamaji for paying customers
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Capsule multi-tenancy configured
|
||||
2. ✅ Kamaji virtual clusters operational
|
||||
3. ⏭️ Create billing/metering for Kamaji tenants
|
||||
4. ⏭️ Add monitoring per tenant
|
||||
5. ⏭️ Configure backup/restore per tenant
|
||||
6. ⏭️ Implement resource quotas enforcement
|
||||
|
||||
---
|
||||
|
||||
## Access Summary
|
||||
|
||||
| Tenant | Type | Access Method | Endpoint |
|
||||
|--------|------|---------------|----------|
|
||||
| dev-team | Capsule | Rancher UI | https://rancher.connectvm.cloud |
|
||||
| prod-team | Capsule | Rancher UI | https://rancher.connectvm.cloud |
|
||||
| qa-team | Capsule | Rancher UI | https://rancher.connectvm.cloud |
|
||||
| customer1 | Kamaji | Kubeconfig | https://160.30.114.10:31443 |
|
||||
|
||||
Reference in New Issue
Block a user