Architecture Overview¶

The RCIIS DevOps platform is built on modern cloud-native principles, implementing a robust GitOps workflow with comprehensive security, observability, and operational excellence.

System Architecture¶

graph TB
    subgraph "Developer Experience"
        DEV[Developers]
        IDE[IDEs/Editors]
        LOCAL[Local Clusters]

        DEV --> IDE
        DEV --> LOCAL
    end

    subgraph "Source Control & CI/CD"
        GIT[Git Repository]
        GHA[GitHub Actions]
        REGISTRY[Harbor Registry]

        IDE --> GIT
        GIT --> GHA
        GHA --> REGISTRY
    end

    subgraph "GitOps Control Plane"
        ARGO[ArgoCD]
        VAULT[Secret Management]

        GIT --> ARGO
        VAULT --> ARGO
    end

    subgraph "Infrastructure Layer"
        K8S[Kubernetes Clusters]
        CNI[Cilium/Calico CNI]
        LB[MetalLB/Cloud LB]
        CERT[cert-manager]

        ARGO --> K8S
        K8S --> CNI
        K8S --> LB
        K8S --> CERT
    end

    subgraph "Application Platform"
        GATEWAY[ApiSIX Gateway]
        KAFKA[Strimzi Kafka]
        CAMELK[Camel K]
        STORAGE[MinIO Storage]

        K8S --> GATEWAY
        K8S --> KAFKA
        K8S --> CAMELK
        K8S --> STORAGE
    end

    subgraph "RCIIS Applications"
        NUCLEUS[Nucleus Core]
        API[REST APIs]
        WEB[Web Interface]
        INTEGRATIONS[Integrations]

        GATEWAY --> NUCLEUS
        GATEWAY --> API
        GATEWAY --> WEB
        CAMELK --> INTEGRATIONS
    end

    subgraph "Observability"
        METRICS[Prometheus]
        LOGS[Loki/ELK]
        TRACES[Jaeger]
        DASH[Grafana]

        K8S --> METRICS
        K8S --> LOGS
        K8S --> TRACES
        METRICS --> DASH
        LOGS --> DASH
        TRACES --> DASH
    end

Core Principles¶

1. GitOps-First Approach¶

Everything is defined as code and version-controlled in Git:

• Infrastructure as Code: Kubernetes manifests, Helm charts
• Configuration as Code: Application configurations, secrets (encrypted)
• Policy as Code: RBAC, network policies, security policies
• Documentation as Code: This documentation, inline comments

2. Security by Design¶

graph LR
    subgraph "Secret Management Flow"
        PLAIN[Plain Secret] --> SOPS[SOPS Encryption]
        SOPS --> ENCRYPTED[Encrypted in Git]
        ENCRYPTED --> KSOPS[KSOPS Plugin]
        KSOPS --> K8S[Kubernetes Secret]
    end

    subgraph "Key Management"
        AGE[Age Keys]
        GPG[GPG Keys]
        CLOUD[Cloud KMS]

        AGE --> SOPS
        GPG --> SOPS
        CLOUD --> SOPS
    end

• Zero Trust Networking: All traffic encrypted, authenticated, and authorized
• Secret Encryption: SOPS with Age/GPG for GitOps-compatible secret management
• RBAC: Fine-grained role-based access control
• Network Segmentation: Kubernetes network policies and service mesh

3. Multi-Environment Strategy¶

graph TD
    DEV[Local Development] --> TEST[Testing Environment]
    TEST --> STAGE[Staging Environment]
    STAGE --> PROD[Production Environment]

    subgraph "Promotion Strategy"
        GIT_DEV[Feature Branch] --> GIT_TEST[Test Branch]
        GIT_TEST --> GIT_STAGE[Staging Branch]
        GIT_STAGE --> GIT_MAIN[Main Branch]
    end

    subgraph "Deployment Strategy"
        APP_SET[ApplicationSet]
        ROLLING[Rolling Sync]
        WAVES[Wave Strategy]

        APP_SET --> ROLLING
        ROLLING --> WAVES
    end

Component Architecture¶

Infrastructure Layer¶

Kubernetes Foundation¶

• Cluster Management: Multi-node clusters with HA control plane
• Container Runtime: containerd with security hardening
• Network CNI: Cilium (preferred) or Calico with eBPF acceleration
• Storage: Local storage with backup strategies

Networking Stack¶

graph TB
    subgraph "External Traffic"
        INTERNET[Internet]
        DNS[DNS]
    end

    subgraph "Ingress Layer"
        LB[Load Balancer]
        INGRESS[Ingress Controller]
        GATEWAY[API Gateway]
    end

    subgraph "Service Mesh"
        PROXY[Envoy Proxies]
        MTL[mTLS]
        POLICY[Traffic Policies]
    end

    subgraph "Application Services"
        NUCLEUS[Nucleus]
        API_SVC[API Services]
        KAFKA_SVC[Kafka]
        STORAGE_SVC[Storage]
    end

    INTERNET --> DNS
    DNS --> LB
    LB --> INGRESS
    INGRESS --> GATEWAY
    GATEWAY --> PROXY
    PROXY --> NUCLEUS
    PROXY --> API_SVC
    PROXY --> KAFKA_SVC
    PROXY --> STORAGE_SVC

Application Platform¶

Message Streaming (Kafka + Strimzi)¶

• Event Streaming: Apache Kafka for real-time data streaming
• Operator Management: Strimzi operator for Kafka lifecycle
• Topics: Structured topics for different data types
• Connectors: Kafka Connect for external system integration

Integration Platform (Camel K)¶

• Enterprise Integration: Apache Camel integration patterns
• Serverless: Knative-based scaling for integration routes
• Cloud-Native: Native Kubernetes integration with operators
• Polyglot: Support for Java, Groovy, Kotlin, XML, YAML routes

API Gateway (ApiSIX)¶

• Traffic Management: Load balancing, rate limiting, circuit breaking
• Security: Authentication, authorization, SSL termination
• Observability: Metrics, logging, distributed tracing
• Plugin Ecosystem: Extensible with Lua and external plugins

Object Storage (MinIO)¶

• S3 Compatibility: AWS S3-compatible API
• Distributed Storage: Clustering and data replication
• Encryption: At-rest and in-transit encryption
• Backup Integration: Automated backup and restore

Application Layer¶

Nucleus Core Application¶

graph TB
    subgraph "Nucleus Architecture"
        API[REST API Layer]
        BIZ[Business Logic]
        DATA[Data Access Layer]
        CACHE[Caching Layer]

        API --> BIZ
        BIZ --> DATA
        BIZ --> CACHE
    end

    subgraph "External Integrations"
        CUSTOMS[Customs Systems]
        BANKS[Banking Systems]
        GOV[Government APIs]

        BIZ --> CUSTOMS
        BIZ --> BANKS
        BIZ --> GOV
    end

    subgraph "Data Persistence"
        DB[Primary Database]
        BACKUP[Backup Systems]
        AUDIT[Audit Logs]

        DATA --> DB
        DB --> BACKUP
        DATA --> AUDIT
    end

Deployment Patterns¶

ApplicationSet Strategy¶

RCIIS uses ArgoCD ApplicationSets for multi-environment deployments:

# Rolling Sync Strategy
strategy:
  type: RollingSync
  rollingSync:
    steps:
      # 1. Testing environment first
      - matchExpressions:
          - key: env
            operator: In
            values: ["testing"]
        maxUpdate: 2
      # 2. Staging environment second  
      - matchExpressions:
          - key: env
            operator: In
            values: ["staging"]
        maxUpdate: 2
      # 3. Production last (when enabled)
      - matchExpressions:
          - key: env
            operator: In
            values: ["production"]
        maxUpdate: 1

Wave-Based Deployment¶

Applications are deployed in dependency order using waves:

• Wave 1: Infrastructure (secrets, configurations)
• Wave 2: Platform services (Kafka, storage, gateway)
• Wave 3: Application services (Nucleus, APIs, UI)

Multi-Source Applications¶

Applications use multiple Git sources for separation of concerns:

sources:
  # Values repository (this repo)
  - repoURL: '[email protected]:MagnaBC/rciis-devops.git'
    targetRevision: master
    ref: values
  # Chart repository (Helm registry or Git)
  - repoURL: 'harbor.devops.africa/rciis'
    chart: 'rciis'
    helm:
      valueFiles:
        - '$values/apps/rciis/nucleus/staging/values.yaml'

Data Flow Architecture¶

Request Processing Flow¶

sequenceDiagram
    participant C as Client
    participant G as API Gateway
    participant N as Nucleus
    participant K as Kafka
    participant D as Database
    participant E as External System

    C->>G: API Request
    G->>G: Authentication & Rate Limiting
    G->>N: Validated Request
    N->>D: Query/Update Data
    N->>K: Publish Event
    K->>E: Trigger Integration
    E-->>K: Response Event
    K-->>N: Process Response
    N->>G: API Response
    G->>C: Final Response

Event Streaming Architecture¶

graph TB
    subgraph "Event Sources"
        NUCLEUS[Nucleus Events]
        EXT[External Events]
        SCHED[Scheduled Events]
    end

    subgraph "Kafka Cluster"
        TOPIC1[customs.transactions]
        TOPIC2[customs.declarations]
        TOPIC3[system.events]
        TOPIC4[integration.events]
    end

    subgraph "Event Consumers"
        ANALYTICS[Analytics Service]
        AUDIT[Audit Service]
        NOTIFY[Notification Service]
        CAMEL[Camel K Routes]
    end

    NUCLEUS --> TOPIC1
    NUCLEUS --> TOPIC2
    EXT --> TOPIC3
    SCHED --> TOPIC4

    TOPIC1 --> ANALYTICS
    TOPIC1 --> AUDIT
    TOPIC2 --> ANALYTICS
    TOPIC2 --> CAMEL
    TOPIC3 --> NOTIFY
    TOPIC4 --> CAMEL

Scalability and Performance¶

Horizontal Scaling¶

• Application Pods: HPA based on CPU/memory/custom metrics
• Kafka Partitions: Distributed across multiple brokers
• Database: Read replicas and connection pooling
• Cache Layer: Distributed Redis/Hazelcast clusters

Performance Optimization¶

• Connection Pooling: Database and HTTP connections
• Caching Strategy: Multi-level caching (L1: local, L2: distributed)
• CDN Integration: Static asset delivery optimization
• Compression: gRPC/HTTP2 with compression

Resource Management¶

# Example resource configuration
resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "2000m"
    memory: "4Gi"

# Pod Disruption Budget
podDisruptionBudget:
  minAvailable: 50%

# Horizontal Pod Autoscaler
hpa:
  minReplicas: 3
  maxReplicas: 50
  targetCPUUtilizationPercentage: 70

Security Architecture¶

Defense in Depth¶

1. Network Security: CNI policies, service mesh mTLS
2. Identity & Access: RBAC, service accounts, external identity providers
3. Secret Management: SOPS encryption, rotation policies
4. Runtime Security: Pod security policies, admission controllers
5. Data Protection: Encryption at rest and in transit

Compliance Framework¶

• Data Sovereignty: Regional data residency requirements
• Audit Logging: Comprehensive audit trails
• Access Controls: Fine-grained permissions
• Encryption Standards: Industry-standard encryption algorithms

Observability Strategy¶

Three Pillars of Observability¶

graph TB
    subgraph "Metrics"
        PROM[Prometheus]
        GRAFANA[Grafana]
        ALERTS[AlertManager]
    end

    subgraph "Logging"
        LOKI[Loki]
        ELASTIC[Elasticsearch]
        KIBANA[Kibana]
    end

    subgraph "Tracing"
        JAEGER[Jaeger]
        ZIPKIN[Zipkin]
        OTLP[OpenTelemetry]
    end

    subgraph "Applications"
        APP[Applications]
        INFRA[Infrastructure]
        PLATFORM[Platform]
    end

    APP --> PROM
    APP --> LOKI
    APP --> JAEGER
    INFRA --> PROM
    INFRA --> LOKI
    PLATFORM --> PROM
    PLATFORM --> LOKI

Next Steps¶

Explore specific aspects of the architecture:

• GitOps Workflow - Detailed GitOps implementation
• Directory Structure - Code organization patterns
• Environment Strategy - Multi-environment management
• Networking Architecture - Network design and security