Secret Management¶

Comprehensive secret management strategy using SOPS (Secrets OPerationS) with Age encryption for GitOps-compatible secret handling.

Overview¶

The RCIIS platform uses SOPS with Age encryption to securely store sensitive configuration data in Git repositories while maintaining GitOps workflows. This approach enables secure, version-controlled secret management that integrates seamlessly with ArgoCD and Kubernetes.

SOPS with Age Encryption¶

Technology Stack¶

SOPS: Mozilla's Secrets OPerationS for file encryption
Age: Modern, secure, and simple encryption tool
KSOPS: Kustomize plugin for SOPS integration
Kubernetes: Native secret resources

Architecture Benefits¶

GitOps Compatible: Encrypted secrets stored in Git
Selective Encryption: Only values encrypted, metadata readable
Key Management: Simple Age key distribution
Audit Trail: Full change history in Git

Age Key Management¶

Key Generation¶

Creating Age Keys:

# Generate new Age key
age-keygen -o ~/.age/rciis-local.key

# Output format
# created: 2024-01-01T00:00:00Z
# public key: age1abcdef1234567890abcdef1234567890abcdef1234567890abcdef123456

# Extract public key
grep "public key:" ~/.age/rciis-local.key

Environment-Specific Keys¶

Key Distribution Strategy:

# Local development
age-keygen -o ~/.age/local.key

# Testing environment
age-keygen -o ~/.age/testing.key

# Staging environment
age-keygen -o ~/.age/staging.key

# Production environment (future)
age-keygen -o ~/.age/production.key

Key Storage and Distribution¶

Secure Key Management: - Local Development: Developer workstation ~/.age/ - CI/CD: GitHub Secrets or similar secure storage - ArgoCD: Kubernetes secrets with Age keys - Backup: Secure offline storage for key recovery

SOPS Configuration¶

SOPS Rules Configuration¶

Repository-wide Configuration (.sops.yaml):

creation_rules:
  # Local development secrets
  - path_regex: apps/rciis/secrets/local/.*\.yaml$
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

  # Testing environment secrets
  - path_regex: apps/rciis/secrets/testing/.*\.yaml$
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age0987654321fedcba0987654321fedcba0987654321fedcba0987654321fedcba

  # Staging environment secrets
  - path_regex: apps/rciis/secrets/staging/.*\.yaml$
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age1111222233334444555566667777888899990000aaaabbbbccccddddeeeeffff

  # Infrastructure secrets
  - path_regex: apps/infra/secrets/.*\.yaml$
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age1111222233334444555566667777888899990000aaaabbbbccccddddeeeeffff

Secret File Structure¶

Application Configuration Secret:

# apps/rciis/secrets/staging/nucleus/appsettings.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nucleus-appsettings
  namespace: nucleus
type: Opaque
stringData:
  appsettings.json: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]

Database Connection Secret:

# apps/rciis/secrets/staging/nucleus/mssql-admin.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nucleus-database
  namespace: nucleus
type: Opaque
stringData:
  connection-string: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]
  username: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]
  password: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]
sops:
  kms: []
  gcp_kms: []
  azure_kv: []
  hc_vault: []
  age:
    - recipient: age1111222233334444555566667777888899990000aaaabbbbccccddddeeeeffff
      enc: LS0tLS1CRUdJTiBBR0UgRU5DUllQVEVEIEZJTEUtLS0tLQ...
  lastmodified: "2024-01-01T00:00:00Z"
  mac: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
  pgp: []
  unencrypted_regex: ^(apiVersion|metadata|kind|type)$
  version: 3.8.1

KSOPS Integration¶

Kustomize Configuration¶

KSOPS Plugin Configuration:

# apps/rciis/nucleus/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: nucleus

resources:
- ../../../secrets/staging/nucleus/

generators:
- secret-generator.yaml

configurations:
- extra/default.conf

transformers:
- ksops-transformer.yaml

replicas:
- name: nucleus-deployment
  count: 2

images:
- name: nucleus-image
  newTag: v1.2.3

Secret Generator¶

KSOPS Secret Generator (secret-generator.yaml):

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: nucleus-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
- ../../../secrets/staging/nucleus/appsettings.yaml
- ../../../secrets/staging/nucleus/mssql-admin.yaml
- ../../../secrets/staging/nucleus/container-registry.yaml
- ../../../secrets/staging/nucleus/config.yaml

ArgoCD Integration¶

Age Key Secret in ArgoCD:

apiVersion: v1
kind: Secret
metadata:
  name: sops-age-key
  namespace: argocd
type: Opaque
data:
  key.txt: |
    # Age private key for SOPS decryption
    AGE-SECRET-KEY-1ABCDEF1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF123456

ArgoCD Repository Configuration:

apiVersion: v1
kind: Secret
metadata:
  name: rciis-devops-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  type: git
  url: [email protected]:MagnaBC/rciis-devops.git
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
    -----END OPENSSH PRIVATE KEY-----

Secret Types and Examples¶

Application Configuration¶

Complete Application Settings:

kindmet   typestr

href="#__codelineno-9-1">apiVersion: v1 class="p">: Secret adata: name: nucleus-config namespace: nucleus class="p">: Opaque ingData: appsettings.json: | { "ConnectionStrings": { "DefaultConnection": "ENC[AES256_GCM,data:...,type:str]", "RedisConnection": "ENC[AES256_GCM,data:...,type:str]" }, "Kafka": { "BootstrapServers": "kafka-cluster-kafka-bootstrap:9092", "GroupId": "nucleus-consumer-group", "SecurityProtocol": "Ssl", "SslCertificateLocation": "/app/certs/client.crt", "SslKeyLocation": "/app/certs/client.key", "SslCaLocation": "/app/certs/ca.crt" }, "MinIO": { "Endpoint": "minio:9000", "AccessKey": "ENC[AES256_GCM,data:...,type:str]", "SecretKey": "ENC[AES256_GCM,data:...,type:str]", "UseSSL": false }, "Authentication": { "JwtKey": "ENC[AES256_GCM,data:...,type:str]", "JwtIssuer": "nucleus-api", "JwtAudience": "nucleus-clients", "JwtExpiryMinutes": 60 }, "ExternalApis": { "CustomsAuthority": { "BaseUrl": "https://api.customs.gov", "ApiKey": "ENC[AES256_GCM,data:...,type:str]", "Certificate": "ENC[AES256_GCM,data:...,type:str]" } } }

Container Registry Credentials¶

Harbor Registry Access:

apiVersion: v1
kind: Secret
metadata:
  name: harbor-registry
  namespace: nucleus
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]

TLS Certificates¶

Custom TLS Certificates:

apiVersion: v1
kind: Secret
metadata:
  name: custom-tls-cert
  namespace: nucleus
type: kubernetes.io/tls
data:
  tls.crt: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]
  tls.key: ENC[AES256_GCM,data:base64encrypted...,iv:...,tag:...,type:str]

API Keys and Tokens¶

External Service Authentication:

apiVersion: v1
kind: Secret
metadata:
  name: external-api-keys
  namespace: nucleus
type: Opaque
stringData:
  slack-webhook-url: ENC[AES256_GCM,data:...,type:str]
  github-token: ENC[AES256_GCM,data:...,type:str]
  monitoring-api-key: ENC[AES256_GCM,data:...,type:str]
  backup-service-token: ENC[AES256_GCM,data:...,type:str]

Secret Operations¶

Creating and Editing Secrets¶

Creating New Secret:

# Set Age key environment
export SOPS_AGE_KEY_FILE=~/.age/staging.key

# Create new secret file
cat > apps/rciis/secrets/staging/nucleus/new-secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: new-secret
  namespace: nucleus
type: Opaque
stringData:
  key1: "plain-text-value"
  key2: "another-value"
EOF

# Encrypt the secret
sops --encrypt --in-place apps/rciis/secrets/staging/nucleus/new-secret.yaml

Editing Existing Secret:

# Edit encrypted secret
sops apps/rciis/secrets/staging/nucleus/appsettings.yaml

# The editor will open with decrypted content
# Make changes and save to re-encrypt

Bulk Secret Operations:

# Encrypt all secrets in a directory
find apps/rciis/secrets/staging -name "*.yaml" -exec sops --encrypt --in-place {} \;

# Decrypt for viewing (don't save)
sops --decrypt apps/rciis/secrets/staging/nucleus/appsettings.yaml

# Update encryption keys for all files
find apps/rciis/secrets/staging -name "*.yaml" -exec sops updatekeys {} \;

CI/CD Integration¶

GitHub Actions Integration¶

Secret Decryption in CI:

name: Deploy Application
on:
  push:
    branches: [master]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Setup SOPS
      uses: mdgreenwald/[email protected]
      with:
        version: '3.8.1'

    - name: Setup Age
      run: |
        curl -Lo age.tar.gz https://github.com/FiloSottile/age/releases/latest/download/age-v1.1.1-linux-amd64.tar.gz
        tar xf age.tar.gz
        sudo mv age/age /usr/local/bin/

    - name: Decrypt secrets
      env:
        SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY_STAGING }}
      run: |
        echo "$SOPS_AGE_KEY" > ~/.age/key.txt
        export SOPS_AGE_KEY_FILE=~/.age/key.txt
        sops --decrypt apps/rciis/secrets/staging/nucleus/appsettings.yaml > /tmp/config.yaml

    - name: Deploy to staging
      run: |
        # Deployment commands here
        kubectl apply -k apps/rciis/nucleus/staging/

ArgoCD Integration¶

SOPS Plugin Configuration:

# ArgoCD ConfigMap for SOPS support
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
  kustomize.path.v1: "/usr/local/bin/kustomize"

Security Best Practices¶

Key Management¶

Key Rotation: Regular rotation of Age keys
Key Distribution: Secure key distribution channels
Access Control: Limit key access to authorized personnel
Backup: Secure backup of encryption keys

Secret Lifecycle¶

Creation: Generate strong, unique secrets
Rotation: Regular secret rotation schedule
Revocation: Immediate revocation of compromised secrets
Cleanup: Remove unused secrets from systems

Operational Security¶

Audit Logging: Track all secret access and modifications
Monitoring: Monitor for unauthorized secret access
Alerting: Alert on secret-related security events
Training: Regular security training for team members

Troubleshooting¶

Common Issues¶

SOPS Decryption Failures:

# Check Age key
echo $SOPS_AGE_KEY_FILE
cat $SOPS_AGE_KEY_FILE

# Verify SOPS configuration
sops --version
sops --decrypt --extract '["metadata"]["name"]' secret.yaml

# Test key access
age-keygen -y ~/.age/key.txt

KSOPS Plugin Issues:

# Check KSOPS installation
which ksops
ksops --version

# Test Kustomize build
kustomize build --enable-alpha-plugins --enable-exec apps/rciis/nucleus/staging/

# Check plugin path
echo $XDG_CONFIG_HOME/kustomize/plugin/viaduct.ai/v1/ksops/ksops

ArgoCD Sync Issues:

# Check ArgoCD application status
argocd app get nucleus-staging

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

# Verify secret deployment
kubectl get secrets -n nucleus
kubectl describe secret nucleus-appsettings -n nucleus

Diagnostic Commands¶

# List encrypted files
find . -name "*.yaml" -exec grep -l "sops:" {} \;

# Verify encryption status
sops --decrypt --extract '["sops"]["version"]' secret.yaml

# Check secret content in cluster
kubectl get secret -o yaml | grep -A 5 -B 5 "stringData\|data"

# Test secret decryption in pod
kubectl exec <pod-name> -- cat /app/config/appsettings.json

Migration and Backup¶

Secret Migration¶

Migrating from Plain Text:

# Backup existing secrets
kubectl get secrets -A -o yaml > secrets-backup.yaml

# Encrypt with SOPS
for file in apps/rciis/secrets/staging/nucleus/*.yaml; do
  sops --encrypt --in-place "$file"
done

# Update Kustomization files
# Add KSOPS generators to kustomization.yaml files

Backup Strategy¶

Secret Backup Procedures:

# Create encrypted backup
tar -czf secrets-backup-$(date +%Y%m%d).tar.gz apps/rciis/secrets/
gpg --cipher-algo AES256 --compress-algo 1 --symmetric secrets-backup-$(date +%Y%m%d).tar.gz

# Store Age keys securely
cp ~/.age/*.key /secure/backup/location/

# Document key fingerprints
for key in ~/.age/*.key; do
  echo "$(basename $key): $(age-keygen -y $key)"
done > key-fingerprints.txt

For advanced SOPS and Age usage, refer to the SOPS documentation and Age documentation.