Skip to content

SOPS Encryption

SOPS (Secrets OPerationS) is the cornerstone of secret management in RCIIS DevOps, enabling GitOps workflows while maintaining security. This guide covers implementation, best practices, and troubleshooting.

Overview

SOPS encrypts YAML, JSON, and other structured data files while preserving their structure and enabling partial encryption. This allows secrets to be stored in Git repositories safely while maintaining the benefits of GitOps workflows.

graph LR
    subgraph "Developer Workflow"
        PLAIN[Plain Secret] --> EDIT[Edit with SOPS]
        EDIT --> ENCRYPT[Encrypted Secret]
        ENCRYPT --> GIT[Git Repository]
    end

    subgraph "Deployment Workflow"
        GIT --> ARGO[ArgoCD]
        ARGO --> KSOPS[KSOPS Plugin]
        KSOPS --> DECRYPT[Decrypt Secret]
        DECRYPT --> K8S[Kubernetes Secret]
    end

    subgraph "Key Management"
        AGE[Age Keys]
        GPG[GPG Keys]
        CLOUD[Cloud KMS]

        AGE -.-> EDIT
        GPG -.-> EDIT
        CLOUD -.-> EDIT
    end

SOPS Configuration

Global Configuration (.sops.yaml)

The repository root contains a .sops.yaml file that defines encryption rules:

creation_rules:
  # Rule for all encrypted files
  - path_regex: '\.enc\.ya?ml$'
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: >-
      age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3290km,
      age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs7890ab

  # Environment-specific rules
  - path_regex: 'apps/rciis/secrets/local/.*\.ya?ml$'
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3290km

  - path_regex: 'apps/rciis/secrets/staging/.*\.ya?ml$'
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: >-
      age1staging1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs456def,
      age1backup1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs789ghi

  - path_regex: 'apps/rciis/secrets/production/.*\.ya?ml$'
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    kms: >-
      arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
    age: >-
      age1prod1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs123abc,
      age1backup1qyqszqgpqyqszqgpqyqszqgpqyqs789ghi,
      age1dr1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs456xyz

Configuration Options

Option Purpose Example
path_regex Files matching this pattern '\.enc\.ya?ml$'
unencrypted_regex Fields to leave unencrypted '^(apiVersion\|metadata\|kind\|type)$'
encrypted_regex Fields to always encrypt '^(data\|stringData)$'
age Age public keys for encryption age1abc...xyz
pgp PGP fingerprints FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
kms Cloud KMS keys arn:aws:kms:region:account:key/key-id

Age Key Management

Key Generation

# Generate new age key pair
age-keygen -o ~/.config/sops/age/keys.txt

# Display public key for sharing
age-keygen -y ~/.config/sops/age/keys.txt

# Generate key for automation
age-keygen -o /tmp/age-key.txt

# Extract public key
export AGE_PUBLIC_KEY=$(age-keygen -y /tmp/age-key.txt)

# Set private key as secret
export SOPS_AGE_KEY_FILE=/tmp/age-key.txt

# Generate with strong entropy
age-keygen -o /secure/location/production-key.txt

# Backup key securely
cp /secure/location/production-key.txt /backup/location/

# Set restrictive permissions
chmod 600 /secure/location/production-key.txt

Key Storage Best Practices

Local Development

# Store keys in standard location
mkdir -p ~/.config/sops/age
chmod 700 ~/.config/sops/age

# Set proper permissions
chmod 600 ~/.config/sops/age/keys.txt

# Environment variable (optional)
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt

Production Environment

  • Hardware Security Modules (HSM): For maximum security
  • Key Management Services: AWS KMS, Azure Key Vault, Google Cloud KMS
  • Secure File Systems: Encrypted filesystems with restricted access
  • Backup Strategy: Multiple secure locations with encryption

Working with Encrypted Secrets

Creating New Secrets

Step 1: Create Plain Secret

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nucleus-config
  namespace: rciis-staging
type: Opaque
stringData:
  database-url: "postgresql://user:password@localhost:5432/nucleus"
  api-key: "super-secret-api-key-12345"
  jwt-secret: "jwt-signing-secret-67890"

Step 2: Encrypt with SOPS

# Encrypt and save as .enc.yaml
sops -e secret.yaml > secret.enc.yaml

# Remove plain file
rm secret.yaml

# Verify encryption
cat secret.enc.yaml

Step 3: Encrypted Result

apiVersion: v1
kind: Secret
metadata:
    name: nucleus-config
    namespace: rciis-staging
type: Opaque
stringData:
    database-url: ENC[AES256_GCM,data:QmVzdCBwcmFjdGljZXM=,iv:abcd1234,tag:efgh5678,type:str]
    api-key: ENC[AES256_GCM,data:c2VjdXJlIGFwaS1rZXk=,iv:ijkl9012,tag:mnop3456,type:str]
    jwt-secret: ENC[AES256_GCM,data:and0LXNpZ25pbmctc2VjcmV0,iv:qrst7890,tag:uvwx1234,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3290km
          enc: |
                -----BEGIN AGE ENCRYPTED FILE-----
                YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxQzJMWE...
                -----END AGE ENCRYPTED FILE-----
    lastmodified: "2024-01-15T10:30:00Z"
    mac: ENC[AES256_GCM,data:abc123,iv:def456,tag:ghi789,type:str]
    pgp: []
    version: 3.8.1

Editing Encrypted Secrets

# Edit encrypted file directly
sops secret.enc.yaml

# SOPS will decrypt, open editor, and re-encrypt on save

Decrypting for Debugging

# View decrypted content
sops -d secret.enc.yaml

# Decrypt to file (be careful!)
sops -d secret.enc.yaml > secret-decrypted.yaml
# Remember to delete after use!
rm secret-decrypted.yaml

KSOPS Integration

KSOPS (Kustomize SOPS) enables Kustomize and ArgoCD to work with SOPS-encrypted files.

Secret Generator Configuration

# secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: nucleus-secrets
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - database-config.enc.yaml
  - api-keys.enc.yaml
  - certificates.enc.yaml

Kustomization Integration

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../base

generators:
  - secret-generator.yaml

generatorOptions:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

Building with KSOPS

# Local build with KSOPS
kustomize build --enable-alpha-plugins --enable-exec .

# Verify secrets are decrypted
kustomize build --enable-alpha-plugins --enable-exec . | grep -A 10 "kind: Secret"

Advanced SOPS Features

Partial Encryption

Use encrypted_regex and unencrypted_regex to control what gets encrypted:

# .sops.yaml
creation_rules:
  - path_regex: '\.enc\.ya?ml$'
    # Only encrypt data and stringData fields
    encrypted_regex: '^(data|stringData)$'
    # Keep metadata readable
    unencrypted_regex: '^(apiVersion|metadata|kind|type)$'
    age: age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3290km

Result: 

apiVersion: v1  # Unencrypted
kind: Secret    # Unencrypted
metadata:       # Unencrypted
  name: my-secret
  namespace: default
type: Opaque    # Unencrypted
stringData:     # Encrypted
  password: ENC[...]
  api-key: ENC[...]

Multiple Recipients

# .sops.yaml - Multiple age keys
creation_rules:
  - path_regex: 'production/.*\.ya?ml$'
    age: >-
      age1admin1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs123abc,
      age1backup1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs456def,
      age1ops1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs789ghi

Key Rotation

# Rotate to new age key
sops updatekeys secret.enc.yaml

# Rotate all files in directory
find . -name "*.enc.yaml" -exec sops updatekeys {} \;

# Add new key to existing file
sops -r -i --add-age age1new... secret.enc.yaml

# Remove old key
sops -r -i --rm-age age1old... secret.enc.yaml

Cloud KMS Integration

AWS KMS

# .sops.yaml
creation_rules:
  - path_regex: 'production/.*\.ya?ml$'
    kms: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
    age: age1backup... # Fallback for local development

Azure Key Vault

creation_rules:
  - path_regex: 'production/.*\.ya?ml$'
    azure_kv: https://myvault.vault.azure.net/keys/mykey/version

Google Cloud KMS

creation_rules:
  - path_regex: 'production/.*\.ya?ml$'
    gcp_kms: projects/myproject/locations/global/keyRings/mykeyring/cryptoKeys/mykey

Security Best Practices

Key Management

  1. Principle of Least Privilege: Only grant access to necessary keys
  2. Key Rotation: Regularly rotate encryption keys (quarterly recommended)
  3. Backup Strategy: Secure backup of keys in multiple locations
  4. Access Logging: Monitor key usage and access patterns
  5. Secure Deletion: Properly destroy old keys and temporary files

File Security

  1. Never Commit Plain Secrets: Always encrypt before committing
  2. Verify Encryption: Check that files are properly encrypted
  3. Audit Trail: Track who has access to decrypt secrets
  4. Environment Separation: Use different keys per environment
  5. Regular Reviews: Periodically review and clean up secrets

Operational Security

# Pre-commit hook to prevent unencrypted secrets
#!/bin/bash
# .git/hooks/pre-commit

# Check for potential secrets in staged files
if git diff --cached --name-only | grep -E "\.(yaml|yml)$" | xargs grep -l "password\|secret\|key" | grep -v "\.enc\."; then
    echo "ERROR: Potential unencrypted secrets detected!"
    echo "Please encrypt secrets with SOPS before committing."
    exit 1
fi

Troubleshooting

Common Issues

SOPS Can't Decrypt Files

# Check age key file exists and has correct permissions
ls -la ~/.config/sops/age/keys.txt

# Verify age key matches recipients
age-keygen -y ~/.config/sops/age/keys.txt

# Check SOPS configuration
sops --version
cat .sops.yaml

KSOPS Plugin Not Found

# Verify KSOPS is installed and in PATH
which ksops
ksops --help

# Install KSOPS if missing
curl -s https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh | bash

Permission Denied Errors

# Fix age key permissions
chmod 600 ~/.config/sops/age/keys.txt
chmod 700 ~/.config/sops/age/

# Check file ownership
ls -la ~/.config/sops/age/keys.txt

ArgoCD Sync Failures

# Check ArgoCD has access to age keys
kubectl get secret sops-age -n argocd -o yaml

# Verify KSOPS is installed in ArgoCD repo server
kubectl logs -n argocd deployment/argocd-repo-server | grep ksops

# Check ArgoCD application logs
kubectl logs -n argocd deployment/argocd-application-controller

Debugging Commands

# Test SOPS encryption/decryption
echo "test: secret" | sops -e --age $(age-keygen -y ~/.config/sops/age/keys.txt) /dev/stdin

# Test KSOPS functionality
echo "apiVersion: v1
kind: Secret
metadata:
  name: test
stringData:
  key: value" | sops -e --age $(age-keygen -y ~/.config/sops/age/keys.txt) /dev/stdin > test.enc.yaml

# Test with KSOPS
cat > test-generator.yaml << EOF
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: test
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - test.enc.yaml
EOF

# Build with kustomize
kustomize build --enable-alpha-plugins --enable-exec .

Monitoring and Alerting

# Example Prometheus rule for SOPS failures
groups:
- name: sops-encryption
  rules:
  - alert: SOPSDecryptionFailure
    expr: increase(argocd_app_sync_total{phase="Failed"}[5m]) > 0
    labels:
      severity: warning
    annotations:
      summary: "ArgoCD sync failed, possibly due to SOPS decryption error"
      description: "Application {{ $labels.name }} failed to sync"

Migration Strategies

From Plain Secrets to SOPS

#!/bin/bash
# migrate-secrets.sh

for file in $(find . -name "*.yaml" -exec grep -l "stringData\|data:" {} \;); do
    if [[ ! "$file" =~ \.enc\.yaml$ ]]; then
        echo "Encrypting $file..."
        sops -e "$file" > "${file%.yaml}.enc.yaml"
        git add "${file%.yaml}.enc.yaml"
        git rm "$file"
    fi
done

Key Migration

# Update all files to use new age key
find . -name "*.enc.yaml" -exec sops updatekeys {} \;

# Verify all files can be decrypted
find . -name "*.enc.yaml" -exec sops -d {} > /dev/null \;

This comprehensive SOPS implementation ensures that secrets are handled securely throughout the RCIIS DevOps lifecycle while maintaining the benefits of GitOps workflows.