A Rails stack on
Rackspace Spot

What a cloudspace is

A cloudspace is Rackspace Spot's name for a managed Kubernetes cluster: Rackspace runs the control plane, you get a kubeconfig, and worker nodes come from node pools you attach to it. Terraform talks to Spot through the rackerlabs/spot provider, authenticated with an access token copied from the Spot console under API Access → Terraform. The guide notes the token has no refresh mechanism, so treat it like any long-lived credential.

The provider gives you four pieces that matter here:

Two data sources help you fill in the blanks: spot_regions lists valid regions and spot_serverclasses lists server classes such as gp.vs1.large-dfw. The official guide only shows a spot pool; the on-demand pool comes from the provider's own documentation.

The spot market and preemption

Spot pools are priced by auction. You bid a maximum hourly price per server for a server class in a region. While your bid wins, you get the servers; when the market moves above your bid, you lose them. The guide's example bids 0.012 for gp.vs1.large-dfw. Read bid_status and won_count on the node pool to see whether your bid is actually holding.

Spot's preemption docs describe the sequence:

1. An “Upcoming Pre-emption” notice goes to the webhook you set as preemption_webhook (Slack webhooks are the supported type), and the affected nodes are cordoned.
2. About a minute later, the nodes are drained so pods can terminate in order.
3. Five minutes after the first notice, a “Pre-emption” notice goes out and the nodes leave the cloudspace.

Another page in the same docs mentions a six-minute grace period to raise your bid before reclaim. Either way, the budget to react is minutes, not hours, and there is no promise that replacement capacity at your price exists afterwards.

For a Rails web pod that's fine: Puma stops cleanly on SIGTERM and a new pod starts elsewhere. For a database primary it means an unplanned failover every time the market moves, and if all your nodes lose the auction at once, there may be nowhere to fail over to.

A topology, service by service

The question for each component is simple: what is lost if its node disappears with five minutes' notice?

So the cloudspace gets two pools. A spot pool autoscales for the stateless work. A small on-demand pool, tainted so that only stateful pods land there, holds Postgres, Elasticsearch and the durable Redis. Labels and taints do the steering; both pool resources accept them.

It's worth saying out loud that Rails 8's defaults shrink this picture. Solid Cache, Solid Queue and Solid Cable keep cache, jobs and pub/sub in the database, and that can remove memcached and Redis entirely. Elasticsearch can sometimes give way to Postgres full-text search. Fewer stateful services means fewer things to protect from preemption. The rest of this page keeps the full stack because that's the question asked, but the cheapest fix for a risky stateful service is to not run it.

The Terraform layout

Keep Terraform to the cluster and its pools. Apply workloads with kubectl or Helm in a second step, using the kubeconfig Terraform produced. Driving Kubernetes resources from the same Terraform run that creates the cluster leads to the familiar chicken-and-egg problem where the Kubernetes provider needs credentials that don't exist yet.

infra/spot/
  versions.tf     # provider pin
  variables.tf    # token, region, bids
  main.tf         # cloudspace + two node pools
  outputs.tf      # kubeconfig (sensitive)
deploy/k8s/
  app.yml  memcached.yml  redis.yml
  postgres.yml  elasticsearch.yml
bin/deploy        # Ruby: terraform apply, then kubectl apply

infra/spot/versions.tf

terraform {
  required_providers {
    spot = {
      source  = "rackerlabs/spot"
      version = "~> 0.1.4"
    }
  }
}

provider "spot" {
  token = var.rackspace_spot_token
}

infra/spot/variables.tf

variable "rackspace_spot_token" {
  type      = string
  sensitive = true   # supplied as TF_VAR_rackspace_spot_token, never in a .tfvars file in git
}

variable "region"             { default = "us-central-dfw-1" }
variable "cloudspace_name"    { default = "jaycorp-rails" }
variable "web_server_class"   { default = "gp.vs1.large-dfw" }
variable "web_bid_price"      { default = 0.012 }
variable "state_server_class" { default = "gp.vs1.large-dfw" }
variable "preemption_webhook" {
  type      = string
  sensitive = true
}

infra/spot/main.tf

resource "spot_cloudspace" "app" {
  cloudspace_name    = var.cloudspace_name
  region             = var.region
  kubernetes_version = "1.31.1"
  cni                = "calico"
  hacontrol_plane    = true    # the guide uses false; a production database wants the HA control plane
  preemption_webhook = var.preemption_webhook
  wait_until_ready   = true
}

# Stateless work: Puma, workers, memcached. Bid-priced, can be preempted.
resource "spot_spotnodepool" "web" {
  cloudspace_name = spot_cloudspace.app.cloudspace_name
  server_class    = var.web_server_class
  bid_price       = var.web_bid_price

  autoscaling = {
    min_nodes = 3
    max_nodes = 8
  }

  labels = {
    "jaycorp.us/pool" = "spot"
    "managed-by"      = "terraform"
  }
}

# Stateful work: Postgres, Elasticsearch, durable Redis. Fixed price.
resource "spot_ondemandnodepool" "state" {
  cloudspace_name      = spot_cloudspace.app.cloudspace_name
  server_class         = var.state_server_class
  desired_server_count = 3

  labels = {
    "jaycorp.us/pool" = "state"
    "managed-by"      = "terraform"
  }

  taints = [{
    key    = "jaycorp.us/stateful"
    value  = "true"
    effect = "NoSchedule"
  }]
}

data "spot_kubeconfig" "app" {
  cloudspace_name = spot_cloudspace.app.cloudspace_name
}

infra/spot/outputs.tf

output "kubeconfig" {
  value     = data.spot_kubeconfig.app.raw
  sensitive = true
}

output "web_bid_status" {
  value = spot_spotnodepool.web.bid_status
}

Two notes on details the guide glosses over. Its example passes spot_cloudspace.example.name to the kubeconfig data source; referencing cloudspace_name as above is the unambiguous choice. And the provider documents kubernetes_version values of 1.29.6, 1.30.10 and 1.31.1 as of 0.1.4, so check the current list before pinning one. Terraform state will contain the kubeconfig, so keep state somewhere private and encrypted, not in the repository.

Kubernetes manifests

Spot's persistent volumes are all ReadWriteOnce: one node mounts a volume at a time. Storage classes differ by datacenter generation (ssdv2-performance is the Gen-2 default, ssd the Gen-1 default), so check which applies to your region and set storageClassName explicitly instead of relying on the default.

Rails app and workers

deploy/k8s/app.yml (abridged)

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      nodeSelector: { jaycorp.us/pool: spot }
      terminationGracePeriodSeconds: 30   # well inside the ~1 minute before drain
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: web } }
      containers:
        - name: puma
          image: registry.example/jaycorp/app:2026-10-17
          command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
          ports: [{ containerPort: 3000 }]
          envFrom: [{ secretRef: { name: app-env } }]
          readinessProbe: { httpGet: { path: /up, port: 3000 } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: web } }
---
# Same image, different command, for the job workers:
#   command: ["bundle", "exec", "sidekiq"]   (or bin/jobs for Solid Queue)
# Run migrations as a Job before rolling out web, not in the Puma entrypoint.

Rails 7.1 and later generate the /up health check, which makes a good readiness probe. A PodDisruptionBudget can't stop a preemption, but it does keep a drain from evicting every web pod at once when it has a choice.

memcached

deploy/k8s/memcached.yml (abridged)

apiVersion: apps/v1
kind: StatefulSet           # stable pod names, so Dalli gets a stable server list
metadata: { name: memcached }
spec:
  serviceName: memcached    # headless Service: memcached-0.memcached, memcached-1.memcached
  replicas: 2
  selector: { matchLabels: { app: memcached } }
  template:
    metadata: { labels: { app: memcached } }
    spec:
      nodeSelector: { jaycorp.us/pool: spot }
      containers:
        - name: memcached
          image: memcached:1.6-alpine
          args: ["-m", "512"]
          ports: [{ containerPort: 11211 }]

No volume: a restart means a cold cache, which is the normal memcached contract.

Redis

If Redis only backs the cache or Action Cable, run it like memcached on the spot pool. If it holds the Sidekiq queue, give it the on-demand pool and a disk:

deploy/k8s/redis.yml (abridged)

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: redis }
spec:
  serviceName: redis
  replicas: 1
  selector: { matchLabels: { app: redis } }
  template:
    metadata: { labels: { app: redis } }
    spec:
      nodeSelector: { jaycorp.us/pool: state }
      tolerations: [{ key: jaycorp.us/stateful, value: "true", effect: NoSchedule }]
      containers:
        - name: redis
          image: redis:7.4
          args: ["--appendonly", "yes", "--appendfsync", "everysec"]
          volumeMounts: [{ name: data, mountPath: /data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: ssdv2-performance
        resources: { requests: { storage: 10Gi } }

With appendfsync everysec, a crash can lose up to about a second of writes. Valkey, the BSD-licensed fork, is a drop-in alternative image if Redis's licensing matters to you.

Postgres

Run Postgres through an operator rather than a hand-written StatefulSet. CloudNativePG (Apache-2.0) handles the primary, a streaming replica, failover, and continuous WAL archiving to any S3-compatible bucket:

deploy/k8s/postgres.yml (abridged)

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: pg }
spec:
  instances: 2                     # primary + one replica, on different nodes
  storage: { size: 50Gi, storageClass: ssdv2-performance }
  affinity:
    nodeSelector: { jaycorp.us/pool: state }
    tolerations: [{ key: jaycorp.us/stateful, value: "true", effect: NoSchedule }]
    enablePodAntiAffinity: true
  backup:
    retentionPolicy: 30d
    barmanObjectStore:
      destinationPath: s3://jaycorp-pg-backups/rails
      endpointURL: https://s3.example        # any S3-API store: R2, B2, Garage, SeaweedFS
      s3Credentials:
        accessKeyId:     { name: pg-backup-s3, key: ACCESS_KEY_ID }
        secretAccessKey: { name: pg-backup-s3, key: SECRET_ACCESS_KEY }

CloudNativePG creates a pg-rw Service pointing at the primary, which is what the app's DATABASE_URL uses. Schedule a daily base backup with its ScheduledBackup resource.

Elasticsearch

The usual route is Elastic's ECK operator, which manages a three-node cluster with one replica per shard and snapshots to an S3 repository. Elasticsearch itself is no longer under an OSI open-source licence by default; if that matters, OpenSearch and its operator fill the same role and the Ruby client gems exist for both.

deploy/k8s/elasticsearch.yml (abridged, ECK)

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata: { name: search }
spec:
  version: 8.15.0
  nodeSets:
    - name: default
      count: 3
      podTemplate:
        spec:
          nodeSelector: { jaycorp.us/pool: state }
          tolerations: [{ key: jaycorp.us/stateful, value: "true", effect: NoSchedule }]
      volumeClaimTemplates:
        - metadata: { name: elasticsearch-data }
          spec:
            accessModes: [ReadWriteOnce]
            storageClassName: ssdv2-performance
            resources: { requests: { storage: 50Gi } }

Three Elasticsearch nodes plus two Postgres instances and Redis is a lot for three on-demand servers; size the pool to the memory these actually need, which you only learn by measuring.

Rails configuration

Every backing service is attached by URL from the environment, so moving Postgres from the cluster to a managed service is a config change, not a code change:

# config/environments/production.rb
config.cache_store = :mem_cache_store, ENV.fetch("MEMCACHE_SERVERS").split(",")
#   MEMCACHE_SERVERS=memcached-0.memcached:11211,memcached-1.memcached:11211

# config/database.yml reads DATABASE_URL       (postgres://…@pg-rw:5432/app)
# config/initializers/sidekiq.rb reads REDIS_URL (redis://redis-0.redis:6379/0)
# the search client reads ELASTICSEARCH_URL     (https://search-es-http:9200)

Those values live in a Kubernetes Secret named app-env, generated from our secrets adapter at deploy time rather than written into a manifest.

Jobs need to tolerate a worker vanishing mid-run. Sidekiq returns in-progress jobs to the queue on a clean shutdown, but a hard kill can still interrupt one, so make jobs idempotent: charging a card or sending an email should check a record first, not trust that it ran exactly once.

A Ruby deploy script

The glue is Ruby, per our conventions. It reads the Spot token through the secrets adapter, applies Terraform, writes the kubeconfig to a private temp file and applies the manifests:

bin/deploy

#!/usr/bin/env ruby
# frozen_string_literal: true

require "open3"
require "tempfile"
require_relative "../secrets/lib/secrets"

def run(*cmd, env: {}, dir: nil)
  out, status = Open3.capture2(env, *cmd, **(dir ? { chdir: dir } : {}))
  abort "failed: #{cmd.join(' ')}" unless status.success?
  out
end

tf_env = {
  "TF_VAR_rackspace_spot_token" => Secrets.fetch("RACKSPACE_SPOT_TOKEN"),
  "TF_VAR_preemption_webhook"   => Secrets.fetch("SPOT_PREEMPTION_WEBHOOK")
}

run("terraform", "apply", "-auto-approve", env: tf_env, dir: "infra/spot")
kubeconfig = run("terraform", "output", "-raw", "kubeconfig", env: tf_env, dir: "infra/spot")

Tempfile.create("kubeconfig") do |file|
  file.chmod(0o600)
  file.write(kubeconfig)
  file.flush
  run("kubectl", "apply", "-f", "deploy/k8s/", env: { "KUBECONFIG" => file.path })
end

In real use you'd want a terraform plan reviewed by a person before any apply that touches the on-demand pool, since removing it removes the database's nodes.

Data durability, honestly

• Spot nodes are not a place for the only copy of anything. If every server in a spot pool loses its bid at once, a database there has nowhere to go until capacity returns at your price.
• On-demand pools remove bid preemption, not node failure. They're still ordinary servers, so Postgres still needs a replica and off-cluster backups.
• Whether a volume reattaches cleanly after preemption isn't documented. Spot's storage page says persistent volumes should survive pod restarts and node failures, and its troubleshooting section notes that attachment can get stuck, with node recycling as the fix. Test it: kill a node holding a Postgres replica in a scratch cloudspace and time the recovery.
• ReadWriteOnce only. No shared volumes, so anything that needs shared files (uploads, for instance) goes to S3-compatible object storage, which is our rule anyway.
• Backups count only once restored. Schedule a restore of the Postgres backup into a fresh cluster, and an Elasticsearch reindex from Postgres, and check the row counts.
• Volumes bill while they exist, by provisioned size, even after the pod using them is gone. Clean up orphaned PVCs.
• The managed option is often the honest answer. For a small team, a managed Postgres outside the cloudspace, reached by DATABASE_URL, buys durability that's hard to match on your own, and the adapter shape keeps it swappable.

How this fits our rules

Spot is a hosted Kubernetes service, but everything here is portable: plain Kubernetes manifests, open operators, Postgres, S3-API backups, and connection strings from the environment. Leaving Spot means pointing the same manifests at another cluster. That matches the project instructions on staying platform agnostic and keeping data exportable.

If we ever deployed this, the cloudspace and each service would get entries in observability/services.yml, the app would export OpenTelemetry to a collector, and preemption notices would go to the same place as our other alerts. None of that exists today; this page describes an option, not a running system.

Sources

1. Deploy your cloudspace via Terraform, Rackspace Spot docs: provider, token, cloudspace, spot node pool and kubeconfig example.
2. rackerlabs/spot provider and its source and docs: full schemas, including spot_ondemandnodepool and taints. Version 0.1.4 at time of writing.
3. Pre-emption explained, Rackspace Spot docs: notice, cordon, drain and ejection timing.
4. Persistent volumes, Rackspace Spot docs: storage classes, ReadWriteOnce, pricing.
5. CloudNativePG and Elastic Cloud on Kubernetes: operators for Postgres and Elasticsearch.