Why I Couldn't Port-Forward a Redis Cluster

The cache I needed to look at lived in a private subnet. No public endpoint, no VPN, no route from my laptop. The supported way in was AWS Systems Manager port forwarding through a bastion:

aws ssm start-session \
  --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["6379"],"localPortNumber":["6379"]}'

That works beautifully — right up until the thing on the other end is a Redis Cluster.

The session opened. The port bound. redis-cli -p 6379 connected and AUTH succeeded. Then the first cluster-aware client I attached sat there doing nothing until it gave up.

The tunnel was fine

Here's what my client saw the moment it asked the configuration endpoint where everything lived:

127.0.0.1:6379> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "10.0.3.11"
      2) (integer) 6379
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "10.0.3.12"
      2) (integer) 6379
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "10.0.3.13"
      2) (integer) 6379

SSM port forwarding is a single mapping: one local port, one remote port. A standalone Redis needs exactly that, which is why everyone's blog post about SSM and Redis works.

A cluster client doesn't want one connection. It connects to the configuration endpoint, runs CLUSTER SLOTS, and then opens a connection per shard — because it hashes the key itself and talks directly to the node that owns that slot. That's the entire point of cluster mode: no proxy hop, the client routes.

So my client dutifully learned about 10.0.3.11, 10.0.3.12 and 10.0.3.13, and tried to reach a private VPC subnet from a laptop on a different continent.

The tunnel wasn't broken. Service discovery was. Every reply was telling my client about a network it wasn't on. And no amount of extra -parameters fixes that, because the problem isn't how many ports I forward — it's that the addresses being handed out are meaningless outside the VPC.

The obvious tool didn't fit

redis-cluster-proxy exists precisely for this. It speaks the cluster protocol, holds the connections to all the shards itself, and presents a single endpoint that behaves like a standalone Redis.

It's also a C program with no supported Windows build, and the bastion I had was Windows Server.

I could have stood up a Linux instance just to run a proxy. But that's provisioning infrastructure — with a change request, a security group review, and a cost line — to solve what was fundamentally a laptop problem.

The bastion is already on that network

Here's the reframe that made it easy.

Those private addresses aren't unroutable. They're unroutable from my laptop. From the bastion, sitting inside the VPC, 10.0.3.11:6379 is just a machine down the hall.

So don't teach the client about the tunnel. Terminate the cluster-ness on the bastion, where discovery actually works, and forward something boring out through SSM.

The trick is that I still need a real cluster client to find the nodes — I just need it to run on the right side of the boundary. So the proxy uses one, once, at startup, and then forgets everything it knows about clustering and becomes a byte pipe.

The script

Roughly forty lines of Node, one entry per environment:

const net = require('net');
const tls = require('tls');
const Redis = require('ioredis');

const CLUSTERS = [
  {
    name: 'dev',
    host: 'clustercfg.my-redis-rg.abc123.euc1.cache.amazonaws.com',
    port: 6379,
    proxyPort: 7777,
    username: 'default',
    password: process.env.DEV_REDIS_PASSWORD
  }
  // stage on 7778, prod on 7779
];

function startClusterProxy(config) {
  const { name, host, port, proxyPort, username, password } = config;

  const discovery = new Redis.Cluster([{ host, port }], {
    dnsLookup: (address, callback) => callback(null, address),
    redisOptions: {
      username,
      password,
      tls: { servername: host }
    },
    clusterRetryStrategy: (times) => {
      if (times > 5) return null;
      return 2000;
    }
  });

  discovery.on('ready', () => {
    const node = discovery.nodes('master')[0];
    const nodeHost = node.options.host;
    const nodePort = node.options.port;
    console.log(`[${name}] proxying to ${nodeHost}:${nodePort}`);

    net.createServer((clientSocket) => {
      const upstream = tls.connect({
        host: nodeHost,
        port: nodePort,
        servername: nodeHost,
        rejectUnauthorized: false
      });

      upstream.on('secureConnect', () => {
        clientSocket.pipe(upstream);
        upstream.pipe(clientSocket);
      });

      upstream.on('error', () => clientSocket.destroy());
      clientSocket.on('error', () => upstream.destroy());
      clientSocket.on('close', () => upstream.destroy());
      upstream.on('close', () => clientSocket.destroy());
    }).listen(proxyPort, '0.0.0.0');
  });
}

CLUSTERS.forEach(startClusterProxy);

Three details matter more than they look.

dnsLookup: (address, callback) => callback(null, address). ElastiCache advertises node hostnames, and by default ioredis resolves them to IPs before connecting. Do that with in-transit encryption on and the TLS servername no longer matches the certificate, so the handshake fails. Passing the address through untouched keeps the hostname intact all the way to the TLS layer. This one line is the difference between "works" and an hour of confusing certificate errors.

The discovery client is scaffolding, not plumbing. It exists to answer one question — what are the real node addresses — and after ready fires, no user traffic ever goes through it. The data path is net and tls sockets and nothing else. That's deliberate: a byte pipe has no opinions about RESP, so MULTI, pipelining, pub/sub and blocking commands all just work, because the proxy never tries to understand them.

AUTH is forwarded, not injected. The password in the config is for discovery only. My local client still authenticates for itself — the proxy just carries the bytes. Which also means TLS is terminated at the bastion: the hop from my laptop is protected by SSM's own encryption, not by Redis TLS. Worth knowing before deciding what to bind to.

On that last point — listen(proxyPort, '0.0.0.0') is how I ran it, and it's the line I'd think hardest about. It exposes a plaintext path to Redis to anything that can reach the bastion. Inside a tight security group that's fine. As a default in a script destined to be copied later, it isn't; 127.0.0.1 is the safer starting point, since SSM connects from the box itself anyway.

Surviving a reboot

The bastion restarts and there's nobody logged in to start a Node script — I reach that box over SSM precisely because I don't have a desktop session on it. So the proxy runs as a scheduled task, as SYSTEM, at startup:

$Action = New-ScheduledTaskAction `
  -Execute "$env:ProgramFiles\nodejs\node.exe" `
  -Argument "C:\redis-proxy\proxy.js" `
  -WorkingDirectory "C:\redis-proxy"

$Trigger = New-ScheduledTaskTrigger -AtStartup

$Principal = New-ScheduledTaskPrincipal `
  -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest

$Settings = New-ScheduledTaskSettingsSet `
  -ExecutionTimeLimit ([TimeSpan]::Zero) `
  -RestartCount 5 -RestartInterval (New-TimeSpan -Minutes 1) `
  -StartWhenAvailable

Register-ScheduledTask -TaskName "RedisClusterProxy" `
  -Action $Action -Trigger $Trigger -Principal $Principal -Settings $Settings -Force

-ExecutionTimeLimit ([TimeSpan]::Zero) is the one to not miss. The default kills a task after three days, which is exactly the kind of thing I would discover on a Friday.

The part nobody mentions

One port is one node. That's not a limitation of the script — it's what a single tunnel can express.

discovery.nodes('master')[0] picks the first master. Any key that hashes into a slot that master doesn't own returns this:

127.0.0.1:7777> SET foo bar
(error) MOVED 12182 10.0.3.13:6379

Which is Redis correctly pointing at somewhere unreachable.

Whether that matters depends entirely on the replication group:

• One shard — common for dev environments and for caches that comfortably fit in a single node — and that master owns all 16384 slots. The proxy covers the whole keyspace and no MOVED ever appears.
• Multiple shards and that port reaches roughly one-Nth of the keyspace. The fix is one listener per master (7777, 7778, 7779), each with its own tunnel — that covers the full keyspace, at the cost of connecting to the right port for a given key.

And a cluster-aware client pointed at these ports will still misbehave, for the original reason: it'll run CLUSTER SLOTS, get private addresses back, and try to dial them directly. Connect in standalone mode. The whole design depends on the client not knowing it's talking to a cluster.

Cleaned up

I'd been re-copying that script between machines for months, so I turned it into something installable: config-driven instead of hardcoded, credentials read from the environment, optional TLS, a --discover flag that just prints the topology, one-port-per-shard as an option, and service installers for systemd and launchd alongside the Windows task.

It's on GitHub as redis-cluster-ssm-proxy, MIT licensed. The e2e test spins up a real three-shard cluster on localhost and drives the proxy through it, including asserting the MOVED behaviour above — so the limitation is documented by a test rather than by a paragraph I hope gets read.

The general lesson I keep relearning: when a connection fails through a tunnel, check what the far end is telling the client about itself before touching the tunnel. Half the time the bytes are flowing perfectly and the problem is that something handed the client an address from a world it cannot see.