An SQS worker for heavy work in a Next.js app on SST

In a Next.js portal, publishing a document has to kick off heavy work: split the text into chunks, generate embeddings, call an LLM to enrich each chunk, and write everything to the search index. That takes minutes and doesn't fit in a request. The answer was an SQS queue with a consumer Lambda, all declared in the same sst.config.ts as the site.

The queue

const dlq = new sst.aws.Queue("IngestDlq");

const queue = new sst.aws.Queue("IngestQueue", {
  visibilityTimeout: "940 seconds",
  dlq: { queue: dlq.arn, retry: 3 },
});

The visibility timeout must exceed the Lambda timeout. The Lambda can run for up to 15 minutes (900 s). If the message becomes visible again before that, another invocation picks up the same document while the first is still working. With a smaller value, SQS itself rejects the Lambda wiring at deploy time.

And without a DLQ, errors loop forever. A document that always fails goes back to the queue until the retention period runs out, and every round it ties up the database again. With retry: 3, after three attempts it stops somewhere visible, where you can inspect and reprocess it.

One document at a time

queue.subscribe(
  {
    handler: "lambdas/ingest.handler",
    timeout: "15 minutes",
    memory: "512 MB",
    architecture: "arm64",
    nodejs: { install: ["@libsql/linux-arm64-gnu"] },
  },
  { batch: { size: 1, partialResponses: true } },
);

A batch of 1 looks wasteful, but writing one document holds the database writer (Turso) for tens of seconds. With large batches, one slow document delays its neighbours, and one that fails drags the others into the DLQ with it. The cost is more invocations, which is cheap next to reindexing.

Fail per message, not per batch

With partialResponses: true, the Lambda returns only the messages that failed, and only those go back to the queue:

export async function handler(event: SQSEvent) {
  const failures: { itemIdentifier: string }[] = [];
  for (const record of event.Records) {
    try {
      await processJob(JSON.parse(record.body));
    } catch (error) {
      console.error("ingest failed", record.messageId, error);
      failures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures: failures };
}

Without it, one malformed document brought down the whole batch, and the ones that had already succeeded were processed again, paying for embeddings and LLM calls twice. The same code works for any batch size.

If you process records in parallel within a batch, use a concurrency limit instead of an unbounded Promise.all. Sending everything at once blows through the LLM gateway's rate limit.

The binary the bundler forgets

The libSQL client uses a native binary, and SST's bundler doesn't package it. Without the nodejs.install line, the Lambda dies with Cannot find module. The package suffix (linux-arm64-gnu) must match the declared architecture.

The same care applies to the site itself. SST creates two internal functions for ISR revalidation (the seeder and the events subscriber), and they default to x86_64. If the bundle was built on arm64, they need to follow:

new sst.aws.Nextjs("Site", {
  link: [queue],
  transform: {
    server(args) {
      args.architecture = "arm64";
    },
    revalidationSeeder(args) {
      args.architecture = "arm64";
    },
    revalidationEventsSubscriber(args) {
      args.architecture = "arm64";
    },
  },
});

The queue right away, the cron as a safety net

link: [queue] on the site is what lets Next.js publish to the queue as soon as a document is approved. Indexing starts immediately, not at the next scheduled run.

There's still an hourly cron that compares the index with the source and enqueues whatever fell behind. If a message gets lost, the document makes it into search on the next cron run.

A bonus environment-variable trap

In the same setup, Langfuse tracing vanished without an error. The code read LANGFUSE_HOST, the .env defined LANGFUSE_BASE_URL, the variable arrived empty, and the SDK fell back to the default EU server with keys from a US account. No traces and no errors. Now the Lambda gets LANGFUSE_BASE_URL ?? LANGFUSE_HOST, and I check the first trace after every deploy.