Building a Production RAG Pipeline in TypeScript with pgvector and Next.js

Retrieval-augmented generation (RAG) means you fetch relevant text from your own data and put it in the model's context before it answers. The pipeline is five steps: ingest documents, split them into chunks, embed the chunks, store the vectors, then at query time retrieve the closest chunks and pass them to the model. This post builds each step in TypeScript using Postgres with the pgvector extension and a Next.js App Router route. The hard parts are not the API calls. They are chunk size, keeping embeddings fresh, and making sure retrieval actually returns relevant text.

I use plain Postgres here on purpose. If you already run Postgres, you do not need a separate vector database. pgvector gives you vector columns, distance operators, and approximate-nearest-neighbour indexes inside the database you already back up and monitor.

Key takeaways

• Postgres with pgvector is enough for most app-sized corpora: vectors live next to your data, backups and monitoring.
• Use an HNSW index with the operator class that matches your distance function (cosine here).
• Embed in batches, and re-ingest documents as delete-then-insert so stale chunks never answer questions.
• Over-fetch, then re-rank, and set a relevance floor so the model can say "I don't know".
• Chunking and freshness decide quality far more than prompt wording.

The schema

Start with the table. You need the source text, some metadata for filtering, and a vector column sized to your embedding model. OpenAI's text-embedding-3-small returns 1536 dimensions, so the column is vector(1536). If you switch models, the dimension usually changes, and you cannot mix dimensions in one column.

create extension if not exists vector;

create table document_chunks (
  id          bigint generated always as identity primary key,
  document_id text        not null,
  content     text        not null,
  metadata    jsonb       not null default '{}',
  embedding   vector(1536) not null,
  created_at  timestamptz not null default now()
);

-- Approximate nearest neighbour index for cosine distance.
create index document_chunks_embedding_hnsw
  on document_chunks
  using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 64);

-- B-tree on document_id so re-ingesting a document is a cheap delete + insert.
create index document_chunks_document_id on document_chunks (document_id);

A note on the index. HNSW builds a graph and gives good recall with low query latency, which is what you want in production. The alternative, IVFFlat, is faster to build but needs the data to exist before you build it and needs tuning of the lists parameter. For most app-sized corpora, HNSW is the better default. The operator class must match the distance you query with. I use cosine here, so vector_cosine_ops and the <=> operator.

Chunking

Chunking is where most RAG quality is won or lost. Embed a whole 40-page PDF as one vector and retrieval becomes useless, because the single vector averages out everything. Chunk too small and you lose the context that makes a passage meaningful.

My default is 500 to 800 tokens per chunk with about 10 to 15 percent overlap, split on natural boundaries (paragraphs, then sentences) rather than a hard character count. Overlap matters because a fact often straddles a boundary; without it, the sentence that answers the question gets cut in half.

// lib/chunk.ts
type Chunk = { content: string; index: number };

// Roughly 4 characters per token for English. Good enough for budgeting;
// use a real tokenizer (tiktoken/js-tiktoken) if you need exact counts.
const CHARS_PER_TOKEN = 4;

export function chunkText(
  text: string,
  { maxTokens = 700, overlapTokens = 100 } = {},
): Chunk[] {
  const maxChars = maxTokens * CHARS_PER_TOKEN;
  const overlapChars = overlapTokens * CHARS_PER_TOKEN;

  // Split on blank lines first so we respect paragraph structure.
  const paragraphs = text
    .split(/\n\s*\n/)
    .map((p) => p.trim())
    .filter(Boolean);

  const chunks: Chunk[] = [];
  let buffer = "";

  for (const paragraph of paragraphs) {
    if (buffer && buffer.length + paragraph.length > maxChars) {
      chunks.push({ content: buffer, index: chunks.length });
      // Carry the tail of the previous chunk for overlap.
      buffer = buffer.slice(-overlapChars) + "\n\n" + paragraph;
    } else {
      buffer = buffer ? buffer + "\n\n" + paragraph : paragraph;
    }
  }

  if (buffer.trim()) chunks.push({ content: buffer, index: chunks.length });
  return chunks;
}

This is deliberately simple. For structured documents, chunk along the structure instead: keep markdown headings with their section, keep table rows together, and store the heading path in metadata so you can show users where an answer came from.

Embeddings

Embed chunks in batches. The API accepts an array of inputs, so one call per batch is far cheaper and faster than one call per chunk. Keep batches under the model's token limit.

// lib/embeddings.ts
import OpenAI from "openai";

const openai = new OpenAI();
const MODEL = "text-embedding-3-small";

export async function embed(texts: string[]): Promise<number[][]> {
  const res = await openai.embeddings.create({
    model: MODEL,
    input: texts,
  });
  // Preserve input order; the API returns results in the same order.
  return res.data.map((d) => d.embedding);
}

Ingesting

Ingestion ties chunking and embedding together and writes to Postgres. I delete existing rows for a document before inserting, so re-ingesting a changed file replaces the old vectors instead of duplicating them. Stale embeddings are a real bug: a user edits a doc, the old chunks stay in the index, and the model answers from text that no longer exists.

I use the pg driver here. pgvector ships a helper that formats a number[] into the literal Postgres expects.

// lib/ingest.ts
import { Pool } from "pg";
import pgvector from "pgvector/pg";
import { chunkText } from "./chunk";
import { embed } from "./embeddings";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

export async function ingestDocument(documentId: string, text: string) {
  const chunks = chunkText(text);
  const vectors = await embed(chunks.map((c) => c.content));

  const client = await pool.connect();
  try {
    await client.query("begin");
    await client.query("delete from document_chunks where document_id = $1", [
      documentId,
    ]);

    for (let i = 0; i < chunks.length; i++) {
      await client.query(
        `insert into document_chunks (document_id, content, metadata, embedding)
         values ($1, $2, $3, $4)`,
        [
          documentId,
          chunks[i].content,
          JSON.stringify({ chunkIndex: chunks[i].index }),
          pgvector.toSql(vectors[i]),
        ],
      );
    }
    await client.query("commit");
  } catch (err) {
    await client.query("rollback");
    throw err;
  } finally {
    client.release();
  }
}

Wrapping the delete and inserts in one transaction means a failed re-ingest never leaves a document with zero chunks.

Retrieval and re-ranking

At query time you embed the user's question with the same model, then ask Postgres for the nearest chunks. The <=> operator is cosine distance, so smaller is closer. I convert it to a similarity score (1 - distance) for readability and filter out weak matches.

// lib/retrieve.ts
import { Pool } from "pg";
import pgvector from "pgvector/pg";
import { embed } from "./embeddings";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

export type Match = { content: string; similarity: number };

export async function retrieve(query: string, k = 8): Promise<Match[]> {
  const [queryVector] = await embed([query]);

  const { rows } = await pool.query(
    `select content,
            1 - (embedding <=> $1) as similarity
       from document_chunks
      order by embedding <=> $1
      limit $2`,
    [pgvector.toSql(queryVector), k],
  );

  return rows as Match[];
}

Pure vector search is a good start but it is not enough on its own. Two fixes pay off the most:

Over-fetch, then re-rank. Retrieve more than you need (say k = 20), then run a cross-encoder re-ranker (Cohere Rerank, or a local model like bge-reranker) to score each chunk against the query and keep the top 4 to 6. Vector similarity finds candidates that are semantically close; a re-ranker judges actual relevance to the specific question, which is a different and harder task.

// lib/rerank.ts
import { CohereClient } from "cohere-ai";
import type { Match } from "./retrieve";

const cohere = new CohereClient({ token: process.env.COHERE_API_KEY });

export async function rerank(
  query: string,
  matches: Match[],
  topN = 5,
): Promise<Match[]> {
  if (matches.length === 0) return matches;

  const res = await cohere.rerank({
    model: "rerank-english-v3.0", // check Cohere's docs for the current model
    query,
    documents: matches.map((m) => m.content),
    topN,
  });

  return res.results.map((r) => matches[r.index]);
}

Add a relevance floor. If the best similarity is still low, the corpus probably does not contain the answer. Returning nothing and letting the model say "I don't know" beats forcing it to answer from irrelevant context.

Grounded generation in a Next.js route

Now the App Router handler. It retrieves, re-ranks, builds a prompt that pins the model to the retrieved context, and streams the answer. The system prompt is doing real work here: it tells the model to answer only from context and to admit when it cannot.

// app/api/chat/route.ts
import { openai } from "@ai-sdk/openai";
import { streamText } from "ai";
import { retrieve } from "@/lib/retrieve";
import { rerank } from "@/lib/rerank";

export const runtime = "nodejs"; // pg needs the Node runtime, not Edge.

export async function POST(req: Request) {
  const { question } = await req.json();

  if (typeof question !== "string" || question.trim().length === 0) {
    return Response.json({ error: "question is required" }, { status: 400 });
  }

  const candidates = await retrieve(question, 20);
  const top = await rerank(question, candidates, 5);

  if (top.length === 0 || top[0].similarity < 0.2) {
    return Response.json({
      answer: "I couldn't find anything relevant in the knowledge base.",
    });
  }

  const context = top
    .map((m, i) => `[Source ${i + 1}]\n${m.content}`)
    .join("\n\n");

  const result = streamText({
    model: openai("gpt-4o-mini"), // any current small model works; keep it in config
    system:
      "You answer using only the provided sources. " +
      "Cite sources as [Source N]. " +
      "If the sources do not contain the answer, say you don't know.",
    prompt: `Sources:\n${context}\n\nQuestion: ${question}`,
  });

  return result.toTextStreamResponse();
}

The nodejs runtime is required because the pg driver opens TCP connections, which the Edge runtime does not support. If you want Edge, use a driver built for it (Neon's serverless driver, for example) and an HTTP-based vector store.

What actually breaks, and how to fix it

Chunk size is wrong. Answers are vague or miss obvious facts. Usually chunks are too large and dilute the signal, or too small and lose context. Re-chunk at 500 to 800 tokens with overlap and re-ingest before touching anything else.

Stale embeddings. The model answers from deleted or outdated content. Make ingestion a delete-then-insert per document, and re-ingest on every source change. If a document can be partially edited, version it and key chunks by content hash so unchanged chunks are not re-embedded.

Retrieval relevance is poor. The right chunk exists but does not surface. Over-fetch and re-rank, add metadata filters (only search the relevant document set), and check that you embed the query with the exact model you embedded the corpus with. Mixing embedding models silently wrecks similarity scores.

Embedding-model drift. You upgrade the embedding model and old vectors no longer compare correctly to new ones. There is no shortcut: re-embed the whole corpus. Keep the model name in a config constant so this is one change, not a hunt across files.

Slow queries as the table grows. Tune HNSW's ef_search at query time (set hnsw.ef_search = 100) to trade latency for recall, and make sure the index actually exists with the operator class that matches your distance function.

FAQ

Do I need a separate vector database for RAG?

Not for most apps. If you already run Postgres, the pgvector extension gives you vector columns, distance operators and approximate nearest-neighbour indexes in the same database you back up and monitor. Dedicated vector databases make sense at very large scale or with specialised filtering needs.

What chunk size works best for RAG?

A common starting point is 500 to 800 tokens with some overlap, then adjusting by testing real questions. Chunks that are too large dilute relevance; chunks that are too small lose the context needed to answer.

Why does my RAG system answer from outdated content?

Usually because old chunks were never removed. Re-ingest each changed document as a delete-then-insert, trigger ingestion on every source change, and re-embed the whole corpus whenever you change embedding models.

Is re-ranking worth the extra call?

Usually yes. Vector search finds semantically similar candidates; a re-ranker scores how well each one answers the specific question. Retrieving 20 candidates and keeping the best 4 to 6 is a reliable way to improve answers without changing anything else.

Next steps

If you want a RAG assistant grounded in your own documents and data, built on the database you already run, I can design and build it with you. Get in touch, or see how I work as a RAG developer.

Sources

• pgvector, GitHub
• pgvector-node, GitHub
• AI SDK: streamText, Vercel
• Rerank, Cohere
• Embeddings, OpenAI