PageIndex

What was covered

• The idea. PageIndex is a fairly new retrieval library that replaces vector search with an LLM that reasons over the document and builds its structure.
• Three ways to process a document. Detect the table of contents and split into sections and pages, then summarise each chunk. Where the contents page has no page numbers, let the LLM find them and build a virtual index. For unstructured documents with no table of contents, generate sections from scratch, which takes longer.
• The output. A JSON file with the structure and a contextual summary per section, which keeps context from earlier sections. The extracted fields are configurable.
• A real use case. Pulling specific values out of very long commercial contracts (100 to 300 pages) that follow no common template. Uploaded contracts go through a pipeline that annotates where each clause is defined. PDF coordinates come from a separate library, and scanned documents work through OCR.
• Against a vector RAG baseline. The earlier vector solution missed many fields. PageIndex found annotations that had been empty.
• Customising it. The library defaults to the OpenAI API. A Copilot proxy server (copilotx) let the pipeline run on an existing Copilot subscription and avoid token spend.

What came up

• No table of contents. The library builds its own sections, at the cost of time. It works best on fully structured documents.
• Cross-references. Terms such as a start date can be defined elsewhere in the contract, so the model loops to check whether it needs more sections in the prompt.
• Similarity versus relevance. Vector similarity works well for classifying many documents. For structured documents such as contracts, relevance matters more and LLM reasoning pays off.
• Hybrid designs. For a large knowledge base (say 10,000 PDFs), PageIndex could act as a new way to chunk, with section summaries as chunk metadata. Its node hierarchy is also a good starting point for a graph database.

Takeaway

Best suited to hybrid setups or to questions limited to a single structured PDF.