The Knowledge Graph Architecture: How AiFiler Connects Data Without Bottlenecks

You've got 5,000 documents. A contract mentions a client. That client appears in email threads. Those emails reference projects. Those projects have budgets in spreadsheets. Finding all of this manually would take hours. A knowledge graph solves that by making these connections explicit and queryable.

But here's the hard part: building a graph that doesn't collapse under its own weight. Most document systems either skip relationships entirely (you get fast searches but lose context) or build dense graphs that slow down as data grows. AiFiler does neither.

The Core Problem: Relationships at Scale

Traditional document management treats files as isolated units. You search, you find, you open. If you need context—who's involved, what projects it touches, what decisions it informed—you're back to manual digging.

A knowledge graph inverts this. Instead of searching for documents, you navigate relationships. But relationships are expensive. Every new connection is a database query. Every query touches multiple tables. At 5,000 documents with even modest relationship density, you hit performance walls fast.

AiFiler's approach: 8 distinct edge types, each optimized for a specific kind of relationship, with query patterns designed to avoid N+1 cascades.

The Edge Type Strategy

Rather than a single generic "relates to" connection, AiFiler defines specific relationship types. This matters because different relationships have different query patterns and cardinality profiles.

The eight edge types:

1. MENTIONS — A document references another by name, ID, or explicit citation
2. CONTAINS — A folder or collection holds a document
3. AUTHORED_BY — A person created or owns the document
4. ASSIGNED_TO — A document is assigned to a user or team
5. TAGGED_WITH — Metadata labels applied to a document
6. RELATES_TO — AI-inferred semantic similarity (generated during ingestion)
7. DEPENDS_ON — A document requires another as input or context
8. REFERENCES_ENTITY — A document mentions a named entity (person, company, project)

Each edge type has its own table in Supabase with indexed columns on both source and target. This means:

• Querying "all documents this contract mentions" is a single indexed lookup
• Finding "all documents assigned to Sarah" doesn't touch the mentions table
• Traversing "projects → documents → people" can use separate, optimized paths

The payoff: O(1) or O(log n) lookups per edge type, not O(n) full-table scans.

How Relationships Get Built

Relationships don't appear magically. They're created at three points in the data lifecycle:

1. Ingestion Time (Structured Relationships)

When you upload a document, AiFiler parses metadata:

File: Q4_Budget_2024.xlsx
Extracted:
  - AUTHORED_BY: [email protected]
  - ASSIGNED_TO: CFO
  - TAGGED_WITH: [budget, 2024, financial]
  - CONTAINS: (if folder context exists)

This happens in lib/ingest/parseFile.ts. The parser extracts author, owner, tags, and folder membership before the document is even indexed. These are deterministic relationships—they don't change unless the metadata changes.

2. Semantic Analysis (AI-Generated Relationships)

During the Claude API processing phase (via lib/ai/client.ts), the system generates RELATES_TO edges by computing semantic similarity. This is where the knowledge graph gets intelligent.

The process:

1. Document is parsed into chunks
2. Chunks are embedded (via Claude or another embedding model)
3. Embeddings are compared against existing documents
4. Similarity scores above a threshold create RELATES_TO edges
5. Edges are stored with a confidence score (0.0–1.0)

This is the expensive operation, but it's asynchronous and batched. It doesn't block document upload. And it's selective—you're not comparing every document to every other document. You're comparing new documents to a curated set of recent/relevant documents, then to the broader corpus only if needed.

3. User-Driven Relationships (Explicit Connections)

Users can create DEPENDS_ON or custom relationships manually via the Knowledge view. When you click "Link to Project X," you're creating an explicit edge that survives even if semantic similarity drops.

The Query Architecture

Here's where the design pays off. Let's trace a real query: "Show me all documents related to the Acme contract."

Step 1: Find the anchor document

SELECT id FROM documents WHERE title ILIKE '%Acme contract%'

Single index lookup. Returns document_id = 42.

Step 2: Fetch all outbound relationships

SELECT target_id, edge_type, confidence FROM edges 
WHERE source_id = 42 
ORDER BY edge_type, confidence DESC

Single table scan on indexed source_id. Returns 15 edges across 5 types.

Step 3: Batch-fetch target documents

SELECT id, title, type FROM documents 
WHERE id IN (list of 15 target_ids)

Single query with IN clause. Returns 15 rows.

Total database round-trips: 3. No N+1 queries. No full-table scans.

The Universal Command (Ctrl+Shift+A) uses this exact pattern when you search for "documents related to X." It's fast because the graph is built to be traversed, not searched.

Avoiding the Density Trap

A naive graph would create edges everywhere. "This document mentions Sarah" → MENTIONS edge. "Sarah is mentioned in this email" → MENTIONS edge. "Sarah's name appears in this spreadsheet" → MENTIONS edge. Suddenly, Sarah has 500 outbound edges, and any query touching her becomes expensive.

AiFiler uses edge pruning:

• MENTIONS edges are created only for explicit, high-confidence references (not every name mention)
• RELATES_TO edges are created only if semantic similarity exceeds a threshold (default: 0.75 cosine similarity)
• REFERENCES_ENTITY edges are deduplicated—one edge per entity per document, regardless of mention count

This keeps the graph sparse. A typical document has 5–15 edges, not 500. That's the difference between fast and slow.

Why This Matters for You

The knowledge graph isn't a feature you see directly. You don't open a graph visualization. But it's why certain workflows are fast:

• Batch operations can move 1,000 documents in minutes because the system understands relationships and can cascade operations intelligently
• Search returns contextually relevant results, not just keyword matches
• Duplicate detection works because semantic relationships surface similar documents before you create them
• Knowledge view can show you a document's full context—who's involved, what it depends on, what depends on it—in milliseconds

The architecture also scales. You can add 50,000 documents without rewriting the schema. The edge types remain constant. The query patterns remain constant. Only the data grows.

The Tradeoff: Freshness vs. Speed

One constraint: semantic relationships are computed asynchronously. When you upload a new contract, it won't immediately appear in "documents related to Q4 planning" until the background job runs (typically within 30 seconds, but potentially longer under load).

This is intentional. Computing relationships synchronously would slow down uploads. Instead, AiFiler prioritizes upload speed and eventual consistency. Most users never notice the delay. For those who need immediate relationships, explicit linking (DEPENDS_ON) works instantly.

What's Next

The current implementation is solid for documents up to ~100,000 with moderate relationship density. Beyond that, you'd need to shard by workspace or add a dedicated graph database (like Neo4j). But that's a problem most teams would be happy to have.

For now, the 8-edge architecture gives you the intelligence of a dense graph with the performance of a sparse one. That's the real trick.