Natural-language graph QA (text-to-Cypher)

What you'll learn

• Explain the text-to-Cypher loop: schema in, query out, rows back, answer from rows.
• Use hand-written reference queries as the expected results for generated ones.
• Diagnose the silent empty result and the right query over a wrong graph.
• Bound a generated query with database permissions, not prompt instructions.

Lesson 2 of T13.2 loaded a graph from a file. A support engineer at Larkspur will not write a two-arm MATCH to ask which plans support export in the EU; they will type the question. Something has to translate.

The failure this page is about looks like success. The assistant answers "No plan currently supports export in the EU" in a confident sentence. The generated query had AVAILABLE_IN pointing the wrong way, matched nothing, and the model phrased an empty result as a fact.

Text-to-Cypher is a loop in which a language model reads the user's question and the graph schema, writes a Cypher query, the database runs it, and a second model call phrases the returned rows as an answer. The schema is the model's only map of the graph: which labels exist, which relationship types connect them, which way they point and which properties they carry. For Larkspur's question the model has to find both SUPPORTS and AVAILABLE_IN and bind them to the same plan.

Figure 1: The text-to-Cypher loop. The answer can only restate what the rows contain.

Each stage hands the next one a different artefact, and each can fail on its own:

Table 1: What moves between the stages of Figure 1.

The schema goes stale when the graph changes. Add a REQUIRES_APPROVAL relationship and a model working from yesterday's schema cannot use it. Refresh the schema used for generation after every schema change, and re-run the questions that touch the changed labels and properties.

What the notebook runs, and what it leaves out

The lesson notebook is narrower than its title. It connects to the healthcare graph from lesson 2 of T13.2 and runs 13 hand-written Cypher queries; no model writes a query. The LangChain Neo4j integration documents GraphCypherQAChain, which adds the generation and answer steps, and schema refresh. That chain is the extension beyond the checked-in notebook, and the module lab uses it.

The 13 queries are still the most useful part of the lesson, because they are the expected answers for a translation test. Here is what they return on the graph that lesson 2 of T13.2's import builds, replayed in Python from the CSV (no Neo4j server was used; the healthcare examples are data-query exercises, not medical guidance):

Table 2: Ten of the notebook's 13 reference queries and their replayed results. The other three list names of specialisations, locations and providers.

Two things stand out. None of the queries uses DISTINCT, and the 368 rows repeat seven names. And every multi-arm query returns everyone, because the import flattened its rows (lesson 2 of T13.2). These queries are written correctly. They return wrong answers because the graph is wrong.

We call this the right query, wrong graph failure. A text-to-Cypher test that only compares generated rows with reference rows passes here, since both queries return the same 368 rows. Grade some answers against the source data as well, or the test certifies the import's mistakes.

When an answer is wrong, the bug is usually in the query, and the answer hides it. Save five fields for every request: the original question, the generated Cypher, its parameters, the returned rows, and the final answer. Then compare the generated pattern with the hand-written reference before reading the prose.

Small mistakes in the pattern return nothing without raising an error:

Table 3: Planted generation mistakes. Each produces a valid query and a misleading result.

This is the silent empty result: Cypher treats an unknown relationship type or property as a pattern that matches nothing (at most with a warning notification), which is a valid answer to the query that was written. The answer step then sees zero rows and phrases them. An empty result should say "no matching facts under this schema and data version"; it cannot establish that a capability is impossible.

A test set for translation

Build the test set from question types, not from one phrasing. For Larkspur:

• Paraphrase: "Which EU plans can export data?" should produce the same rows as the reference.
• Unknown entity: a plan that does not exist should return no rows and say so.
• Ambiguous name: "Team" could be several plans, which needs a clarifying answer rather than a guess.
• Multi-relationship: export and EU together, the join that a single arm gets wrong.
• Empty result: a feature no plan has, which checks the wording of a "nothing found" answer.

Grade the rows against the reference query as well as grading the answer.

The database is the guard

A generated query runs with whatever authority its database user has. A prompt that says "generate read-only queries" restricts nothing; a model can still emit a DELETE. Give the chain a read-only database user, a query timeout and a result limit. Then a bad query fails at the database.

Key distinction: LangChain's chain requires an explicit allow_dangerous_requests opt-in. Its security note explains that the flag records that you accept the risk; it does not enforce any permission.

Lesson 4 of T13.2 adds the case where the user does not know the exact entity name, and the graph needs a vector index to find a starting point.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Text-to-Cypher generates a query from the question and the schema, runs it and phrases the rows. Refresh the schema after every schema change and re-run the questions it affects.
• Log question, Cypher, parameters, rows and answer separately. A wrong relationship name, direction or property key returns no rows and no error.
• Hand-written queries are the expected results for generated ones, but a correct query over a flawed graph still returns wrong rows. Enforce safety with a read-only user, timeouts and result limits.