Fully local RAG with Hugging Face and FAISS

What you'll learn

• Check a "local" pipeline component by component for data that leaves the machine.
• Compare the notebook's pipeline with a fully offline one.
• Explain what keeping inference local costs: hardware, re-embedding, and the risk of loading a saved index.
• Diagnose two small reuse bugs: a double retrieval and an answer split.

Lessons 1 to 4 of T6.4 called a hosted embedding model and a hosted chat model. Some Larkspur customers sign contracts that say support tickets and the handbook may not leave their network. For them, every model call is a question of where the data goes.

Local inference means a model runs on hardware you control, so its inputs never cross the network. Notebook 05a replaces every OpenAI call with open models:

Table 1: 05a's configuration.

"Local" has to be checked per component, because one remote call is enough to send the data out. Here the embedding model and the index run on your machine. The generator does not: the endpoint client calls the Hugging Face Inference API, and the notebook checks for an API token before it starts. Every prompt, with the three retrieved chunks inside it, leaves the machine:

Figure 1: 05a as written. Indexing is local; generation is not.

We call this the half-local pipeline. The notebook's own comparison table lists data privacy as "Stays local", which is true only for indexing. Its next steps name the fix: swap the endpoint client for a local pipeline that loads the model weights and generates in your process.

Table 2: Half-local against fully offline.

Keeping everything local trades the API bill for three costs:

• Hardware and latency. A 7B model generating on your own CPU or GPU replaces a network call. Measure tokens per second on the target machine before promising a latency.
• One embedding model on both sides. The query must be embedded by the model that embedded the chunks. Switching from OpenAI's 1,536-dimension vectors to bge's 768 means re-embedding the whole corpus, not only new documents.
• Loading a saved index runs pickle. FAISS's loader needs an explicit dangerous-deserialisation flag because the docstore is saved as a pickle file, and loading a pickle can run code. Load only index files your own pipeline wrote.

Pitfall: a borrowed index file. A FAISS index downloaded from a shared drive or a model hub is code as well as data. Rebuild it from source documents, or load only files your pipeline wrote and stored where only it can write.

Two smaller things to change when you reuse 05a's code:

Table 3: Two reuse bugs in 05a.

On a fixed FAISS index the two retrievals return the same list, so the double retrieval only costs an extra search. On an index that updates between the calls, the sources shown could differ from the context the model read.

05a's fallback evaluation counts expected keywords in each answer, such as "carbon dioxide" and "methane". Substring matching rewards any answer that mentions the words, right or wrong. Treat it as a smoke test, and use the golden-set metrics from T5.1 for decisions.

1. Audit every component with Table 2: embeddings, index, reranker (T6.3), generator, and any grader.
2. Generate in-process on hardware sized from a measured tokens-per-second figure.
3. Re-embed the whole handbook with the local embedding model, and evaluate it on the golden set before switching.
4. Write and load the index yourself, never from a shared location.

This completes the T6.4 system: grounded, cited, explainable, table-aware and deployable where the data must stay.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Check "local" per component. One remote call sends the data out: 05a's endpoint client ships every prompt, chunks included, to the Inference API (the half-local pipeline), while a local pipeline generates in your process.
• Changing embedding models means re-embedding the whole corpus. Measure tokens per second on the target hardware before promising a latency.
• Load only FAISS index files your own pipeline wrote, because the docstore is a pickle. Retrieve once and reuse the result, so the sources shown match the context.