NumPy, pandas and Matplotlib

A NumPy array holds numbers of one dtype in a fixed shape, so arithmetic runs on the whole array in compiled code instead of a Python loop. The notebook covers creation (np.array, np.zeros, np.arange, np.linspace), indexing and boolean masks, reshape and .T, ufuncs, aggregation along an axis, broadcasting, @ for matrix products and np.linalg. Retrieval code uses a handful of these on one object: the embedding matrix.

An index of the twelve Larkspur handbook chunks, embedded with a 384-dimension model, is an array E of shape (12, 384): one row per chunk, one column per dimension. A question embeds to a vector q of shape (384,). Cosine similarity is the dot product of unit-length vectors, so retrieval is three lines:

With three chunks and three dimensions you can check every number by hand:

E = np.array([[0.9, 0.1, 0.0],    # refund-window
              [0.7, 0.3, 0.1],    # refund-timing
              [0.0, 0.2, 0.9]])   # sla
q = np.array([0.8, 0.2, 0.1])     # "How long does a refund take?"

E_n = E / np.linalg.norm(E, axis=1, keepdims=True)
scores = E_n @ (q / np.linalg.norm(q))   # [0.984, 0.987, 0.170]
top = np.argsort(-scores)[:2]            # [1, 0]: refund-timing, then refund-window

keepdims=True is what makes the division correct. np.linalg.norm(E, axis=1) returns the row lengths as a flat array of shape (3,). Broadcasting lines shapes up from the last axis, so a flat (3,) divides each column by a different row's length. On this toy matrix that runs without an error and gives scores of 0.989, 0.852 and 0.180, which puts refund-window first. With keepdims=True the norms have shape (3, 1) and stretch across the columns, one length per row. On the real (12, 384) matrix the flat version fails with ValueError: operands could not be broadcast together with shapes (12,384) (12,), which is the better outcome, because square test data hides the bug.

Three more habits from the notebook pay off later:

• Check .shape, .ndim and .dtype before you trust a result. A (384,) query gives scores of shape (12,); the same query as a (384, 1) column gives (12, 1), and np.argsort on that sorts each one-element row instead of the twelve scores.
• Choose the dtype for memory. astype converts it. A million 1,536-dimension vectors take 6,144,000,000 bytes (6.1 GB) as float32 and twice that as float64, the default for arrays built from Python floats. Most vector libraries, FAISS among them, expect float32.
• Top-k does not need a full sort. np.argsort sorts all n scores. On millions of rows, np.argpartition(-scores, k)[:k] finds the k best without ordering the rest; sort only those k afterwards.

A DataFrame is a table of named columns, each a Series with its own dtype, plus an index that labels the rows. NumPy holds the vectors; pandas holds everything around them: chunk ids and text, metadata, golden questions, and the results of every evaluation run.

The notebook opens with pd.read_csv('../assets/churn_prediction.csv'). Run df.info() on it before anything else: it has 15,929 rows and 21 columns, and dependents and city load as float64 although they hold whole numbers, because 1,348 and 460 of their values are missing and NaN is a float. gender has 288 missing values, occupation 47, and last_transaction stays a string column (object) until pd.to_datetime converts it. The same thing happens to chunk metadata loaded from a CSV: a page-number column with one blank becomes floats, and a date column stays text.

The operations the notebook teaches map onto evaluation work directly:

• Select and filter: df[df["recall"] < 1] lists the questions that missed evidence; .loc selects by label, .iloc by position.
• Add columns: df["tokens"] = df["text"].str.len() / 4 for a rough token count; .apply and .map for anything a vectorised expression cannot do.
• Join: pd.merge(results, chunks, on="chunk_id") brings chunk text next to the scores. Merge on an id, never on row order.
• Group and pivot: the baseline row is a groupby.

T5.1's top-k lab scores three golden questions at four values of k. As a long table, one row per question and k, its baseline is one line:

runs.groupby("k")[["recall", "precision"]].mean().round(2)

runs.pivot_table(index="question", columns="k", values="recall") turns the same table into one row per question, which shows where the mean comes from: the export question reaches 1.0 at k = 2, the refund question stays at 0.5 and the outage question at 0.0. runs.groupby("k")["recall"].mean().plot(marker="o") draws the curve through Matplotlib, which the optional notebook covers.

Missing values deserve a check of their own. mean() skips NaN by default. If the outage question errors at k = 2 and its recall is recorded as NaN, mean recall@2 becomes 0.75 instead of 0.50, with no warning. Compare count() (2) with size() (3) per group, and record failures in a status column instead of a missing number.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Retrieval is normalise, dot product, sort. Divide by row norms with keepdims=True, because square test data hides the broadcasting bug that a (12, 384) matrix exposes.
• Check .shape, .ndim and .dtype before you trust a result. Store vectors as float32, and use np.argpartition for top-k on millions of rows.
• Run df.info() first, merge on an id rather than row order, and compare count() with size() per group, since mean() skips NaN without a warning.