Why your AI parts assistant finds the family but misses the code

Find where the code was lost

Imagine a counter colleague asking an internal assistant for a bearing by its complete catalogue code. The answer names the right family but selects a neighbouring variant. Before replacing the language model, capture the query, the ranked search results and the passages actually sent into the answer. If the required record never reached that last set, a more fluent answer generator has not addressed the missing evidence. If the record did reach it, investigate answer selection separately.

Anthropic’s Contextual Retrieval article explains that semantic embeddings can miss exact identifiers, while lexical search such as BM25 can help retrieve them. It also describes how splitting documents can remove useful context. These are reasons to test your retrieval pipeline; they are not evidence that hybrid search will improve this distributor’s catalogue.

Check the underlying catalogue before tuning rankings. Inspect whether an imported PDF changed a zero into the letter O, whether a table row lost its header, and whether discontinued and current records were merged. Keep each code with its supplier, variant attributes, catalogue revision and source location. A matching string is only useful when it identifies the intended record; different suppliers may reuse a code.

Build a small test with agreed answers

Ask the catalogue owner and a counter colleague to label a manageable sample of actual requests. A proposed starting set is thirty queries, split across complete codes, staff abbreviations and descriptions; this is a test design, not a measured result or a statistical guarantee. Include English code strings inside Traditional Chinese requests, close variants, unknown codes and obsolete records. For each query, record acceptable catalogue IDs, the supporting source passage, and whether clarification is required. Reserve some cases for a final check after tuning.

These are invented test inputs, not product specifications or compatibility advice.

Compare three retrieval paths

Run the existing semantic search as the baseline. Next try a dedicated exact-code field lookup, followed by the existing semantic search when no verified exact record is available. Store approved aliases separately from the original code. Removing punctuation, changing letter case or expanding abbreviations should follow catalogue rules: stripping a suffix can collapse two different parts into one. Treat an ambiguous abbreviation as several candidates, not as an exact match.

For the third path, run lexical and semantic retrieval together and merge their ranked results, removing duplicate records. Keep the same catalogue snapshot, final passage limit, answer model and prompt across comparisons. If you alter document boundaries to retain row context, apply that repair to all paths before comparing them. Inspect how the lexical index tokenises hyphenated codes and Chinese text; BM25 ranking is not the same as a strict equality check on a code field.

Read the retrieved passages before scoring the final answer. For each query group, count how often the labelled record appears in the first position and anywhere within the fixed passage limit. Separately count wrong-variant answers and whether ambiguous or unknown requests receive the expected clarification. Keep the denominator visible, and record response time and staff checking effort. A strong average can hide complete-code failures that matter most at the counter.

Decide whether hybrid search earns its upkeep

The catalogue owner should agree acceptance conditions before seeing results. For this pilot, a useful proposed gate is no wrong-variant answer on the labelled complete-code cases, plus correct clarification on the ambiguous cases. Passing a small sample does not guarantee future correctness. Add a local response-time limit and an affordable maintenance budget. Hybrid retrieval brings a second index, ranking settings and catalogue-update checks; its benefit must survive the reserved cases and justify that work.

Choose the exact-code path if it fixes identifier failures while preserving useful descriptive search. Keep the existing approach if the experiment exposes only catalogue import defects that can be repaired directly. Adopt hybrid search only if it resolves additional labelled failures without unacceptable regressions or operating effort. If the correct passage is already present but the answer still chooses the wrong part, take that evidence into a separate answer-selection fix.

The next action is a working session with one catalogue owner and one counter colleague: choose the sample, agree the labels and save the current retrieval trace. Use the AI application planning guide below to turn the findings into a scoped change request. If implementation support is needed, bring that evidence to an AI integration discussion so the work starts with a demonstrated retrieval failure.