Count the URL space before changing rules
A filter interface is useful to buyers because it narrows a catalogue by attributes such as colour, size, brand and availability. The same controls can create many addressable combinations. If a shop has a category, several brands and multiple sort orders, the number of URLs can grow much faster than the number of genuinely different buying decisions. Google’s current guidance says faceted navigation can cause overcrawling and slower discovery of useful pages. Start with a small inventory of the actual filter parameters, path segments, sort controls and pagination patterns in your site. Record which URLs your interface links to and which a crawler can reach from those links. Separate stable category and product pages from ephemeral combinations such as “blue, size medium, lowest price, in stock today”. The goal is not to erase useful customer controls. It is to decide whether a filter state also deserves to become a search landing page. That decision should be recorded before technical rules are written.
Choose a policy for each filter family
A merchant can usually start by asking whether a filtered page answers a persistent search question better than the unfiltered category. A dedicated “waterproof work boots” landing page may have a stable assortment, useful explanation and a clear internal path. A page for a temporary stock toggle and arbitrary sort order is less likely to have independent value. Mark each family as a candidate for crawling, a customer-only navigation state, or a case needing more evidence. Google recommends preventing crawling when filtered URLs do not need to appear in its products; it also gives a separate set of practices for URLs that should remain potentially crawlable. Do not treat a canonical tag as a magic block. Google says it may reduce crawling of noncanonical variants over time, but describes it as generally less effective for this purpose than blocking faceted URL crawling or using fragments. This is a product and content decision with technical consequences, so the store owner should approve the shortlist of meaningful landing pages.
Protect useful pages while limiting combinations
For filter states that do not need independent search visibility, Google describes robots.txt rules as one way to prevent crawling. It also describes URL fragments for filters, while noting that Google generally does not support fragments in crawling and indexing. Neither method should be copied blindly. A broad rule matching every question mark could accidentally cover a category pagination URL, campaign link or other page that the business actually wants discovered. Make a list of the exact parameter names and URL patterns, then test the rule against representative allowed and disallowed addresses. Keep product detail and core category pages reachable through ordinary links. If a selected filter page is meant to be searchable, make sure the broad rule does not swallow it. A robots rule controls crawler access; it is not a privacy control for unpublished products or customer information. Those require their own access decisions. Also avoid claiming a blocked URL can be read for a noindex directive that the crawler cannot fetch.
Make chosen URLs finite and consistent
If a shop keeps selected filtered URLs potentially crawlable, consistency matters. Google recommends the standard ampersand separator for query parameters and a logical, fixed order when filters are encoded in paths. Duplicate filters and nonsensical combinations should not create another valid page. For a filtered combination with no results, Google’s faceted-navigation guidance says to return HTTP 404 at that URL, rather than redirecting to a shared error address; it gives the same advice for nonexistent pagination. The shop still owes a helpful human explanation and a route back to a broader category. A 404 response is about the status of the nonexistent result page, not a reason to show an empty browser screen. Decide whether an empty result is genuinely a nonexistent combinational page under your implementation, and test the response code with the rendered message. Avoid a rule that rewrites every filtered URL to the unfiltered category while leaving the visible filter controls selected; that disconnects the page address from the customer’s state.
Test a small retail scenario
Imagine a stationery shop with a “notebooks” category, filters for paper size and cover colour, and a sort menu. The owner selects one useful page about A5 refillable notebooks because it answers a durable buying question and can carry an original buying explanation. Colour plus stock status plus sorting remain customer navigation only. The developer lists sample URLs for the base category, the chosen A5 page, an ordinary product detail page, a colour combination, a repeated size parameter and a page number beyond the results. For each address, the team records expected crawl access, expected HTTP status, canonical preference where relevant, visible content and internal links. They then request the pages and compare the actual responses with that sheet. These are illustrative categories, not a statement that Google will index the chosen page. Indexing is a separate outcome and depends on more than allowing a crawl. The test protects both the customer journey and the finite URL plan.
Review the policy when the catalogue changes
A filter policy can decay as the catalogue grows. A new brand, seasonal category or location-specific assortment may create a useful new landing question; a discontinued range may make an old curated page thin or empty. Assign a person to review the small set of allowed pages when merchandise changes, and compare the page’s actual products with its explanatory text. Check server logs or search tooling where available for unexpected parameter patterns, but do not treat a measurement outage as proof that no problem exists or as proof of an opportunity. Google’s troubleshooting guidance distinguishes crawling from indexing and warns that unnecessary URLs can consume serving resources. For a small business, the immediate test is simpler: can a buyer still filter normally, can a crawler reach the intended category and product pages, and do impossible combinations produce the planned response? Keep those results beside the filter policy so a later developer does not undo the decision by adding one more parameter without review.
Match canonical intent to visible content
A canonical preference can help identify the representative URL among similar pages, but it should reflect what the page actually shows. If a shop points every filtered page to the broad category, it is signalling that those combinations are not independent landing pages. That may fit colour and sort variations, but it conflicts with an editorial decision to make a carefully written category subset a distinct destination. Record the canonical preference beside each selected URL family and test it in the rendered output, because templates sometimes emit a self-reference on one version and a category reference on another. Google’s faceted guidance treats canonical as a less effective long-term crawl control than its primary approaches, so it should be considered a preference signal rather than the whole plan. Where filters change visible products, headings and explanatory copy, compare the pages as a reader would. If the intended representative page does not answer the same decision, revise the grouping. This protects the store from hiding a useful page or promoting many near-identical ones by accident.
Give merchandising and development one rulebook
The final deliverable should be a short URL decision sheet that people outside the search team can maintain. For each parameter, state its customer purpose, whether it changes a crawlable address, whether combinations can be linked internally, the expected status of an impossible state and the owner who may add new values. Include a few real example URLs from the shop, while removing any customer-specific tokens. The merchandise team can then flag a new high-value category before a developer launches another generic filter. The developer can check that a new sort choice does not create an unexpected series of crawlable pages. A release check should navigate from a category to a selected filter, open a product, return to the results and verify that the the buyer’s chosen state remains understandable. Crawl controls should not make the interface unusable. Revisit the sheet after a redesign, a new locale, a campaign with persistent landing pages or a change in catalogue size. This is an editorial control for useful answers as well as an engineering control for URL growth.
Before you proceed
- Each filter family has a documented crawl decision.
- Representative allowed, blocked and impossible URLs return the expected result.
- Customer filtering still works after crawler controls are applied.
