How to Cluster Keywords With AI and Keep Search Intent Intact
Compare embedding and shared-ranking-URL keyword clustering, set thresholds from vendor and open-source docs, and run an overlap check that keeps two intents off one page.
On this page
AI keyword clustering without losing search intent nuance comes down to a division of labor: embeddings propose which keywords belong near each other, and the results page decides which of them can share a URL. An embedding measures how alike two phrases read. Intent shows up in which pages a search engine ranks for them, and the two can disagree, so a definition query and a product query may sit close together in vector space and still need different pages. One similarity score behind one button is a weak basis for a content brief.
Should meaning, shared results, or both decide a cluster?
Three signals are in use, and the documentation behind each is candid about its limits.
| Method | What it compares | How its own documentation describes it | Where intents get merged |
|---|---|---|---|
| Embedding similarity | Distance between vectors of the keyword text | A threshold is set and "Clusters below that threshold are merged" (Sentence Transformers) | Phrases that read alike and want different pages |
| Shared ranking URLs | The pages ranking for each keyword | Top 7 results, clustered at "40% or more URLs in common" (Keyword Insights) | Thin or unstable results, or the wrong country setting |
| Top result only | The top traffic keyword of the first-ranked page | Parent Topic, taken from "the #1 ranking page for your keyword" (Ahrefs) | One strong page ranking first for several intents |
| Both, in two layers | Shared URLs for pages, language similarity for hubs | Standard clusters from results, "topical clusters" from NLP (Keyword Insights) | Fewer places, for the price of a lookup per keyword |
Sentence Transformers documents k-means, agglomerative clustering and a fast community-detection routine, and its note on k-means explains why that algorithm suits keyword lists poorly: "K-Means requires that the number of clusters is specified beforehand. The sentences are clustered in groups of about equal size." Nobody knows the page count before clustering, and intents come in very unequal sizes. Agglomerative clustering takes a threshold instead of a cluster count.
Keyword Insights works from live results instead: "In the keyword clustering process, we examine each keyword in the top 7 search results. If a keyword shares 40% or more URLs in common with others (a percentage you can modify in the settings), we classify them as a cluster." Ahrefs takes a shortcut through its Parent Topic metric, for which "we take the #1 ranking page for your keyword and find the keyword responsible for sending the most traffic to that page." The same article, first published in March 2024, concedes that "Keywords Explorer is arguably less accurate than some keyword clustering tools because it clusters by Parent Topic." It judges similarity from the top-ranking result where other tools read the whole results page, and speed is what it gets in return.
The detail worth copying is how Keyword Insights combines the two signals. Page-level clusters come from shared URLs. A second layer, which the Keyword Insights help center calls topical clusters, groups those clusters with natural language processing, and its settings are described in numbers: soft groups clusters that are "about 70% alike", medium "around 88% alike" and hard "about 93% alike". Even inside one of those topical clusters, the page says of its own example, "Each will need its own separate article." A vendor that runs both layers uses language similarity to build the hub and results overlap to decide the page.
So embeddings alone are enough for removing near-duplicate phrasings, sketching hubs and internal links, and sizing a topic before you decide how to split a content budget between SEO and AI answer visibility. Shared URLs alone suit a focused list that only needs a page map. For a list that will turn into briefs, run both: embeddings first to form hubs and shrink the number of comparisons, then overlap inside each hub.
How should the thresholds be set?
The semantic clustering best practices that matter most for AI keyword clustering and search intent in SEO concern thresholds, because a threshold is where two intents stay apart or get folded into one page.
State overlap as shared URLs out of a known depth. When we read its help center in October 2026, Keyword Insights described its default at least three ways: 40% of URLs in common on the overview, 3 shared URLs, shown against the top 10, on the Keyword Insights grouping accuracy page, and 4 shared URLs from the top 7 on the Keyword Insights algorithm page. Those are different bars, since 3 of 10 is 30% and 4 of 7 is about 57%, so read the value in your own account before trusting a cluster. The vendor narrowed its overlap setting from a 1-10 range to 1-7 because, it says, "Google no longer consistently shows 10 traditional links in the top 10 results." It also compares "the full ranking URL when clustering and not the root domain", which means two different pages on one domain do not count as a match.
How members are compared matters as much as the number. The algorithm page describes a centroid method that groups keywords around the highest-volume term, where members "won't necessarily have common URLs with each other", and an agglomerative method in which "All keywords are compared against one another". It says the second produces smaller, tighter clusters and recommends it for most customers. The same choice exists on the embedding side as linkage. In scikit-learn, distance_threshold is "The linkage distance threshold at or above which clusters will not be merged." Complete linkage "uses the maximum distances between all observations of the two sets", so the threshold caps the least similar pair in a cluster. Single linkage uses the minimum, which lets a chain of near neighbors pull two unrelated keywords into one group. For intent work, choose complete or average.
Minimum cluster size is the default that quietly removes intents. The community_detection function in Sentence Transformers defaults to a similarity threshold of 0.75 and a min_community_size of 10, and its reference says it "Returns only communities that are larger than min_community_size." A sound page can have three keywords behind it, and under those defaults that group never reaches the spreadsheet.
We found no primary source that names a cosine similarity at which two keywords share an intent, and the right number moves with the embedding model. Our suggestion is to label a sample of keyword pairs by results overlap, then pick the loosest embedding threshold that still keeps the failed pairs in separate clusters.
The overlap check to run before any brief is written
- For each cluster with two or more keywords, pull the top 7 organic URLs per keyword for the target country and language. Keyword Insights calls the correct location "essential for precise SERP clustering".
- Count the full URLs each member shares with the cluster's head term, then with every other member.
- Keep the members at or above your bar. Move the rest out, as their own page or into a cluster they do overlap with.
- For anything within one URL of the bar, read the page types that rank before deciding, since product pages and explainers are different jobs.
- If your tool labels intent per keyword, treat mixed labels inside one cluster as a flag. Keyword Insights says its per-keyword labels let you "easily spot keywords within a cluster that have a different intent."
The following example is hypothetical. An embedding pass has grouped five keywords that contain "keyword clustering", and the counts are invented to show the arithmetic against a bar of 4 shared URLs out of 7.
| Keyword in the embedding cluster | URLs shared with the head term (of 7) | Page types ranking | Decision |
|---|---|---|---|
| keyword clustering tool (head term) | n/a | Product pages | Page A |
| best keyword clustering software | 5 | Product pages and roundups | Page A |
| free keyword clustering tool | 4 | Product pages | Page A, once it also clears the bar against the row above |
| what is keyword clustering | 1 | Guides and glossaries | Page B |
| keyword clustering python script | 0 | Code tutorials and repositories | Page C |
Five results lookups turn one muddled brief into three clear ones: a product page, an explainer and a technical tutorial. The pairwise step matters for the third row, because sharing four URLs with the head term says nothing about what that keyword shares with the second.
What else belongs on a best-practice list?
The best practices for AI keyword clustering that protect search intent in SEO fit on a short list, and most are habits around the tooling.
- Let embeddings build hubs and let shared ranking URLs decide pages.
- Write every threshold as shared URLs out of a stated depth, on full URLs, for one country and language.
- Use all-pairs comparison whenever the output becomes briefs.
- Keep small clusters, because a three-keyword intent is still a page.
- Treat clusters as snapshots of live results. Rerun the check before a brief ships and when rankings for a cluster move.
- Store the keyword-to-page map so each URL owns one intent, which also gives AI share of voice tracking a stable unit to report against.
The same discipline carries into AI search. Google Search Central says AI Overviews and AI Mode may use what it calls a "query fan-out" technique, issuing "multiple related searches across subtopics and data sources", and that "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." Our reading is that a page scoped to one intent is the natural candidate for one of those narrower searches, which is the structure that GEO work and the tactics for getting cited by ChatGPT and AI Overviews both assume.
Our AI search optimization team takes on this kind of keyword-to-page mapping, and the AI Visibility Checker and the AI search playbook are further resources once the pages are settled.
Hand a writer a cluster with its evidence attached (the shared URLs, the depth they were counted over, the country, and the date the results were pulled), and the brief that follows has one intent to serve.
Sources
- Sentence Transformers, clustering examples
- Sentence Transformers, community_detection reference
- scikit-learn, AgglomerativeClustering reference
- Keyword Insights, keyword clustering overview
- Keyword Insights, keyword grouping accuracy
- Keyword Insights, clustering types
- Keyword Insights, topical cluster creation method
- Ahrefs, Keyword Clustering in Seconds
- Google Search Central, AI Features and Your Website