What we shipped
- An agent that builds a keyword universe from a client profile by using recursive loops to expand positive keyword matches and contract negative keyword matches until it settles on a perfect list of transactional, commercial and informational keywords for the clients project.
- Agents for each primitive of the keyword research process including: URL chunking, intent mapping, keyword research, keyword mapping, content gapping and content update briefs.
Why we built it this way
Hitting an API to get a list of keywords based off a seed list has 2 opposite issues:
- it introduces too much noise
- it misses important keywords
Keyword research as a process also has major built in operational issues:
- scale complicates it
- intent and meaning can be ambiguous
- client data is rarely good enough to be used to compare against (if the client is coming to an SEO agency, it's likely the website doesn't serve as a canonical example of keyword targets)
- LLM context windows massively degrade accuracy over time
- it's a manual and tedious job but it's too important to give to a LLM to get wrong as its the inflection point for the entire campaign
Our process breaks each task inside a keyword research deliverables down into primitives that an LLM can ingest and understand.
We then built a test and a spec for each primitive output for both the human in the loop to confirm and the LLM to sweep over. This meant that each stage of research was being driven by a Type A specialist but the heavy lifting around pulling, cleaning and sorting data was being done by the LLM.
Each human edited primitive became the input for the next stage of the LLMs task so the further the LLM advanced, the better the data was that they were working with. This, in the end, produced near perfect keyword mapping, gapping and content quality issues that were then easily scheduled into our project management software to be updated.
How it benefits clients
- Keyword research can now be done in a day instead of weeks.
- We get a full view of the entire keyword landscape to understand costs of producing new content, editing old content and optimising existing pages.
- We can then forecast cost to benefit for the client and focus on the highest revenue drivers first instead of falling into the SEO trap of fixing something "just because"
- Once the system is built it allows for scale. Because the negative and positive keyword boundaries are so dialled in, we can run keyword expansion very easily as the guardrails are already in place.
What's next
Tightening the cleanup stage, which is the hardest part, so the mapping from keyword cluster to URL comes out clean more often on the first pass. It's likely that a non-keyword research primitive will be used in future iterations and the client research and seed keyword stage will be much more involved.
Adding some pre-flight checks to the process to stop agents running off and completing primitives without the requisite information to do the job well.
Deeper specifications for each primitive to check against.
Technical innovations
It runs as a phased pipeline, each phase independently runnable and auditable. Keywords are embedded and filtered with FAISS, then grouped into variant clusters only when three signals agree at once: topic cosine similarity, token Jaccard overlap, and identical search intent. Crucially, the maths only narrows the candidates. The final keyword-to-URL decision goes to the model, with the site's real structure as context, and every decision keeps a written reasoning trail.
How it fits the agentic journey
Our REST framework naturally adapted to becoming agentic as it fits into our AI manifesto beliefs that: it's all just databases and the smaller and more specific the step, the better so the model and the human can make easy judgement calls and keep the process moving.