This post is a duplicate of my post on distilled.net and it is canonicalised to that address.
In my experience, the most common reasons a website doesn’t rank for a specific keyword are:
- The site isn’t actually relevant
- There isn’t any single page which actually uses that keyword
- There are too many pages which use that keyword.
It’s true that a site might not rank because of something like authority or some kind of technical issue but for the average SEO team, increasing authority or fixing technical problems are far harder and take much longer than simply looking at the site and making sure we’re not just misusing content.
So often, I’ve started working with teams who have spent months or years trying to perfect their technical setup, who’ve invested thousands in link building campaigns but haven’t made sure they include the keyword they are targeting on the relevant page.
I will explain in more depth below but here are the first eight things to do to find out why you’re not ranking;
- If you used to rank – check what changed
- Do the quickest and easiest technical checks
- Ignore link building until you know there is no other explanation
- Make sure the keywords are actually relevant to your site
- Make sure you have a page targeting the keywords
- Make sure you don’t have loads of pages targeting the keywords
- Make sure the page you are targeting the keyword with is the right kind
- Do more in-depth technical checks
- Only now look at the harder solutions
That list is deliberately ordered to start with easy checks that could save all of your site traffic, steadily getting more granular and time intensive but always with a focus on the easiest thing we can do to make the biggest impact on the problem at hand.
1. If you used to rank – check what changed
If you don’t have any evidence that you used to consistently rank well then skip to step two. As a rule of thumb – you should only really focus on this section if you know that you consistently ranked top 20 for at least a couple months.
As an aside – if the keyword you care about is a “money term” like “Valentines Day flowers” and you seem to have lost traction just as the season is arriving, you might want to read Tom Capper’s post on how to rank for head terms for some insight. If you’re a smaller brand – the rankings may return to normal just after the peak season (which may be small consolation but could get people off your back while you do the foundational work).
Assuming you’ve seen rankings change for a selected number of keywords which you know you used to rank for, and it’s not a matter of “money terms” fluctuating around peak season, we can simplify things by splitting changes into three groups;
- Someone changed something on your site so now you’re not doing as well
- A competitor started doing something better so now you’re not doing as well
- Google changed their algorithm so now you’re not doing as well
Someone changed something on your site so now you’re not doing as well
It can be really hard to keep track of all the changes on a site so it’s quite possible someone made a change you didn’t know about and that’s why you’re no longer ranking.
You could ask your devs, particularly if you already have a good idea of the dates to look into. Devs keep impressively detailed logs of what they’ve changed so that could answer your question quickly. However, it might not answer your question, and it could have been someone else changing something through the CMS who made no record of the change whatsoever.
If you don’t have a tool like Deepcrawl running regular crawls of your site but do have an old Screaming Frog crawl of your site then you can use my free Change Detection Google Sheet to help get an idea of what might have changed. If you can do that – have a look through the results and try to work out what changes might have caused these issues. In particular look for pages being removed or indexation commands, then expand your search to things like keyword changes.
If that doesn’t turn anything up, read through the rest of this blog post.
A competitor started doing something better so now you’re not doing as well
If a competitor started producing better content, or fixed something technical on their site, that could mean they jump up in rankings, pushing you down.
The key thing here is to look at historic data and see who might be doing better now.If you have historic rank tracking in a tool like Stat, start by looking for which competitors jumped up for the keywords you care about. You could even create a tag for the keywords you care about in specific and use the competitive landscape report to see what changed. If you don’t have historic data in something like Stat you could try your luck with Ahrefs to see if they happen to have historic data for the keywords you care about.
If you have no luck in either, the fact that your company cares about this keywords, could mean that someone on your team just knows which site is appearing which wasn’t there before.
If you can identify the site or sites which are doing better, look at the pages they’ve made, see what differences there are between theirs and yours (do they have more information? Are they a homepage? Are they featured in the nav?). Then as much as you can, without just copying the page – steal their tactics.
If you can’t find any sites in particular which seem to have jumped up, if you don’t have historic data or if your site just seems to have gone down then read through the rest of this blog post.
Google changed their algorithm
Marie Haynes keeps a great, comprehensive list of algorithm changes. If you used to rank – look at when you used to rank, and when you just stopped ranking. If that lines up (like, to the day) with a non-trivial algorithm change then that could be the culprit. If the algorithm change was recent, it’s worth leaving it a couple weeks to see if things settle down.
Google’s algorithm is, understandably a source of frequent stress for website owners because at any point they can turn a dial and it becomes as if we’re working in a completely different search engine. For this investigation, that is exactly how we should think about Google before and after an algorithm change. It is a different search engine. That simplifies our question because we’re no longer talking about keywords we used to rank for – we’re talking about keywords we never ranked for in this new search engine. That means we can follow a lot of the same principles for working out what’s going wrong.
Go through all of the steps in sections 2-9 in order as if you never ranked.
2. Do the quickest and easiest technical checks
As I said, it’s easy to become convinced that some mysterious technical issue is preventing you from appearing. In the vein of that stereotype about mechanics – it’s difficult to know when you ask an SEO agency if technical fixes are actually needed or if they are just blinding you with jargon (I’ve been on the receiving end of that myself when I was in-house).
I’m here to tell you that you don’t need to start with those expensive agency conversations. You can start with some really simple checks and I’m going to give the complete layman’s description of each one below. If any of the checks below return a result that you’re not expecting, hop over to my colleague Ben Estes’ great technical SEO checklist.
Check that Google has seen and saved some of your site
Have you ever noticed that Google lists how many results it found when we do a search? We can use that to get a rough idea of if it has seen and saved the pages on our site and to make sure we haven’t, for some reason, been removed from Google.
Go to Google and then type; site:<your website domain>
So for example, if I wasn’t to check the Distilled site, an individual page might be https://www.distilled.net/resources/seo-ab-testing-whitepaper/, to check the site, I take just “distilled.net” and write site:distilled.net
If the number of results Google lists is much higher or lower than you’d expect, then I’d check Ben’s list. But I mean much higher or lower. If you think you’ve got about 500-1000 pages on your website and Google says it found a million pages, something is probably wrong. Likewise, if Google has only found 10 pages and you haven’t just launched your site, something is probably wrong.
You can also use this check for specific subdomains. For instance, if you have a blog at blog.yoursite.com you can write;
Check that you rank for your own brand name
As long as your brand isn’t totally new, and isn’t just a competitive term (think “Car Rentals” or “Injury Lawyers”) your site should appear when you type in your brand name.
This bit is pretty simple. I feel like I should write more but that’s it – search for your brand name. Are you coming up? If so then great, on to the next step. If not – check Ben’s list.
Check that your pages rank for their exact content
Go to some of your most important pages. For each, select one of the top paragraphs on the page, copy about a sentence and then paste that whole sentence into Google. If you don’t appear anywhere then there could be some reason Google hasn’t seen the page or it’s been removed. If you see other sites appearing with that exact content, that might be your problem!
3. Ignore link building until you know there is no other explanation
Believe me, I understand the appeal and I often manage link building campaigns for my clients. Producing a large, impressive link-building piece often lets a team sidestep internal politics and dev queues to actually get something published. They also often look great and can sometimes secure TV coverage. I’ve even seen big Creative pieces as a way for the SEO team to get the attention and approval of the CEO to increase internal clout which makes other things easier. For some sites, links are that piece that’s missing which would allow them to rank. However;
Links are almost never the answer.
Links are almost never the answer.
Links are almost never the answer.
Links are almost never the answer.
Proper link building is hard, it’s expensive, and link volume is often not what’s stopping you from ranking.
You know what’s worse than having your CEO breathing down your neck because you don’t rank for one specific keyword? Having your CEO breathing down your neck because you don’t rank for one specific keyword and you’ve just spent £60,000 on link building campaigns which haven’t had an impact.
If you decide early on that you need links, the only way to find out you’re wrong is after you spent all that time and money. There are much cheaper and easier things you should do and check before you even touch link building. If you get to the end of this list and you’ve done everything, then you can consider link building pieces.
4. Make sure the keywords are actually relevant to your site
It’s very easy to assume that everyone thinks and talks the same way we do, that because we refer to our core product as “widgets”, that everyone searching for “widgets” wants us.
Google is a business – their continued success is dependent on giving people answers to what they are actually searching for, so Google doesn’t care what you think is relevant, Google cares what 90% of people are searching for.
The easiest way to check what Google thinks something means, is to Google it (shocker). Start by getting the list of keywords you want to rank for. Then either Google each of them yourself, or use a tool like Stat, SEMRush, or Ahrefs to get the top 10 results for each.
If direct business competitors appear in the top 10, then that’s a sign Google thinks you can be relevant. When I say direct business competitors I mean sites which are trying to do the same thing as you. So sites using the same word to mean a different thing are definitely out, but you should also think again if all of the results are things like Wikipedia, or a listing site.
If all of the results are things like Wikipedia you might still be able to target the term with an informational blog post, if it’s all listing sites things become trickier and I’d start by focusing on other things and then come back to these listings-heavy keywords.
5. Make sure you have a page targeting the keyword
Believe me, the impact of just creating a page targeting a keyword can be stark. Below is a screenshot showing the difference before and after launching a targeted page. For the previous years the site was not ranking at all – having a page clearly targeting the keyword pushed them to position two within the space of a couple weeks.
It could be argued that this should be the first check but it’s quite intentionally not. If it was, a person might never ask themselves “is it right for us to rank?” or “have I accidentally blocked Google from my site?” and those are more important questions in terms of saving your time and traffic.
There are a few ways to check this. You could use my keyword checker Google sheet to check if you’re ranking for specific keywords. Get the keyword checker sheet here:
- I’d like to receive content like this via email.
The easiest if you’re only checking a few keywords is to reuse the site: search we mentioned above. This time, as well as searching for your domain, search for your keyword too, in quotes. So for example:
site:distilled.net “technical audit checklist”
That will give you a list of all the pages on your site which specifically mention the keyword. There are a few possible scenarios here.
Loads of pages show up
Important – if loads of pages come up then your problem might be that you have too many pages competing for this keyword. Or a combination of that, plus having no one page focused enough. Instead of working your way down the list of pages, follow the steps in section six.
A handful of pages show up
You might still have issues with too many pages competing, so still take a look at section six. Before you do that – starting with the page at the top, work your way down the list, visiting each page, and try to work out if the keyword you want to rank for is the main focus of the page.
No pages show up
If no pages appear then you don’t have any pages, which Google knows about, which are targeting the keyword. Find a page which you would expect to target this keyword and make sure that Google knows about it. Then, just add this keyword in the title or meta description and monitor your progress.
If that’s not enough add some (valuable) content to the page targeting this keyword.
If that’s not enough then try creating a page or blog post specifically about this topic. Consider removing the keyword from the page you just changed to avoid cannibalisation issues which we discuss in section 6. If you don’t know what kind of page to create, read section 7 of this blog post.
6. Make sure you don’t have loads of pages targeting the keywords
Targeting a keyword with too many pages can and does hurt your traffic. Sometimes as badly as having no page at all.
The below is a modest example of a page jumping from around position 40 to position 7 because we removed the keyword in question from a bunch of other pages.
When too many pages are competing for the same keyword we often refer to that as “cannibalisation”. To check for cannibalisation, look for any combination of the following things;
- A few pages are ranking for the keyword, but poorly. For instance, they’re all in the bottom half of page 2, or worse.
- One page ranks well for a few days, but then for a day or so, another page starts ranking below it (i.e. one is in position 11, the other is in position 12). Then they both disappear entirely.
- No page is ranking consistently well and Google keeps switching between different pages.
You can check for cannibalisation by using Search Console, a paid keyword tracker like Stat, or manual searches. I’ve given details for each below.
Paid keyword tracker
If you have access to a keyword tracker like Stat, track the keywords for a little while. If you’re using Stat, in the keywords report select the keyword you’re interested in.
Use the “Overview” tab to track rankings over time and look for things like rankings jumping up and down.
Use the “Archived SERPs” tab to check if multiple pages from your domain are ranking at the same time (they’ll be highlighted in yellow)
Search Console is a great, free source of cannibalisation data with a few drawbacks;
- Search Console reports won’t show keywords that you haven’t had any impressions for. So if you’re doing so badly for a keyword that no one even saw your site, you won’t see it at all in Search Console data.
- Search Console data is sampled, so you can’t guarantee you’ll get every keyword.
- Search Console data doesn’t make the distinction between multiple pages ranking, and getting site links so
For this result, Search Console would show the homepage, About Us, Jobs, Blog, and Resources pages as ranking, even though this isn’t a case of cannibalisation.
With that in mind, you can use this Google Sheet to check your recent Search Console data for keywords where you have a few pages ranking. There are instructions on the first tab of the sheet for how to use it.Get the keyword checker sheet here:
- I’d like to receive content like this via email.
If you only have a handful of keywords you want to check for, you don’t have access to paid tools, and Search Console isn’t turning anything up then follow the steps in section 5. If you see a lot of pages coming up for a keyword, or even a handful of pages which are prominently targeting the keyword that could be your issue.
Try removing the site: search and flick through the first ten pages of results to see if multiple pages appear. Bonus points if multiple pages from your site are appearing very close together (i.e. positions 27 and 28).
Whichever method you use to work out whether you are cannibalising on keywords, once you know, the next step is to fix it. Choose the best page to target the keyword – for a way to go about that, read section 7. of this post which is about making sure you’re targeting a keyword with the right kind of page.
Once you’ve chosen your page, you have two levers you can move. You can make your chosen page more relevant for the keyword, or you can make the other pages less relevant for the keyword. I’d advise first making the page you choose a little more relevant, then making other pages less relevant until either you see a result or you run out of things to do. If you’ve run out of things to do, start making your chosen page more relevant until you run out of options there too.
Making your chosen page more relevant can be as simple as adding content. If the keyword isn’t mentioned in your title, meta descriptions, etc. try adding it. If you’ve already done that, consider adding a short paragraph about the topic you want to rank for. Resist the urge to keyword stuff and don’t add content which doesn’t make sense on the page.
Making other pages less relevant can be as simple as removing content. If the keyword is in your title, meta descriptions etc. try removing it or using different words. If there’s specific on-page content which is about this keyword, consider moving and combining all of that content on the page you want to rank. If the whole page is about the keyword, maybe the page you’re trying to remove is actually the one that should rank? If you’re certain that this page isn’t the one that should rank, first check for other keywords it might be ranking for or, to make sure you’re not throwing anything away, then canonicalise it or redirect it to the main page you want to rank.
7. Make sure the page you are targeting the keyword with is the right kind
Sometimes, even if a site is the right kind of site, and a page is clearly the one you want to rank, Google won’t let it rank because it’s the wrong kind of page.
The main ways we can categorise a page on a site are;
- Strength (Is this the homepage? Linked to in the nav? Or an old forgotten blog post?)
- Specificity (Is this a broad page which just mentions the keyword or is it all about that term?)
- Type (Is it a blog post or product page? This relates to search intent which we’ll cover below)
What you should do here is look at what is already ranking in the top ten.
Is it mainly sites homepages? In which case, the norm is strong (homepages get most of the focus of a site) and broad (homepages don’t tend to be terribly focused).
Is it mainly pages which don’t appear in those sites internal navigation? Which specifically dedicated to this keyword? You can often tell because the keyword will be in the URL of the page or a lot of the page content will be about the term. In which case the norm is probably weaker but more specific.
Then we need to work out whether the norm is for product pages or blog posts? Here is where we start down the rabbit hole of “search intent”. “Search intent” is the catch-all for “what the person thinks they’re going to get when they search”.
An easy way to see this in action is to search “ski” and “skis”. When we search “ski” Google thinks that we either want to know more about skiing or that we want to do something (in this case go skiing). When we search for “skis” Google thinks we want to buy skis.
A site which wants to rank for “ski” will not be able to do so with a page selling skis, no matter how strong and well-optimised that page is. So if most of the pages that are ranking are for holidays – your page should be about holidays. If they are mostly blog posts and Wikipedia articles, it should be an informational page – don’t try to sell something. If they’re all specific product pages – create a product page. While just following the herd isn’t usually the ideal strategy, looking at the category of content which is currently performing well is the best way to get a sense of direction, then you can tweak other things.
While you should be able to get what you need, at least for small lists of keywords, by manually checking, if you’d like to dive in to a more technical solution to checking search intent Rory Truesdale has been doing some excellent work on this and has written about it in Search Engine Journal.
Once you know the intent, strength, and specificity of content already ranking, you can start to play around with the strength and specificity. If you have a weak page which is also quite broad – experiment with making that page more specific by adding more content, or stronger by linking to it internally. If all of the ranking sites are doing so with strong, broad pages, for instance a page linked to in the top nav, experiment with making one of your strong pages more specific or choose an even stronger page, like your homepage for example, and see how specific you can make that.
A general rule of thumb is that you most likely won’t be able to go against the intent of a search, but dialling up either the strength or specificity of how you are targeting a term will make you more likely to rank.
If you have no pages on your site which target the search term and the right search intent – try changing one to target it, or creating one even if you have other pages (of the wrong intent) already targeting the term. If that causes cannibalisation problems you can then deal with them, if it doesn’t then you’ve got an easy result. To be clear here – you do not have to have one page for every keyword. You can have a page targeting more than one keyword, but if there is a topic you want to rank for, which would fill a page by itself, and it has enough value to your business to justify a whole page then make that!
If you have lots of pages on your site targeting the search term and only one of them has the right search intent, select that page as the primary page to target this term, increase how specific and relevant this primary page is (by adding content), and decrease the specificity and relevance of other pages (by removing content or removing the page altogether).
If you have lots of pages on your site targeting the search term and any of them could target the right search intent, start with the strongest page. Ask yourself – would my moodiest website visitors (and my boss) be happy if I put a paragraph on this page about this topic. If not, the topic isn’t important enough to go on that page. Repeat the process working your way down the list until you find the strongest page you can make more specific, then follow the steps in section 6. If you’ve done everything in section 6 and you’re still not seeing any improvement, maybe the page you chose was too weak. Try linking to it more internally or decide if you’d prefer to choose a stronger page.
8. Do more in-depth technical checks
If you’ve made sure you have, well written, strong, specific, relevant pages targeting the keywords you want to rank for, and you’re still not getting anywhere, you might be able to tip the scales by making some technical improvements to your site.
While a lot of technical improvements will improve overall site health (and there are lots of graphs of those) the screenshot below is directly from a report to another client where we made technical improvements bumped up page rankings for a business-priority keyword from page 2, to the middle of page 1. Since this point the page has consistently ranked around position 6-8.
Technical checks can be intimidating. Fortunately, Ben Estes has produced a great technical SEO checklist which will lead you through a lot of the most common technical issues.
9. Only now look at harder solutions
Just as no two businesses are the same, no two sites are the same. Google, quite intentionally, works in mysterious ways and sometimes when we’ve covered all of our bases we still don’t have a good idea of why we aren’t ranking. At this point you could look into deeper technical issues, using tools like log file analysis, you could compare site speed with competitors to see if that’s what giving them the edge, or you could try building links.
I hope you haven’t had to get this far, but if you have there are still options out there, and you can investigate them, or hire to solve them, in the knowledge that you’ve covered the core stuff already.