Website Crawling
In this article, you will learn how website crawling works and why it’s important for your website’s SEO.
A 404 not found error happens when a website’s server can’t locate a specific URL or page that you’re trying to access via your browser or through a hyperlink. This URL is known in other words as a broken or dead link.
More specifically, the 404 error code comes from the set of HTTP codes (Hypertext Transfer Protocol) that browsers and servers use to communicate with each other.
To give you some context, the 404 error comes from the set of 4xx codes which are client-side errors, meaning from the user’s or browser’s request caused an error. There are a total of 5 classes of HTTP codes:
In this article, you’ll be learning specifically about the HTTP status 404 not found error. It's a technical SEO issue that needs to be fixed to prevent creating a negative user experience for the user and causing search engine crawling issues.
404 not found errors happen in a few different circumstances:
Let’s dive into each reason a bit more…
This situation happens very commonly when the user misspells the URL and that page doesn’t exist.
For example, if the page the user wanted to go to was hikeseo.co/learn/onsite/ but they typed in hikeseo.co/learn/oniste/ by accident, the server would send a 404 not found error to the browser.
This problem happens when webmasters have redirected a page URL to another URL, but the URL was incorrect.
For example, if we had an old page hikeseo.co/on-site/ and we redirected it to hikeseo.co/learn/on-site/ we would get a 404 error and realize that we mistyped the new URL.
This is also a common occurrence where a webmaster deletes a page and doesn’t update other links on the website that link to that page. What also happens is that links from other websites will still be pointing to the deleted page URL, which will cause a 404 error.
For example, if we had a page hikeseo.co/promo and then deleted it, any link on the website or externally that still pointed to that URL would create a 404 not found error.
Another situation where a 404 can occur is if a page URL is edited or changed, but the old URL hasn’t been redirected to the updated one.
For example, if we had a page hikeseo.co/learn-seo/ but then decided to change the URL to hikeseo.co/learn/ and forgot to redirect the old URL, then the old URL would cause a 404 for anyone trying to access it.
Soft 404s occur when the server doesn’t send a 404 error to the browser, yet the page still doesn’t exist. Google still views these pages as 404 errors, but it’s best practice to make sure the website sends proper 404 codes to missing or nonexistent pages.
There are many reasons why it’s important to fix 404 errors:
Improves User Experience - When a visitor encounters a 404 it can frustrate or confuse them, which leaves a negative experience with them. By fixing the 404 error, you ensure that the user finds the content they intended to find in the first place.
Maintain SEO - Search engines do not like broken links and it’s been known that websites with too many 404s can be penalized. Broken links also hamper the search engine’s ability to crawl the website efficiently and can lead to pages that are orphaned (no links to them).
Avoid Losing Traffic - If a visitor encounters a 404, they may leave and never come back again. This means losing a potential customer or even someone who might have linked to that content if it was there. By fixing it, you create the opportunity for the visitor to come back to the website again, which increases your traffic and potential customers.
Protect Brand Reputation - Links that don’t work create an unprofessional and unreliable experience, affecting the brand image, reputation, and credibility. Fixing the 404 errors protects your brand reputation and builds trust with your visitors.
Ensure Accurate Analytics - 404 errors can cause inaccuracies in analytics tracking, such as increased pageviews due to users refreshing the page, trying to access the broken link multiple times, and inflating bounce rates due to users leaving immediately.
By ensuring these 404s are fixed, analytics tracking can be more accurate in the data that is collected.
There are many ways in which 404 error pages can be discovered:
If you use a tool such as Screaming Frog, DeepCrawl, or Hike, these platforms will crawl your website pages and identify any 404 errors that need fixing.
If you have Google Search Console set up for your website, then any 404s or soft 404s will be flagged within the coverage section under “Index”.
These tools specifically check for broken links on your website, as opposed to a full website crawl that finds pages and highlights other issues. Some recommended tools include W3C Link Checker, Dead Link Checker, or Broken Link Check.
Fixing 404 errors is easy - the fastest way to do this is to create a redirect of the broken URL to point to a new, working URL. However, before you do this, there are two important considerations to keep in mind:
There are two types of redirects, each with their purpose, depending on the situation: 301 permanent redirects, and 302 temporary redirects.
301 redirects are to be used for page URLs that are not intended to be used again. The benefit of 301 redirects is that they pass on most (if not all) of the page authority of the broken URL to the new URL. Page authority is built from backlinks from external websites pointing to your webpage. This ensures that the new page URL has a high chance of ranking at the same level in the search results as the old URL, assuming the topic is the same, or if not, then at least boosting the authority and rankings of the new page.
302 redirects are ideally used for page URLs that will be used again soon. It could be that the page that is causing a 404 error is temporarily offline for maintenance or updates, and in the meantime, is temporarily redirected to another page. Once the edited page is live again, the 302 redirect can be removed and that page will resume working once again.
If the new page is the same but with a different URL, then the redirect is straightforward.
If, however, the page has been deleted and there’s no exact alternative, the goal is to find the next best match, meaning that the topic of the new page should match as closely as possible with the broken page topic. If there is no old content to reference, a hint as to what the broken page was about could lie within the URL text.
For example, if the broken page was hikeseo.co/learn/onsite/technical/404-examples/ and then the closest matching live page would be hikeseo.co/learn/onsite/technical/404-not-found/ (this page). You can see that the topic focuses on 404s, so it’s the closest match across all other live pages.
If the broken page is within a nested hierarchy and doesn’t have a close-match alternative, the next best option would be to redirect it to the parent page.
For example, if the broken page was hikeseo.co/learn/seo-in-1999/ then the parent page to redirect to would be hikeseo.co/learn/.
Yes, personalizing your 404 error page enhances user experience as it can include elements of humor, and/or helpful prompts or alternative links.
Instead of leading visitors to a blank, unhelpful 404 page, you can add value by making them smile, laugh, and pointing them in the next-best direction to help them find what they are looking for.
As you’ve learned in this article, 404 not found errors can be detrimental to both user experience and search engine optimization, so it’s important to address these issues as soon as possible on your website.
Using Hike, it’s easy to spot 404s on your website, and then action them on your CMS, either using a plugin or by using redirect settings on your domain or web host. Hike will automatically highlight action items, one 404 page per action that you can quickly fix and check off your list in no time.
Get your SEO in top shape with Hike, even if you don’t have much time or little to no technical knowledge as a small business owner.
We built Hike and Kit to help agencies deliver SEO without the staff overhead. Whether you’re reselling or scaling your own services, we make it easy to offer SEO that drives real results — no big SEO team required.
Love it or get a full refund
No long term commitments
14-day money back guarantee