Using Screaming Frog, A Great Free Webmaster Tool

I was late to the party when it came to Screaming Frog, what I consider a great free SEO tool that all Webmasters should use at some point. I started using it just to find broken links within my larger sites, then moved on to more complex technical SEO issues, that there is just not enough space to write about here.

I introduce Screaming Frog within my SEO training, to all students, as it is free (well in most parts), it often identifies things that I think are in place and fine. Of course this should be used in conjunctionwith the Google Search Console.

Some Screaming Frog Features and Benefits

1. Website Crawling: It works by crawling websites just like search engines do, allowing you to analyze various aspects such as URL structure, HTTP status codes, meta descriptions, page titles, and more.
2. SEO Audit: This enables you to perform a comprehensive SEO audit of your website, checking for issues like broken links, duplicate content, missing meta tags, and other problems.
3. XML Sitemap Generation: The tool can also generate XML sitemaps for your website, which helps search engines understand the structure and organization of your content. This improves your site’s visibility and indexing.
4. Hreflang Tag Analysis: If you are working in different languages, this tool can help you identify and audit hreflang tags on your international websites. This feature is useful for ensuring that search engines show the correct localized versions of your webpages to users.
5. Page Speed Insights: You can analyze the loading speed of your webpages and identify potential performance issues. This can help you optimize your site’s speed, leading to improved user experience and search engine rankings. although I go to Page Speed Insights direct.
6. Analytics Integration: You can integrate with various analytics platforms such as Google Analytics, enabling you to retrieve data and insights about your website’s performance in one central location.
7. Custom Extraction: The tool offers advanced options for extracting specific data from webpages, which can be useful for tasks such as competitor analysis, content auditing, or extracting structured data.

An Example Task: Locating 301 Redirects

Screaming Frog can be a helpful tool for managing 301 redirects on your website. Here’s how you can use it for that purpose:

1. Crawl your website: Launch Screaming Frog and enter the URL of your website in the “Enter URL to spider” field. Click on the “Start” button to initiate the crawl. This will help you gather information about your website’s URLs, including any existing 301 redirects.
2. Identify URL redirects: After the crawl is complete, go to the “Response Codes” tab, and filter by “3XX (Redirection)” status codes. This will show you a list of URLs that have redirect responses.
3. Review existing 301 redirects: In the “Inlinks” tab, filter the list further by “Status Code” to only show URLs with a 301 status code. This will give you a detailed overview of your existing 301 redirects, including the source URLs and the destination URLs.
4. Create a redirect map: Compile the information from the crawl and create a redirect map. The redirect map should include the old URLs (source URLs) and the new URLs (destination URLs) for the 301 redirects. This will serve as your reference when implementing the redirects.
5. Implement the redirects: Once you have your redirect map, you can proceed to implement the 301 redirects on your website. This typically involves modifying your website’s .htaccess file or using server-side configuration options. Ensure that each old URL is redirected to the new corresponding URL using a 301 status code.
6. Test the redirects: After you’ve implemented the redirects, it’s essential to test them to ensure they are working correctly. Use the Screaming Frog’s “Response Codes” tab again to check if the old URLs now show a 200 (OK) status code and redirect to the new URLs.
7. Submit the changes to search engines: Once you have successfully implemented and tested the 301 redirects, it’s important to update the search engines about the changes. You can do this by submitting an updated XML sitemap or using the respective search engine’s URL submission tool

Scan Your Website To Find 404s

This popular SEO tool can be used to crawl websites and identify various issues, including 404 errors. Here are the steps to use Screaming Frog to locate 404s:

1. Click on the “Start” button to begin the crawl. The software will start scanning the website for various URLs, links, and content.
2. Once the crawl is complete, you will see a summary of the crawl in the main window. Look for the “Status Codes” column and filter it to show only the 404 errors.
3. Click on the “Filter” drop-down menu and select “Not 200 OK.” This will show all URLs that return a status code other than 200, which includes 404 errors.
4. In the main window, scroll through the list of URLs to locate the 404 errors. You can see the specific URLs that are returning 404 errors in the “Address” column.
5. If you want to export the list of 404 errors to a separate file, go to the “Bulk Export” tab at the bottom of the window and choose the desired export format (e.g., CSV, Excel, or Google Drive). This will save the list of 404 errors as a separate file for further analysis or fixing.

So this is all about the technical stuff relating to your website, but I would guess that any SEO strategist or agency would have used Screaming Frog at some point. It isn’t the only tool that does this type of scan. But it is a least a starting point for those starting out, and for many others including me, a key tool I use reguarly. Click on this URL is you want to download and use. Screaming Frog SEO Spider Website Crawler

Just search fro Screaming Fro