How to Use Screaming Frog

Screaming Frog SEO Spider is an indispensable tool for any SEO professional. It’s a desktop-based website crawler that allows you to audit your website from an SEO perspective, identifying common and complex issues that could be hindering your search engine rankings. This comprehensive guide will walk you through how to use Screaming Frog, from initial setup to in-depth analysis.

What is Screaming Frog?

At its core, Screaming Frog SEO Spider is a powerful website crawler. It mimics how search engine bots crawl the web, allowing you to discover a wealth of information about your website's structure, content, and technical SEO health. It's available in both a free version (with limitations) and a paid version with full functionality.

The tool collects key on-page elements for each URL it crawls, including:

Page Titles
Meta Descriptions
Headings (H1-H6)
Content
URLs
Image Alt Text
Canonical Tags
Robots.txt Directives
Redirects
Response Codes (200, 301, 404, etc.)

This data is presented in an easy-to-understand interface, making it a go-to for identifying technical SEO issues, conducting site audits, and planning website migrations.

Getting Started with Screaming Frog

Before you can dive into the analysis, you need to download and install Screaming Frog.

Installation

Download: Visit the official Screaming Frog website and download the installer for your operating system (Windows, macOS, or Ubuntu).
Install: Follow the on-screen instructions to complete the installation process.
License (Optional): If you have a license key, you can enter it upon launching the application to unlock the full features.

Initial Configuration

Once installed, it’s crucial to configure Screaming Frog to best suit your needs.

Basic Spider Configuration

When you first open Screaming Frog, you'll see a prompt to enter a URL. You can:

Enter a URL: Type in the URL of the website you want to crawl.
Upload a List: If you only want to crawl specific URLs, you can upload a list of them. This is useful for auditing specific sections of a large site or for recrawling.
Mode Selection:
- Spider Mode: This is the default and most common mode, used for crawling entire websites.
- List Mode: Useful for crawling a predefined list of URLs.
- API Mode: For integrating with APIs, such as Google Analytics or Search Console.

Important Settings to Consider

Before starting a crawl, adjust a few key settings for optimal results:

Crawl Speed: Under Configuration > Speed, you can adjust the crawl rate. For smaller sites or to avoid overwhelming a server, start with a lower speed. For larger sites, you might increase it, but be mindful of server resources.
User-Agent: Under Configuration > User-Agent, you can select which user agent Screaming Frog mimics. It defaults to a generic bot, but you can change it to mimic Googlebot or other search engine bots.
Crawl Limits: Under Configuration > Crawl-Limits, you can set limits on the number of URLs to crawl. This is helpful for testing configurations on smaller sections of a large website.
Respect robots.txt: By default, Screaming Frog respects robots.txt files. You can toggle this under Configuration > Robots.txt > Ignore Robots.txt. However, it's generally recommended to respect robots.txt unless you have a specific reason not to.
Include/Exclude: Under Configuration > Exclude and Configuration > Include, you can specify patterns to exclude or include during the crawl. This is useful for targeting specific subdomains or directories.

Performing Your First Crawl

With the basic configuration in place, you're ready to initiate your first crawl.

Enter URL: Type the website's domain into the search bar.
Start: Click the "Start" button.

Screaming Frog will begin crawling your website, and you'll see data populate in real-time across various tabs. The crawl process can take anywhere from a few minutes to several hours, depending on the size and complexity of the website.

Understanding the Interface and Tabs

The Screaming Frog interface is divided into several key tabs, each providing different insights.

Internal

This is the primary tab, showing a list of all the internal URLs discovered during the crawl. It displays essential information like:

URL: The web address of the page.
Status Code: The HTTP status code (e.g., 200 OK, 301 Moved Permanently, 404 Not Found).
Status Text: A description of the status code.
Page Title (Element): The title tag of the page.
Meta Description (Element): The meta description of the page.
H1 (Element): The main heading of the page.
H2 (Element): The second-level heading.
Word Count: The number of words on the page.
Inlinks/Outlinks: The number of internal links pointing to and from the URL.
Response Time: How long it took to retrieve the page.

You can filter this tab extensively to find specific issues. For example, filtering by "Status Code" and selecting "4xx" will show you all broken links.

External

This tab lists all the external URLs that your website links to. It's useful for identifying:

Orphaned Pages: Pages that have no incoming internal links but might still be indexed.
Broken External Links: External links that return a 404 error.

Page Titles

This tab focuses specifically on the title tags of your pages. It highlights:

Missing Titles: Pages without a title tag.
Duplicate Titles: Multiple pages with the same title.
Titles Too Long/Short: Titles that exceed recommended character limits (which can lead to truncation in search results).
Titles with Common/Weak Words: Identifying opportunities to improve title relevance.

Meta Descriptions

Similar to the Page Titles tab, this section analyzes your meta description tags, looking for:

Missing Descriptions: Pages without meta descriptions.
Duplicate Descriptions: Identical descriptions across multiple pages.
Descriptions Too Long/Short: Descriptions that are too long or too short for optimal display.

H1s and H2s

This tab is crucial for understanding your on-page content structure. It helps identify:

Missing H1s: Pages without a primary heading.
Duplicate H1s: Multiple pages with the same H1.
Multiple H1s: Pages with more than one H1 tag (generally discouraged).
Missing H2s: Pages lacking secondary headings.

Images

This tab provides details about all images found on your site, helping you identify:

Missing Alt Text: Images without descriptive alt attributes, which are important for accessibility and SEO.
Duplicate Alt Text: Images with the same alt text.
Large Images: Images that might be slowing down page load times.

Canonicalization

The Canonicalization tab is vital for managing duplicate content issues. It helps identify:

Canonicalized URLs: Pages that have been marked as canonical to another page.
Non-Canonical URLs: Pages that are canonicalized to another page but are not the canonical version themselves.
Canonical Loops: Situations where pages point to each other as canonical.

Redirects

This tab details all redirects found on your site, including:

301 (Permanent) Redirects: Permanent moves of content.
302 (Temporary) Redirects: Temporary moves.
Redirect Chains: Sequences of redirects (e.g., URL A -> URL B -> URL C), which can dilute link equity and negatively impact user experience.

Response Codes

This tab categorizes URLs by their HTTP response codes, making it easy to spot:

Client Errors (4xx): Most importantly, 404 Not Found errors, indicating broken links.
Server Errors (5xx): Issues with your server.
Redirection Errors (3xx): Redirects that might be incorrectly configured.

Key SEO Audits You Can Perform

Screaming Frog is a powerful tool for performing various types of SEO audits.

Technical SEO Audit

This is perhaps the most common use case for Screaming Frog. A technical SEO audit helps uncover issues that directly impact how search engines crawl and index your site.

Crawl your website.
Focus on the "Response Codes" tab: Filter for 4xx errors to find broken links and fix them. Look for 5xx errors to address server issues.
Analyze the "Internal" tab:
- Filter for "Status Code" is "200 OK".
- Check for "Word Count" is 0 or very low – these might be thin content pages.
- Examine "Inlinks" and "Outlinks" to identify potential orphaned pages or pages with few internal links.
Review the "Page Titles" and "Meta Descriptions" tabs: Identify missing, duplicate, too long, or too short tags. This is a fundamental step in how to create an SEO plan.
Inspect the "H1s" and "H2s" tabs: Ensure each page has a single, descriptive H1 and that H2s are used appropriately for content structure.
Check the "Canonicalization" tab: Ensure canonical tags are implemented correctly to prevent duplicate content issues.

Content Audit

While Screaming Frog isn't a content creation tool, it provides the data needed to perform a content audit.

Crawl your site.
Export the "Internal" tab.
Filter by "Word Count": Identify pages with very low word counts. These might require content expansion or consolidation.
Analyze "Page Titles" and "Meta Descriptions": Are they compelling and keyword-relevant?
Examine "H1s": Do they accurately reflect the page's content and include target keywords?
Look for Duplicate Content: While Screaming Frog doesn't flag duplicate content verbatim, it can identify duplicate title tags, meta descriptions, and H1 tags, which often indicate duplicate content issues. You'll need to cross-reference this with manual review or other tools for true content duplication.

Link Audit

Understanding your internal and external linking structure is crucial for SEO.

Internal Links:
- Use the "Internal" tab to see the number of "Inlinks" and "Outlinks" for each page.
- Filter for pages with zero "Inlinks" – these are orphaned pages that search engines might not find.
- Identify pages with a very high number of "Outlinks" – these can dilute link equity.
- Use the "Redirects" tab to check for redirect chains and broken redirects.
External Links:
- The "External" tab shows all outbound links.
- Filter for "Status Code" 4xx to find broken external links.

Crawl Budget Optimization

For large websites, optimizing the crawl budget is essential. Screaming Frog helps identify crawl inefficiencies.

Crawl your site using "List Mode": Start by crawling only your most important URLs (e.g., product pages, category pages).
Analyze Redirects: Identify excessive redirect chains that waste crawl budget.
Find Duplicate Content: Use canonical tags and redirects to consolidate duplicate pages, reducing the number of URLs search engines need to crawl.
Fix Broken Links: Ensure search engines aren't wasting time on 404 pages.
Check robots.txt: Ensure you aren't accidentally blocking important pages from being crawled.

Advanced Features and Tips

Screaming Frog offers many advanced features to enhance your audits.

Custom Extraction

This feature allows you to extract specific data from HTML using CSS Path, XPath, or Regex. This is incredibly useful for pulling custom data points, such as:

Schema markup data
Product prices
Specific content elements

API Integrations

Connect Screaming Frog to APIs like Google Analytics, Google Search Console, and PageSpeed Insights. This allows you to pull metrics directly into your crawl data, such as:

Pageviews
Sessions
Bounce Rate
Average Session Duration
Core Web Vitals scores

This unified data set provides a much richer context for your SEO analysis. For example, you can identify pages with low traffic but high bounce rates, or pages with poor performance in Search Console that also have technical issues.

Bulk Exporting

Screaming Frog allows you to export almost any data you see in the interface. This is crucial for detailed analysis, reporting, and sharing findings. You can export:

All internal URLs
Response codes
Page titles and meta descriptions
Hreflang data
Redirect chains

Visualizations

The "Site Structure" visualization offers a tree-like view of your website's internal linking. This helps you understand how pages are connected and identify potential issues like:

Deeply buried pages (pages far from the homepage)
Disconnected sections of your site

Crawling Specific Sections

To crawl only a specific part of a website, you can use the "Include" and "Exclude" functionalities under Configuration. For instance, to crawl only pages within a /blog/ directory, you would add /blog/ to the "Include" section. This is extremely useful when you're focusing on how to optimize product filtering on a large e-commerce site.

Analyzing Competitors

While Screaming Frog is primarily for auditing your own site, you can use it to crawl competitor sites (ethically and within their robots.txt limits) to understand their site structure and identify potential SEO opportunities or issues they might have. This can be a valuable part of how to analyze competitors.

Common Issues Found with Screaming Frog

404 Errors: Broken links leading to non-existent pages.
Redirect Chains: Multiple redirects that slow down users and dilute link equity.
Missing or Duplicate Title Tags/Meta Descriptions: Affecting click-through rates and search engine understanding.
Missing H1 Tags or Multiple H1 Tags: Harming content structure and relevance.
Thin Content: Pages with very little content, which may not provide enough value to users or search engines.
Orphaned Pages: Pages that are not linked to from anywhere else on the site, making them difficult for search engines to discover.
Incorrect Canonicalization: Leading to duplicate content indexing issues.
Large Image Files: Slowing down page load speed.
Non-Indexable Pages: Pages that are blocked from search engine indexing by robots.txt or noindex tags, when they should be indexed.

Best Practices for Using Screaming Frog

Crawl Regularly: SEO is dynamic. Schedule regular crawls to catch new issues as they arise.
Use the Paid Version: For serious SEO work, the paid version offers unlimited crawls and advanced features that are essential.
Understand HTTP Status Codes: Know what each code means and how to address it.
Filter and Sort Effectively: Master the filtering and sorting options in each tab to quickly find specific issues.
Export Data for Analysis: Don't just look at the interface; export data to spreadsheets for deeper analysis and reporting.
Combine with Other Tools: Screaming Frog is powerful, but it's often best used in conjunction with tools like Google Analytics, Google Search Console, and Ahrefs/Semrush for a holistic SEO strategy. For instance, understanding how contrast ratios affect user experience can be a separate but related optimization, like how to optimize contrast ratio.
Be Mindful of Server Load: Especially on smaller or shared hosting environments, crawl at a reasonable speed to avoid impacting website performance.

Frequently Asked Questions About Screaming Frog

What is the main purpose of Screaming Frog?

The main purpose of Screaming Frog SEO Spider is to crawl websites and collect key on-page and technical SEO data, allowing users to identify and fix issues that affect search engine visibility and user experience.

Is Screaming Frog free?

Screaming Frog offers a free version with limitations on the number of URLs you can crawl (500 URLs per crawl). For comprehensive audits and larger websites, a paid license is required.

How often should I run a Screaming Frog audit?

The frequency depends on your website's size and how often it changes. For active websites, monthly audits are recommended. For smaller or less frequently updated sites, quarterly audits might suffice.

Can Screaming Frog find duplicate content?

Screaming Frog can help identify duplicate on-page elements like title tags, meta descriptions, and H1 tags. For identifying actual duplicate content, you might need to combine its findings with other tools or manual checks.

What are the most critical issues to fix after a Screaming Frog crawl?

The most critical issues typically include 404 errors (broken links), 5xx server errors, redirect chains, missing critical page elements (titles, H1s), and incorrect canonicalization, as these directly impact crawlability, indexability, and user experience.

Conclusion

Screaming Frog SEO Spider is an essential tool for anyone serious about SEO. Its ability to quickly crawl and analyze vast amounts of data makes it invaluable for identifying technical SEO issues, conducting content audits, and understanding your website's structure. By mastering its features and regularly performing audits, you can significantly improve your website's performance in search engine results and provide a better experience for your users.

If you're looking to improve your website's search engine rankings and need expert assistance with your SEO strategy, we at ithile can help. We offer comprehensive SEO consulting services to help you achieve your online goals.

How to Optimize for Local SEO

What is Custom Metrics