Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Mozscape API
    • Free SEO Tools
      • Competitive Research
      • Link Explorer
      • Keyword Explorer
      • Domain Analysis
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your All-In-One Suite of SEO Tools

        The essential SEO toolset: keyword research, link building, site audits, page optimization, rank tracking, reporting, and more.

        Learn more
        Try Moz Pro free
        Illustration of Moz Pro
      • Moz Local

        Complete Local SEO Management

        Raise your local SEO visibility with easy directory distribution, review management, listing updates, and more.

        Learn more
        Check my presence
        Illustration of Moz Local
      • STAT

        Enterprise Rank Tracking

        SERP tracking and analytics for SEO experts, STAT helps you stay competitive and agile with fresh insights.

        Learn more
        Book a demo
        Illustration of STAT
      • Mozscape API

        The Power of Moz Data via API

        Power your SEO with the proven, most accurate link metrics in the industry, powered by our index of trillions of links.

        Learn more
        Get connected
        Illustration of Mozscape API
      • Compare SEO Products
    • Free SEO Tools
      • Competitive Research

        Competitive Intelligence to Fuel Your SEO Strategy

        Gain intel on your top SERP competitors, keyword gaps, and content opportunities.

        Find competitors
        Illustration of Competitive Research
      • Link Explorer

        Powerful Backlink Data for SEO

        Explore our index of over 40 trillion links to find backlinks, anchor text, Domain Authority, spam score, and more.

        Get link data
        Illustration of Link Explorer
      • Keyword Explorer

        The One Keyword Research Tool for SEO Success

        Discover the best traffic-driving keywords for your site from our index of over 500 million real keywords.

        Search keywords
        Illustration of Keyword Explorer
      • Domain Analysis

        Free Domain SEO Analysis Tool

        Get top competitive SEO metrics like Domain Authority, top pages, ranking keywords, and more.

        Analyze domain
        Illustration of Domain Analysis
      • MozBar

        Free, Instant SEO Metrics As You Surf

        Using Google Chrome, see top SEO metrics instantly for any website or search result as you browse the web.

        Try MozBar
        Illustration of MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

        Read the Beginner's Guide
      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

        See All SEO Guides
      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

        Visit the Learning Center
      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

        Explore the Catalog
      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

        View All Webinars
      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

        Find SEO Answers
      The Impact of Local Business Reviews
      SEO Industry Report

      The Impact of Local Business Reviews

      Learn more
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

        Grow Your Business
      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

        Read Our Story
      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

        Drive Client Success
      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

        See What's Possible
      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

        Scale Your SEO
      • New Releases

        Get the scoop on the latest and greatest from Moz.

        See What’s New
      Surface actionable competitive intel
      New Feature: Moz Pro

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Mozscape API
      • Mozscape API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Research & Trends
    3. Search Behavior
    4. Blocking links from being crawled

    Blocking links from being crawled

    Search Behavior
    6
    16
    2131
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • uesat
      uesat last edited by

      For my blog, should I block category, "read more" and author links from being crawled by search engine robots?  Is it good for SEO?

      Thanks

      1 Reply Last reply Reply Quote 0
      • evolvingSEO
        evolvingSEO @uesat last edited by

        Hi, sorry... I just meant if you did have a few other spammy links from other sites, those could hurt you. I doubt the woorank ones are hurting, but perhaps the other 3 spammy sites you mentioned could have been.

        1 Reply Last reply Reply Quote 0
        • uesat
          uesat @evolvingSEO last edited by

          Thanks Dan.  Just to be clear, what do you mean by this "It may not be those links, but if there were even a few others, it could potentially hurt the site."  You mean blogroll links in other sites?  I have 82 links from Woorank.  I did not ask them to put me there and I didnt even know who they were until I saw them on webmaster tools.  They're unrelated to my niche but they're not a bad quality site.  Could they hurt my site?

          I did have spammy unrelated links from 3 other websites.  But that's it.  I have a few links from general topic websites and most of my links are from relevant sites.  I'm also on another blogroll from a relevant site, just not as big as the one I mentioned.

          evolvingSEO 1 Reply Last reply Reply Quote 0
          • evolvingSEO
            evolvingSEO @uesat last edited by

            Thanks for the details. It may not be those links, but if there were even a few others, it could potentially hurt the site. I had a client who had done some spammy link building in 2011ish with another agency and just those few links hurt the site. I would try removing or disavowing anything you can find that's spammy, and not helping, and see if they helps anything.

            The types of things to de-optimize on your own site would be more like keyword stuffing in the titles, URLs, headers and internal anchors - not so much blocking pages from being crawled. Blocking crawling it typically only if you want to improve crawl efficiency on really large sites or you just don't want Google to page attention to that content, but it won't fix a Penguin issue if it's on-site.

            -Dan

            uesat 1 Reply Last reply Reply Quote 1
            • uesat
              uesat @ALLee last edited by

              Alicia,

              I was hit by penguin 2.0.  There's no question about it

              1 Reply Last reply Reply Quote 0
              • uesat
                uesat @evolvingSEO last edited by

                Dan,

                The website's topic is very similar to mine.  The link was part of a blogroll of about 15 other sites.  The anchor text was my website name.  For example, if my website were www.cleanbluecars.com, the anchor was Clean Blue Cars.  Maybe Google is considering that to be an exact match commercial anchor text?

                The websites on that blogroll were not part of a link exchange, but some of the websites on that blogroll mention the main website on their blog.  The main website hosts a blog carnival every Monday so people naturally mention it.  The main website was who put me on its blogroll because I normally participate on their weekly blog carnival.  The carnival links are and have always been no-follow.

                evolvingSEO 1 Reply Last reply Reply Quote 0
                • ALLee
                  ALLee @evolvingSEO last edited by

                  agree with Dan. If it's not any of the above and you are sure none of the other links are spam, you might not have been hit with penguin 2.0, but some other penalty.

                  I would check with other tools as well. Use as many tools you have at your disposal and create a comprehensive list of backlinks.

                  uesat 1 Reply Last reply Reply Quote 0
                  • evolvingSEO
                    evolvingSEO @uesat last edited by

                    It depends. If the sitewide links were;

                    • off topic from the site it was on, not as relevant
                    • among a giant list of other "blogroll" links
                    • on a site that may have a penalty against it
                    • exact match commercial anchor text <--big one
                    • if it's part of a "link exchange"

                    Those issues could hurt you coming from one site. If the links were exact match commercial anchors I'd see about changing them to brand or domain name anchors. If the site linking to you is a random topic, or you're among a big giant list of links, or part of a link exchange get the links removed or disavow.

                    -Dan

                    ALLee uesat 2 Replies Last reply Reply Quote 0
                    • uesat
                      uesat @evolvingSEO last edited by

                      I used Google Webmaster tools and I used Site Explorer to try and find the spam.  From Webmaster tools was where I found that 50% of all my backlinks came from just 1 site.  But this also happens to be the strongest relevant site I could get backlinks from.  Is this something that Penguin penalizes you for?  The fact that most links are coming from just 1 website?

                      evolvingSEO 1 Reply Last reply Reply Quote 0
                      • evolvingSEO
                        evolvingSEO @David_ODonnell last edited by

                        I agree with David here. Someone made a custom category plugin for WordPress which will do what David is suggesting. I would not block anything, but rather noindex what you don't want in the index, which is typically for me tags, subpages, author and date archives.

                        -Dan

                        1 Reply Last reply Reply Quote 1
                        • evolvingSEO
                          evolvingSEO @uesat last edited by

                          Penguin 2.0 is definitely from webspam and backlinks. I suggest using Google Webmaster Tools as your backlink dataset. They provide links that are more likely to be causing problems. It's extremely unlikely the internal links are hurting you if it is Penguin 2.0.

                          uesat 1 Reply Last reply Reply Quote 1
                          • Carla_Dawson
                            Carla_Dawson last edited by

                            I agree with Alice I would not block them. If the content is unique, they can't hurt you. Category sections show up an SERP's all the time and will increase your CTR (click thru rate)

                            1 Reply Last reply Reply Quote 1
                            • uesat
                              uesat @ALLee last edited by

                              I'm concerned because my website lost 50% of traffic from Penguin 2 and I'm trying to find out why.  My website only has about 150 posts.  According to the Moz Campaign Link Analysys, I have 168 followed external links and more than 11,000 internal links, which I find to be odd.  I thought that may be hurting my rankings

                              I've read a bunch of articles on Penguin about spam and my website had less than 1% of those types of links.  Most of my links are from relevant websites.  The only link that I think might have hurt me is from a PR4 relevant website that had me on their blogroll.  Over 50% of the links to my site came from just that one website because I was on their blogroll.  But this still doesn't make sense to me because that link was 100% natural.

                              evolvingSEO 1 Reply Last reply Reply Quote 0
                              • MoosaHemani
                                MoosaHemani Banned last edited by

                                Category, you can block it, if you see they are creating duplicate content to the website.

                                Read More: Why block this? I mean it is link to your post and this is very much ok so I personally will not recommend you this!

                                Author Link: Hello No, Infect making it more interactive can allow you inspire new readership!

                                1 Reply Last reply Reply Quote 1
                                • David_ODonnell
                                  David_ODonnell last edited by

                                  If possible add unique content to your category pages, and use unique post extracts. Then no need to block.

                                  Don't block "read more" as these link to full posts.

                                  evolvingSEO 1 Reply Last reply Reply Quote 2
                                  • ALLee
                                    ALLee last edited by

                                    Hi,

                                    I wouldn't block these pages from being crawled by search engines. Category pages are great for making sure more link juice flows to your deeper blog pages and making sure they get indexed. I believe author links give authorrank to the corresponding blog post too. I'm not sure about what you mean by 'read more' links. May I ask why you are concerned that these pages hurt SEO?

                                    uesat 1 Reply Last reply Reply Quote 1
                                    • 1 / 1
                                    • First post
                                      Last post

                                    Got a burning SEO question?

                                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                    Start my free trial


                                    Browse Questions

                                    Explore more categories

                                    • Moz Tools

                                      Chat with the community about the Moz tools.

                                    • SEO Tactics

                                      Discuss the SEO process with fellow marketers

                                    • Community

                                      Discuss industry events, jobs, and news!

                                    • Digital Marketing

                                      Chat about tactics outside of SEO

                                    • Research & Trends

                                      Dive into research and trends in the search industry.

                                    • Support

                                      Connect on product support and feature requests.

                                    • See all categories

                                    Related Questions

                                    • DannyHoodless

                                      Help Me With These Horrible Crawl Errors

                                      I am currently using Opencart to run a fairly popular ecommerce website. I have done quite well in terms of rankings so far. However, the Opencart platform only goes so far. There are endless amounts of crawl errors, from duplicate content down to missing meta tags. There is no easy way to add canonical tags. The pages that are being listed in the crawl errors seem to be mainly search URL's that i have no idea how Moz found them. An example of the page is: .....com/index.php?route=product/search&tag=home+office+flooring Which has ten duplicate pages all missing meta tags. I have used Google webmaster tools and htaccess to deny access to these pages with the slug "route=product/search" but it doesn't seem to work. So my question to you guys is as follows: Is it worth trying to fix these errors and do they have an effect on SEO? If so, how do I prevent these errors that seem to grow at every crawl? Cheers, Danny

                                      Search Behavior | | DannyHoodless
                                      0
                                    • WilliamKammer

                                      What should go in a "Link Juice" cocktail?

                                      The office just got some of those fancy Google Partner bottles, and I want to fill them up with a tasty cocktail that is link juice themed. Any ideas on how to tie Google and/or SEO into a fancy cocktail?

                                      Search Behavior | | WilliamKammer
                                      1
                                    • xoffie

                                      Personalised Geo-targeted results - How does Google pass link juice?

                                      Hello, Many websites now serve specific home page offers based on the location of the customer, my question is, how does link juice flow around a site when the links (this case from the homepage) are served up based on a visitors location? Internal links from your homepage are valuable for ranking that product well in the SERPs so how does Google deal with this? So, for example, a car hire website based in the UK. If you arrive on the care hire website sat in Manchester (Northern UK city), on the homepage the website serves offers of car hire deals in Manchester, Leeds, London and international destinations. If you arrived on this website from London (Southern UK City), you would not see the Manchester link at all but London, and other cities in the South. In this case, when Google crawls the car hire website, it will see internal links but a)which version and b) is there any way of sharing this link value around? Basically, we want to understand if Manchester in this case will get the benefit of an internal homepage link from Google even though we only show Manchester to people FROM Manchester, OR, do Google only give juice based on one version of the website, a generic UK version? Or to put it another way, is there any way of cashing in on both geo-targetting the customer based on their location AND getting link juice from those geo-specific home page links? Perhaps there is some code or way of telling Google that people from Manchester (a certain % of our visitors) will see a homepage internal link for Manchester that will pass some small % link value?

                                      Search Behavior | | xoffie
                                      0
                                    • mbulox

                                      What to expect after removing external and sitewide links?

                                      Hello, I've been working on removing some sitewide links and some directory links and I would like to know what to expect to see on my rankings based on your own experience if you've dealt with this before. Cheers 🙂

                                      Search Behavior | | mbulox
                                      0
                                    • CruiseControl

                                      Is putting/removing Adsense ads on the site affects crawling for seo?

                                      Because I noticed when we totally remove Adsense ads on our site, the pages crawled per day on the google webmastertools suddenly dropped into noticable amount and we tested again to turn it on for a singe day, it jumped up again. So if Adsense affects crawling then, if we have adsense ad on all pages of the site the more chances it gets crawled by the bot? WS76R8w

                                      Search Behavior | | CruiseControl
                                      0
                                    • funktiongolf

                                      Is it my back link profile that is affecting my rankings ?

                                      my website www.funktiongolf.co.uk had been ranking well until January and i'm nearly sure its due to low quality links and poor back-link profile i.e over-optimized money keywords can someone guide me? thanks for anyone that replies. I really do appreciate the
                                       help.

                                      Search Behavior | | funktiongolf
                                      0
                                    • ShaMenz

                                      Gathering Data on Link Removal and Disavow Tool

                                      Hi, There is so much opinion out there about Link Removal and Google's Disavow Tool, but it's really hard to get hold of reliable data about what people are actually doing in this area. Through my involvement in rmoov, I've become really interested in participation and cleanup rates when it comes to Link Removal campaigns. So, I've put together a quick survey aimed at getting a real indication of what is happening out there. The survey is here The recent introduction of Google's Disavow Tool has opened up another area I'd  really like to understand more, so there is a question on this included. I'd love to pull together a blog post with a decent snapshot of participation, success or failure for those trying to remove links and hope Moz Members might help out by answering this quick survey. Sha P.S. I'll update this post once results are available for those that are interested.

                                      Search Behavior | | ShaMenz
                                      1
                                    • ChrisHolgate

                                      Google Webmaster Tools notice of detected unnatural links to our website

                                      Hi, We have been using SEOmoz for a good few months now and have found it incredibly useful.  Unfortunately however I think we may have slipped up a little bit as we have just received the following message in our Google Webmaster Tools: Google Webmaster Tools notice of detected unnatural links to http://www.refreshcartridges.co.uk/ I have a funny feeling I might know what the problem is but I'd like a confirmation before I potentially go off on a wild goose chase. We also own the domain name www.computerarticles.co.uk and recently started introducing the author at the bottom of the post along with both a link to the website homepage and a link to a deep, popular page on our website.  For example, check out http://www.computerarticles.co.uk/twitter-time-saving-tools/ We genuinely didn't realise this was a bad thing and thought it would just provide Google with a way of deep linking straight in to popular areas of our site.  I would really appreciate advice as to whether you think it might be these links that are causing the problem or whether there is something else that could be causing the problem. In the event it is these links that are causing the issue would you recommend removing the entire 'about the author' page (as we have published around 500 pages on that site) and simply put a link in the blog roll or simplify it to just link to the homepage. I appreciate in advance any help you could give. Regards Chris

                                      Search Behavior | | ChrisHolgate
                                      0
                                    Moz logo
                                    • Contact
                                    • Community
                                    • Free Trial
                                    • Terms & Privacy
                                    • Jobs
                                    • Help
                                    • News & Press
                                    • Mozcon
                                    © 2021 - 2023 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

                                    Looks like your connection to Moz was lost, please wait while we try to reconnect.