Are 404 Pages Always Bad for SEO?

There are some very different schools of thought out there regarding 404 error code pages. Some SEOs recommend:

• Never allowing them - 301'ing every error page back to the home page or an internal category level page to preserve the maximum amount of link juice (in case someone links to a broken URL)
• Letting any erroneous/mistyped URL 404.
• Something in between - 301 some kinds of 404 pages and not others.

I'm generally in this last group. I think there are times when it pays dividends to let a URL 404, both for accessibility and search engine reasons. I also don't think it's intuitive or semantically accurate to 301 every 404 page on the site - it certainly pays to build great custom 404s (good piece with examples on that here), but to simply have your homepage appear when a URL is mistyped or a link breaks doesn't send the right message to users or search engines.

When faced with 404s, my thinking is that unless the page:

A) Receives important links to it from external sources (Google Webmaster Tools is great for this)
B) Is receiving a substantive quantity of visitor traffic
and/or C) Has an obvious URL that visitors/links intended to reach

It's OK to let it 404.

Recently, though, Lindsay and I were faced with a tough call on a consulting project. The client has a site that receives a ton of search queries, many of which map to their category and subcategory level pages (which are more landing pages than search query pages, but also serve to address the search keywords). The client also has a number of search pages that have no content (either because they're for mis-typed, nonsense or mis-spelled searches or because they simply don't have content for those terms). Some of these pages earn links, some get a moderate amount of traffic and up until recently, they've essentially existed as error pages that resolve with a 200 code.

What to do?

Our conundrum contained a few critical elements. We don't want the search engines wasting bandwidth crawling and indexing junk pages (especially since the site is monstrous and needs that crawl/index power to flow to the right sections). We also don't want users to have a bad experience and while the error pages effectively communicate the right message (there's no results for this query), semantically the pages should really 404. Finally, of course, we don't want to waste any of that precious link juice that's flowing to some of them.

The solution turned out to be a compromise - we'd 404 the pages, but keep track of those that earned links and any substantive level of traffic and try to build better experiences for those pages (sometimes a 301 to a sub-category page, sometimes to a results listing and sometimes we'll actually add content to those pages and make them resolve). We hope that this lets us have our cake and eat it, too.

We'd love to hear your thoughts around 404s and SEO in general, as well as on this specific scenario (and others like it). 1000s of SEOs are smarter than 2 :-)