SEO Has a Younger Sibling: It’s On-Site Search, and It Deserves Attention

On-site search, also known as internal search, is a critical yet undervalued and underrepresented tactic in the search industry. In 2014,eConsultancy released a report that showed that 15% of companies dedicate resources to optimizing the on-site search experience, 42% fold on-site search into other online measurement responsibilities, and 42% of companies ignore on-site search. This report was released more than two years ago and after some extensive Googling, some Duck Duck Go-ing, and some slight Bing-ing, I couldn't find any reputable site releasing similar, more recent, reports.

Wait a minute... Nearly 84% of companies don't actively optimize or measure their on-site search?

Consider the following benefits of active on-site search optimization:

• On average, 30% of visitors will perform an on-site search
• When comparing revenue gained from people who performed an on-site search vs. people who did not perform an on-site search, the people who performed an on-site search generated more revenue than those who did not.
• Performing an on-site search is a strong behavioral predictor of intent to convert:
  • People who perform a site-search are twice as likely to convert.
  • People who perform an on-site search are more likely to return to the site with an intent to purchase.
  • In the research, companies had an average overall ecommerce conversion rate of 2.77%. However, the conversion rate nearly doubled to 4.63% from people who used on-site search and found what they were looking for.

Data sources: eConsultancy, ConversionXL

It's honestly surprising that, in an industry that focuses so intently on data-driven marketing and conversion optimization, we don't put more effort into on-site search, harnessing users who are engaged and interested in our content and serves. My point is that for a tactic that can become an incredible force multiplier, there's so little information on it.

Okay, okay, you're kinda starting to rant. What are my first steps to understanding on-site search?

The first thing you need to understand before starting to optimize your on-site search is what data is being collected by analytics from your on-site search platform. For this exercise, it's irrelevant whether you're using Solr, the Google Search Appliance, or a baked-in search platform that came with your content management system.

Ultimately, the goal is to understand “Search Quality” by quantifying it:

When discussing quantifying on-site search, I tend to use the metaphor of resolution. I started playing video games as a kid and that’s continued to today. So, seeing little blocky characters evolve into something incredibly complex is a part of my experience. Clever designers were able to create experiences using only 8-bit resolution. As hardware grew more capable, 16-, 32-, and 64-bit consoles became more mainstream. Each generation of hardware supported greater resolution and the amount of visual information become more complex, more immersive, and more descriptive.

This is how I view measuring and quantifying on-site search. Start by understanding the basic metrics. Start with 8-bit resolution. This is where you’ll start seeing patterns emerge and behaviors evolve. As your experience grows, add more complexity to your metrics, look at the interplays between different types of content, look at how your on-site search results change as your content changes, look at the time it takes for people to find what they’re looking for. It won’t take too long to where you’re looking at your on-site search data at 64-bit resolution.

Image source

As your measurement resolution improves, you can understand the story deeper and with more clarity, and can use that understanding to impact greater change.

Search behavior

Search behavior is the measure of the quantitative actions a user takes on your website. This includes keywords, clicks, and calculated measurements like CTR. Measuring search behavior allows the search professional to see if any changes to the way on-site search works results in a net benefit to on-site search.

How to quanitify search behavior

The simplest metric to use to examine user behavior is CTR.

First of all, there’s an implicit assumption that clicks signal that the user has found something useful and the click is a signal of that found value. Not all clicks are indicators of search success, and there are users that click multiple links on a SERP as a way to refine their search. For now, in our 8-bit world, we need to understand that assumption, yet it’s important to measure overall CTR as a signal of search quality.

When you first start looking at SERP CTR, it’s likely yours will be low. When I first started measuring on-site search, we used the Google Search Appliance and never surpassed a 24% SERP CTR. The first question I was asked was “What’s a good on-site search CTR?” I was able to answer this by examining our different content types (blogs, videos, articles, etc). What I found through customer surveys was that a higher CTR generally correlated with a higher satisfaction with the search experience. The surprising part? Fewer people than expected had a “neutral” experience. This showed me that when it comes to on-site search, people are happy (and remember) a good search experience and people are frustrated (and remember) a poor search experience. The room for a neutral feeling has a very small window. If we had a greater than 65% CTR on our search results, we would be able to show a positive search experience and a greater chance for conversion, assuming that a good search experience meant that the user was able to find what they were looking for.

Higher-resolution metrics on CTR can include different elements of the SERP:

• Filter Section CTR
• Promoted Section CTR
• Category Result CTR
• Sponsored Result CTR
• Natural Search CTR

SERP Source

Understanding how all of these SERP elements influence clicks can identify opportunities for CTR optimization with either better results or a better UI. This is probably the most superficial metric to measure behavior — I haven’t even mentioned keyword refinements, revenue per search, revenue per keyword, or conversions gained through search. All are very important to measure the overall success of on-site search, but probably deserve their own article.

Search findability

Findability is the measurement of how a user finds the content they’re looking for. Findability is directly tied to the content’s rank for particular keywords. When the on-site SERP shows the results, knowing which piece of content the user clicks on, combined with the average rank of that content, can reveal if good content is being clicked, thus making it “findable.”

How to quantify findability

I take a fairly simple approach to measuring findability. I take a look at it from a content-level perspective. For each piece of content, I measure the SERP impressions, the amount of times a piece of content shows up in the top ten results. I look at the average rank, then measure the CTR of the piece of content. Using this information I can identify pieces of content that show up a lot in the SERP and have a low CTR. Looking at the average rank tells me if, on average, the content is seen. If it has an average rank of less than 10 (such as “content 2” in the table below) there’s a reason why it shows up frequently but rarely gets clicked on, despite being in a good ranking position. Conversely, when I look at “content 5,” I see that it has high impressions, low clicks, high rank, and a low CTR. I then take a look at this piece of content to see if I can make it rank a bit higher in our on-site search. If I see the CTR improve, then I’ve increased its findability as well as its value.

A higher-resolution version of findability would include the number of keywords that trigger an impression and a visibility scoring model that would describe how “visible” a piece of content is. Factoring in clicks and CTR into the visibility model would give you an overall findability score that could quickly identify valuable content that is not found and non-valuable content that is frequently found.

Result set quality

Determining result set quality is answering the question, “How good are the results for given keywords?” For example, if a common search term on your site is “shoes” and your top result is the page where the user can filter down to the right kind of shoe, and that page is in the pathway to a good conversion rate, then that’s a good result for the keyword. However, if the results for “shoes” are more general — “athletic shoes,” “red shoes,” “ballet shoes” — and the results for this keyword have a poor conversion rate, then the result set quality can be improved.

I use a fairly simple method to make sure that my results are performing according to the best interests of the company. I look at the number of times a SERP appeared and then measure the number of times that SERP resulted in a conversion or a consumption (a user reading a page that doesn’t have a call-to-action on it). Then, looking at the successful events that occur after a SERP occurs, I have a metric that I can track, trend, and show the value of on-site search.

Which metrics are core to measuring on-site search?

At the most basic level, these are the metrics we'll need to start measuring on-site search. Many of the nuances we’re measuring in SEO applies to onsite-search as well. While most of these metrics we likely already know, I want to describe them in the context of on-site search.

• Query
  • A query is a package of information that is sent to the search platform. While "query" and "keyword" can generally be interchanged, keywords and keyword phrases have meaning because they're a part of language. A query is a packet of information because it is the "thing" that is sent to the search engine to be parsed, recognized, and compared against a body of documents.
• Keyword
  • A keyword or keyword phrase is the representation of user intent, expressed in words. A query is an abstraction of the keyword phrase. Keywords and phrases are essentially bags of words to be compared against other, larger bags of words.
• Instance/Impression
  • An instance (also impression) is the number of times or the frequency that a keyword or keyword phrase is entered into the search bar and sent to the search engine.
• Click
  • In the context of on-site search, a click occurs when a piece of content returned by the search engine is clicked.
• CTR
  • The click-through rate of on-site search is measured by clicks/instances. CTR is a calculated metric, but critical to understand the success of individual keywords as well as the overall health of the search engine.
• Rank
  • Rank is a measurement of the position of content in the SERP. Rank can be expressed in two different ways. Rank for keyword is the number where a piece of content sits for a given keyword. Average Rank measures the average of where a particular piece of content ranks over a set of keywords.
  • Examples:
    • For keyword "x," document "y" is at rank 4
    • For document "y," the average rank is 2.3 for keywords "a, b, c, d, & e"
• Conversion
  • For content found through on-site search, a conversion is the number of times the user completed an action such as a purchase, lead gen, or consumption.
  • Conversion is a quantitative metric that is binary. The user either converted or they didn't.
• Consumption
  • Consumption is a qualitative measurement for content that doesn't have a clear conversion point. For sites or pages that are information-oriented (rather than conversion-oriented), it's generally described as "it is likely that the user consumed this document."
  • Examples:
    • A user scrolled to the bottom of the content and spent more than "x" time on page
    • A user shared the document via social media
    • A user commented on the document
    • A user viewed this page after search and before conversion.
• Search with No Results/SNR
  • A search with no results is a search that didn't match any documents, resolving in 0 results on the SERP.

• Unicorn Searches
  • A keyword query that has an instance of 1 for a given time range.

That's a long list of ways to measure on-site search. How can I apply these to understanding it?

While measuring the "On-Site Search Pillars" of search behavior, search findability, & result set quality is the ultimate goal to truly understanding on-site search, the most basic pillar and arguably the most important one is search behavior. Understanding customer behavior as they interact with on-site search is likely the very first thing we'd want to start measuring to get actionable information.

Here are five ways to start understanding and optimizing your on-site search.

1: Keymatch critical keywords

Keymatching is a system that’s available in most enterprise search platforms that generally puts up to three recommended results at the very top of the SERP, right above the natural results. Keymatching is non-algorithmic and often includes a way (like Google AdWords) to match keywords to content. Usually there’s a visual element to separate it from the natural results, such as “recommended results.” Think of keymatching as Adwords for your own site. You can promote new products or increase the visibility of highly converting products. See the keymatch for “heartbleed” on Red Hat’s Customer Portal SERP. Right below the recommended results (keymatch) are the natural results:

For example, when the heartbleed vulnerability was identified, I created a keymatch for specific terms that, when searched, would trigger our keymatch system to display our Heartbleed Detector app. Based upon our search volume, we knew that people would want to check to see if they were vulnerable first, then look for a solution. We used our keymatch system to make the top of the funnel very clear.

Another strategy that’s more focused on ecommerce is to keymatch terms that match to promotions or sales you’re having. For a site that’s selling apparel where all their socks are on sale, keymatching will give the user the information that all socks are discounted.

2: Get a good overview of the health of your on-site search.

To do this, look at the overall on-site search CTR. Don't worry about what it is at the moment; this is about understanding how well things are performing in the real world. Use this metric to get a strategic view of how search is performing. If you see that your on-site search CTR is lower than you want it to be, look at the top keywords searched, then repeat the searches on your on-site search.

It's very likely that your top searches are "high volume but low intent" keywords. You might be looking at top-of-funnel keywords like "shoes," "insurance plans," or "sci-fi movies." Is there a good landing page on your site so that the user can funnel down to what they're looking for? Generally, these low-intent keywords are from people who may not know what they're looking for, but will know it when they see it.

Use either a keymatch system or a boosting system to push top-of-funnel pages to the top of the search results. If you don't have these types of pages, you'll know that you're going to need them. What’s a boosting system, you ask? Boosting is a term that adds relevancy to a piece of content or even content types to rank higher in the natural SERP.

For example, imagine that the term “red shoes” resulted in on-site natural results for different types of red shoes. However, data shows that people looking for “red shoes” convert at a much higher rate for the imaginary “Red” brand that has a shoe line. One way we can improve the natural search results for this scenario is to boost the brand “Red” for queries including “red shoes.” While you can force content to the very top, keep in mind that boosting is a broadsword solution that may cause collateral search damage and unintended consequences to your search results. However, if the data supports such a broad change, using a boost in your on-site search platform is a good tool to use smartly.

3: Look at individual pieces of content in the context of on-site search performance.

From the perspective of an individual piece of content, it will likely show up for a variety of keywords. Looking at the data from on-site search at the content level will reveal what keywords trigger that piece of content to show up in the top 10 results.

The first step is to look at how much traffic that piece of content gets from on-site search. Secondly, look at the clicks and CTR. Notice that in the following example, most keywords deliver a decent CTR except one. This piece of content delivers no clicks, no revenue, and ultimately no value to the user. This is a powerful signal to the business to look at why this piece of content ranks for that keyword. Sometimes, on-site search can deliver poor or irrelevant results. The solution here is to either fix the content so it delivers value or use your ability to de-boost that piece of content for that keyword.

Clearly, this content works with other keywords, so the problem may not be the content, but the fact that it ranks for a keyword that has no relevance to the content. To understand on-site search’s weight in generating revenue, compare the revenue and conversion rates with other marketing channels such as SEO, email, direct, or social.

4: Use searches with no results to enhance content

If you have people searching for things that deliver no results and the frequency is significant, this is a great way to give your content team information on what to write about next. For example, the site ThinkGeek had an April Fool's joke about a Tauntaun sleeping bag. I talked to someone at ThinkGeek and they mentioned that even though the product wasn't real, they started seeing several hundreds of searches for that product on their site. What was surprising to them was that the searches not only increased, but continued well after the April Fool’s joke. As a result, they actually made that sleeping bag and it became a customer hit. While it wasn't only found via on-site search, the consistency of mentions in social media, on-site search, and customer service convinced them to actually make it.

5: When it comes to UI/UX, test snippet UI, test SERP UI, test everything

Don't accept the out-of-the-box search UI. Test putting images or embedded videos into the snippet. Test putting linkable "top level" breadcrumb navigation into the snippet. Create good A/B tests to see what will increase clicks and conversions. Talk to internal stakeholders, talk to your users, and find out how they use search. Once you know what your critical business goals are around on-site search, testing the UI can significantly increase CTR and conversions.

Great, thanks… Now I want to do all the things. Where do I start?

I know that I’ve covered a lot of topics that really require a lot of nuance, a lot of industry-specific idiosyncrasies, and other considerations to on-site search. Because there’s not a lot of current information or even communities that surround on the non-development side of on-site search, I can understand that there might be questions about how do I start using this information.

My advice: start by understanding what your search platform is. Do you use IBM Websphere for your ecommerce platform? Do you use Apache Solr? Google Search Appliance? Elasticsearch? Look deeply to what the features of your ecommerce or search platform has. If you haven’t already, hook up your analytics platform to your on-site search and start collecting data. I’ve found that many times, when companies have set up their analytics, they don’t apply basic metrics to their on-site search. Look at the data you have and start setting up experiments. Try to make a piece of content rank higher in your on-site search. Experiment with meta-keyword tags (yes… Google and other search engines ignore it, but generally, on-site search engines still love meta keywords). Start small experiments to prove that you can increase revenue by making small changes. Once you understand your search platform, once you’re seeing small changes you’ve made that have positive results, dig deeper. Share your experiences (I’d love to hear them) and find people to collaborate with. I cannot claim that I know everything, but what I hope to have done is created a small group of enthusiastic people who can work together to create even more information on this oft-ignored, but oh-so-useful subject.

Some acknowledgements:

Red Hat isn’t just about open source technology, it’s about being an open organization. With that open and collaborative mindset, I want to share my thanks and deep appreciation to some people who helped me with this article. Linda Caplinger, Jairus Mitchell, & Thomas Stokes are brilliant people who provided feedback, suggestions, and welcome critiques.