A guide to visual search

Sam Colebrook

Content Strategy Director

Google now understands human language with 95% accuracy. Its ability to decipher the intent behind our words has led to it becoming market leader of the search world. Job done? Not quite.

With reports that Google image searches now account for almost 30% of all US queries, the industry giant is looking to capitalise on this by turning its attention to non-text searches.

But it’s not just Google that’s branching out from word-based searches. Bing is sitting up and taking notice too, having invested heavily in visual search with new features such as its deep image tool - more on this later. And with over 175m monthly users, it makes sense that Pinterest is looking to capitalise on image-based search as well. As CEO Ben Silbermann explained, “a lot of the future of search is going to be about pictures instead of keywords”.

Unsurprisingly, image search is highly-relevant to sectors like fashion and beauty, where visuals sell. But these aren’t the only verticals that stand to benefit; homeware, food, travel and retail also have the potential to harness the power of image-search to strengthen their brands. Here we’ll explore the different types of visual searches, their benefits and, most importantly, how you can optimise for them. But first, what do we mean by visual search and image search?

Image search vs visual search

Put simply, image search is the act of retrieving images from a search engine based on a user’s input. As computers become better at understanding visual information, these inputs have become more varied. For example, developments in technology mean search engines can now interpret colour, texture and shapes within an image in a way that’s not been done before.

And it’s not just searches with image-based results that are on the rise. We’re now seeing the emergence of searches using image inputs, too. In other words, users can now search with images, not just for them.

This has led to the coining of the umbrella term ‘visual search’ to refer to all visual data inputs and retrieval – including new reverse image search technology and traditional keyword-in/image-out model.

Types of visual search

There are many different types of visual search – here’s a breakdown of what each means:

1. Meta search

In the early days, the only information search engines could see was metadata – keyword-rich elements within the code. So searches for ‘red trench coat’ would return images that included this keyword in the file name or alt text tag. These meta elements are still important, but developments in image recognition technology mean search engines can now understand the images themselves, too. 

2. Reverse image search

Pioneered by Google, the image-in, similar images out (or reverse image search) model initially became popular with brands trying to identify uncited uses of their company images. But marketers are now also adopting the technology in consumer-facing ways – more on these later. 

3. Related search

Pinterest leads the way in related image searches, which follow a similar logic to the Google Suggest function in that they show users queries and common prepositions relating to what they’re looking for.

4. Filtered search

Another Pinterest initiative. Building on related search, this functionality suggests filters – like colour and size – to help users focus their search. The success of this has led to most search engines adopting it.

5. Augmented reality search

Thanks to smartphone camera integration, users can now search using visual inputs from the palm of their hand. Google and Pinterest’s Lens apps allow users to capture objects in real-life and return related images.

6. Image repository

Pinterest was the first to realise that users may want to save their image search results, especially when looking for inspiration. This lead to the creation of Pinterest boards and has paved the way for more repositories – i.e. places to save/collect images.

7. Deep image search

Pioneered by Bing, deep image search lets you select objects within an image using a crop tool and find related images.

The key thing to note here is that image search is no longer limited to Google, (or even Bing and Yahoo). Just as YouTube started as a video-sharing site and evolved into a video search engine, we’re seeing a variety of platforms, like Pinterest, gaining image ‘search engine’ status.

What are the benefits of visual search?

Needless to say, these developments are great for users. But visual search also provides real opportunities for marketers, too. Here are some of them:

  • We shop with our eyes. 93% of consumers said they consider visual factors above all else when making a purchase, and image-based searches mean they can get to these quicker and more easily than ever before.   

  • More social-generated sales. 93% of active pinners said they use Pinterest to plan purchases, and 87% said they’ve bought something because of Pinterest. Visual search features, like the Pinterest Lens, make finding and shopping based on a single image easier – so there’s more opportunity for ecommerce brands to drive sales from the platform.

  • More buying-intent behind search. Shoppers searching with images tend to be looking for a specific item or product. With visual search, they’ll find your product faster and when they do, they’ll be less likely to abandon basket.

  • More opportunity for cross-sell. If a user searches for ‘gold bag’ and finds your product, they might be tempted to buy other items featured in the same image, like additional accessories or clothes.

  • Easier path to purchase. Visual search saves users time and makes finding (and buying) products easier. This is particularly true for brands who incorporate visual search into their apps, like Amazon.

  • Low competition. Relatively speaking, visual search is in its infancy and as such, rarely features in marketing strategies. This presents a big opportunity for brands who are willing to invest in featuring prominently in visual search results (VSERPs).

How to research consumer phrases and keywords for visual search

While identifying keywords for image searches isn’t as straightforward as it is for text-based queries – at least until Google introduce ads into image SERPs and AdWords – there are a few tricks you can use to find the right terms to use:

 1. Image stack

Google often includes an image stack in a keyword’s organic search results if a lot of users re-enter the keyword in image search immediately after – an indication they were looking for images to begin with.

So to see whether a keyword has image-based demand, search for the term and see if an image stack appears in the organic search results.

2. Autocomplete

In the absence of any real image keyword data, autocomplete is the next best thing.

Google has confirmed that search ‘popularity’ and ‘freshness’ are key factors in its autocomplete algorithm, so if it suggests a keyword in image search, chances are it has demand.

Type your root keyword into the image search bar to see which variations Google suggests. Be careful, though – your previous searches have a heavy influence on autocomplete results, so make sure you’re in Incognito Mode.

How to optimise images for visual search

Keyword research sorted? Here are some simple tricks you can action right now to harness the power of image search, help search engines understand your images and push them up the search results:

1. Embed images in HTML

First, make sure your images are embedded in HTML and not in CSS using the background-image property, as this method isn’t crawlable.


                    body {

                        background-image: url("filename.jpg");



                    <img src=“filename.jpg”>

2. Optimise your metadata

We know this is often where a search engine will begin to extract meaning from an image, so it’s a sensible place to start optimisation. Make sure you include your target keywords in the image’s file name. For example, image-01.jpeg doesn’t give a search engine any information, whereas red-trench coat.jpeg is far clearer.

Example:      <img src=“red-trench-coat.jpg”>

3. Next, make the most of your image alt tags.

Originally created to help accessibility software for the visually impaired, it’s common SEO knowledge that search engines also use them to understand images. Optimise these tags with your target keywords, to provide them with additional relevancy signals.

Example:      <img src=“red-trench-coat.jpg” alt=“Red trench coat”>

Captions are the descriptions found beneath an image and are another great way to provide users and search engines with context.

Example:      <figure>

                     <img src="red-trench-coat.jpg" alt="Red " />

                     <figcaption> Red Trench coat</figcaption>


You can also use your metadata to distinguish between images showing multiple angles of the same object.

Example:      <img src=“red-trench-coat-front.jpg” alt=“Red trench coat front”>

                     <img src=“red-trench-coat-back.jpg” alt=“Red trench coat back”>

4. Use structured data

Another way to help search engines understand your images is to use structured data. Include keyword-rich microdata – such as the search engine approved schema markup – to give your images an additional relevancy boost. This one is especially important for product images.

5. Create an image XML sitemap

Search engines love sitemaps. It’s always risky to rely on crawlers discovering your pages through internal linking alone, and the same goes for images.

Creating an XML sitemap for your site’s images will help search engines crawl images that may otherwise be inaccessible – such as images blocked by JavaScript.

It also offers another opportunity to give search engines more context to your images, such as their type, subject matter, title, caption, and location. Crucially, you can also attribute a priority score to each image to help search engines understand which are the most important.

6. Think about image contrast

As we’ve seen, search engines are getting better at understanding the shapes, colours, and objects within an image, but it still makes sense to do what you can to help them out.

Any objects you want to be easily identifiable – like products, for example – should be clearly visible within an image. To give search engines the best chance of seeing your object/product, it should be against a plain contrasting background. So set your light objects against dark backgrounds and dark objects against light backgrounds.

Visual search innovators: which brands are raising the bar?

Here’s who’s making waves in the world of visual search right now:

  • ASOS – the ecommerce fashion giant’s Style Match tool is essentially a reverse image search for clothes. Now available on iOS and Android, the in-app feature lets users take or upload a photo and find similar products for sale.

  • eBay – rather than rely on its sellers adding relevant tags and descriptions to products, the online marketplace has launched Image Search, will allows shoppers to take or upload a photo from their camera roll and use it to search the site’s listings. Another new feature – Find It On eBay – goes one step further and lets users start a search with any image they find online (including from social media).

  • Amazon – the online retailer first introduced visual search into its iOS app back in 2014. Again, users can search for products with their smartphone camera. This feature has captured the attention of ‘showrooming’ shoppers, who visit a physical store, then see how much they can save by buying the product elsewhere.

What does the future hold for visual search?

The key to the success of visual search is the ability of search engines to attribute context to image-based content. Today, this is achieved by understanding shape, colour, texture, and data labels, but as understanding of visual inputs and outputs improves, so will the search experience for users.

A key driver in this innovation is the ImageNet large-scale visual recognition challenge (ILSVRC). Now in its eighth year, it’s the largest academic challenge in computer vision, putting leading image recognition algorithms to the test. One interesting criteria that it assess is an algorithm’s ability to describe a complex scene by accurately locating and identifying many objects in it.

This is the next big challenge for visual search: to not only recognise objects within visual content, but to fully understand context through its composition.

The search engine that achieves this will be able connect us with a wealth of information based on our surroundings. It will be able to understand the interior design theme of your living room and recommend complimentary products, or suggest recipes based on an image of ingredients, for example.

Beyond commercial applications, these algorithms will become more impactful once they begin to establish visual blueprints of the world around us. They will be able to compare visual inputs with these and identify subtle differences – like the faulty wiring in an image of a car engine, or a hairline fracture in an X-ray.

Search engines have already made a huge impact on the way we live, through their understanding of language, but that’s just the beginning. We’re on the verge of a visual revolution that will change the way we interact with the world for good.


To find out more about visual search and how it can help your business, get in touch today.

Continue reading


Are you ready to make a digital step-change?

We believe that moving too slowly in digital is the biggest risk your business faces. If you are ready to move faster in digital, we are here to help.

Get In touch