MindReadr
Questions and answers are divided into two basic sections:
"Help me use MindReadr!" which explains how the buttons work and
"What’s it all about?" which answers some of the behind-the-scenes questions.
? …the 
?
Click the “Reset” button along the top bar, middle of page.
Or, if you really like clicking, go to Advanced Version and click every image in the
My Feedback area until that area is empty. This essentially eliminates all the feedback which is like
starting all over.
We start by pseudo randomly displaying all of the images. Each time it is different.
Search refinement is done in two major ways. Like other image collections, you can do it by just selecting images that already have certain keywords associated with them. This demonstration does not yet have a keyword search box. Instead, we have used particular keywords and gotten several thousand images of each from Flickr. When you use the pull-down list at the top left of the page, you can select a subset of All of the images by selecting one of the given keywords.
The most interesting search refinement mechanism, and the one this demo is all about, is using the displayed images to let our image search engine, called MindReader, know which images captivate the imagery you want to see and which ones do not.
In Basic Mode (default), the image re-sorting is immediate upon clicking the or
below an image. Selections are accumulated until you click on the
HINT:
Your results will improve quickest once you have selected at least one positive and two negative examples.
In the Advanced Version, click on the orange label in the upper right corner, the searcher has two ways to provide feedback to the engine. One is similar to the Basic version, click on the + and – icons below the images. The difference is that the re-sorting is not immediate. To submit this feedback to the MindReader engine, click on the Refine Search button.
The second way to refine your search is to use the SuperFeedback box in the lower right-hand corner. After an initial search iteration, these images are specifically selected to provide the richest feedback to the engine relative to refining your search, that is relative to separating the ones it thinks you want from the ones it thinks you don’t want.
These images are pre-selected as “Less Like These” as noted by the red bar below. If none of them are what you want, immediately click the “Go” button. Otherwise, click the bar or icons below the image and it will be selected as positive feedback. Do this for ALL of the images in the SuperFeedback box that are positive examples, then click “Go”.
The best way to reset your current selections, without resetting the whole search, is to select the image again and it will toggle back to neutral. In Advanced mode, you can change earlier iteration selections by clicking on them in the My Feedback buckets.
?
…the ?
In Basic mode, you are immediately submitting one more clue to the search engine about how to re-sort the image pool.
In the Advanced Version, you are staging another clue, but the search engine will not “see” it until you submit the feedback using the Refine Search button.
These bars represent the current label of those images. Green is “More Like This”; Red is “Less like This”. The green bar shows up on subsequent iterations to show you which images you selected. The red bar rarely shows up later because these images are at the end of the search results.
This page is a demonstration of how searching by visual clues can speed up your search or add fun to your browsing. This is the very first time this capability has ever been shown on the Internet interacting with public image repositories such as Flickr, Google, Yahoo! Images, and Ask Images. This version uses slightly over 13,000 images from Flickr.
The Big Deal is that instead of typing more words to refine your image search, you can simply click on more pictures! Watch how quickly the results reflect your idea of a good picture for your topic. Until you actually "Reset" the search, each click is combined with previous clicks to learn your specific concept.
Except for the initial selection of keyword (the first version restricts it to a pull-down list, but it will eventually be replaced by the typical keyword box) and restricting the search to that domain defined by the keyword, keywords are NOT USED AT ALL in the search refinement. The only information used to refine the search is from your "labeling" images as good (+) or bad (-) examples of what you have in mind.
The only reason there is no keyword search box is because this is foremost a demonstration of how visual search can help image search. We have not collected all of the images for all of the keywords yet, but we wanted to get a representative sample into the websphere sooner, rather than later.
In the mean time, you know exactly what keywords we do have. We want you to try it and let us know if you think it would be useful to expand to other keywords.
Obviously we had to start somewhere. We decided to start with a few searches that we knew would show off our visual search functionality. nbsp;Also, another search demonstration just came out ( http://www.xcavator.net/index.html ) and we though, "What the heck. Let’s use the same categories so users can compare."
How many images is Google searching? Well, they say they are searching some trabillion gazillion, but rather the same 981 or so images keep popping up and that’s all you can get for your efforts.
Basically, we’ve downloaded the same images you would have gotten from Flickr, processed them to extract our proprietary signature, and then created this demonstration to show you how much fun it is to find several images of what you want to see without having to do the scrolling, scrolling.
Originally I wanted to have our site show “1-20 images of 2,310,000” as sort of a spoof on Google and the other sites. But, others prevailed and we’re showing the actual number of images searched with each iteration.
At one point, we also thought we might also tell you how fast we are (more chuckles), but suffice it to say it’s well under a second, even a half second, even a quarter of a second. But who cares? The real question is, did you have to wait? If you did, let us hear about it! Although, if you had to wait, I’ll bet you that your wait is on the download, not the search.
It’s hard-coded – Oops – no, not really. For a minute there, I was thinking of a competitor’s demo (http://corbis.ltutech.com/).
It’s practice – No, not really. More like “Trial and Error”. VIMA’s Ph’D-laden researchers have been working since early 2001 to come up with the right feature set to extract from images where the feature set is small enough to be fast (i.e., calculations are related to number of features), yet large enough to be perceptually accurate (i.e., accuracy and measure of similarity are related to number of features…if they are the right features).
I think we’ve got it! The features we extract (trade secret) combined with some public domain machine-learning algorithms, modified with some careful kernel selection (trade secret), and some clever innovations addressing this application (patents) allow us to combine features from several images that the searcher selects – both positive and negative examples – and the system appears to learn what each individual searcher has in mind.
There are other examples of image similarity search on the Internet, but except for Xcavator (http://www.xcavator.net/index.html), these are all single click application only! With these single-click "solutions", if you don’t find what you want on the first click...too bad!
In the real world, how can I accurately and consistently show you similar pictures if you only show me one picture? Beautiful sunset picture. How do you like these? Oh, you don’t want the sun in the picture? How about these? Oh, you want a variety of colors, not just the colors in your original? How about these? Well, your original had silhouetted mountains, I thought that was important. How about these…
Now imagine if you could look at a bunch of pictures, click a few times, and let the computer figure it out. How about stop just imagining it and go back to the search page and do it for real?!?
Pilot error! No, seriously. There are a number of reasons the results can be unsatisfactory to you.
My absolute favorite comes from the fact that if no good, similar images exist in the few thousand or so images we have for each keyword then, it doesn’t matter how many clicks or how many times you keep scrolling down the page, you will not find a similar image BECAUSE IT IS NOT HERE.
If you were to tell us, "I know there are several that are very similar, but I can’t seem to get them to all come up when I start with one of them."e, well, now we can talk…what’s going "wrong". Possible explanations are:
Labeling (+,-) is inconsistent. Perhaps you started a new search (you thought!), but actually you are adding more labeling/refinement to your previous search. (Solution: Click “Reset” to start a new search. Or, use the Advanced Version and check the My Feedback boxes to see what negative and positive examples you have already selected…and eliminate the ambiguous ones.)
Your concept is inconsistent. This is really a variation of "Labeling is inconsistent".
If you do not have a clear image in your mind, then your clicks are at risk for being inconsistent and
your evaluation of the results may be based on your like or dislike for the consistency of the
results.
(Solution: Get a clear concept in your mind, better yet, actually decide to try to find similar of
a picture that is already there.)
Your concept is not visual distinct. Perhaps the concept you are looking for, say a pretty face,
may not be visually distinct from other faces. The visual search engine is trying to figure out the
dominant and the unnecessary features. Some features may be too subtle or not distinct.
[Solution: Got none. Well, were this part of a large image collection with keywords, I would say,
try to refine the search a tad more with keywords, then start using the visual search tool.]
In other words, are these searches canned or is the actual search refinement happening with each click? It’s happening with each click! The only thing that is "canned" is that we have gotten copies of the Flickr images ahead of time so that we can extract the visual features ahead of time. It would be too slow (~100 msec/image) to do on the fly every time and then do the search.
You can start with any image – including your own – and the image similarity sorting is live, real, genuine.
Well, other than knowing some folks at each of these companies, we are not connected to these companies in a business sense.
In a demonstration sense, we are connected to the search results by a link to the page with these pictures when you double-click on the thumbnail that we display. In Flickr’s case, it is a requirement of using their API. For the other companies (not yet shown) we thought it would be good to show you how many pages you would have otherwise had to review to find that same image.
For the Basic Version, it’s just like any other search result page – more images. Except with visual search, the images are ordered by decreasing similarity. The further you get from the first image, the more dissimilar the search engine believes the image is to the image feedback you have given.
In the Advanced Version, the answer is the same, but a wrinkle is introduced with the different image selection process of the Advanced Version. See the next question.
We have two versions because the big search engine companies do not believe their audience can handle the Advanced Version. They are probably right for the casual user that would probably not even notice the extra buttons, etc. anyway.
The Basic Version is meant to be as helpful as possible while being as transparent as possible – "possible" as in what we think is helpful and what we think is transparent.
The Advanced Version gives you more control and more insight. The Basic Version automatically re-sorts the image pool with every click. The Advanced Version lets you click several images at lets you control when to re-sort the images.
The Advanced Version lets you see the images you have "labeled" by having
image buckets labeled My Feedback "More Like These" and "Less Like These", filled
with the images you have clicked as or , respectively.
Then there’s the Super Feedback area. Okay, this is only for the real image search power users. Others, back away slowly, no fast movements, keep moving, keep your hands in plain sight...
This is one of the real breakthroughs with VIMA’s MindReader Search Engine. VIMA’s technology not only takes in any number of positively and negatively labeled images and return the most similar…it will also let the searcher know which images left in the pile – if labeled – will give the re-sorting image the maximally effective (is that a word?) feedback.
Maximally effective in two senses: 1) number of clicks and 2) images that will separate the good from the not-so-good.
1) &nbps;These SuperFeedback images are already already labeled as "Not Like These" (notice the red bar below?). With only one click (on the "Go" button), you have submitted 8 clicks of feedback. You can click on any or all of the images to change their selection to positive and they you are giving positive and negative feedback.
2) The second sense is that these images are specially selected as having the features in them at just the right levels that were you to label them, the search engine can better separate similar from dissimilar.
Another, less glamorous way to describe these images is that they are the most ambiguous. These are the images on the borderline between what the search engine believes are similar and what the search engine believes are not similar. By labeling these images – in contrast to labeling images the search engine already believes are similar or dissimilar – the boundary between Similar and Dissimilar widens.