{"content":[{"text":"BU CS TR98-004. To appear in IEEE Workshop on Content-based Access of Image and Video Libraries, June 1998.","bbox":{"left":0.099804,"top":0.037159,"width":0.791326,"height":0.017601000000000006}},{"text":"Combining Textual and Visual Cues for Content-based Image \n Retrieval on the World Wide Web \n Marco La Cascia, Saratendu Sethi, and Stan Sclaroff \n Image and Video Computing Group \n Computer Science Department \n Boston University \n Boston, MA 02215","bbox":{"left":0.148433,"top":0.100662,"width":0.693707,"height":0.16177699999999998}},{"text":"Abstract \n A system is proposed that combines textual and visual \n statistics in a single index vector for content-based search \n of a WWW image database. Textual statistics are captured \n in vector form using latent semantic indexing (LSI) based \n on text in the containing HTML document. Visual statistics \n are captured in vector form using color and orientation \n histograms. By using an integrated approach, it becomes \n possible to take advantage of possible statistical couplings \n between the content of the document (latent semantic con- \n tent) and the contents of images (visual statistics). The \n combined approach allows improved performance in con- \n ducting content-based search. Search performance experi- \n ments are reported for a database containing 100,000 im- \n ages collected from the WWW. \n 1 Introduction \n The growing importance of the world wide web has led to \n the birth of a number of image search engines [6, 7, 11, 12]. \n The web's staggering scale puts severe limitations on the \n types of indexing algorithms that can be employed. Luck- \n ily, due to the scale and unstructured nature of the WWW, \n even the most basic indexing tools would be welcome. Ex- \n isting image engines allow users to search for images via \n an SQL keywords interface [6, 12] and/or via query by im- \n age example (QBE) [7, 11, 12]. \n In QBE, the system presents an initial page of represen- \n tative (or randomly selected) images thumbnails to the user \n [5]. The user then marks one or more images as relevant to \n the search. The visual statistics for these images are then \n used in deﬁning a query. The user's success in locating \n images in the database depends in great part on which im- \n ages appear within this initial group of thumbnails. Given \n the numerous and diverse images available in an WWW \n index, it is difﬁcult to guarantee that there will be even one \n relevant image shown in the initial page. We call this the \n page zero problem.","bbox":{"left":0.095297,"top":0.291605,"width":0.381055,"height":0.567245}},{"text":"Other WWW image engines allow the user to form a \n query in terms of SQL keywords [6, 12]. This alleviates \n the page zero problem, since the user can give text infor- \n mation that narrows the scope of possible images displayed \n on page zero. To build the image index, keywords are \n extracted heuristically from HTML documents containing \n each image, and/or from the image URL. Unfortunately, \n it is difﬁcult to include visual cues within an SQL frame- \n work. This results in systems that force inherently visual \n information into a textual form, or systems that treat textual \n and visual cues disjointly.","bbox":{"left":0.514904,"top":0.294128,"width":0.38068100000000005,"height":0.167904}},{"text":"2 New Approach \n By truly unifying textual and visual statistics, one would \n expect to get better results than either used separately. In \n this paper, we propose an approach that allows the combi- \n nation of visual statistics with textual statistics in the vec- \n tor space representation commonly used in query by image \n content systems. Text statistics are captured in vector form \n using latent semantic indexing (LSI) [3]. \n Text documents are represented by low-dimensional \n vectors that can be matched against user queries in the \n LSI “semantic” space. The LSI index for an HTML docu- \n ment is then associated with each of the images contained \n therein. Visual statistics (e.g., color, orientedness) are also \n computed for each image. The LSI vector and visual statis- \n tics vector are then combined into a uniﬁed vector that can \n be used for content-based search of the resulting image \n database. \n In previous systems, visual information has been used in \n a form that is compatible with standard DBMS frameworks \n [6] or only to reﬁne the query [12]. By using an integrated \n approach, we are able to take advantage of possible statis- \n tical couplings between the content of the document (latent \n semantic content) and the contents of images (image statis- \n tics). Furthermore, LSI implicitly addresses problems with \n synonyms, word sense, lexical matching, and term omis- \n sion.","bbox":{"left":0.514904,"top":0.481909,"width":0.380892,"height":0.40679000000000004}}],"images_bbox":[],"images_bbox_not_text_overlap":[]}