Document Citation

Title:
An Interactive Image Retrieval Framework For Biomedical Articles Based On Visual Region-Of-Interest (ROI) Identification and Classification.

Author(s):
Md Mahmudur Rahman, Daekeun You, Matthew S. Simpson, Sameer K. Antani, Dina Demner-Fushman, and George R. Thoma.

Institution(s):
1) National Library of Medicine, Bethesda, MD 20894

Source:
The 2nd IEEE Conference on Healthcare Informatics, Imaging, and Systems Biology Analyzing Big Data for Healthcare and Biomedical Sciences (HISB 2012). La Jolla, CA. September 2012.

Abstract:
Scientific articles in the biomedical domain convey information using multiple modalities, including text and images. Authors of biomedical publications frequently use images to elucidate the text, to illustrate the medical concepts or to highlight special cases. These images often convey essential information in context and can be very valuable for clinical decision support (CDS) and education. Authors also often use annotation markers (pointers) such as arrows, letters or symbols overlaid on figures and illustrations to highlight the regions-of-interest (ROIs). These annotations are then referenced and correlated with concepts in the caption text or figure descriptions in the article text. This association creates a bridge between the visual characteristics of important regions within an image and their semantic interpretation. For example, an area of a computed tomography (CT) scan having a slightly bright and hazy appearance can be mapped to the pattern ground-glass opacity. This paper presents an interactive biomedical image retrieval system based on automatic visual ROI extraction and classification into visual concepts. The goal is to develop a retrieval system that finds images that contain patterns similar to an interactively marked ROI. Our proposed method first localizes and recognizes the annotations by utilizing a combination of rule-based and statistical image processing techniques. Identifying these, assists in extracting ROIs that are likely to be highly relevant to the discussion in the article text. Eight complementary texture-related features are extracted from each ROI. Finally, the above features are combined to form a 487-dimensional feature vector for input to the classifier. The image regions are then annotated for classification (SVM) using biomedical concepts (such as cyst, bronchiectasis, honeycomb, etc.) obtained from a glossary of imaging terms entitled "Fleishner Society: Glossary of Terms for Thoracic Imaging by Hansell et al." [1]. Note that because this glossary is a source for RSNA's RadLex, any ROIs that might correspond to these terms can be directly mapped onto an existing biomedical ontology. We chose to limit our annotation effort to thoracic CT scans and their associated captions [2]. Such images exhibit high regularity and account for a large portion of the images publicly available as part of the 2010 ImageCLEF medical retrieval track data set [3]. With an online concept classification scheme, the retrieval system can map the visual characteristics of a query region (a user may interactively mark an ROI) to textual concepts, and then use these concepts to search image captions. The relevance to a clinical query is improved by this addition of semantic information to image features extracted for retrieval. In addition, the user can toggle the search process from purely visual to a textual one (cross-modal) or integrate both visual and textual search in a single process (multi-modal) based on user feedback. The hypothesis that such approaches would improve biomedical image retrieval is validated through experiments on a biomedical article dataset of thoracic CT scans from the collection used in the 2010 ImageCLEF medical retrieval track. Preliminary results show the effectiveness of the proposed retrieval approach and are promising for our larger goal of creating a visual ontology of biomedical imaging entities, and utilizing this resource for effective retrieval.

Publication Type: CONFERENCE

More about this article:

Abstract | Full Text (PDF)