This project seeks to improve information retrieval from collections of full-text biomedical
articles, images, and patient cases, by moving beyond conventional text-based searching to
combining both text and visual features to:
- Build text processing and image processing tools to index images and image-related text,
and enable searching of the literature by textual, visual and hybrid search queries.
- Build tools employing a combination of text and image features to enrich traditional
bibliographic citations with relevant biomedical images, charts, graphs, diagrams and
other illustrations, as well as with patient-oriented outcomes from the literature.
In addition to developing these tools, we intend to test them in two related initiatives that seek
to:
- Improve the retrieval of the biomedical literature by targeting the visual content in
articles. Within this broad goal, we initially focus on finding information relevant to a
patient's medical case in the literature, and then linking it to the health record, and
clinical question answering.
- Improve the retrieval of semantically similar images from the literature and from image
databases, with the goal of reducing the "semantic gap" that is a significant hindrance to
the use of image retrieval for practical clinical purposes.
See the technical report to the LHNCBC Board
of Scientific Counselors (September 2010) for details.
- Text processing:
- Caption extraction and segmentation
- Mention extraction
- Biomedical terminology extraction
- Image processing:
- multi-panel figure segmentation
- text and symbols localization
- color and texture features computation
- Image classification using supervised machine learning
- Image annotation:
- automatic UMLS-based medium-level annotation using text references to image regions of interest and mark-yp
- Simpson M, Demner-Fushman D, Thoma G. Evaluating the Importance of Image-related Text for Ad-hoc and Case-based Biomedical Article Retrieval. Proceedings of the 2010 Annual Symposium of the American Medical Information Association (AMIA 2010), Washington, DC, November 2010
- Simpson M, Sneiderman C, Demner-Fushman D, Thoma G, Knowledge Acquisition from Clinical Case Reports: Quality and Utility for Case-based Biomedical Article Retrieval. Proceedings of the 2010 Annual Symposium of the American Medical Information Association (AMIA 2010), Washington, DC, November 2010
- Stanley RJ, De S, Demner-Fushman D, Antani S, Thoma GR.
An image feature-based approach to automatically find images for application to clinical decision support.
Comput Med Imaging Graph. 2010 Dec 6.
- Rahman MM, Antani SK, Long LR, Demner-Fushman D, Thoma GR.
Multi-Modal Query Expansion Based On Local Analysis For Medical Image Retrieval.
Lecture Notes in Computer Science. First MICCAI International Workshop on Medical Content-Based Retrieval for Clinical Decision Support, International Conference on Medical Image Computing and Computer Assisted Interventio February 2010;5853/2010(doi: 10.1007/978-3-642-11769-5):110-9.
- You D, Antani S, Demner-Fushman D, Rahman M, Govindaraju V, Thoma G.
Biomedical Article Retrieval Using Multimodal Features and Image Annotations In Region-based CBIR.
Document Recognition and Retrieval XVII. Proceedings of the SPIE. San Jose, CA. January 2010;7534:75340V-75340V-12.
- Simpson M, Demner-Fushman D, Antani S, Thoma GR. Design and
Implementation of a Biomedical Image Retrieval System. NIH Research
Festival. October 6-9, 2009
- Simpson M, Rahman MM, Demner-Fushman D, Antani S, Thoma GR. Text- and
Content-based Approaches to Image Retrieval for the ImageCLEF
2009 Medical Retrieval Track. CLEF2009 Working Notes. CLEF 2009
Workshop 30 September - 2 October, Corfu, Greece, in conjunction with
ECDL2009.
- Demner-Fushman D, Antani S, Simpson M, Thoma GR. Annotation and
retrieval of clinically relevant images. Int J Med Inform. 2009 Jun 20.
- Apostolova E, Demner-Fushman D. Towards Automatic Image Region
Annotation - Image Region Textual Coreference Resolution. In Proceedings
of the 2009 Annual Conference of the North American Chapter of the
Association for Computational Linguistics. June 2009. Boulder, Colorado.
Association for Computational Linguistics, p 41--44.
- You D, Apostolova E, Antani S, Demner-Fushman D, Thoma G. Figure
content analysis for improved biomedical article retrieval. Proc. SPIE,
Vol. 7247, 72470V (2009)
- Simpson M, Demner-Fushman D, Sneiderman C, Antani SK Thoma GR. Using
Non-lexical Features to Identify Effective Indexing Terms for Biomedical
Illustrations Proceedings of the 12th Conference of the European
Chapter of the Association for Computational Linguistics (EACL-09),Athens, Greece, April 2009
- Sneiderman C, Demner-Fushman D, Fung KW, Bray B. UMLS-based Automatic Image Indexing.
Proceedings of the 2008 Annual Symposium of the American Medical Information Association (AMIA 2008), Washington, DC, November 2008
- Demner-Fushman D, Antani S, Simpson M, Thoma GR.
Combining Medical Domain Ontological Knowledge and Low-level Image Features for Multimedia Indexing
LREC 2008 (Sixth International Conference on Language Resources and Evaluation),
OntoImage Workshop, Marrakech, Morocco, May 2008
- Antani S, Demner-Fushman D, Li J, Srinivasan BV, Thoma GR.
Exploring use of images in clinical articles for decision support in Evidence-Based Medicine.
Proc. SPIE-IS&T Electronic Imaging. San Jose, CA. January 2008;6815:68150Q(1-10)
- Demner-Fushman D, Antani SK, Thoma GR.
Automatically Finding Images for Clinical Decision Support.
Proceedings of IEEE International Workshop on Data Mining in Medicine (DM-Med 2007). Omaha, NE. October 2007;:139-44
- Li J, Demner-Fushman D, Antani SK, Thoma GR.
Localizing Text and Symbols in Images from Biomedical Journal Articles.
Poster at 20th NIH Research Festival. National Institutes of Health September 2007
- Srinivasan B, Antani SK, Demner-Fushman D, Thoma GR.
Identification and Segmentation of Multi-Panel Images in Biomedical Journal Articles
Poster at 20th NIH Research Festival. National Institutes of Health. September 2007
