Zhiyun Xue, Sameer Antani, L. Rodney Long, George R. Thoma
National Library of Medicine, National Institutes of Health
xuez@mail.nih.gov
Content-based indexing and retrieval is gaining increasing interest in the medical domain with the growing size of medical image databases. We present here a Web-accessible retrieval system for searching for similar uterine cervix images based on their visual characteristics. The system operates on a subset of a large database created for archiving patient records collected by two key projects in cervical cancer research. It was developed to bridge the "gaps" that hold back the practical adoption of most CBIR systems. This collaboration between engineers and gynecological experts promises to provide a new biomedical resource beyond current text-based searching tools.
Uterine cervical cancer is one of the most common gynecological cancers, with nearly 500,000 new cases diagnosed worldwide each year [1]. It especially affects women with low socio-economic status and those with limited access to preventive care. Cervicography is one cost-effective technique for cervical cancer screening and is often used in low-resource regions. In cervicography, the cervix is moistened with a small amount of acetic acid. Many abnormal cervical lesions may turn white on the acetic acid exposure, compared with the normal pinkish color of the normal squamous epithelium of the cervix [2]. This effect is termed acetowhitening and is commonly visible to the naked eye. Based on the acetowhite appearance and other visual signs in the cervix, gynecologists assess the degree of abnormality, and biopsies are taken from suspicious areas when appropriate. Cervicographic images (cervigrams) are color photos taken with a special macro-lens strobe-flash camera (cerviscope) after the cervix is swabbed with acetic acid. These images provide valuable aid to the diagnosis, research, and training for cervical cancer prevention.
For cervical cancer research, the Guanacaste [3] and ALTS Projects [4] are two important studies organized and funded by the National Cancer Institute (NCI). The Guanacaste Project is a population-based natural history study of HPV and cervical neoplasia. It was conducted in a rural area of Costa Rica with a very high incidence rate of cervical cancer. About 10,000 women were enrolled and followed for up to seven years. The ALTS (ASCUS/LSIL Triage Study) Project investigated the clinical management of low grade cervical cytologic abnormalities. It was conducted at four medical centers in the United States. Approximately 5,000 women with atypical ASCUS or LSIL were enrolled and followed at half year intervals for a total of two years. During these projects, large amounts of visual and textual information including approximately 100,000 cervigrams were collected from the screening tests involving cervicography, colposcopy, cytology, and histology. To manage and analyze this data, the National Library of Medicine (NLM), in collaboration with oncological gynecologists from the NCI, has been developing a set of Web-accessible software. Two of these programs are closely related to the research topic of this paper: the Boundary Marking Tool (BMT) 1 and the Multimedia Database Tool (MDT) 2 , whose user interfaces are shown in Figures 1 and 2. The BMT [5] allows medical experts to manually draw the boundaries of important anatomical regions in cervigrams and to record region labels, as well as to enter a detailed set of textual data relevant to evaluation and diagnosis. The MDT [6] serves as the central database tool for accessing the Guanacaste and ALTS Project data. It is a text-based searching tool. Though valuable, searching by text alone is deficient in retrieving images exhibiting particular visual features. This observation has motivated the research and development in Content-Based Image Retrieval (CBIR) for cervigram databases. CBIR is a set of techniques based on visual features for indexing, querying, and retrieving images from a database.
Figure 1. Boundary Marking Tool
CBIR is the basis for the CervigramFinder, a prototype online system we developed for searching and retrieving cervigrams based on their visual attributes from a subset of the cervigram repository collected from the Guanacaste and ALTS projects. It is expected that CervigramFinder will play an increasingly important role for providing content-based access to the cervigram databases, as a complementary module to the MDT. The paper is organized as follows: Section 2 describes the details of CervigramFinder, including algorithm development, system architecture, usability, and system use. The discussion and conclusion are given in Section 3 and Section 4, respectively.
Figure 2. Multimedia Database Tool
To the best of our knowledge, CervigramFinder is the first Web-accessible CBIR system for searching and retrieving cervigrams. It addresses the uniqueness of cervigrams and incorporates user knowledge. It tries to reduce the gaps that prevent the inclusion of CBIR into medical applications.
One distinguishing aspect of CervigramFinder is that it is implemented as a region-based searching tool that retrieves similar pathology-important regions in cervigrams from the database. Most CBIR approaches operate on the image as a whole [7,8]. This global approach does not ensure satisfactory results for our application, since physicians look for visual characteristics within particular regions, such as the acetowhite area, the transformation zone, the columnar epithelium, and the os, when examining the cervix surface to identify abnormality and decide if a biopsy is necessary. CervigramFinder operates on an expert-annotated subset of the cervigram database, in which more than 2000 medically-significant regions have been manually marked and labeled by oncological gynecologists using the BMT software. It is capable of including results from automatic labeling approaches from our ongoing research into indexing such large image collections. CervigramFinder tries to incorporate as much user knowledge as possible. It implements a user-in-the-loop approach to aid in reducing the "semantic gap". In this approach, the query region is delineated by the user and the visual attributes of the marked region are then extracted by automatic processing algorithms. The extracted feature vector of the query region is then compared with those precomputed feature vectors of all the regions in the database to find the most similar ones.
CervigramFinder incorporates fundamental CBIR functions, such as feature extraction, feature normalization, feature combination, feature dimension reduction, and similarity measures [9]. We did an extensive study on evaluating and identifying key techniques among those found in the technical literature by comparing their performance for the retrieval of cervigrams. We developed a convenient tool for this extensive study [10]. One outstanding issue in CBIR systems is to represent image visual features effectively. CervigramFinder uses color, texture, size and location attributes to describe tissue regions and retrieve similar regions from the database. These attributes are also crucial visual signs considered by physicians when they analyze the cervix and decide on the severity of abnormality. Shape is not considered because it is significantly less important for identifying or distinguishing regions in our application. We implemented and examined various popular algorithms for representing color and texture properties. We also proposed a new and effective location descriptor which takes into account the visual characteristics of cervix lesions. In addition, we implemented various distance-based similarity measures as well as optional approaches for combining and normalizing features which are important components of CBIR for further improving retrieval performance. For more details on this comprehensive algorithm study and retrieval performance analysis, please refer to our papers [10, 11].
Figure 3. System architecture
The CervigramFinder is developed using the platform-independent Java language and can be accessed via the Web. It employs a system architecture which adheres to the principles of modular and extensible software development. The software architecture is shown in Figure 3. The system has four key components: a client, a gateway, an indexing and retrieval server, and the database. The client is a Java applet that provides a graphical user interface for users to create queries and interact with retrieval results. The gateway uses a Java servlet to manage Web client-server communications and user authentication. The indexing and retrieval server calculates the feature signatures and does similarity matching. The image-text database contains "ground truth" cervigrams with regions marked and labeled by experts, and associated text data. This modular and expandable distributed architecture allows us to alter the user interface or add new indexing/retrieval algorithms conveniently without changing other components throughout our process of ongoing improvement. The client/server distributed processing system architecture is expected to be a significant benefit not only for users to evaluate the system, but also in its eventual use at multiple research sites.
Recently, several papers have addressed the possible obstacles to the use of CBIR in the medical domain and have identified a set of gaps/limitations in existing CBIR applications [12-14]. Bridging these gaps could better position a CBIR system for use by the biomedical community. In system testing, we made efforts to reduce one of the gaps, namely the usability gap, which measures the ease of use of the system. This gap is rarely addressed during the design and development of CBIR systems. However, it is one critical issue which directly influences the user acceptance of the system. To improve the usability of CervigramFinder, we collaborated with a usability study expert and several gynecologists who have different levels of exposure to CBIR techniques to carry out a preliminary testing of the effectiveness and the user satisfaction of the interface. Based on their inputs, we redesigned the interface of CervigramFinder to make it more appealing and easier to use for both engineers and physicians. Figure 4 shows a snapshot of the user interface before and after the usability study. Besides the layout, other significant differences between the two versions include the additions of location feature and editing functions of query image in the second version.
The system can be accessed from anywhere through a Web browser. Using the graphical user interface built into a Web page shown in Figure 4 (b), users may select a query image, draw a region on the image, and specify which visual features of the region are to be used in the search. They can also specify the relative importance of these features. The system then computes the feature vector of the query region and compares it with the pre-computed feature vectors of regions for images in the database. The returned regions are ranked by the degree of similarity to the query feature vector, and the images containing these regions are displayed along with fragments of the associated patient data. To facilitate users in creating a query, the system also supports zooming in/out, switching between contrast-enhanced image and original image, editing the query region, and displaying the query image in a large window. For each returned image, a magnified view is also available. In addition, help information on the system is provided.
(a)version 1
(b)version 2
Figure 4. User interface
Practical use of CBIR systems relies on many techniques and there are still a number of research issues that need to be addressed [15]. CervigramFinder addresses some of those issues, but more work needs to be done in the area of, for example, reliable image segmentation, fast image indexing, effective feature combination, and user feedback exploitation. Nonetheless, CBIR potentially provides new and exciting alternatives to the limitations imposed by the traditional retrieval methods for medical image database management. We look forward to pushing this technology further toward meaningful applications in medicine.
The need for searching large medical image repositories by their visual content for specific applications has greatly increased in recent years. This paper reports the design and implementation of CervigramFinder, a prototype CBIR system for providing access to the NLM cervigram database by the image characteristics of particular regions on the cervix. It addresses several aspects that affect the practical use of CBIR systems in the biomedical world. This research and development represents our efforts toward building a support tool for uterine cervical cancer diagnosis and education.
The authors thank Anne R. Haake, Ph.D. of Rochester Institute of Technology, for her valuable help in the usability study.
This research was supported by the Intramural Research Program of the National Institutes of Health, National Library of Medicine, and Lister Hill National Center for Biomedical Communications.
[1] NCI Women's Health Report FY2005-2006. http://women.cancer.gov/planning/whr0506/whr0506.pdf
[2] J.W. Sellors, R. Sankaranarayanan, "Colposcopy and Treatment of Cervical Intraepithelial Neoplasia - A Beginner's Manual," Edited by J.W. Sellors and R. Sankaranarayanan, Published by the International Agency for Research on Cancer, France, 2003.
[3] R. Herrero, M. Schiffman, C. Bratti et al., "Design and methods of a population-based natural history study of cervical neoplasia in a rural province of Costa-Rica: the Guanacaste Project," Rev Panam Salud Publica vol. 1, no. 5, pp.362-375, 1997.
[4] M. Schiffman, M. E. Adrianza, "ASCUS-LSIL Triage Study: Design, methods and characteristics of trial participants," Acta Cytol, vol.44, no.5, pp. 726-742, Sep-Oct 2000.
[5] J. Jeronimo, R. Long, L. Neve, B. Michael, S. Antani, and M. Schiffman, "Digital tools for collecting data from cervigrams for research and training in colposcopy," Journal of Lower Genital Tract Disease, vol. 10, no. 1, pp. 16-25, January 2006.
[6] M. Bopf, T. Coleman, L. R. Long, S. Antani, G. R. Thoma, "An architecture for streamlining the implementation of biomedical text/image databases on the web," Proceedings of the 17th IEEE Symposium on Computer-Based Medical Systems, pp. 563-568, June 2004.
[7] T. M. Lehmann et al., "Automatic categorization of medical images for content-based retrieval and data mining," Computerized Medical Imaging and Graphics, vol.29, pp. 143–155, 2005.
[8] J. Wang, J. Li, and G. Wiederhold, "SIMPLIcity: semantics-sensitive integrated matching for picture libraries," IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 947–963, 2001.
[9] Z. Xue, L. R. Long, S. Antani, G. R. Thoma, "A Web-accessible content-based cervicographic image retrieval system," Proc. SPIE Medical Imaging, vol. 6919, pp. 07-1-9, April 2008.
[10] Z. Xue, S. Antani, L. R. Long, J. Jeronimo, G. R. Thoma, "Investigating CBIR techniques for cervicographic images," Proc AMIA Annu Fall Symp, pp.826-830, 2007.
[11] Z. Xue, S. Antani, L. R. Long, J. Jeronimo, G. R. Thoma, "Cervicographic image retrieval by spatial similarity of lesions," Proceedings of 19th International Conference on Pattern Recognition, pp. 1-4, December 2008.
[12] T. M. Deserno, S. Antani, L. R. Long, "Ontology of gaps in content-based image retrieval", Journal of Digital Imaging, vol. 22, no. 1, pp. 202-215, April 2009.
[13] L. R. Long, S. Antani, T. M. Deserno, G. R. Thoma, "Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions," International Journal of Healthcare Information Systems and Informatics, vol. 4, no. 1, pp. 1-17, January 2009.
[14] S. Antani, L. R. Long, G. R. Thoma, "Bridging the gap: Enabling CBIR in medical applications," Proc. 21st IEEE International Symposium on Computer-Based Medical Systems, pp. 4-6, Jyväskylä, Finland. June 2008.
[15] H. Muller, N. Michoux, D. Bandon, and A. Geissbuhler, "A review of content-based image retrieval systems in medical applications—clinical benefits and future directions", International Journal of Medical Informatics, vol. 73, no. 1, pp. 1-23, 2004.