imaging modality is an important aspect of the image for medical retrieval [ 16 ] .  in user studies , clinicians have indicated that modality is one of the most important filters that they would like to be able to limit their search by .
many image retrieval websites ( goldminer , yottalook ) allow users to limit the search results to a particular modality
. however , this modality is typically extracted from the caption and is often not correct or present  .
some works have shown that image modality can be extracted from the image itself using visual features [ 810 ] .
therefore , in this paper , we propose to use both visual and textual features for medical image representation , and combine the different features using normalized kernel function in svm . in computer vision
, studies have shown that the simple global features such as histogram of edge , gray or color intensity , can represent images , and give the acceptable performance in image retrieval or recognition research fields .
based on the success of the above - mentioned visual features for general image recognition , we also use them as medical image representation for modality classification . recently ,
using local visual feature for image representation has become very popular , and been proved to be very effective for image categorization or retrieval .
the most famous approach for image representation using local visual feature is bag of keypoints [ 12 , 13 ] .
the basic idea of bag of keypoints is that a set of local image patches is sampled using some method ( e.g. , densely , randomly , or using a keypoint detector ) and a vector of visual descriptors is evaluated on each patch independently ( e.g. , sift descriptor , normalized pixel values ) .
the resulting distribution of descriptors in descriptor space is then quantified in some way ( e.g. , by using vector quantization against a prespecified codebook to convert it to a histogram of votes for ( i.e. , patches assigned to codebook centres ) and the resulting global descriptor vector is used as a characterization of the image ( e.g. , as feature vector on which to learn an image classification rule based on an svm classifier ) .
furthermore , according to the visual properties of medical images , we also calculate a histogram of small - block variance as visual feature for image representation . for textual feature , we predefine 90 vocabulary words somewhat according to the statistical properties of training samples ' captions and our knowledge about medical modality , and calculate a binary histogram for any medical image using their captions . after obtaining the different features for image representation ,
we combine them together using kernel function for svm classifier . because different features maybe have deferent scale and dimension , in order to allow each individual feature to contribute equally for modality classification , we normalize the distance between two samples using mean distance of all training samples , and
the final kernel for svm classification is the mean of individual kernel , which can be called joint kernel equal contribution ( jkec ) .
furthermore , for some easy misclassified modalities such as ct and mr or pet and nm , a global classifier , which deals with all modalities in the used database , may not be effective in distinguishing the local modalities from each other . therefore , after the global classification , a local classifier is used in the easy misclassified modality pairs to refine the classification results .
finally , the proposed algorithm is evaluated on the modality dataset of imageclef 2010 , and almost achieve the expected accuracy rate expected by the modality classification task of imageclef 2010 , which is about 97% classification rate .
in this section we describe how we extract a feature representation , which is somewhat robust to the high variability inherent in medical images and includes enough discriminative information for modality category .
some previous studies showed that it is difficult to correctly classify image categorization with only one type of image feature [ 14 , 15 ] .
so in this paper , we represent images with different images features including gray and color intensity histogram , block - based edge and variance histogram , popular bag - of - words model as visual feature , and a binary histogram of the predefined vocabulary words from image captions as textual feature .
they are easy to compute and tend to be robust against small changes of camera viewpoints . for gray intensity histogram , we can calculate the number of each intensity ( 0255 ) for all image pixel , and normalize it using pixel number .
given an image i in some color space ( e.g. , red , green , and blue ) , to calculate color histogram the color channels are quantized into a coarser space with k bins for red , m bins for green , and l bins for blue .
therefore the color histogram is a vector h = ( h1,h2,  ,hn ) , where n = kml , and each element hi represents the number of pixels of the discretized color in the image .
we assume that all images have been scaled to the same size . otherwise , we normalize histogram elements as 
										 ( 1)hj=yjj=0nyj . we firstly segment the image into several blocks , and calculate edge histogram weighted by gradient intensity in each block . in experiment , we grid - segment an image into 4-by-4 block , and calculate a 20-bin edge histogram in each block .
so we have 320-(2016-)dimensional edge histogram feature for medical image representation . for each pixel in an image , a small patch centered by the specific pixel are used for calculating the local variation of the pixel .
after obtaining the local variation of all pixels in the image , a histogram of variation intensity is calculated for the image representation . in computer vision ,
local descriptors ( i.e. , features computed over limited spatial support ) have proved well - adapted to matching and recognition tasks , as they are robust to partial visibility and clutter . in this paper
, we use grid - sampling patches , and then compute appearance - based descriptors on the patches .
in contrast to the interest points from the detector , these points can also fall onto very homogeneous areas of the image .
after the patches are extracted , the sift   descriptor is applied to represent the local features .
for each of 8 orientation planes , the gradient image is sampled over a 4-by-4 grid of locations , thus resulting in a 128-dimensional feature vector for each region .
a gaussian window function is used to assign a weight to the magnitude of each sample point .
this makes the descriptor less sensitive to small changes in the position of the support region and puts more emphasis on the gradients that are near the center of the region . to obtain robustness to illumination changes ,
the descriptors are made invariant to illumination transformations of the form ai(x ) + b by scaling the norm of each descriptor to unity .
these sift features are then clustered with a k - means algorithm using the euclidean distance .
then we discard all information for each patch except its corresponding closest cluster center identifier . for the test data ,
this identifier is determined by evaluating the euclidean distance to all cluster centers for each patch .
thus , the clustering assigns a cluster c(x)(c = 1,  c ) to each image patch x and allows us to create histograms of cluster frequencies by counting how many of the extracted patches belong to each of the clusters . the histogram representation h(x ) with c bins
is then determined by counting and normalization such that 
										 ( 2)hc(x)=1lxl=1lx(c , c(xl ) ) , 
									 where  denotes the kronecker delta function .  
figure 1 shows the procedure of bag - of - words ( bow ) feature extraction and the extracted histogram feature of an example image .
furthermore , the geometrical relation between the extracted patches is completely neglected in the approach presented here .
while this relation could be used to improve classification accuracy , it remains difficult to achieve an effective reduction of the error rate in various situations by doing so .
according to the statistical properties of word occurrence in each training modality image 's captions and our prior knowledge about the classifying modalities , we select 90 key - words , such as ct , curve , mr , urethrogram , and pet , as the vocabulary for forming a binary histogram for each medical image .
the binary histogram for image representation is 90-dimension vector , where each dimension is correspond to one selected keyword .
if one keyword appeared one or more than one time in an image 's caption , the value of the corresponding dimension in its represented binary histogram will be 1 , otherwise it will be 0 .
,n of n instances consisting of an image xi   and a class label yi  1,2 ,  , c , and given a set of f image features fm :    , m = 1,2 , 
, f , where dm denotes the dimensionality of the mth feature , the problem of learning a classification function y :   1,2 ,  , c from the features and training set is called feature combination problem . in computer vision , the problem of learning a multiclass classifier from training data is often addressed by means of kernel methods .
kernel methods make use of kernel functions defining a measure of similarity between pairs of instances . in the context of feature combination
it is useful to associate a kernel to each image feature as the following equation ( 3 ) , and combine the kernels of different features together . for a kernel function k of each feature between real vectors
we define the short - hand notation : 
						 ( 3)km(ii , ij,)=k(fm(ii),fm(ij))=k(s(fm(ii),fm(ij ) ) ) , 
					 where ii and ij are two samples , fm(ii ) is the mth extracted feature from the sample ii , and s(fm(ii ) , fm(ij ) ) is the similarity measure between the mth features of the samples ii and ij .
then the image kernel km :      only considers similarity with respect to image feature fm .
if the image feature is specific to a certain aspect , say , it only considers color information , then the kernel measures similarity only with regard to this aspect .
the subscript m of the kernel can then be understood as indexing into the set of features .
because different features maybe have different scale and dimension , in order to allow each individual feature to contribute equally for modality classification , we normalize the distance between two samples using mean distance of all training samples , and then , obtain the kernel function for each individual feature fm .
the final kernel for svm classification is the mean of individual kernel , which can be called joint kernel equal contribution ( jkec ) . for the feature similarity calculation of two samples
, we use several distances : euclidean distance ( l2 distance ) , l1 distance , and  distance , for evaluating the classification performance .
the  distance for two samples can be calculated as follows : 
						 ( 4)smi , j = s(fm(ii),fm(ij))=1l(xlyl)2xl+yl , 
					 where x and y represent the mth features fm(ii ) , fm(ij ) of samples i and j , respectively , and xl is the lth element of the vector x. sm is the similarity measure of the mth feature between the ith and jth training samples .
then , the rbf function is used for calculating the kernel : 
						 ( 5)kmi , j = km(ii , ij,)=exp  ( s(fm(ii),fm(ij))m ) , 
					 where m is the normalized item for kernel function of the mth feature . here , we use the distance mean of all training samples as m = 1/n(iism ) ( n is the training sample number ) , which will lead to similar contribution of each feature to kernel .
then the final combined kernel function can be obtained by 
						 ( 6)ki , j=1mimkmi , j , 
					 where m is the feature number for image representation .
the proposed algorithm is evaluated on the modality training dataset of imageclef 2010 , which expects about 97% classification rate on the released evaluated and test datasets . because the ground - truths of the evaluated and test dataset are not released , we cross - validate our proposed strategy with the released training dataset firstly .
the classification rate with our experiment on training dataset almost approximated the required goal of the modality classification task .
in the released medical database by imageclef 2011 , some modalities have a lot of visual similarity such as pet and nm modality . therefore , it is difficult to distinguish them in the global modality classification , which deals with all modalities in the database . in this section , after the global conventional classification , we design local classifiers to refine the classification results in easy - misclassified modalities .
next , we firstly explain the used dataset , and then , introduce how to design the local classifier according to evaluation results . the database released for the imageclef-2010 medical modality classification in medical retrieval task includes 2390 annotated modality images ( ct : 314 ; gx : 355 ; mr : 299 ; nm : 204 ; pet : 285 ; px : 330 ; us : 307 ; xr : 296 ) for training and a separate evaluated set consisting of 2620 images .
the aim is to automatically classify the evaluated set using 8 different modality label sets including ct , mr , and pet .
a more detailed explanation of the database and the tasks can be found in . for validating the discriminant properties of different modalities , we randomly select 180 samples from each medical modality in imageclef 2010 training dataset , and the remainder for testing .
we combine all visual and textual features using the jkec fusion strategy introduced in section 3 , for modality classification .
the confusion matrix of one run is shown in table 1 . from table 1 , it can be seen that 92.537% ct sample images are correctly recognized as ct modality , and 3.9851% and 2.2388% are classified as mr and xr modalities , respectively . on the other hand ,
about 2 - 3% mr or xr sample images are also misrecognized as ct modality . at the same time
, it is obvious that nm and pet or gx and px modalities are also easily misclassified from each other .
therefore , we design three local classifiers for the limited easy - misclassified modalities , which are ct , mr , and xr group , nm and pet group , gx and px group , to refine the classification results in local regions .
the compared experimental results with or without refinement procedure are shown in figure 5 . from figure 5
, it can be seen that the recognition rates for ct , mr , pet , px , and xr modalities with local classifier refinement can be improved more than 1% compared to those without refinement .
in this section , we validate the recognition rates of different features with three types of similarity measures : euclidean distance ( l2 distance ) , l1 distance and  distance , and do the cross - validation experiments using the combined visual and textual features on imageclef 2010 training dataset .
then , the submitted runs to medical modality classification of imageclef 2010 and the released results will be introduced .
the recognition rates of different features with three types of similarity measures : in order to validate what kind of distance measure is adaptive to each extracted feature for image representation , we apply three types of similarity measures : euclidean distance ( l2 distance ) , l1 distance , and  distance , for calculating the kernel function as in ( 5 ) of svm classifier . in the experiments , with the imageclef 2010 training dataset ,
180 images are randomly selected for training from each modality , the remainder are for test .
the compared recognition rates are shown in figure 6 , where  kai2  means  distance . from figure 6
, it can be seen that l1 and  distance can obtain much better performance than l2 distance for most features , and  distance can achieve a little better than or similar results to l1 distance . then , in the next experiments , we utilize  distance for a similarity measure of all features to calculate svm kernel functions .
cross - validation experiments : in the experiments , we firstly divide the training dataset of imageclef 2010 into 5 groups , and use 4 groups as training and 1 group as test to cross - validate the performance of different features with  distance as similarity measure .
the recognition rates are shown , figure 7 , where visual means with the combined features of all visual ones , visual + textual means with the combined features of all visual and textual ones , visual + textual + refine means using refinement procedure after classification with all combined features . from figure 7
, it can be seen that the average recognition rate can be improved about 1% after the refinement procedure introduced in section 4 .
submitted runs : as medical image processing group ( mipg ) of our intelligent image processing laboratory ( iipl ) in ritsumeikan university , we prepared four runs for evaluation image set , which used combine visual feature , textual feature , both visual and textual features , and weighted visual and textual features .
we submitted two runs using textual , combined textual and visual features by on - line - system , respectively .
our results are ranked second among 6 participating teams , and the result of one run is also ranked second among about 50 runs . at the same time , the recognition rates ( submitted run : 93.36% , unsubmitted run : 93.89% ) of our methods using mixed feature ( visual plus textual ) are similar to the first ranking results 94% by xerox research centre europe .
in this paper , we proposed to extract different visual and textual features for medical image representation , and use jkec strategy to fusion them for modality classification . to extract visual features from the images
, we used histogram descriptor of edge , gray , or color intensity and block - based variation as global features and sift histogram as local feature , and the binary histogram of some predefined vocabulary words for image captions is used for textual feature . because different features maybe have different scale and dimension , in order to allow each individual feature to contribute equally for modality classification , we proposed to use joint kernel equal contribution ( jkec ) for kernel fusion of different features .
furthermore , for some easy misclassified modality pairs such as ct and mr or pet and nm modalities , a local classifier is used for distinguishing samples in the pair modality to improve performance .
 we describe an approach for the automatic modality classification in medical image retrieval task of the 2010 clef cross - language image retrieval campaign ( imageclef ) . 
 this paper is focused on the process of feature 
 extraction from medical images and fuses the different extracted visual features and textual feature for modality classification . to extract visual features from the images , we used histogram descriptor of edge , gray , or color intensity and block - based variation as global features and sift histogram as local feature . for textual feature of image representation , 
 the binary histogram of some predefined vocabulary words from image captions is used . 
 then , we combine the different features using normalized kernel functions for svm classification . furthermore , for some easy misclassified modality pairs such as ct and mr or pet and nm modalities , a local classifier is used for distinguishing samples in the pair modality to improve performance . 
 the proposed strategy is evaluated with the provided modality dataset by imageclef 2010 . 
1. Introduction
2. Feature Extraction for Image Representation
3. Feature Fusion
4. Refinement Procedure for Easy Misclassified  Modalities
5. Experiments
6. Conclusions