Composition

of

a

DIDS

by

Integrating

Heterogeneous

IDSs

on

Grids

Paulo

F.

Silva

and

Carlos

B.

Westphall

and

Carla

M.

Westphall

Network

and

Management

Laboratory

Department

of

Computer

Science

and

Statistics

Federal

University

of

Santa

Catarina,

Florianópolis,

Brazil

Marcos

D.

Assunção

Grid

Computing

and

Distributed

Systems

Laboratory

and

NICTA

Victoria

Laboratory

Department

of

Computer

Science

and

Software

Engineering

The

University

of

Melbourne,

Victoria,

Australia

{paulo,westphal,assuncao,carla}@lrg.ufsc.br

ABSTRACT

This

paper

considers

the

composition

of

a

DIDS

(Distributed

Intrusion

Detection

System)

by

integrating

heterogeneous

IDSs

(Intrusion

Detection

Systems).

A

Grid

middleware

is

used

for

this

integration.

In

addition,

an

architecture

for

this

integration

is

proposed

and

validated

through

simulation.

Categories

and

Subject

Descriptors

[Distributed

Systes]:

Client/Server,

Distributed

Applications.

INTRODUCTION

Solutions

for

integrating

heterogeneous

IDSs

(Intrusion

Detection

Systems)

have

been

proposed

by

several

groups

Some

reasons

for

integrating

IDSs

are

described

by

the

IDWG

(Intrusion

Detection

Working

Group)

from

the

IETF

(Internet

Engineering

Task

Force)

as

follows:

Many

IDSs

available

in

the

market

have

strong

and

weak

points,

which

generally

make

necessary

the

deployment

of

more

than

one

IDS

to

provided

an

adequate

solution.

Attacks

and

intrusions

generally

originate

from

multiple

networks

spanning

several

administrative

domains;

these

domains

usually

utilize

different

IDSs.

The

integration

of

IDSs

is

then

needed

to

correlate

information

from

multiple

networks

to

allow

the

identification

of

distributed

attacks

and

or

intrusions.

The

interoperability/integration

of

different

IDS

components

would

benefit

the

research

on

intrusion

detection

and

speed

up

the

deployment

of

IDSs

as

commercial

products.

DIDSs

(Distributed

Intrusion

Detection

Systems)

therefore

started

to

emerge

in

early

to

allow

the

correlation

of

intrusion

information

from

multiple

hosts,

networks

or

domains

to

detect

distributed

attacks.

Research

on

DIDSs

has

then

received

much

interest,

mainly

because

centralised

IDSs

are

not

able

to

provide

the

information

needed

to

prevent

such

attacks

However,

the

realization

of

a

DIDS

requires

a

high

degree

of

coordination.

Computational

Grids

are

appealing

as

they

enable

the

development

of

distributed

application

and

coordination

in

a

distributed

environment.

Grid

computing

aims

to

enable

coordinate

resource

sharing

in

dynamic

groups

of

individuals

and/or

organizations.

Moreover,

Grid

middleware

provides

means

for

secure

access,

management

and

allocation

of

remote

resources;

resource

information

services;

and

protocols

and

mechanisms

for

transfer

of

data

According

to

Foster

et

al.

Grids

can

be

viewed

as

a

set

of

aggregate

services

defined

by

the

resources

that

they

share.

OGSA

(Open

Grid

Service

Architecture)

provides

the

foundation

for

this

service

orientation

in

computational

Grids.

The

services

in

OGSA

are

specified

through

well-defined,

open,

extensible

and

platformindependent

interfaces,

which

enable

the

development

of

interoperable

applications.

This

article

proposes

a

model

for

integration

of

IDSs

by

using

computational

Grids.

The

proposed

model

enables

heterogeneous

IDSs

to

work

in

a

cooperative

way;

this

integration

is

termed

DIDSoG

(Distributed

Intrusion

Detection

System

on

Grid).

Each

of

the

integrated

IDSs

is

viewed

by

others

as

a

resource

accessed

through

the

services

that

it

exposes.

A

Grid

middleware

provides

several

features

for

the

realization

of

a

DIDSoG,

including

decentralized

coordination

of

resources;

use

of

standard

protocols

and

interfaces;

and

the

delivery

of

optimized

QoS

(Quality

of

Service).

The

service

oriented

architecture

followed

by

Grids

(OGSA)

allows

the

definition

of

interfaces

that

are

adaptable

to

different

platforms.

Different

implementations

can

be

encapsulated

by

a

service

interface;

this

virtualisation

allows

the

consistent

access

to

resources

in

heterogeneous

environments

The

virtualisation

of

the

environment

through

service

interfaces

allows

the

use

of

services

without

the

knowledge

of

how

they

are

actually

implemented.

This

characteristic

is

important

for

the

integration

of

IDSs

as

the

same

service

interfaces

can

be

exposed

by

different

IDSs.

Grid

middleware

can

thus

be

used

to

implement

a

great

variety

of

services.

Some

functions

provided

by

Grid

middleware

are

(i)

data

management

services,

including

access

services,

replication,

and

localisation;

(ii)

workflow

services

that

implement

coordinate

execution

of

multiple

applications

on

multiple

resources;

(iii)

auditing

services

that

perform

the

detection

of

frauds

or

intrusions;

(iv)

monitoring

services

which

implement

the

discovery

of

sensors

in

a

distributed

environment

and

generate

alerts

under

determined

conditions;

(v)

services

for

identification

of

problems

in

a

distributed

environment,

which

implement

the

correlation

of

information

from

disparate

and

distributed

logs.

These

services

are

important

for

the

implementation

of

a

DIDSoG.

A

DIDS

needs

services

for

the

location

of

and

access

to

distributed

data

from

different

IDSs.

Auditing

and

monitoring

services

take

care

of

the

proper

needs

of

the

DIDSs

such

as:

secure

storage,

data

analysis

to

detect

intrusions,

discovery

of

distributed

sensors,

and

sending

of

alerts.

The

correlation

of

distributed

logs

is

also

relevant

because

the

detection

of

distributed

attacks

depends

on

the

correlation

of

the

alert

information

generated

by

the

different

IDSs

that

compose

the

DIDSoG.

The

next

sections

of

this

article

are

organized

as

follows.

Section

presents

related

work.

The

proposed

model

is

presented

in

Section

Section

describes

the

development

and

a

case

study.

Results

and

discussion

are

presented

in

Section

Conclusions

and

future

work

are

discussed

in

Section

RELATED

WORK

DIDMA

is

a

flexible,

scalable,

reliable,

and

platformindependent

DIDS.

DIDMA

architecture

allows

distributed

analysis

of

events

and

can

be

easily

extended

by

developing

new

agents.

However,

the

integration

with

existing

IDSs

and

the

development

of

security

components

are

presented

as

future

work

The

extensibility

of

DIDS

DIDMA

and

the

integration

with

other

IDSs

are

goals

pursued

by

DIDSoG.

The

flexibility,

scalability,

platform

independence,

reliability

and

security

components

discussed

in

are

achieved

in

DIDSoG

by

using

a

Grid

platform.

More

efficient

techniques

for

analysis

of

great

amounts

of

data

in

wide

scale

networks

based

on

clustering

and

applicable

to

DIDSs

are

presented

in

The

integration

of

heterogeneous

IDSs

to

increase

the

variety

of

intrusion

detection

techniques

in

the

environment

is

mentioned

as

future

work

DIDSoG

thus

aims

at

integrating

heterogeneous

IDSs

Ref.

presents

a

hierarchical

architecture

for

a

DIDS;

information

is

collected,

aggregated,

correlated

and

analysed

as

it

is

sent

up

in

the

hierarchy.

The

architecture

comprises

of

several

components

for:

monitoring,

correlation,

intrusion

detection

by

statistics,

detection

by

signatures

and

answers.

Components

in

the

same

level

of

the

hierarchy

cooperate

with

one

another.

The

integration

proposed

by

DIDSoG

also

follows

a

hierarchical

architecture.

Each

IDS

integrated

to

the

DIDSoG

offers

functionalities

at

a

given

level

of

the

hierarchy

and

requests

functionalities

from

IDSs

from

another

level.

The

hierarchy

presented

in

integrates

homogeneous

IDSs

whereas

the

hierarchical

architecture

of

DIDSoG

integrates

heterogeneous

IDSs.

There

are

proposals

on

integrating

computational

Grids

and

IDSs

Ref.

and

propose

the

use

of

Globus

Toolkit

for

intrusion

detection,

especially

for

DoS

(Denial

of

Service)

and

DDoS

(Distributed

Denial

of

Service)

attacks;

Globus

is

used

due

to

the

need

to

process

great

amounts

of

data

to

detect

these

kinds

of

attack.

A

two-phase

processing

architecture

is

presented.

The

first

phase

aims

at

the

detection

of

momentary

attacks,

while

the

second

phase

is

concerned

with

chronic

or

perennial

attacks.

Traditional

IDSs

or

DIDSs

are

generally

coordinated

by

a

central

point;

a

characteristic

that

leaves

them

prone

to

attacks.

Leu

et

al.

point

out

that

IDSs

developed

upon

Grids

platforms

are

less

vulnerable

to

attacks

because

of

the

distribution

provided

for

such

platforms.

Leu

et

al.

have

used

tools

to

generate

several

types

of

attacks

including

TCP,

ICMP

and

UDP

flooding

and

have

demonstrated

through

experimental

results

the

advantages

of

applying

computational

Grids

to

IDSs.

This

work

proposes

the

development

of

a

DIDS

upon

a

Grid

platform.

However,

the

resulting

DIDS

integrates

heterogeneous

IDSs

whereas

the

DIDSs

upon

Grids

presented

by

Leu

et

al.

do

not

consider

the

integration

of

heterogeneous

IDSs.

The

processing

in

phases

is

also

contemplated

by

DIDSoG,

which

is

enabled

by

the

specification

of

several

levels

of

processing

allowed

by

the

integration

of

heterogeneous

IDSs.

The

DIDS

GIDA

(Grid

Intrusion

Detection

Architecture)

targets

at

the

detection

of

intrusions

in

a

Grid

environment

GridSim

Grid

simulator

was

used

for

the

validation

of

DIDS

GIDA.

Homogeneous

resources

were

used

to

simplify

the

development

However,

the

possibility

of

applying

heterogeneous

detection

systems

is

left

for

future

work

Another

DIDS

for

Grids

is

presented

by

Choon

and

Samsudim

Scenarios

demonstrating

how

a

DIDS

can

execute

on

a

Grid

environment

are

presented.

DIDSoG

does

not

aim

at

detecting

intrusions

in

a

Grid

environment.

In

contrast,

DIDSoG

uses

the

Grid

to

compose

a

DIDS

by

integrating

specific

IDSs;

the

resulting

DIDS

could

however

be

used

to

identify

attacks

in

a

Grid

environment.

Local

and

distributed

attacks

can

be

detected

through

the

integration

of

traditional

IDSs

while

attacks

particular

to

Grids

can

be

detected

through

the

integration

of

Grid

IDSs.

THE

PROPOSED

MODEL

DIDSoG

presents

a

hierarchy

of

intrusion

detection

services;

this

hierarchy

is

organized

through

a

two-dimensional

vector

defined

by

Scope:Complexity.

The

IDSs

composing

DIDSoG

can

be

organized

in

different

levels

of

scope

or

complexity,

depending

on

its

functionalities,

the

topology

of

the

target

environment

and

expected

results.

Figure

presents

a

DIDSoG

composed

by

different

intrusion

detection

services

(i.e.

data

gathering,

data

aggregation,

data

correlation,

analysis,

intrusion

response

and

management)

provided

by

different

IDSs.

The

information

flow

and

the

relationship

between

the

levels

of

scope

and

complexity

are

presented

in

this

figure.

Information

about

the

environment

(host,

network

or

application)

is

collected

by

Sensors

located

both

in

user

and

user

computers

in

domain

The

information

is

sent

to

both

simple

Analysers

that

act

on

the

information

from

a

single

host

(level

and

to

aggregation

and

correlation

services

that

act

on

information

from

multiple

hosts

from

the

same

domain

(level

Simple

Analysers

in

the

first

scope

level

send

the

information

to

more

complex

Analysers

in

the

next

levels

of

complexity

(level

N).

When

an

Analyser

detects

an

intrusion,

it

communicates

with

Countermeasure

and

Monitoring

services

registered

to

its

scope.

An

Analyser

can

invoke

a

Countermeasure

service

that

replies

to

a

detected

attack,

or

informs

a

Monitoring

service

about

the

ongoing

attack,

so

the

administrator

can

act

accordingly.

Aggregation

and

correlation

resources

in

the

second

scope

receive

information

from

Sensors

from

different

users"

computers

(user

and

user

in

the

domain

These

resources

process

the

received

information

and

send

it

to

the

analysis

resources

registered

to

the

first

level

of

complexity

in

the

second

scope

(level

The

information

is

also

sent

to

the

aggregation

and

correlation

resources

registered

in

the

first

level

of

complexity

in

the

next

scope

(level

User

Domain

Analysers

Level

Local

Sensors

Analysers

Level

Aggreg.

Correlation

Level

User

Domain

Local

Sensors

Analysers

Level

Analysers

Level

Aggreg.

Correlation

Level

Domain

Monitor

Level

Monitor

Level

Analysers

Level

Analysers

Level

Monitor

Level

Response

Level

Response

Level

Response

Level

Fig.

How

DIDSoG

works.

The

analysis

resources

in

the

second

scope

act

like

the

analysis

resources

in

the

first

scope,

directing

the

information

to

a

more

complex

analysis

resource

and

putting

the

Countermeasure

and

Monitoring

resources

in

action

in

case

of

detected

attacks.

Aggregation

and

correlation

resources

in

the

third

scope

receive

information

from

domains

and

These

resources

then

carry

out

the

aggregation

and

correlation

of

the

information

from

different

domains

and

send

it

to

the

analysis

resources

in

the

first

level

of

complexity

in

the

third

scope

(level

The

information

could

also

be

sent

to

the

aggregate

service

in

the

next

scope

in

case

of

any

resources

registered

to

such

level.

The

analysis

resources

in

the

third

scope

act

similar

to

the

analysis

resources

in

the

first

and

second

scopes,

except

that

the

analysis

resources

in

the

third

scope

act

on

information

from

multiple

domains.

The

functionalities

of

the

registered

resources

in

each

of

the

scopes

and

complexity

level

can

vary

from

one

environment

to

another.

The

model

allows

the

development

of

N

levels

of

scope

and

complexity.

Figure

presents

the

architecture

of

a

resource

participating

in

the

DIDSoG.

Initially,

the

resource

registers

itself

to

GIS

(Grid

Information

Service)

so

other

participating

resources

can

query

the

services

provided.

After

registering

itself,

the

resource

requests

information

about

other

intrusion

detection

resources

registered

to

the

GIS.

A

given

resource

of

DIDSoG

interacts

with

other

resources,

by

receiving

data

from

the

Source

Resources,

processing

it,

and

sending

the

results

to

the

Destination

Resources,

therefore

forming

a

grid

of

intrusion

detection

resources.

Grid

Resource

BaseNative

IDS

Grid

Origin

Resources

Grid

Destination

Resources

Grid

Information

Service

Descri

ptor

Connec

tor

Fig.

Architecture

of

a

resource

participating

of

the

DIDSoG.

A

resource

is

made

up

of

four

components:

Base,

Connector,

Descriptor

and

Native

IDS.

Native

IDS

corresponds

to

the

IDS

being

integrated

to

the

DIDSoG.

This

component

process

the

data

received

from

the

Origin

Resources

and

generates

new

data

to

be

sent

to

the

Destination

Resources.

A

Native

IDS

component

can

be

any

tool

processes

information

related

to

intrusion

detection,

including

analysis,

data

gathering,

data

aggregation,

data

correlation,

intrusion

response

or

management.

The

Descriptor

is

responsible

for

the

information

that

identifies

a

resource

and

its

respective

Destination

Resources

in

the

DIDSoG.

Figure

presents

the

class

diagram

of

the

stored

information

by

the

Descriptor.

The

ResourceDescriptor

class

has

Feature,

Level,

DataType

and

Target

Resources

type

members.

Feature

class

represents

the

functionalities

that

a

resource

has.

Type,

name

and

version

attributes

refer

to

the

functions

offered

by

the

Native

IDS

component,

its

name

and

version,

respectively.

Level

class

identifies

the

level

of

target

and

complexity

in

which

the

resource

acts.

DataType

class

represents

the

data

format

that

the

resource

accepts

to

receive.

DataType

class

is

specialized

by

classes

Text,

XML

and

Binary.

Class

XML

contains

the

DTDFile

attribute

to

specify

the

DTD

file

that

validates

the

received

XML.

-ident

-version

-description

ResourceDescriptor

-featureType

-name

-version

Feature

-type

-version

DataType

-escope

-complex

Level

Text

Binary

-DTDFile

XML

TargetResources

-featureType

Fig.

Class

Diagram

of

the

Descriptor

component.

TargetResources

class

represents

the

features

of

the

Destination

Resources

of

a

determined

resource.

This

class

aggregates

Resource.

The

Resource

class

identifies

the

characteristics

of

a

Destination

Resource.

This

identification

is

made

through

the

featureType

attribute

and

the

Level

and

DataType

classes.

A

given

resource

analyses

the

information

from

Descriptors

from

other

resources,

and

compares

this

information

with

the

information

specified

in

TargetResources

to

know

to

which

resources

to

send

the

results

of

its

processing.

The

Base

component

is

responsible

for

the

communication

of

a

resource

with

other

resources

of

the

DIDSoG

and

with

the

Grid

Information

Service.

It

is

this

component

that

registers

the

resource

and

the

queries

other

resources

in

the

GIS.

The

Connector

component

is

the

link

between

Base

and

Native

IDS.

The

information

that

Base

receives

from

Origin

Resources

is

passed

to

Connector

component.

The

Connector

component

performs

the

necessary

changes

in

the

data

so

that

it

is

understood

by

Native

IDS,

and

sends

this

data

to

Native

IDS

for

processing.

The

Connector

component

also

has

the

responsibility

of

collecting

the

information

processed

by

Native

IDS,

and

making

the

necessary

changes

so

the

information

can

pass

through

the

DIDSoG

again.

After

these

changes,

Connector

sends

the

information

to

the

Base,

which

in

turn

sends

it

to

the

Destination

Resources

in

accordance

with

the

specifications

of

the

Descriptor

component.

IMPLEMENTATION

We

have

used

GridSim

Toolkit

for

development

and

evaluation

of

the

proposed

model.

We

have

used

and

extended

GridSim

features

to

model

and

simulate

the

resources

and

components

of

DIDSoG.

Figure

presents

the

Class

diagram

of

the

simulated

DIDSoG.

The

class

starts

the

simulation

components.

The

class

represents

a

user

of

DIDSoG.

This

class"

function

is

to

initiate

the

processing

of

a

resource

Sensor,

from

where

the

gathered

information

will

be

sent

to

other

resources.

keeps

a

registry

of

the

DIDSoG

resources.The

class

implements

the

Base

component

(see

Figure

interacts

with

class,

which

represents

the

Descriptor

component.

The

class

is

created

from

an

XML

file

that

specifies

a

resource

descriptor

(see

Figure

GridInformationService

GridSim

GridResource

Fig.

Class

Diagram

of

the

simulatated

DIDSoG.

A

Connector

component

must

be

developed

for

each

Native

IDS

integrated

to

DIDSoG.

The

Connector

component

is

implemented

by

creating

a

class

derived

from

The

new

class

will

implement

new

functionalities

in

accordance

with

the

needs

of

the

corresponding

Native

IDS.

In

the

simulation

environment,

data

collection

resources,

analysis,

aggregation/correlation

and

generation

of

answers

were

integrated.

Classes

were

developed

to

simulate

the

processing

of

each

Native

IDS

components

associated

to

the

resources.

For

each

simulated

Native

IDS

a

class

derived

from

was

developed.

This

class

corresponds

to

the

Connector

component

of

the

Native

IDS

and

aims

at

the

integrating

the

IDS

to

DIDSoG.

A

XML

file

describing

each

of

the

integrated

resources

is

chosen

by

using

the

Connector

component.

The

resulting

relationship

between

the

resources

integrated

to

the

DIDSoG,

in

accordance

with

the

specification

of

its

respective

descriptors,

is

presented

in

Figure

The

and

resources

generate

simulated

data

in

the

TCPDump

format.

The

generated

data

is

directed

to

and

resources,

in

the

case

of

and

to

in

the

case

of

according

to

the

specification

of

their

descriptors.

Level

Level

Level

Level

TCPDump

TCPDump

TCPDumpAg

TCPDumpAg

IDMEF

IDMEF

IDMEF

TCPDump

Level

Level

Fig.

Flow

of

the

execution

of

the

simulation.

The

Native

IDS

of

generates

alerts

for

any

attempt

of

connection

to

port

The

data

received

from

had

presented

such

features,

generating

an

IDMEF

(Intrusion

Detection

Message

Exchange

Format)

alert

The

generated

alert

was

sent

to

resource,

where

a

warning

was

dispatched

to

the

administrator

informing

him

of

the

alert

received.

The

resource

received

the

information

generated

by

sensors

and

Its

processing

activities

consist

in

correlating

the

source

IP

addresses

with

the

received

data.

The

resultant

information

of

the

processing

of

was

directed

to

the

resource.

The

Native

IDS

component

of

the

generates

alerts

when

a

source

tries

to

connect

to

the

same

port

number

of

multiple

destinations.

This

situation

is

identified

by

the

in

the

data

received

from

and

an

alert

in

IDMEF

format

is

then

sent

to

the

resource.

In

addition

to

generating

alerts

in

IDMEF

format,

also

directs

the

received

data

to

the

in

the

level

of

complexity

The

Native

IDS

component

of

generates

alerts

when

the

transmission

of

ICMP

messages

from

a

given

source

to

multiple

destinations

is

detected.

This

situation

is

detected

in

the

data

received

from

and

an

IDMEF

alert

is

then

sent

to

the

resource.

The

resource

receives

the

alerts

generated

by

analysers

and

in

accordance

with

the

implementation

of

its

Native

IDS

component.

Warnings

on

alerts

received

are

dispatched

to

the

administrator.

The

simulation

carried

out

demonstrates

how

DIDSoG

works.

Simulated

data

was

generated

to

be

the

input

for

a

grid

of

intrusion

detection

systems

composed

by

several

distinct

resources.

The

resources

carry

out

tasks

such

as

data

collection,

aggregation

and

analysis,

and

generation

of

alerts

and

warnings

in

an

integrated

manner.

EXPERIMENT

RESULTS

The

hierarchic

organization

of

scope

and

complexity

provides

a

high

degree

of

flexibility

to

the

model.

The

DIDSoG

can

be

modelled

in

accordance

with

the

needs

of

each

environment.

The

descriptors

define

data

flow

desired

for

the

resulting

DIDS.

Each

Native

IDS

is

integrated

to

the

DIDSoG

through

a

Connector

component.

The

Connector

component

is

also

flexible

in

the

DIDSoG.

Adaptations,

conversions

of

data

types

and

auxiliary

processes

that

Native

IDSs

need

are

provided

by

the

Connector.

Filters

and

generation

of

Specific

logs

for

each

Native

IDS

or

environment

can

also

be

incorporated

to

the

Connector.

If

the

integration

of

a

new

IDS

to

an

environment

already

configured

is

desired,

it

is

enough

to

develop

the

Connector

for

the

desired

IDS

and

to

specify

the

resource

Descriptor.

After

the

specification

of

the

Connector

and

the

Descriptor

the

new

IDS

is

integrated

to

the

DIDSoG.

Through

the

definition

of

scopes,

resources

can

act

on

data

of

different

source

groups.

For

example,

scope

can

be

related

to

a

given

set

of

hosts,

scope

to

another

set

of

hosts,

while

scope

can

be

related

to

hosts

from

scopes

and

Scopes

can

be

defined

according

to

the

needs

of

each

environment.

The

complexity

levels

allow

the

distribution

of

the

processing

between

several

resources

inside

the

same

scope.

In

an

analysis

task,

for

example,

the

search

for

simple

attacks

can

be

made

by

resources

of

complexity

whereas

the

search

for

more

complex

attacks,

that

demands

more

time,

can

be

performed

by

resources

of

complexity

With

this,

the

analysis

of

the

data

is

made

by

two

resources.

The

distinction

between

complexity

levels

can

also

be

organized

in

order

to

integrate

different

techniques

of

intrusion

detection.

The

complexity

level

could

be

defined

for

analyses

based

on

signatures,

which

are

simpler

techniques;

the

complexity

level

for

techniques

based

on

behaviour,

that

require

greater

computational

power;

and

the

complexity

level

for

intrusion

detection

in

applications,

where

the

techniques

are

more

specific

and

depend

on

more

data.

The

division

of

scopes

and

the

complexity

levels

make

the

processing

of

the

data

to

be

carried

out

in

phases.

No

resource

has

full

knowledge

about

the

complete

data

processing

flow.

Each

resource

only

knows

the

results

of

its

processing

and

the

destination

to

which

it

sends

the

results.

Resources

of

higher

complexity

must

be

linked

to

resources

of

lower

complexity.

Therefore,

the

hierarchic

structure

of

the

DIDSoG

is

maintained,

facilitating

its

extension

and

integration

with

other

domains

of

intrusion

detection.

By

carrying

out

a

hierarchic

relationship

between

the

several

chosen

analysers

for

an

environment,

the

sensor

resource

is

not

overloaded

with

the

task

to

send

the

data

to

all

the

analysers.

An

initial

analyser

will

exist

(complexity

level

to

which

the

sensor

will

send

its

data,

and

this

analyser

will

then

direct

the

data

to

the

next

step

of

the

processing

flow.

Another

feature

of

the

hierarchical

organization

is

the

easy

extension

and

integration

with

other

domains.

If

it

is

necessary

to

add

a

new

host

(sensor)

to

the

DIDSoG,

it

is

enough

to

plug

it

to

the

first

hierarchy

of

resources.

If

it

is

necessary

to

add

a

new

analyser,

it

will

be

in

the

scope

of

several

domains,

it

is

enough

to

relate

it

to

another

resource

of

same

scope.

The

DIDSoG

allows

different

levels

to

be

managed

by

different

entities.

For

example,

the

first

scope

can

be

managed

by

the

local

user

of

a

host.

The

second

scope,

comprising

several

hosts

of

a

domain

can

be

managed

by

the

administrator

of

the

domain.

A

third

entity

can

be

responsible

for

managing

the

security

of

several

domains

in

a

joint

way.

This

entity

can

act

in

the

scope

independently

from

others.

With

the

proposed

model

for

integration

of

IDSs

in

Grids,

the

different

IDSs

of

an

environment

(or

multiple

IDSs

integrated)

act

in

a

cooperative

manner

improving

the

intrusion

detection

services,

mainly

in

two

aspects.

First,

the

information

from

multiple

sources

are

analysed

in

an

integrated

way

to

search

for

distributed

attacks.

This

integration

can

be

made

under

several

scopes.

Second,

there

is

a

great

diversity

of

data

aggregation

techniques,

data

correlation

and

analysis,

and

intrusion

response

that

can

be

applied

to

the

same

environment;

these

techniques

can

be

organized

under

several

levels

of

complexity.

CONCLUSION

The

integration

of

heterogeneous

IDSs

is

important.

However,

the

incompatibility

and

diversity

of

IDS

solutions

make

such

integration

extremely

difficult.

This

work

thus

proposed

a

model

for

composition

of

DIDS

by

integrating

existing

IDSs

on

a

computational

Grid

platform

(DIDSoG).

IDSs

in

DIDSoG

are

encapsulated

as

Grid

services

for

intrusion

detection.

A

computational

Grid

platform

is

used

for

the

integration

by

providing

the

basic

requirements

for

communication,

localization,

resource

sharing

and

security

mechanisms.

The

components

of

the

architecture

of

the

DIDSoG

were

developed

and

evaluated

using

the

GridSim

Grid

simulator.

Services

for

communication

and

localization

were

used

to

carry

out

the

integration

between

components

of

different

resources.

Based

on

the

components

of

the

architecture,

several

resources

were

modelled

forming

a

grid

of

intrusion

detection.

The

simulation

demonstrated

the

usefulness

of

the

proposed

model.

Data

from

the

sensor

resources

was

read

and

this

data

was

used

to

feed

other

resources

of

DIDSoG.

The

integration

of

distinct

IDSs

could

be

observed

through

the

simulated

environment.

Resources

providing

different

intrusion

detection

services

were

integrated

(e.g.

analysis,

correlation,

aggregation

and

alert).

The

communication

and

localization

services

provided

by

GridSim

were

used

to

integrate

components

of

different

resources.

Various

resources

were

modelled

following

the

architecture

components

forming

a

grid

of

intrusion

detection.

The

components

of

DIDSoG

architecture

have

served

as

base

for

the

integration

of

the

resources

presented

in

the

simulation.

During

the

simulation,

the

different

IDSs

cooperated

with

one

another

in

a

distributed

manner;

however,

in

a

coordinated

way

with

an

integrated

view

of

the

events,

having,

thus,

the

capability

to

detect

distributed

attacks.

This

capability

demonstrates

that

the

IDSs

integrated

have

resulted

in

a

DIDS.

Related

work

presents

cooperation

between

components

of

a

specific

DIDS.

Some

work

focus

on

either

the

development

of

DIDSs

on

computational

Grids

or

the

application

of

IDSs

to

computational

Grids.

However,

deals

with

the

integration

of

heterogeneous

IDSs.

In

contrast,

the

proposed

model

developed

and

simulated

in

this

work,

can

shed

some

light

into

the

question

of

integration

of

heterogeneous

IDSs.

DIDSoG

presents

new

research

opportunities

that

we

would

like

to

pursue,

including:

deployment

of

the

model

in

a

more

realistic

environment

such

as

a

Grid;

incorporation

of

new

security

services;

parallel

analysis

of

data

by

Native

IDSs

in

multiple

hosts.

In

addition

to

the

integration

of

IDSs

enabled

by

a

grid

middleware,

the

cooperation

of

heterogeneous

IDSs

can

be

viewed

as

an

economic

problem.

IDSs

from

different

organizations

or

administrative

domains

need

incentives

for

joining

a

grid

of

intrusion

detection

services

and

for

collaborating

with

other

IDSs.

The

development

of

distributed

strategy

proof

mechanisms

for

integration

of

IDSs

is

a

challenge

that

we

would

like

to

tackle.

REFERENCES

Sulistio,

A.;

Poduvaly,

G.;

Buyya,

R;

and

Tham,

CK,

Constructing

A

Grid

Simulation

with

Differentiated

Network

Service

Using

GridSim,

Proc.

of

the

International

Conference

on

Internet

Computing

June

Las

Vegas,

USA.

Choon,

O.

T.;

Samsudim,

A.

Grid-based

Intrusion

Detection

System.

The

IEEE

Asia-Pacific

Conference

Communications,

September

Foster,

I.;

Kesselman,

C.;

Tuecke,

S.

The

Physiology

of

the

Grid:

An

Open

Grid

Service

Architecture

for

Distributed

System

Integration.

Draft

June

Available

at

Access

Feb.

Foster,

Ian;

Kesselman,

Carl;

Tuecke,

Steven.

The

anatomy

of

the

Grid:

enabling

scalable

virtual

organizations.

International

Journal

of

Supercomputer

Applications,

Kannadiga,

P.;

Zulkernine,

M.

DIDMA:

A

Distributed

Intrusion

Detection

System

Using

Mobile

Agents.

Proceedings

of

the

IEEE

Sixth

International

Conference

on

Software

Engineering,

Artificial

Intelligence,

Networking

and

Parallel/Distributed

Computing,

May

Leu,

Fang-Yie,

et

al.

Integrating

Grid

with

Intrusion

Detection.

Proceedings

of

IEEE

March

Leu,

Fang-Yie,

et

al.

A

Performance-Based

Grid

Intrusion

Detection

System.

Proceedings

of

the

IEEE

July

McCanne,

S.;

Leres,

C.;

Jacobson,

V.;

TCPdump/Libpcap,

Snapp,

S.

R.

et

al.

DIDS

(Distributed

Intrusion

Detection

System)

Motivation,

Architecture

and

An

Early

Prototype.

Proceeding

of

the

Fifteenth

IEEE

National

Computer

Security

Conference.

Baltimore,

MD,

October

Sterne,

D.;

et

al.

A

General

Cooperative

Intrusion

Detection

Architecture

for

MANETs.

Proceedings

of

the

Third

IEEE

March

Tolba,

M.

F.

et

al.

GIDA:

Toward

Enabling

Grid

Intrusion

Detection

Systems.

IEEE

International

Symposium

on

Cluster

Computing

and

the

Grid,

May

Wood,

M.

Intrusion

Detection

message

exchange

requirements.

October

Available

at

Access

March

Zhang,

Yu-Fang;

Xiong,

Z.;

Wang,

X.

Distributed

Intrusion

Detection

Based

on

Clustering.

Proceedings

of

IEEE

International

Conference

Machine

Learning

and

Cybernetics,

August

Curry,

D.;

Debar,

H.

Intrusion

Detection

Message

exchange

format

data

model

and

Extensible

Markup

Language

(XML)

Document

Type

Definition.

March

Available

at

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O B O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O B O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B O B I I I O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

heterogeneous intrusion detection system

distributed intrusion detection system
grid middleware
open grid service architecture
computational grid
grid intrusion detection architecture
gridsim grid simulator
intrusion detection system
grid
system integration


A New Approach for Evaluating Query Expansion: Query-Document Term Mismatch Tonya Custis Thomson Corporation 610 Opperman Drive St. Paul, MN tonya.custis@thomson.com Khalid Al-Kofahi Thomson Corporation 610 Opperman Drive St. Paul, MN khalid.al-kofahi@thomson.com ABSTRACT The effectiveness of information retrieval (IR) systems is influenced by the degree of term overlap between user queries and relevant documents. Query-document term mismatch, whether partial or total, is a fact that must be dealt with by IR systems. Query Expansion (QE) is one method for dealing with term mismatch. IR systems implementing query expansion are typically evaluated by executing each query twice, with and without query expansion, and then comparing the two result sets. While this measures an overall change in performance, it does not directly measure the effectiveness of IR systems in overcoming the inherent issue of term mismatch between the query and relevant documents, nor does it provide any insight into how such systems would behave in the presence of query-document term mismatch. In this paper, we propose a new approach for evaluating query expansion techniques. The proposed approach is attractive because it provides an estimate of system performance under varying degrees of query-document term mismatch, it makes use of readily available test collections, and it does not require any additional relevance judgments or any form of manual processing. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval General Terms Measurement, Experimentation 1. INTRODUCTION In our domain,1 and unlike web search, it is very important for attorneys to find all documents (e.g., cases) that are relevant to an issue. Missing relevant documents may have non-trivial consequences on the outcome of a court proceeding. Attorneys are especially concerned about missing relevant documents when researching a legal topic that is new to them, as they may not be aware of all language variations in such topics. Therefore, it is important to develop information retrieval systems that are robust with respect to language variations or term mismatch between queries and relevant documents. During our work on developing such systems, we concluded that current evaluation methods are not sufficient for this purpose. {Whooping cough, pertussis}, {heart attack, myocardial infarction}, {car wash, automobile cleaning}, {attorney, legal counsel, lawyer} are all examples of things that share the same meaning. Often, the terms chosen by users in their queries are different than those appearing in the documents relevant to their information needs. This query-document term mismatch arises from two sources: (1) the synonymy found in natural language, both at the term and the phrasal level, and (2) the degree to which the user is an expert at searching and/or has expert knowledge in the domain of the collection being searched. IR evaluations are comparative in nature (cf. TREC). Generally, IR evaluations show how System A did in relation to System B on the same test collection based on various precision- and recall-based metrics. Similarly, IR systems with QE capabilities are typically evaluated by executing each search twice, once with and once without query expansion, and then comparing the two result sets. While this approach shows which system may have performed better overall with respect to a particular test collection, it does not directly or systematically measure the effectiveness of IR systems in overcoming query-document term mismatch. If the goal of QE is to increase search performance by mitigating the effects of query-document term mismatch, then the degree to which a system does so should be measurable in evaluation. An effective evaluation method should measure the performance of IR systems under varying degrees of query-document term mismatch, not just in terms of overall performance on a collection relative to another system. 1 Thomson Corporation builds information based solutions to the professional markets including legal, financial, health care, scientific, and tax and accounting. In order to measure that a particular IR system is able to overcome query-document term mismatch by retrieving documents that are relevant to a user"s query, but that do not necessarily contain the query terms themselves, we systematically introduce term mismatch into the test collection by removing query terms from known relevant documents. Because we are purposely inducing term mismatch between the queries and known relevant documents in our test collections, the proposed evaluation framework is able to measure the effectiveness of QE in a way that testing on the whole collection is not. If a QE search method finds a document that is known to be relevant but that is nonetheless missing query terms, it shows that QE technique is indeed robust with respect to query-document term mismatch. 2. RELATED WORK Accounting for term mismatch between the terms in user queries and the documents relevant to users" information needs has been a fundamental issue in IR research for almost 40 years [38, 37, 47]. Query expansion (QE) is one technique used in IR to improve search performance by increasing the likelihood of term overlap (either explicitly or implicitly) between queries and documents that are relevant to users" information needs. Explicit query expansion occurs at run-time, based on the initial search results, as is the case with relevance feedback and pseudo relevance feedback [34, 37]. Implicit query expansion can be based on statistical properties of the document collection, or it may rely on external knowledge sources such as a thesaurus or an ontology [32, 17, 26, 50, 51, 2]. Regardless of method, QE algorithms that are capable of retrieving relevant documents despite partial or total term mismatch between queries and relevant documents should increase the recall of IR systems (by retrieving documents that would have previously been missed) as well as their precision (by retrieving more relevant documents). In practice, QE tends to improve the average overall retrieval performance, doing so by improving performance on some queries while making it worse on others. QE techniques are judged as effective in the case that they help more than they hurt overall on a particular collection [47, 45, 41, 27]. Often, the expansion terms added to a query in the query expansion phase end up hurting the overall retrieval performance because they introduce semantic noise, causing the meaning of the query to drift. As such, much work has been done with respect to different strategies for choosing semantically relevant QE terms to include in order to avoid query drift [34, 50, 51, 18, 24, 29, 30, 32, 3, 4, 5]. The evaluation of IR systems has received much attention in the research community, both in terms of developing test collections for the evaluation of different systems [11, 12, 13, 43] and in terms of the utility of evaluation metrics such as recall, precision, mean average precision, precision at rank, Bpref, etc. [7, 8, 44, 14]. In addition, there have been comparative evaluations of different QE techniques on various test collections [47, 45, 41]. In addition, the IR research community has given attention to differences between the performance of individual queries. Research efforts have been made to predict which queries will be improved by QE and then selectively applying it only to those queries [1, 5, 27, 29, 15, 48], to achieve optimal overall performance. In addition, related work on predicting query difficulty, or which queries are likely to perform poorly, has been done [1, 4, 5, 9]. There is general interest in the research community to improve the robustness of IR systems by improving retrieval performance on difficult queries, as is evidenced by the Robust track in the TREC competitions and new evaluation measures such as GMAP. GMAP (geometric mean average precision) gives more weight to the lower end of the average precision (as opposed to MAP), thereby emphasizing the degree to which difficult or poorly performing queries contribute to the score [33]. However, no attention is given to evaluating the robustness of IR systems implementing QE with respect to querydocument term mismatch in quantifiable terms. By purposely inducing mismatch between the terms in queries and relevant documents, our evaluation framework allows us a controlled manner in which to degrade the quality of the queries with respect to their relevant documents, and then to measure the both the degree of (induced) difficulty of the query and the degree to which QE improves the retrieval performance of the degraded query. The work most similar to our own in the literature consists of work in which document collections or queries are altered in a systematic way to measure differences query performance. [42] introduces into the document collection pseudowords that are ambiguous with respect to word sense, in order to measure the degree to which word sense disambiguation is useful in IR. [6] experiments with altering the document collection by adding semantically related expansion terms to documents at indexing time. In cross-language IR, [28] explores different query expansion techniques while purposely degrading their translation resources, in what amounts to expanding a query with only a controlled percentage of its translation terms. Although similar in introducing a controlled amount of variance into their test collections, these works differ from the work being presented in this paper in that the work being presented here explicitly and systematically measures query effectiveness in the presence of query-document term mismatch. 3. METHODOLOGY In order to accurately measure IR system performance in the presence of query-term mismatch, we need to be able to adjust the degree of term mismatch in a test corpus in a principled manner. Our approach is to introduce querydocument term mismatch into a corpus in a controlled manner and then measure the performance of IR systems as the degree of term mismatch changes. We systematically remove query terms from known relevant documents, creating alternate versions of a test collection that differ only in how many or which query terms have been removed from the documents relevant to a particular query. Introducing query-document term mismatch into the test collection in this manner allows us to manipulate the degree of term mismatch between relevant documents and queries in a controlled manner. This removal process affects only the relevant documents in the search collection. The queries themselves remain unaltered. Query terms are removed from documents one by one, so the differences in IR system performance can be measured with respect to missing terms. In the most extreme case (i.e., when the length of the query is less than or equal to the number of query terms removed from the relevant documents), there will be no term overlap between a query and its relevant documents. Notice that, for a given query, only relevant documents are modified. Non-relevant documents are left unchanged, even in the case that they contain query terms. Although, on the surface, we are changing the distribution of terms between the relevant and non-relevant documents sets by removing query terms from the relevant documents, doing so does not change the conceptual relevancy of these documents. Systematically removing query terms from known relevant documents introduces a controlled amount of query-document term mismatch by which we can evaluate the degree to which particular QE techniques are able to retrieve conceptually relevant documents, despite a lack of actual term overlap. Removing a query term from relevant documents simply masks the presence of that query term in those documents. It does not in any way change the conceptual relevancy of the documents. The evaluation framework presented in this paper consists of three elements: a test collection, C; a strategy for selecting which query terms to remove from the relevant documents in that collection, S; and a metric by which to compare performance of the IR systems, m. The test collection, C, consists of a document collection, queries, and relevance judgments. The strategy, S, determines the order and manner in which query terms are removed from the relevant documents in C. This evaluation framework is not metric-specific; any metric (MAP, P@10, recall, etc.) can be used to measure IR system performance. Although test collections are difficult to come by, it should be noted that this evaluation framework can be used on any available test collection. In fact, using this framework stretches the value of existing test collections in that one collection becomes several when query terms are removed from relevant documents, thereby increasing the amount of information that can be gained from evaluating on a particular collection. In other evaluations of QE effectiveness, the controlled variable is simply whether or not queries have been expanded or not, compared in terms of some metric. In contrast, the controlled variable in this framework is the query term that has been removed from the documents relevant to that query, as determined by the removal strategy, S. Query terms are removed one by one, in a manner and order determined by S, so that collections differ only with respect to the one term that has been removed (or masked) in the documents relevant to that query. It is in this way that we can explicitly measure the degree to which an IR system overcomes query-document term mismatch. The choice of a query term removal strategy is relatively flexible; the only restriction in choosing a strategy S is that query terms must be removed one at a time. Two decisions must be made when choosing a removal strategy S. The first is the order in which S removes terms from the relevant documents. Possible orders for removal could be based on metrics such as IDF or the global probability of a term in a document collection. Based on the purpose of the evaluation and the retrieval algorithm being used, it might make more sense to choose a removal order for S based on query term IDF or perhaps based on a measure of query term probability in the document collection. Once an order for removal has been decided, a manner for term removal/masking must be decided. It must be determined if S will remove the terms individually (i.e., remove just one different term each time) or additively (i.e., remove one term first, then that term in addition to another, and so on). The incremental additive removal of query terms from relevant documents allows the evaluation to show the degree to which IR system performance degrades as more and more query terms are missing, thereby increasing the degree of query-document term mismatch. Removing terms individually allows for a clear comparison of the contribution of QE in the absence of each individual query term. 4. EXPERIMENTAL SET-UP 4.1 IR Systems We used the proposed evaluation framework to evaluate four IR systems on two test collections. Of the four systems used in the evaluation, two implement query expansion techniques: Okapi (with pseudo-feedback for QE), and a proprietary concept search engine (we"ll call it TCS, for Thomson Concept Search). TCS is a language modeling based retrieval engine that utilizes a subject-appropriate external corpus (i.e., legal or news) as a knowledge source. This external knowledge source is a corpus separate from, but thematically related to, the document collection to be searched. Translation probabilities for QE [2] are calculated from these large external corpora. Okapi (without feedback) and a language model query likelihood (QL) model (implemented using Indri) are included as keyword-only baselines. Okapi without feedback is intended as an analogous baseline for Okapi with feedback, and the QL model is intended as an appropriate baseline for TCS, as they both implement language-modeling based retrieval algorithms. We choose these as baselines because they are dependent only on the words appearing in the queries and have no QE capabilities. As a result, we expect that when query terms are removed from relevant documents, the performance of these systems should degrade more dramatically than their counterparts that implement QE. The Okapi and QL model results were obtained using the Lemur Toolkit.2 Okapi was run with the parameters k1=1.2, b=0.75, and k3=7. When run with feedback, the feedback parameters used in Okapi were set at 10 documents and 25 terms. The QL model used Jelinek-Mercer smoothing, with λ = 0.6. 4.2 Test Collections We evaluated the performance of the four IR systems outlined above on two different test collections. The two test collections used were the TREC AP89 collection (TIPSTER disk 1) and the FSupp Collection. The FSupp Collection is a proprietary collection of 11,953 case law documents for which we have 44 queries (ranging from four to twenty-two words after stop word removal) with full relevance judgments.3 The average length of documents in the FSupp Collection is 3444 words. 2 www.lemurproject.org 3 Each of the 11,953 documents was evaluated by domain experts with respect to each of the 44 queries. The TREC AP89 test collection contains 84,678 documents, averaging 252 words in length. In our evaluation, we used both the title and the description fields of topics 151200 as queries, so we have two sets of results for the AP89 Collection. After stop word removal, the title queries range from two to eleven words and the description queries range from four to twenty-six terms. 4.3 Query Term Removal Strategy In our experiments, we chose to sequentially and additively remove query terms from highest-to-lowest inverse document frequency (IDF) with respect to the entire document collection. Terms with high IDF values tend to influence document ranking more than those with lower IDF values. Additionally, high IDF terms tend to be domainspecific terms that are less likely to be known to non-expert user, hence we start by removing these first. For the FSupp Collection, queries were evaluated incrementally with one, two, three, five, and seven terms removed from their corresponding relevant documents. The longer description queries from TREC topics 151-200 were likewise evaluated on the AP89 Collection with one, two, three, five, and seven query terms removed from their relevant documents. For the shorter TREC title queries, we removed one, two, three, and five terms from the relevant documents. 4.4 Metrics In this implementation of the evaluation framework, we chose three metrics by which to compare IR system performance: mean average precision (MAP), precision at 10 documents (P10), and recall at 1000 documents. Although these are the metrics we chose to demonstrate this framework, any appropriate IR metrics could be used within the framework. 5. RESULTS 5.1 FSupp Collection Figures 1, 2, and 3 show the performance (in terms of MAP, P10 and Recall, respectively) for the four search engines on the FSupp Collection. As expected, the performance of the keyword-only IR systems, QL and Okapi, drops quickly as query terms are removed from the relevant documents in the collection. The performance of Okapi with feedback (Okapi FB) is somewhat surprising in that on the original collection (i.e., prior to query term removal), its performance is worse than that of Okapi without feedback on all three measures. TCS outperforms the QL keyword baseline on every measure except for MAP on the original collection (i.e., prior to removing any query terms). Because TCS employs implicit query expansion using an external domain specific knowledge base, it is less sensitive to term removal (i.e., mismatch) than the Okapi FB, which relies on terms from the top-ranked documents retrieved by an initial keywordonly search. Because overall search engine performance is frequently measured in terms of MAP, and because other evaluations of QE often only consider performance on the entire collection (i.e., they do not consider term mismatch), the QE implemented in TCS would be considered (in an0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 MeanAveragePrecision(MAP) Okapi FB Okapi TCS QL FSupp: Mean Average Precision with Query Terms Removed Figure 1: The performance of the four retrieval systems on the FSupp collection in terms of Mean Average Precision (MAP) and as a function of the number of query terms removed (the horizontal axis). 0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 Precisionat10Documents(P10) Okapi FB Okapi TCS QL FSupp: P10 with Query Terms Removed Figure 2: The performance of the four retrieval systems on the FSupp collection in terms of Precision at 10 and as a function of the number of query terms removed (the horizontal axis). other evaluation) to hurt performance on the FSupp Collection. However, when we look at the comparison of TCS to QL when query terms are removed from the relevant documents, we can see that the QE in TCS is indeed contributing positively to the search. 5.2 The AP89 Collection: using the description queries Figures 4, 5, and 6 show the performance of the four IR systems on the AP89 Collection, using the TREC topic descriptions as queries. The most interesting difference between the performance on the FSupp Collection and the AP89 collection is the reversal of Okapi FB and TCS. On FSupp, TCS outperformed the other engines consistently (see Figures 1, 2, and 3); on the AP89 Collection, Okapi FB is clearly the best performer (see Figures 4, 5, and 6). This is all the more interesting, based on the fact that QE in Okapi FB takes place after the first search iteration, which 0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall Okapi FB Okapi TCS Indri FSupp: Recall at 1000 documents with Query Terms Removed Figure 3: The Recall (at 1000) of the four retrieval systems on the FSupp collection as a function of the number of query terms removed (the horizontal axis). 0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 MeanAveragePrecision(MAP) Okapi FB Okapi TCS QL AP89: Mean Average Precision with Query Terms Removed (description queries) Figure 4: MAP of the four IR systems on the AP89 Collection, using TREC description queries. MAP is measured as a function of the number of query terms removed. 0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 Precisionat10Documents(P10) Okapi FB Okapi TCS QL AP89: P10 with Query Terms Removed (description queries) Figure 5: Precision at 10 of the four IR systems on the AP89 Collection, using TREC description queries. P at 10 is measured as a function of the number of query terms removed. 0 1 2 3 4 5 6 7 Number of Query Terms Removed from Relevant Documents 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Recall Okapi FB Okapi TCS QL AP89: Recall at 1000 documents with Query Terms Removed (description queries) Figure 6: Recall (at 1000) of the four IR systems on the AP89 Collection, using TREC description queries, and as a function of the number of query terms removed. we would expect to be handicapped when query terms are removed. Looking at P10 in Figure 5, we can see that TCS and Okapi FB score similarly on P10, starting at the point where one query term is removed from relevant documents. At two query terms removed, TCS starts outperforming Okapi FB. If modeling this in terms of expert versus non-expert users, we could conclude that TCS might be a better search engine for non-experts to use on the AP89 Collection, while Okapi FB would be best for an expert searcher. It is interesting to note that on each metric for the AP89 description queries, TCS performs more poorly than all the other systems on the original collection, but quickly surpasses the baseline systems and approaches Okapi FB"s performance as terms are removed. This is again a case where the performance of a system on the entire collection is not necessarily indicative of how it handles query-document term mismatch. 5.3 The AP89 Collection: using the title queries Figures 7, 8, and 9 show the performance of the four IR systems on the AP89 Collection, using the TREC topic titles as queries. As with the AP89 description queries, Okapi FB is again the best performer of the four systems in the evaluation. As before, the performance of the Okapi and QL systems, the non-QE baseline systems, sharply degrades as query terms are removed. On the shorter queries, TCS seems to have a harder time catching up to the performance of Okapi FB as terms are removed. Perhaps the most interesting result from our evaluation is that although the keyword-only baselines performed consistently and as expected on both collections with respect to query term removal from relevant documents, the performances of the engines implementing QE techniques differed dramatically between collections. 0 1 2 3 4 5 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 MeanAveragePrecision(MAP) Okapi FB Okapi TCS QL AP89: Mean Average Precision with Query Terms Removed (title queries) Figure 7: MAP of the four IR systems on the AP89 Collection, using TREC title queries and as a function of the number of query terms removed. 0 1 2 3 4 5 Number of Query Terms Removed from Relevant Documents 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Precisionat10Documents(P10) Okapi FB Okapi TCS QL AP89: P10 with Query Terms Removed (title queries) Figure 8: Precision at 10 of the four IR systems on the AP89 Collection, using TREC title queries, and as a function of the number of query terms removed. 0 1 2 3 4 5 Number of Query Terms Removed from Relevant Documents 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Recall Okapi FB Okapi TCS QL AP89: Recall at 1000 documents with Query Terms Removed (title queries) Figure 9: Recall (at 1000) of the four IR systems on the AP89 Collection, using TREC title queries and as a function of the number of query terms removed. 6. DISCUSSION The intuition behind this evaluation framework is to measure the degree to which various QE techniques overcome term mismatch between queries and relevant documents. In general, it is easy to evaluate the overall performance of different techniques for QE in comparison to each other or against a non-QE variant on any complete test collection. Such an approach does tell us which systems perform better on a complete test collection, but it does not measure the ability of a particular QE technique to retrieve relevant documents despite partial or complete term mismatch between queries and relevant documents. A systematic evaluation of IR systems as outlined in this paper is useful not only with respect to measuring the general success or failure of particular QE techniques in the presence of query-document term mismatch, but it also provides insight into how a particular IR system will perform when used by expert versus non-expert users on a particular collection. The less a user knows about the domain of the document collection on which they are searching, the more prevalent query-document term mismatch is likely to be. This distinction is especially relevant in the case that the test collection is domain-specific (i.e., medical or legal, as opposed to a more general domain, such as news), where the distinction between experts and non-experts may be more marked. For example, a non-expert in the medical domain might search for whooping cough, but relevant documents might instead contain the medical term pertussis. Since query terms are masked only the in relevant documents, this evaluation framework is actually biased against retrieving relevant documents. This is because non-relevant documents may also contain query terms, which can cause a retrieval system to rank such documents higher than it would have before terms were masked in relevant documents. Still, we think this is a more realistic scenario than removing terms from all documents regardless of relevance. The degree to which a QE technique is well-suited to a particular collection can be evaluated in terms of its ability to still find the relevant documents, even when they are missing query terms, despite the bias of this approach against relevant documents. However, given that Okapi FB and TCS outperformed each other on two different collection sets, further investigation into the degree of compatibility between QE expansion approach and target collection is probably warranted. Furthermore, the investigation of other term removal strategies could provide insight into the behavior of different QE techniques and their overall impact on the user experience. As mentioned earlier, our choice of the term removal strategy was motivated by (1) our desire to see the highest impact on system performance as terms are removed and (2) because high IDF terms, in our domain context, are more likely to be domain specific, which allows us to better understand the performance of an IR system as experienced by expert and non-expert users. Although not attempted in our experiments, another application of this evaluation framework would be to remove query terms individually, rather than incrementally, to analyze which terms (or possibly which types of terms) are being helped most by a QE technique on a particular test collection. This could lead to insight as to when QE should and should not be applied. This evaluation framework allows us to see how IR systems perform in the presence of query-document term mismatch. In other evaluations, the performance of a system is measured only on the entire collection, in which the degree of query-term document mismatch is not known. By systematically introducing this mismatch, we can see that even if an IR system is not the best performer on the entire collection, its performance may nonetheless be more robust to query-document term mismatch than other systems. Such robustness makes a system more user-friendly, especially to non-expert users. This paper presents a novel framework for IR system evaluation, the applications of which are numerous. The results presented in this paper are not by any means meant to be exhaustive or entirely representative of the ways in which this evaluation could be applied. To be sure, there is much future work that could be done using this framework. In addition to looking at average performance of IR systems, the results of individual queries could be examined and compared more closely, perhaps giving more insight into the classification and prediction of difficult queries, or perhaps showing which QE techniques improve (or degrade) individual query performance under differing degrees of querydocument term mismatch. Indeed, this framework would also benefit from further testing on a larger collection. 7. CONCLUSION The proposed evaluation framework allows us to measure the degree to which different IR systems overcome (or don"t overcome) term mismatch between queries and relevant documents. Evaluations of IR systems employing QE performed only on the entire collection do not take into account that the purpose of QE is to mitigate the effects of term mismatch in retrieval. By systematically removing query terms from relevant documents, we can measure the degree to which QE contributes to a search by showing the difference between the performances of a QE system and its keywordonly baseline when query terms have been removed from known relevant documents. Further, we can model the behavior of expert versus non-expert users by manipulating the amount of query-document term mismatch introduced into the collection. The evaluation framework proposed in this paper is attractive for several reasons. Most importantly, it provides a controlled manner in which to measure the performance of QE with respect to query-document term mismatch. In addition, this framework takes advantage and stretches the amount of information we can get from existing test collections. Further, this evaluation framework is not metricspecific: information in terms of any metric (MAP, P@10, etc.) can be gained from evaluating an IR system this way. It should also be noted that this framework is generalizable to any IR system, in that it evaluates how well IR systems evaluate users" information needs as represented by their queries. An IR system that is easy to use should be good at retrieving documents that are relevant to users" information needs, even if the queries provided by the users do not contain the same keywords as the relevant documents. 8. REFERENCES [1] Amati, G., C. Carpineto, and G. Romano. Query difficulty, robustness and selective application of query expansion. In Proceedings of the 25th European Conference on Information Retrieval (ECIR 2004), pp. 127-137. [2] Berger, A. and J.D. Lafferty. 1999. Information retrieval as statistical translation. In Research and Development in Information Retrieval, pages 222-229. [3] Billerbeck, B., F. Scholer, H. E. Williams, and J. Zobel. 2003. Query expansion using associated queries. In Proceedings of CIKM 2003, pp. 2-9. [4] Billerbeck, B., and J. Zobel. 2003. When Query Expansion Fails. In Proceedings of SIGIR 2003, pp. 387-388. [5] Billerbeck, B. and J. Zobel. 2004. Questioning Query Expansion: An Examination of Behaviour and Parameters. In Proceedings of the 15th Australasian Database Conference (ADC2004), pp. 69-76. [6] Billerbeck, B. and J. Zobel. 2005. Document Expansion versus Query Expansion for ad-hoc Retrieval. In Proceedings of the 10th Australasian Document Computing Symposium. [7] Buckley, C. and E.M. Voorhees. 2000. Evaluating Evaluation Measure Stability. In Proceedings of SIGIR 2000, pp. 33-40. [8] Buckley, C. and E.M. Voorhees. 2004. Retrieval evaluation with incomplete information. In Proceedings of SIGIR 2004, pp. 25-32. [9] Carmel, D., E. Yom-Tov, A. Darlow, D. Pelleg. 2006. What Makes A Query Difficult? In Proceedings of SIGIR 2006, pp. 390-397. [10] Carpineto, C., R. Mori and G. Romano. 1998. Informative Term Selection for Automatic Query Expansion. In The 7th Text REtrieval Conference, pp.363:369. [11] Carterette, B. and J. Allan. 2005. Incremental Test Collections. In Proceedings of CIKM 2005, pp. 680-687. [12] Carterette, B., J. Allan, and R. Sitaraman. 2006. Minimal Test Collections for Retrieval Evaluation. In Proceedings of SIGIR 2006, pp. 268-275. [13] Cormack, G.V., C. R. Palmer, and C.L. Clarke. 1998. Efficient Construction of Large Test Collections. In Proceedings of SIGIR 1998, pp. 282-289. [14] Cormack, G. and T.R. Lynam. 2006. Statistical Precision of Information Retrieval Evaluation. In Proceedings of SIGIR 2006, pp. 533-540. [15] Cronen-Townsend, S., Y. Zhou, and W.B. Croft. 2004. A Language Modeling Framework for Selective Query Expansion, CIIR Technical Report. [16] Efthimiadis, E.N. Query Expansion. 1996. In Martha E. Williams (ed.), Annual Review of Information Systems and Technology (ARIST), v31, pp 121- 187. [17] Evans, D.A. and Lefferts, R.G. 1995. CLARIT-TREC Experiments. Information Processing & Management. 31(3): 385-295. [18] Fang, H. and C.X. Zhai. 2006. Semantic Term Matching in Axiomatic Approaches to Information Retrieval. In Proceedings of SIGIR 2006, pp. 115-122. [19] Gao, J., J. Nie, G. Wu and G. Cao. 2004. Dependence language model for information retrieval. In Proceedings of SIGIR 2004, pp. 170-177. [20] Harman, D.K. 1992. Relevance feedback revisited. In Proceedings of ACM SIGIR 1992, pp. 1-10. [21] Harman, D.K., ed. 1993. The First Text REtrieval Conference (TREC-1): 1992. [22] Harman, D.K., ed. 1994. The Second Text REtrieval Conference (TREC-2): 1993. [23] Harman, D.K., ed. 1995. The Third Text REtrieval Conference (TREC-3): 1994. [24] Harman, D.K., 1998. Towards Interactive Query Expansion. In Proceedings of SIGIR 1998, pp. 321-331. [25] Hofmann, T. 1999. Probabilistic latent semantic indexing. In Proceedings of SIGIR 1999, pp 50-57. [26] Jing, Y. and W.B. Croft. 1994. The Association Thesaurus for Information Retrieval. In Proceedings of RIAO 1994, pp. 146-160 [27] Lu, X.A. and R.B. Keefer. Query expansion/reduction and its impact on retrieval effectiveness. In: D.K. Harman, ed. The Third Text REtrieval Conference (TREC-3). Gaithersburg, MD: National Institute of Standards and Technology, 1995,231-239. [28] McNamee, P. and J. Mayfield. 2002. Comparing Cross-Language Query Expansion Techniques by Degrading Translation Resources. In Proceedings of SIGIR 2002, pp. 159-166. [29] Mitra, M., A. Singhal, and C. Buckley. 1998. Improving Automatic Query Expansion. In Proceedings of SIGIR 1998, pp. 206-214. [30] Peat, H. J. and P. Willett. 1991. The limitations of term co-occurrence data for query expansion in document retrieval systems. Journal of the American Society for Information Science, 42(5): 378-383. [31] Ponte, J.M. and W.B. Croft. 1998. A language modeling approach to information retrieval. In Proceedings of SIGIR 1998, pp.275-281. [32] Qiu Y., and Frei H. 1993. Concept based query expansion. In Proceedings of SIGIR 1993, pp. 160-169. [33] Robertson, S. 2006. On GMAP - and other transformations. In Proceedings of CIKM 2006, pp. 78-83. [34] Robertson, S.E. and K. Sparck Jones. 1976. Relevance Weighting of Search Terms. Journal of the American Society for Information Science, 27(3): 129-146. [35] Robertson, S.E., S. Walker, S. Jones, M.M. Hancock-Beaulieu, and M. Gatford. 1994. Okapi at TREC-2. In D.K. Harman (ed). 1994. The Second Text REtrieval Conference (TREC-2): 1993, pp. 21-34. [36] Robertson, S.E., S. Walker, S. Jones, M.M. Hancock-Beaulieu, and M. Gatford. 1995. Okapi at TREC-3. In D.K. Harman (ed). 1995. The Third Text REtrieval Conference (TREC-2): 1993, pp. 109-126 [37] Rocchio, J.J. 1971. Relevance feedback in information retrieval. In G. Salton (Ed.), The SMART Retrieval System. Prentice-Hall, Inc., Englewood Cliffs, NJ, pp. 313-323. [38] Salton, G. 1968. Automatic Information Organization and Retrieval. McGraw-Hill. [39] Salton, G. 1971. The SMART Retrieval System: Experiments in Automatic Document Processing. Englewood Cliffs NJ; Prentice-Hall. [40] Salton,G. 1980. Automatic term class construction using relevance-a summary of work in automatic pseudoclassification. Information Processing & Management. 16(1): 1-15. [41] Salton, G., and C. Buckley. 1988. On the Use of Spreading Activation Methods in Automatic Information Retrieval. In Proceedings of SIGIR 1998, pp. 147-160. [42] Sanderson, M. 1994. Word sense disambiguation and information retrieval. In Proceedings of SIGIR 1994, pp. 161-175. [43] Sanderson, M. and H. Joho. 2004. Forming test collections with no system pooling. In Proceedings of SIGIR 2004, pp. 186-193. [44] Sanderson, M. and Zobel, J. 2005. Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability. In Proceedings of SIGIR 2005, pp. 162-169. [45] Smeaton, A.F. and C.J. Van Rijsbergen. 1983. The Retrieval Effects of Query Expansion on a Feedback Document Retrieval System. Computer Journal. 26(3):239-246. [46] Song, F. and W.B. Croft. 1999. A general language model for information retrieval. In Proceedings of the Eighth International Conference on Information and Knowledge Management, pages 316-321. [47] Sparck Jones, K. 1971. Automatic Keyword Classification for Information Retrieval. London: Butterworths. [48] Terra, E. and C. L. Clarke. 2004. Scoring missing terms in information retrieval tasks. In Proceedings of CIKM 2004, pp. 50-58. [49] Turtle, Howard. 1994. Natural Language vs. Boolean Query Evaluation: A Comparison of Retrieval Performance. In Proceedings of SIGIR 1994, pp. 212-220. [50] Voorhees, E.M. 1994a. On Expanding Query Vectors with Lexically Related Words. In Harman, D. K., ed. Text REtrieval Conference (TREC-1): 1992. [51] Voorhees, E.M. 1994b. Query Expansion Using Lexical-Semantic Relations. In Proceedings of SIGIR 1994, pp. 61-69.

O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O

query-document term mismatch
relevant document

query expansion
information retrieval
information search
document expansion
document processing
evaluation