Robustness

of

Adaptive

Filtering

Methods

In

a

Cross-benchmark

Evaluation

Yiming

Yang,

Shinjae

Yoo,

Jian

Zhang,

Bryan

Kisiel

School

of

Computer

Science,

Carnegie

Mellon

University

Forbes

Avenue,

Pittsburgh,

PA

USA

ABSTRACT

This

paper

reports

a

cross-benchmark

evaluation

of

regularized

logistic

regression

(LR)

and

incremental

Rocchio

for

adaptive

filtering.

Using

four

corpora

from

the

Topic

Detection

and

Tracking

(TDT)

forum

and

the

Text

Retrieval

Conferences

(TREC)

we

evaluated

these

methods

with

non-stationary

topics

at

various

granularity

levels,

and

measured

performance

with

different

utility

settings.

We

found

that

LR

performs

strongly

and

robustly

in

optimizing

(a

TREC

utility

function)

while

Rocchio

is

better

for

optimizing

Ctrk

(the

TDT

tracking

cost),

a

high-recall

oriented

objective

function.

Using

systematic

cross-corpus

parameter

optimization

with

both

methods,

we

obtained

the

best

results

ever

reported

on

and

Relevance

feedback

on

a

small

portion

of

the

test

documents

yielded

significant

performance

improvements,

measuring

up

to

a

reduction

in

Ctrk

and

a

increase

in

(with

compared

to

the

results

of

the

top-performing

system

in

without

relevance

feedback

information.

Categories

and

Subject

Descriptors

[Information

Search

and

Retrieval]:

Information

filtering,

Relevance

feedback,

Retrieval

models,

Selection

process;

[Design

Methodology]:

Classifier

design

and

evaluation

General

Terms

Algorithms,

Measurement,

Performance,

Experimentation

INTRODUCTION

Adaptive

filtering

(AF)

has

been

a

challenging

research

topic

in

information

retrieval.

The

task

is

for

the

system

to

make

an

online

topic

membership

decision

(yes

or

no)

for

every

document,

as

soon

as

it

arrives,

with

respect

to

each

pre-defined

topic

of

interest.

Starting

from

in

the

Topic

Detection

and

Tracking

(TDT)

area

and

in

the

Text

Retrieval

Conferences

(TREC),

benchmark

evaluations

have

been

conducted

by

NIST

under

the

following

A

very

small

number

to

of

positive

training

examples

was

provided

for

each

topic

at

the

starting

point.

Relevance

feedback

was

available

but

only

for

the

systemaccepted

documents

(with

a

yes

decision)

in

the

TREC

evaluations

for

AF.

Relevance

feedback

(RF)

was

not

allowed

in

the

TDT

evaluations

for

AF

(or

topic

tracking

in

the

TDT

terminology)

until

was

the

first

time

that

TREC

and

TDT

metrics

were

jointly

used

in

evaluating

AF

methods

on

the

same

benchmark

(the

corpus)

where

non-stationary

topics

dominate.

The

above

conditions

attempt

to

mimic

realistic

situations

where

an

AF

system

would

be

used.

That

is,

the

user

would

be

willing

to

provide

a

few

positive

examples

for

each

topic

of

interest

at

the

start,

and

might

or

might

not

be

able

to

provide

additional

labeling

on

a

small

portion

of

incoming

documents

through

relevance

feedback.

Furthermore,

topics

of

interest

might

change

over

time,

with

new

topics

appearing

and

growing,

and

old

topics

shrinking

and

diminishing.

These

conditions

make

adaptive

filtering

a

difficult

task

in

statistical

learning

(online

classification),

for

the

following

reasons:

it

is

difficult

to

learn

accurate

models

for

prediction

based

on

extremely

sparse

training

data;

it

is

not

obvious

how

to

correct

the

sampling

bias

(i.e.,

relevance

feedback

on

system-accepted

documents

only)

during

the

adaptation

process;

it

is

not

well

understood

how

to

effectively

tune

parameters

in

AF

methods

using

cross-corpus

validation

where

the

validation

and

evaluation

topics

do

not

overlap,

and

the

documents

may

be

from

different

sources

or

different

epochs.

of

these

problems

is

addressed

in

the

literature

of

statistical

learning

for

batch

classification

where

all

the

training

data

are

given

at

once.

The

first

two

problems

have

been

studied

in

the

adaptive

filtering

literature,

including

topic

profile

adaptation

using

incremental

Rocchio,

Gaussian-Exponential

density

models,

logistic

regression

in

a

Bayesian

framework,

etc.,

and

threshold

optimization

strategies

using

probabilistic

calibration

or

local

fitting

techniques

Although

these

works

provide

valuable

insights

for

understanding

the

problems

and

possible

solutions,

it

is

difficult

to

draw

conclusions

regarding

the

effectiveness

and

robustness

of

current

methods

because

the

third

problem

has

not

been

thoroughly

investigated.

Addressing

the

third

issue

is

the

main

focus

in

this

paper.

We

argue

that

robustness

is

an

important

measure

for

evaluating

and

comparing

AF

methods.

By

robust

we

mean

consistent

and

strong

performance

across

benchmark

corpora

with

a

systematic

method

for

parameter

tuning

across

multiple

corpora.

Most

AF

methods

have

pre-specified

parameters

that

may

influence

the

performance

significantly

and

that

must

be

determined

before

the

test

process

starts.

Available

training

examples,

on

the

other

hand,

are

often

insufficient

for

tuning

the

parameters.

In

for

example,

there

is

only

one

labeled

training

example

per

topic

at

the

start;

parameter

optimization

on

such

training

data

is

doomed

to

be

ineffective.

This

leaves

only

one

option

(assuming

tuning

on

the

test

set

is

not

an

alternative),

that

is,

choosing

an

external

corpus

as

the

validation

set.

Notice

that

the

validation-set

topics

often

do

not

overlap

with

the

test-set

topics,

thus

the

parameter

optimization

is

performed

under

the

tough

condition

that

the

validation

data

and

the

test

data

may

be

quite

different

from

each

other.

Now

the

important

question

is:

which

methods

(if

any)

are

robust

under

the

condition

of

using

cross-corpus

validation

to

tune

parameters?

Current

literature

does

not

offer

an

answer

because

no

thorough

investigation

on

the

robustness

of

AF

methods

has

been

reported.

In

this

paper

we

address

the

above

question

by

conducting

a

cross-benchmark

evaluation

with

two

effective

approaches

in

AF:

incremental

Rocchio

and

regularized

logistic

regression

(LR).

Rocchio-style

classifiers

have

been

popular

in

AF,

with

good

performance

in

benchmark

evaluations

(TREC

and

TDT)

if

appropriate

parameters

were

used

and

if

combined

with

an

effective

threshold

calibration

strategy

Logistic

regression

is

a

classical

method

in

statistical

learning,

and

one

of

the

best

in

batch-mode

text

categorization

It

was

recently

evaluated

in

adaptive

filtering

and

was

found

to

have

relatively

strong

performance

(Section

Furthermore,

a

recent

paper

reported

that

the

joint

use

of

Rocchio

and

LR

in

a

Bayesian

framework

outperformed

the

results

of

using

each

method

alone

on

the

corpus.

Stimulated

by

those

findings,

we

decided

to

include

Rocchio

and

LR

in

our

crossbenchmark

evaluation

for

robustness

testing.

Specifically,

we

focus

on

how

much

the

performance

of

these

methods

depends

on

parameter

tuning,

what

the

most

influential

parameters

are

in

these

methods,

how

difficult

(or

how

easy)

to

optimize

these

influential

parameters

using

cross-corpus

validation,

how

strong

these

methods

perform

on

multiple

benchmarks

with

the

systematic

tuning

of

parameters

on

other

corpora,

and

how

efficient

these

methods

are

in

running

AF

on

large

benchmark

corpora.

The

organization

of

the

paper

is

as

follows:

Section

introduces

the

four

benchmark

corpora

and

and

used

in

this

study.

Section

analyzes

the

differences

among

the

TREC

and

TDT

metrics

(utilities

and

tracking

cost)

and

the

potential

implications

of

those

differences.

Section

outlines

the

Rocchio

and

LR

approaches

to

AF,

respectively.

Section

reports

the

experiments

and

results.

Section

concludes

the

main

findings

in

this

study.

BENCHMARK

CORPORA

We

used

four

benchmark

corpora

in

our

study.

Table

shows

the

statistics

about

these

data

sets.

was

the

evaluation

benchmark

for

adaptive

filtering

in

TREC

consisting

of

roughly

Reuters

news

stories

from

August

to

August

with

topic

labels

(subject

The

first

two

weeks

(August

to

of

documents

is

the

training

set,

and

the

remaining

½

months

(from

September

to

August

is

the

test

set.

was

the

evaluation

benchmark

for

adaptive

filtering

in

TREC

consisting

of

the

same

set

of

documents

as

those

in

but

with

a

slightly

different

splitting

point

for

the

training

and

test

sets.

The

topics

are

quite

different

from

those

in

they

are

queries

for

retrieval

with

relevance

judgments

by

NIST

assessors

was

the

evaluation

benchmark

in

the

dry

The

tracking

part

of

the

corpus

consists

of

news

stories

from

multiple

sources

in

English

and

Mandarin

(AP,

NYT,

CNN,

ABC,

NBC,

MSNBC,

Xinhua,

Zaobao,

Voice

of

America

and

PRI

the

World)

in

the

period

of

October

to

December

Machine-translated

versions

of

the

non-English

stories

(Xinhua,

Zaobao

and

VOA

Mandarin)

are

provided

as

well.

The

splitting

point

for

training-test

sets

is

different

for

each

topic

in

TDT.

was

the

evaluation

benchmark

in

The

tracking

part

of

the

corpus

consists

of

news

stories

in

the

period

of

April

to

September,

from

news

agents

or

broadcast

sources

in

English,

Arabic

and

Mandarin,

with

machine-translated

versions

of

the

non-English

stories.

We

only

used

the

English

versions

of

those

documents

in

our

experiments

for

this

paper.

The

TDT

topics

differ

from

TREC

topics

both

conceptually

and

statistically.

Instead

of

generic,

ever-lasting

subject

categories

(as

those

in

TREC),

TDT

topics

are

defined

at

a

finer

level

of

granularity,

for

events

that

happen

at

certain

times

and

locations,

and

that

are

born

and

die,

typically

associated

with

a

bursty

distribution

over

chronologically

ordered

news

stories.

The

average

size

of

TDT

topics

(events)

is

two

orders

of

magnitude

smaller

than

that

of

the

topics.

Figure

compares

the

document

densities

of

a

TREC

topic

(Civil

Wars)

and

two

TDT

topics

(Gunshot

and

APEC

Summit

Meeting,

respectively)

over

a

time

period,

where

the

area

under

each

curve

is

normalized

to

one.

The

granularity

differences

among

topics

and

the

corresponding

non-stationary

distributions

make

the

cross-benchmark

evaluation

interesting.

For

example,

algorithms

favoring

large

and

stable

topics

may

not

work

well

for

short-lasting

and

nonstationary

topics,

and

vice

versa.

Cross-benchmark

evaluations

allow

us

to

test

this

hypothesis

and

possibly

identify

the

weaknesses

in

current

approaches

to

adaptive

filtering

in

tracking

the

drifting

trends

of

topics.

Table

Statistics

of

benchmark

corpora

for

adaptive

filtering

evaluations

N(tr)

is

the

number

of

the

initial

training

documents;

N(ts)

is

the

number

of

the

test

documents;

n+

is

the

number

of

positive

examples

of

a

predefined

topic;

is

an

average

over

all

the

topics.

Week

P(topic|week)

Gunshot

APEC

Summit

Meeting

Civil

Figure

The

temporal

nature

of

topics

METRICS

To

make

our

results

comparable

to

the

literature,

we

decided

to

use

both

TREC-conventional

and

TDT-conventional

metrics

in

our

evaluation.

metrics

Let

A,

B,

C

and

D

be,

respectively,

the

numbers

of

true

positives,

false

alarms,

misses

and

true

negatives

for

a

specific

topic,

and

DCBAN

be

the

total

number

of

test

documents.

The

TREC-conventional

metrics

are

defined

as:

Precision

BAA

Recall

CAA

CABA

A

F

β

β

β

η

ηηβ

ηβ

),/()(max

CABA

SUT

where

parameters

β

and

η

were

set

to

and

respectively

in

and

For

evaluating

the

performance

of

a

system,

the

performance

scores

are

computed

for

individual

topics

first

and

then

averaged

over

topics

(macroaveraging).

TDT

metrics

The

TDT-conventional

metric

for

topic

tracking

is

defined

as:

famisstrk

PTPwPTPwTC

where

P(T)

is

the

percentage

of

documents

on

topic

T,

missP

is

the

miss

rate

by

the

system

on

that

topic,

faP

is

the

false

alarm

rate,

and

and

are

the

costs

(pre-specified

constants)

for

a

miss

and

a

false

alarm,

respectively.

The

TDT

benchmark

evaluations

(since

have

used

the

settings

of

=w

=w

and

=TP

for

all

topics.

For

evaluating

the

performance

of

a

system,

Ctrk

is

computed

for

each

topic

first

and

then

the

resulting

scores

are

averaged

for

a

single

measure

(the

topic-weighted

Ctrk).

To

make

the

intuition

behind

this

measure

transparent,

we

substitute

the

terms

in

the

definition

of

Ctrk

as

follows:

N

CA

TP

N

DB

TP

CA

C

Pmiss

DB

B

Pfa

BwCw

N

DB

B

N

DB

w

CA

C

N

CA

wTCtrk

Clearly,

trkC

is

the

average

cost

per

error

on

topic

T,

with

and

controlling

the

penalty

ratio

for

misses

vs.

false

alarms.

In

addition

to

trkC

also

employed

=βSUT

as

a

utility

metric.

To

distinguish

this

from

the

=βSUT

in

we

call

former

in

the

rest

of

this

paper.

Corpus

#Topics

N(tr)

N(ts)

Avg

n+

(tr)

Avg

n+

(ts)

Max

n+

(ts)

Min

n+

(ts)

#Topics

per

doc

(ts)

The

correlations

and

the

differences

From

an

optimization

point

of

view,

and

are

both

utility

functions

while

Ctrk

is

a

cost

function.

Our

objective

is

to

maximize

the

former

or

to

minimize

the

latter

on

test

documents.

The

differences

and

correlations

among

these

objective

functions

can

be

analyzed

through

the

shared

counts

of

A,

B,

C

and

D

in

their

definitions.

For

example,

both

and

are

positively

correlated

to

the

values

of

A

and

D,

and

negatively

correlated

to

the

values

of

B

and

C;

the

only

difference

between

them

is

in

their

penalty

ratios

for

misses

vs.

false

alarms,

i.e.,

in

and

in

The

Ctrk

function,

on

the

other

hand,

is

positively

correlated

to

the

values

of

C

and

B,

and

negatively

correlated

to

the

values

of

A

and

D;

hence,

it

is

negatively

correlated

to

and

More

importantly,

there

is

a

subtle

and

major

difference

between

Ctrk

and

the

utility

functions:

and

That

is,

Ctrk

has

a

very

different

penalty

ratio

for

misses

vs.

false

alarms:

it

favors

recall-oriented

systems

to

an

extreme.

At

first

glance,

one

would

think

that

the

penalty

ratio

in

Ctrk

is

since

=w

and

=w

However,

this

is

not

true

if

=TP

is

an

inaccurate

estimate

of

the

on-topic

documents

on

average

for

the

test

corpus.

Using

as

an

example,

the

true

percentage

is:

N

n

TP

where

N

is

the

average

size

of

the

test

sets

in

and

n+

is

the

average

number

of

positive

examples

per

topic

in

the

test

sets.

Using

=TP

as

an

(inaccurate)

estimate

of

enlarges

the

intended

penalty

ratio

of

to

roughly

speaking.

To

wit:

BC

NN

B

N

C

B

N

C

DB

B

N

CA

N

C

faPTPwmissPTPw

faPwmissPwTtrkC

ρρ

where

)(ˆ

TP

TP

ρ

is

the

factor

of

enlargement

in

the

estimation

of

P(T)

compared

to

the

truth.

Comparing

the

above

result

to

formula

we

can

see

the

actual

penalty

ratio

for

misses

vs.

false

alarms

was

in

the

evaluations

on

using

Ctrk.

Similarly,

we

can

compute

the

enlargement

factor

for

using

the

statistics

in

Table

as

follows:

)(ˆ

TP

TP

ρ

which

means

the

actual

penalty

ratio

for

misses

vs.

false

alarms

in

the

evaluation

on

using

Ctrk

was

approximately

The

implications

of

the

above

analysis

are

rather

significant:

Ctrk

defined

in

the

same

formula

does

not

necessarily

mean

the

same

objective

function

in

evaluation;

instead,

the

optimization

criterion

depends

on

the

test

corpus.

Systems

optimized

for

Ctrk

would

not

optimize

(and

because

the

former

favors

high-recall

oriented

to

an

extreme

while

the

latter

does

not.

Parameters

tuned

on

one

corpus

(e.g.,

might

not

work

for

an

evaluation

on

another

corpus

(say,

unless

we

account

for

the

previously-unknown

subtle

dependency

of

Ctrk

on

data.

Results

in

Ctrk

in

the

past

years

of

TDT

evaluations

may

not

be

directly

comparable

to

each

other

because

the

evaluation

collections

changed

most

years

and

hence

the

penalty

ratio

in

Ctrk

varied.

Although

these

problems

with

Ctrk

were

not

originally

anticipated,

it

offered

an

opportunity

to

examine

the

ability

of

systems

in

trading

off

precision

for

extreme

recall.

This

was

a

challenging

part

of

the

evaluation

for

AF.

Comparing

the

metrics

in

TDT

and

TREC

from

a

utility

or

cost

optimization

point

of

view

is

important

for

understanding

the

evaluation

results

of

adaptive

filtering

methods.

This

is

the

first

time

this

issue

is

explicitly

analyzed,

to

our

knowledge.

METHODS

Incremental

Rocchio

for

AF

We

employed

a

common

version

of

Rocchio-style

classifiers

which

computes

a

prototype

vector

per

topic

(T)

as

follows:

TD

d

TD

d

TqTp

TDdTDd

rr

rr

rr

γβα

The

first

term

on

the

RHS

is

the

weighted

vector

representation

of

topic

description

whose

elements

are

terms

weights.

The

second

term

is

the

weighted

centroid

of

the

set

)(TD+

of

positive

training

examples,

each

of

which

is

a

vector

of

withindocument

term

weights.

The

third

term

is

the

weighted

centroid

of

the

set

)(TD−

of

negative

training

examples

which

are

the

nearest

neighbors

of

the

positive

centroid.

The

three

terms

are

given

pre-specified

weights

of

βα,

and

γ

controlling

the

relative

influence

of

these

components

in

the

prototype.

The

prototype

of

a

topic

is

updated

each

time

the

system

makes

a

yes

decision

on

a

new

document

for

that

topic.

If

relevance

feedback

is

available

(as

is

the

case

in

TREC

adaptive

filtering),

the

new

document

is

added

to

the

pool

of

either

)(TD+

or

)(TD−

and

the

prototype

is

recomputed

accordingly;

if

relevance

feedback

is

not

available

(as

is

the

case

in

TDT

event

tracking),

the

system"s

prediction

(yes)

is

treated

as

the

truth,

and

the

new

document

is

added

to

)(TD+

for

updating

the

prototype.

Both

cases

are

part

of

our

experiments

in

this

paper

(and

part

of

the

TDT

evaluations

for

AF).

To

distinguish

the

two,

we

call

the

first

case

simply

Rocchio

and

the

second

case

PRF

Rocchio

where

PRF

stands

for

pseudorelevance

feedback.

The

predictions

on

a

new

document

are

made

by

computing

the

cosine

similarity

between

each

topic

prototype

and

the

document

vector,

and

then

comparing

the

resulting

scores

against

a

threshold:

))),((cos(

no

yes

dTpsign

new

θ

rr

Threshold

calibration

in

incremental

Rocchio

is

a

challenging

research

topic.

Multiple

approaches

have

been

developed.

The

simplest

is

to

use

a

universal

threshold

for

all

topics,

tuned

on

a

validation

set

and

fixed

during

the

testing

phase.

More

elaborate

methods

include

probabilistic

threshold

calibration

which

converts

the

non-probabilistic

similarity

scores

to

probabilities

(i.e.,

dTP

r

for

utility

optimization

and

margin-based

local

regression

for

risk

reduction

It

is

beyond

the

scope

of

this

paper

to

compare

all

the

different

ways

to

adapt

Rocchio-style

methods

for

AF.

Instead,

our

focus

here

is

to

investigate

the

robustness

of

Rocchio-style

methods

in

terms

of

how

much

their

performance

depends

on

elaborate

system

tuning,

and

how

difficult

(or

how

easy)

it

is

to

get

good

performance

through

cross-corpus

parameter

optimization.

Hence,

we

decided

to

use

a

relatively

simple

version

of

Rocchio

as

the

baseline,

i.e.,

with

a

universal

threshold

tuned

on

a

validation

corpus

and

fixed

for

all

topics

in

the

testing

phase.

This

simple

version

of

Rocchio

has

been

commonly

used

in

the

past

TDT

benchmark

evaluations

for

topic

tracking,

and

had

strong

performance

in

the

evaluations

for

adaptive

filtering

with

and

without

relevance

feedback

(Section

Results

of

more

complex

variants

of

Rocchio

are

also

discussed

when

relevant.

Logistic

Regression

for

AF

Logistic

regression

(LR)

estimates

the

posterior

probability

of

a

topic

given

a

document

using

a

sigmoid

function

xw

ewxyP

rrrr

where

x

r

is

the

document

vector

whose

elements

are

term

weights,

w

r

is

the

vector

of

regression

coefficients,

and

−+∈y

is

the

output

variable

corresponding

to

yes

or

no

with

respect

to

a

particular

topic.

Given

a

training

set

of

labeled

documents

nn

yxyxD

r

L

r

the

standard

regression

problem

is

defined

as

to

find

the

maximum

likelihood

estimates

of

the

regression

coefficients

(the

model

parameters):

)|(logmaxarg)|(maxarg

ii

xwyn

i

w

wDP

w

wDP

w

mlw

rr

r

r

r

r

r

r

This

is

a

convex

optimization

problem

which

can

be

solved

using

a

standard

conjugate

gradient

algorithm

in

O(INF)

time

for

training

per

topic,

where

I

is

the

average

number

of

iterations

needed

for

convergence,

and

N

and

F

are

the

number

of

training

documents

and

number

of

features

respectively

Once

the

regression

coefficients

are

optimized

on

the

training

data,

the

filtering

prediction

on

each

incoming

document

is

made

as:

no

yes

wxyPsign

optnew

θ

rr

Note

that

w

r

is

constantly

updated

whenever

a

new

relevance

judgment

is

available

in

the

testing

phase

of

AF,

while

the

optimal

threshold

optθ

is

constant,

depending

only

on

the

predefined

utility

(or

cost)

function

for

evaluation.

If

is

the

metric,

for

example,

with

the

penalty

ratio

of

for

misses

and

false

alarms

(Section

the

optimal

threshold

for

LR

is

for

all

topics.

We

modified

the

standard

(above)

version

of

LR

to

allow

more

flexible

optimization

criteria

as

follows:

μλ

rrr

rr

r

weysw

n

i

xwy

i

w

map

ii

where

iys

is

taken

to

be

α

β

and

γ

for

query,

positive

and

negative

documents

respectively,

which

are

similar

to

those

in

Rocchio,

giving

different

weights

to

the

three

kinds

of

training

examples:

topic

descriptions

(queries),

on-topic

documents

and

off-topic

documents.

The

second

term

in

the

objective

function

is

for

regularization,

equivalent

to

adding

a

Gaussian

prior

to

the

regression

coefficients

with

mean

μ

r

and

covariance

variance

matrix

Ι⋅λ2/1

where

Ι

is

the

identity

matrix.

Tuning

λ

is

theoretically

justified

for

reducing

model

complexity

(the

effective

degree

of

freedom)

and

avoiding

over-fitting

on

training

data

How

to

find

an

effective

μ

r

is

an

open

issue

for

research,

depending

on

the

user"s

belief

about

the

parameter

space

and

the

optimal

range.

The

solution

of

the

modified

objective

function

is

called

the

Maximum

A

Posteriori

(MAP)

estimate,

which

reduces

to

the

maximum

likelihood

solution

for

standard

LR

if

EVALUATIONS

We

report

our

empirical

findings

in

four

parts:

the

official

evaluation

results,

the

cross-corpus

parameter

optimization

results,

and

the

results

corresponding

to

the

amounts

of

relevance

feedback.

benchmark

results

The

evaluations

for

adaptive

filtering

were

conducted

by

NIST

in

November

Multiple

research

teams

participated

and

multiple

runs

from

each

team

were

allowed.

Ctrk

and

were

used

as

the

metrics.

Figure

and

Figure

show

the

results;

the

best

run

from

each

team

was

selected

with

respect

to

Ctrk

or

respectively.

Our

Rocchio

(with

adaptive

profiles

but

fixed

universal

threshold

for

all

topics)

had

the

best

result

in

Ctrk,

and

our

logistic

regression

had

the

best

result

in

All

the

parameters

of

our

runs

were

tuned

on

the

corpus.

Results

for

other

sites

are

also

listed

anonymously

for

comparison.

Ctrk

Ours

Metric

Ctrk

(the

lower

the

better)

Ours

Figure

results

in

Ctrk

of

systems

using

true

relevance

feedback.

(Ours

is

the

Rocchio

method.)

We

also

put

the

and

quartiles

as

sticks

for

each

Ours

Metric

(the

higher

the

better)

Ours

Figure

results

in

of

systems

using

true

relevance

feedback.

(Ours

is

LR

with

r

and

CTrk

Ours

Primary

Topic

Traking

Results

in

Ours

Ctrk

Figure

results

in

Ctrk

of

systems

without

using

true

relevance

feedback.

(Ours

is

PRF

Rocchio.)

Adaptive

filtering

without

using

true

relevance

feedback

was

also

a

part

of

the

evaluations.

In

this

case,

systems

had

only

one

labeled

training

example

per

topic

during

the

entire

training

and

testing

processes,

although

unlabeled

test

documents

could

be

used

as

soon

as

predictions

on

them

were

made.

Such

a

setting

has

been

conventional

for

the

Topic

Tracking

task

in

TDT

until

Figure

shows

the

summarized

official

submissions

from

each

team.

Our

PRF

Rocchio

(with

a

fixed

threshold

for

all

the

topics)

had

the

best

performance.

We

use

quartiles

rather

than

standard

deviations

since

the

former

is

more

resistant

to

outliers.

Cross-corpus

parameter

optimization

How

much

the

strong

performance

of

our

systems

depends

on

parameter

tuning

is

an

important

question.

Both

Rocchio

and

LR

have

parameters

that

must

be

prespecified

before

the

AF

process.

The

shared

parameters

include

the

sample

weightsα

β

and

γ

the

sample

size

of

the

negative

training

documents

(i.e.,

)(TD−

the

term-weighting

scheme,

and

the

maximal

number

of

non-zero

elements

in

each

document

vector.

The

method-specific

parameters

include

the

decision

threshold

in

Rocchio,

and

μ

r

λ

and

MI

(the

maximum

number

of

iterations

in

training)

in

LR.

Given

that

we

only

have

one

labeled

example

per

topic

in

the

training

sets,

it

is

impossible

to

effectively

optimize

these

parameters

on

the

training

data,

and

we

had

to

choose

an

external

corpus

for

validation.

Among

the

choices

of

and

we

chose

(c.f.

Section

because

it

is

most

similar

to

in

terms

of

the

nature

of

the

topics

(Section

We

optimized

the

parameters

of

our

systems

on

and

fixed

those

parameters

in

the

runs

on

for

our

submissions

to

We

also

tested

our

methods

on

and

for

further

analysis.

Since

exhaustive

testing

of

all

possible

parameter

settings

is

computationally

intractable,

we

followed

a

step-wise

forward

chaining

procedure

instead:

we

pre-specified

an

order

of

the

parameters

in

a

method

(Rocchio

or

LR),

and

then

tuned

one

parameter

at

the

time

while

fixing

the

settings

of

the

remaining

parameters.

We

repeated

this

procedure

for

several

passes

as

time

allowed.

Threshold

Figure

Performance

curves

of

adaptive

Rocchio

Figure

compares

the

performance

curves

in

for

Rocchio

on

and

when

the

decision

threshold

varied.

These

curves

peak

at

different

locations:

the

is

closest

to

the

while

the

and

are

quite

far

away

from

the

If

we

were

using

or

instead

of

as

the

validation

corpus

for

or

if

the

corpus

were

not

available,

we

would

have

difficulty

in

obtaining

strong

performance

for

Rocchio

in

The

difficulty

comes

from

the

ad-hoc

(non-probabilistic)

scores

generated

by

the

Rocchio

method:

the

distribution

of

the

scores

depends

on

the

corpus,

making

cross-corpus

threshold

optimization

a

tricky

problem.

Logistic

regression

has

less

difficulty

with

respect

to

threshold

tuning

because

it

produces

probabilistic

scores

of

xy

upon

which

the

optimal

threshold

can

be

directly

computed

if

probability

estimation

is

accurate.

Given

the

penalty

ratio

for

misses

vs.

false

alarms

as

in

in

and

in

Ctrk

(Section

the

corresponding

optimal

thresholds

(t)

are

and

respectively.

Although

the

theoretical

threshold

could

be

inaccurate,

it

still

suggests

the

range

of

near-optimal

settings.

With

these

threshold

settings

in

our

experiments

for

LR,

we

focused

on

the

crosscorpus

validation

of

the

Bayesian

prior

parameters,

that

is,

μ

r

and

λ.

Table

summarizes

the

results

We

measured

the

performance

of

the

runs

on

and

using

and

the

performance

of

the

runs

on

and

using

For

comparison

we

also

include

the

best

results

of

Rocchio-based

methods

on

these

corpora,

which

are

our

own

results

of

Rocchio

on

and

and

the

best

results

reported

by

NIST

for

and

From

this

set

of

results,

we

see

that

LR

significantly

outperformed

Rocchio

on

all

the

corpora,

even

in

the

runs

of

standard

LR

without

any

tuning,

i.e.

This

empirical

finding

is

consistent

with

a

previous

report

for

LR

on

although

our

results

of

LR

in

are

stronger

than

the

results

for

standard

LR

and

for

LR

using

Rocchio

prototype

as

the

prior)

in

that

report.

More

importantly,

our

cross-benchmark

evaluation

gives

strong

evidence

for

the

robustness

of

LR.

The

robustness,

we

believe,

comes

from

the

probabilistic

nature

of

the

system-generated

scores.

That

is,

compared

to

the

ad-hoc

scores

in

Rocchio,

the

normalized

posterior

probabilities

make

the

threshold

optimization

in

LR

a

much

easier

problem.

Moreover,

logistic

regression

is

known

to

converge

towards

the

Bayes

classifier

asymptotically

while

Rocchio

classifiers"

parameters

do

not.

Another

interesting

observation

in

these

results

is

that

the

performance

of

LR

did

not

improve

when

using

a

Rocchio

prototype

as

the

mean

in

the

prior;

instead,

the

performance

decreased

in

some

cases.

This

observation

does

not

support

the

previous

report

by

but

we

are

not

surprised

because

we

are

not

convinced

that

Rocchio

prototypes

are

more

accurate

than

LR

models

for

topics

in

the

early

stage

of

the

AF

process,

and

we

believe

that

using

a

Rocchio

prototype

as

the

mean

in

the

Gaussian

prior

would

introduce

undesirable

bias

to

LR.

We

also

believe

that

variance

reduction

(in

the

testing

phase)

should

be

controlled

by

the

choice

of

λ

(but

not

μ

r

for

which

we

conducted

the

experiments

as

shown

in

Figure

Table

Results

of

LR

with

different

Bayesian

priors

Corpus

Best

Rocchio

The

LR

results

on

in

this

table

are

better

than

our

official

result

because

parameter

optimization

has

been

improved

afterwards.

The

result

by

Oracle)

is

only

available

in

which

is

not

directly

comparable

to

the

scores

in

just

indicative.

μ

r

was

set

to

the

Rocchio

prototype

Lambda

Performance

Ctrk

on

on

on

on

Figure

LR

with

varying

lambda.

The

performance

of

LR

is

summarized

with

respect

to

λ

tuning

on

the

corpora

of

and

The

performance

on

each

corpus

was

measured

using

the

corresponding

metrics,

that

is,

for

the

runs

on

and

and

and

Ctrk

for

the

runs

on

In

the

case

of

maximizing

the

utilities,

the

safe

interval

for

λ

is

between

and

meaning

that

the

performance

of

regularized

LR

is

stable,

the

same

as

or

improved

slightly

over

the

performance

of

standard

LR.

In

the

case

of

minimizing

Ctrk,

the

safe

range

for

λ

is

between

and

and

setting

λ

between

and

yielded

relatively

large

improvements

over

the

performance

of

standard

LR

because

training

a

model

for

extremely

high

recall

is

statistically

more

tricky,

and

hence

more

regularization

is

needed.

In

either

case,

tuning

λ

is

relatively

safe,

and

easy

to

do

successfully

by

cross-corpus

tuning.

Another

influential

choice

in

our

experiment

settings

is

term

weighting:

we

examined

the

choices

of

binary,

TF

and

TF-IDF

(the

ltc

version)

schemes.

We

found

TF-IDF

most

effective

for

both

Rocchio

and

LR,

and

used

this

setting

in

all

our

experiments.

Percentages

of

labeled

data

How

much

relevance

feedback

(RF)

would

be

needed

during

the

AF

process

is

a

meaningful

question

in

real-world

applications.

To

answer

it,

we

evaluated

Rocchio

and

LR

on

TDT

with

the

following

settings:

Basic

Rocchio,

no

adaptation

at

all

PRF

Rocchio,

updating

topic

profiles

without

using

true

relevance

feedback;

Adaptive

Rocchio,

updating

topic

profiles

using

relevance

feedback

on

system-accepted

documents

plus

documents

randomly

sampled

from

the

pool

of

systemrejected

documents;

LR

with

rr

=μ

and

threshold

All

the

parameters

in

Rocchio

tuned

on

Table

summarizes

the

results

in

Ctrk:

Adaptive

Rocchio

with

relevance

feedback

on

of

the

test

documents

reduced

the

tracking

cost

by

over

the

result

of

the

PRF

Rocchio,

the

best

system

in

the

evaluation

for

topic

tracking

without

relevance

feedback

information.

Incremental

LR,

on

the

other

hand,

was

weaker

but

still

impressive.

Recall

that

Ctrk

is

an

extremely

high-recall

oriented

metric,

causing

frequent

updating

of

profiles

and

hence

an

efficiency

problem

in

LR.

For

this

reason

we

set

a

higher

threshold

instead

of

the

theoretically

optimal

threshold

in

LR

to

avoid

an

untolerable

computation

cost.

The

computation

time

in

machine-hours

was

for

the

run

of

adaptive

Rocchio

and

for

the

run

of

LR

on

when

optimizing

Ctrk.

Table

summarizes

the

results

in

adaptive

LR

was

the

winner

in

this

case,

with

relevance

feedback

on

of

the

test

documents

improving

the

utility

by

over

the

results

of

PRF

Rocchio.

Table

AF

methods

on

(Performance

in

Ctrk)

Base

Roc

PRF

Roc

Adp

Roc

LR

of

RF

Ctrk

(baseline)

Table

AF

methods

on

(Performance

in

Base

Roc

PRF

Roc

Adp

Roc

of

RF

(baseline)

Evidently,

both

Rocchio

and

LR

are

highly

effective

in

adaptive

filtering,

in

terms

of

using

of

a

small

amount

of

labeled

data

to

significantly

improve

the

model

accuracy

in

statistical

learning,

which

is

the

main

goal

of

AF.

Summary

of

Adaptation

Process

After

we

decided

the

parameter

settings

using

validation,

we

perform

the

adaptive

filtering

in

the

following

steps

for

each

topic:

Train

the

LR/Rocchio

model

using

the

provided

positive

training

examples

and

randomly

sampled

negative

examples;

For

each

document

in

the

test

corpus:

we

first

make

a

prediction

about

relevance,

and

then

get

relevance

feedback

for

those

(predicted)

positive

documents.

Model

and

IDF

statistics

will

be

incrementally

updated

if

we

obtain

its

true

relevance

feedback.

CONCLUDING

REMARKS

We

presented

a

cross-benchmark

evaluation

of

incremental

Rocchio

and

incremental

LR

in

adaptive

filtering,

focusing

on

their

robustness

in

terms

of

performance

consistency

with

respect

to

cross-corpus

parameter

optimization.

Our

main

conclusions

from

this

study

are

the

following:

Parameter

optimization

in

AF

is

an

open

challenge

but

has

not

been

thoroughly

studied

in

the

past.

Robustness

in

cross-corpus

parameter

tuning

is

important

for

evaluation

and

method

comparison.

We

found

LR

more

robust

than

Rocchio;

it

had

the

best

results

(in

ever

reported

on

and

without

extensive

tuning.

We

found

Rocchio

performs

strongly

when

a

good

validation

corpus

is

available,

and

a

preferred

choice

when

optimizing

Ctrk

is

the

objective,

favoring

recall

over

precision

to

an

extreme.

For

future

research

we

want

to

study

explicit

modeling

of

the

temporal

trends

in

topic

distributions

and

content

drifting.

Acknowledgments

This

material

is

based

upon

work

supported

in

parts

by

the

National

Science

Foundation

(NSF)

under

grant

by

the

DoD

under

award

and

by

the

Defense

Advanced

Research

Project

Agency

(DARPA)

under

Contract

No.

Any

opinions,

findings

and

conclusions

or

recommendations

expressed

in

this

material

are

those

of

the

author(s)

and

do

not

necessarily

reflect

the

views

of

the

sponsors.

REFERENCES

J.

Allan.

Incremental

relevance

feedback

for

information

filtering.

In

J.

Callan.

Learning

while

filtering

documents.

In

J.

Fiscus

and

G.

Duddington.

Topic

detection

and

tracking

overview.

In

Topic

detection

and

tracking:

event-based

information

organization,

J.

Fiscus

and

B.

Wheatley.

Overview

of

the

TDT

Evaluation

and

Results.

In

T.

Hastie,

R.

Tibshirani

and

J.

Friedman.

Elements

of

Statistical

Learning.

Springer,

S.

Robertson

and

D.

Hull.

The

filtering

track

final

report.

In

S.

Robertson

and

I.

Soboroff.

The

filtering

track

final

report.

In

S.

Robertson

and

I.

Soboroff.

The

TREC

filtering

track

report.

In

S.

Robertson

and

S.

Walker.

Microsoft

Cambridge

at

In

R.

Schapire,

Y.

Singer

and

A.

Singhal.

Boosting

and

Rocchio

applied

to

text

filtering.

In

Y.

Yang

and

B.

Kisiel.

Margin-based

local

regression

for

adaptive

filtering.

In

Y.

Zhang

and

J.

Callan.

Maximum

likelihood

estimation

for

filtering

thresholds.

In

Y.

Zhang.

Using

Bayesian

priors

to

combine

classifiers

for

adaptive

filtering.

In

J.

Zhang

and

Y.

Yang.

Robustness

of

regularized

linear

classification

methods

in

text

categorization.

In

T.

Zhang,

F.

J.

Oles.

Text

Categorization

Based

on

Regularized

Linear

Classification

Methods.

Inf.

Retr.

B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O B O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I I I I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O B I O O O B I O O O O O O O O B O O O O O O B I O O B I B O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O B O O O B O B O O O O O O O O O O O O O O O O O B O B O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O B O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O B O O O O O O B O O O O O O B I O O B O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O B O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

granularity difference
cross-benchmark evaluation
rocchio-style method
cross-corpus parameter optimization
adaptive filter
topic track

systematic method for parameter tuning across multiple corpora