Automatic

Extraction

of

Titles

from

General

Documents

using

Machine

Learning

Yunhua

Computer

Science

Department

Xi"an

Jiaotong

University

No

Xianning

West

Road

Xi'an,

China,

yunhuahu@mail.xjtu.edu.cn

Hang

Li,

Yunbo

Cao

Microsoft

Research

Asia

Sigma

Center,

No.

Zhichun

Road,

Haidian,

Beijing,

China,

{hangli,yucao}@microsoft.com

Qinghua

Zheng

Computer

Science

Department

Xi"an

Jiaotong

University

No

Xianning

West

Road

Xi'an,

China,

qhzheng@mail.xjtu.edu.cn

Dmitriy

Meyerzon

Microsoft

Corporation

One

Microsoft

Way

Redmond,

WA,

USA,

dmitriym@microsoft.com

ABSTRACT

In

this

paper,

we

propose

a

machine

learning

approach

to

title

extraction

from

general

documents.

By

general

documents,

we

mean

documents

that

can

belong

to

any

one

of

a

number

of

specific

genres,

including

presentations,

book

chapters,

technical

papers,

brochures,

reports,

and

letters.

Previously,

methods

have

been

proposed

mainly

for

title

extraction

from

research

papers.

It

has

not

been

clear

whether

it

could

be

possible

to

conduct

automatic

title

extraction

from

general

documents.

As

a

case

study,

we

consider

extraction

from

Office

including

Word

and

PowerPoint.

In

our

approach,

we

annotate

titles

in

sample

documents

(for

Word

and

PowerPoint

respectively)

and

take

them

as

training

data,

train

machine

learning

models,

and

perform

title

extraction

using

the

trained

models.

Our

method

is

unique

in

that

we

mainly

utilize

formatting

information

such

as

font

size

as

features

in

the

models.

It

turns

out

that

the

use

of

formatting

information

can

lead

to

quite

accurate

extraction

from

general

documents.

Precision

and

recall

for

title

extraction

from

Word

is

and

respectively,

and

precision

and

recall

for

title

extraction

from

PowerPoint

is

and

respectively

in

an

experiment

on

intranet

data.

Other

important

new

findings

in

this

work

include

that

we

can

train

models

in

one

domain

and

apply

them

to

another

domain,

and

more

surprisingly

we

can

even

train

models

in

one

language

and

apply

them

to

another

language.

Moreover,

we

can

significantly

improve

search

ranking

results

in

document

retrieval

by

using

the

extracted

titles.

Categories

and

Subject

Descriptors

[Information

Storage

and

Retrieval]:

Information

Search

and

Retrieval

Search

Process;

[Information

Systems

Applications]:

Office

Automation

Word

processing;

[Software

Engineering]:

Metrics

complexity

measures,

performance

measures

General

Terms

Algorithms,

Experimentation,

Performance.

INTRODUCTION

Metadata

of

documents

is

useful

for

many

kinds

of

document

processing

such

as

search,

browsing,

and

filtering.

Ideally,

metadata

is

defined

by

the

authors

of

documents

and

is

then

used

by

various

systems.

However,

people

seldom

define

document

metadata

by

themselves,

even

when

they

have

convenient

metadata

definition

tools

Thus,

how

to

automatically

extract

metadata

from

the

bodies

of

documents

turns

out

to

be

an

important

research

issue.

Methods

for

performing

the

task

have

been

proposed.

However,

the

focus

was

mainly

on

extraction

from

research

papers.

For

instance,

Han

et

al.

proposed

a

machine

learning

based

method

to

conduct

extraction

from

research

papers.

They

formalized

the

problem

as

that

of

classification

and

employed

Support

Vector

Machines

as

the

classifier.

They

mainly

used

linguistic

features

in

the

In

this

paper,

we

consider

metadata

extraction

from

general

documents.

By

general

documents,

we

mean

documents

that

may

belong

to

any

one

of

a

number

of

specific

genres.

General

documents

are

more

widely

available

in

digital

libraries,

intranets

and

the

internet,

and

thus

investigation

on

extraction

from

them

is

sorely

needed.

Research

papers

usually

have

well-formed

styles

and

noticeable

characteristics.

In

contrast,

the

styles

of

general

documents

can

vary

greatly.

It

has

not

been

clarified

whether

a

machine

learning

based

approach

can

work

well

for

this

task.

There

are

many

types

of

metadata:

title,

author,

date

of

creation,

etc.

As

a

case

study,

we

consider

title

extraction

in

this

paper.

General

documents

can

be

in

many

different

file

formats:

Microsoft

Office,

PDF

(PS),

etc.

As

a

case

study,

we

consider

extraction

from

Office

including

Word

and

PowerPoint.

We

take

a

machine

learning

approach.

We

annotate

titles

in

sample

documents

(for

Word

and

PowerPoint

respectively)

and

take

them

as

training

data

to

train

several

types

of

models,

and

perform

title

extraction

using

any

one

type

of

the

trained

models.

In

the

models,

we

mainly

utilize

formatting

information

such

as

font

size

as

features.

We

employ

the

following

models:

Maximum

Entropy

Model,

Perceptron

with

Uneven

Margins,

Maximum

Entropy

Markov

Model,

and

Voted

Perceptron.

In

this

paper,

we

also

investigate

the

following

three

problems,

which

did

not

seem

to

have

been

examined

previously.

Comparison

between

models:

among

the

models

above,

which

model

performs

best

for

title

extraction;

Generality

of

model:

whether

it

is

possible

to

train

a

model

on

one

domain

and

apply

it

to

another

domain,

and

whether

it

is

possible

to

train

a

model

in

one

language

and

apply

it

to

another

language;

Usefulness

of

extracted

titles:

whether

extracted

titles

can

improve

document

processing

such

as

search.

Experimental

results

indicate

that

our

approach

works

well

for

title

extraction

from

general

documents.

Our

method

can

significantly

outperform

the

baselines:

one

that

always

uses

the

first

lines

as

titles

and

the

other

that

always

uses

the

lines

in

the

largest

font

sizes

as

titles.

Precision

and

recall

for

title

extraction

from

Word

are

and

respectively,

and

precision

and

recall

for

title

extraction

from

PowerPoint

are

and

respectively.

It

turns

out

that

the

use

of

format

features

is

the

key

to

successful

title

extraction.

We

have

observed

that

Perceptron

based

models

perform

better

in

terms

of

extraction

accuracies.

We

have

empirically

verified

that

the

models

trained

with

our

approach

are

generic

in

the

sense

that

they

can

be

trained

on

one

domain

and

applied

to

another,

and

they

can

be

trained

in

one

language

and

applied

to

another.

We

have

found

that

using

the

extracted

titles

we

can

significantly

improve

precision

of

document

retrieval

(by

We

conclude

that

we

can

indeed

conduct

reliable

title

extraction

from

general

documents

and

use

the

extracted

results

to

improve

real

applications.

The

rest

of

the

paper

is

organized

as

follows.

In

section

we

introduce

related

work,

and

in

section

we

explain

the

motivation

and

problem

setting

of

our

work.

In

section

we

describe

our

method

of

title

extraction,

and

in

section

we

describe

our

method

of

document

retrieval

using

extracted

titles.

Section

gives

our

experimental

results.

We

make

concluding

remarks

in

section

RELATED

WORK

Document

Metadata

Extraction

Methods

have

been

proposed

for

performing

automatic

metadata

extraction

from

documents;

however,

the

main

focus

was

on

extraction

from

research

papers.

The

proposed

methods

fall

into

two

categories:

the

rule

based

approach

and

the

machine

learning

based

approach.

Giuffrida

et

al.

for

instance,

developed

a

rule-based

system

for

automatically

extracting

metadata

from

research

papers

in

Postscript.

They

used

rules

like

titles

are

usually

located

on

the

upper

portions

of

the

first

pages

and

they

are

usually

in

the

largest

font

sizes.

Liddy

et

al.

and

Yilmazel

el

al.

performed

metadata

extraction

from

educational

materials

using

rule-based

natural

language

processing

technologies.

Mao

et

al.

also

conducted

automatic

metadata

extraction

from

research

papers

using

rules

on

formatting

information.

The

rule-based

approach

can

achieve

high

performance.

However,

it

also

has

disadvantages.

It

is

less

adaptive

and

robust

when

compared

with

the

machine

learning

approach.

Han

et

al.

for

instance,

conducted

metadata

extraction

with

the

machine

learning

approach.

They

viewed

the

problem

as

that

of

classifying

the

lines

in

a

document

into

the

categories

of

metadata

and

proposed

using

Support

Vector

Machines

as

the

classifier.

They

mainly

used

linguistic

information

as

features.

They

reported

high

extraction

accuracy

from

research

papers

in

terms

of

precision

and

recall.

Information

Extraction

Metadata

extraction

can

be

viewed

as

an

application

of

information

extraction,

in

which

given

a

sequence

of

instances,

we

identify

a

subsequence

that

represents

information

in

which

we

are

interested.

Hidden

Markov

Model

Maximum

Entropy

Model

Maximum

Entropy

Markov

Model

Support

Vector

Machines

Conditional

Random

Field

and

Voted

Perceptron

are

widely

used

information

extraction

models.

Information

extraction

has

been

applied,

for

instance,

to

part-ofspeech

tagging

named

entity

recognition

and

table

extraction

Search

Using

Title

Information

Title

information

is

useful

for

document

retrieval.

In

the

system

Citeseer,

for

instance,

Giles

et

al.

managed

to

extract

titles

from

research

papers

and

make

use

of

the

extracted

titles

in

metadata

search

of

papers

In

web

search,

the

title

fields

(i.e.,

file

properties)

and

anchor

texts

of

web

pages

(HTML

documents)

can

be

viewed

as

‘titles"

of

the

pages

Many

search

engines

seem

to

utilize

them

for

web

page

retrieval

Zhang

et

al.,

found

that

web

pages

with

well-defined

metadata

are

more

easily

retrieved

than

those

without

well-defined

metadata

To

the

best

of

our

knowledge,

no

research

has

been

conducted

on

using

extracted

titles

from

general

documents

(e.g.,

Office

documents)

for

search

of

the

documents.

MOTIVATION

AND

PROBLEM

SETTING

We

consider

the

issue

of

automatically

extracting

titles

from

general

documents.

By

general

documents,

we

mean

documents

that

belong

to

one

of

any

number

of

specific

genres.

The

documents

can

be

presentations,

books,

book

chapters,

technical

papers,

brochures,

reports,

memos,

specifications,

letters,

announcements,

or

resumes.

General

documents

are

more

widely

available

in

digital

libraries,

intranets,

and

internet,

and

thus

investigation

on

title

extraction

from

them

is

sorely

needed.

Figure

shows

an

estimate

on

distributions

of

file

formats

on

intranet

and

internet

Office

and

PDF

are

the

main

file

formats

on

the

intranet.

Even

on

the

internet,

the

documents

in

the

formats

are

still

not

negligible,

given

its

extremely

large

size.

In

this

paper,

without

loss

of

generality,

we

take

Office

documents

as

an

example.

Figure

Distributions

of

file

formats

in

internet

and

intranet.

For

Office

documents,

users

can

define

titles

as

file

properties

using

a

feature

provided

by

Office.

We

found

in

an

experiment,

however,

that

users

seldom

use

the

feature

and

thus

titles

in

file

properties

are

usually

very

inaccurate.

That

is

to

say,

titles

in

file

properties

are

usually

inconsistent

with

the

‘true"

titles

in

the

file

bodies

that

are

created

by

the

authors

and

are

visible

to

readers.

We

collected

Word

and

PowerPoint

documents

from

an

intranet

and

the

internet

and

examined

how

many

titles

in

the

file

properties

are

correct.

We

found

that

surprisingly

the

accuracy

was

only

(cf.,

Section

for

details).

A

number

of

reasons

can

be

considered.

For

example,

if

one

creates

a

new

file

by

copying

an

old

file,

then

the

file

property

of

the

new

file

will

also

be

copied

from

the

old

file.

In

another

experiment,

we

found

that

Google

uses

the

titles

in

file

properties

of

Office

documents

in

search

and

browsing,

but

the

titles

are

not

very

accurate.

We

created

queries

to

search

Word

and

PowerPoint

documents

and

examined

the

top

results

of

each

query

returned

by

Google.

We

found

that

nearly

all

the

titles

presented

in

the

search

results

were

from

the

file

properties

of

the

documents.

However,

only

of

them

were

correct.

Actually,

‘true"

titles

usually

exist

at

the

beginnings

of

the

bodies

of

documents.

If

we

can

accurately

extract

the

titles

from

the

bodies

of

documents,

then

we

can

exploit

reliable

title

information

in

document

processing.

This

is

exactly

the

problem

we

address

in

this

paper.

More

specifically,

given

a

Word

document,

we

are

to

extract

the

title

from

the

top

region

of

the

first

page.

Given

a

PowerPoint

document,

we

are

to

extract

the

title

from

the

first

slide.

A

title

sometimes

consists

of

a

main

title

and

one

or

two

subtitles.

We

only

consider

extraction

of

the

main

title.

As

baselines

for

title

extraction,

we

use

that

of

always

using

the

first

lines

as

titles

and

that

of

always

using

the

lines

with

largest

font

sizes

as

titles.

Figure

Title

extraction

from

Word

document.

Figure

Title

extraction

from

PowerPoint

document.

Next,

we

define

a

‘specification"

for

human

judgments

in

title

data

annotation.

The

annotated

data

will

be

used

in

training

and

testing

of

the

title

extraction

methods.

Summary

of

the

specification:

The

title

of

a

document

should

be

identified

on

the

basis

of

common

sense,

if

there

is

no

difficulty

in

the

identification.

However,

there

are

many

cases

in

which

the

identification

is

not

easy.

There

are

some

rules

defined

in

the

specification

that

guide

identification

for

such

cases.

The

rules

include

a

title

is

usually

in

consecutive

lines

in

the

same

format,

a

document

can

have

no

title,

titles

in

images

are

not

considered,

a

title

should

not

contain

words

like

‘draft",

‘whitepaper",

etc,

if

it

is

difficult

to

determine

which

is

the

title,

select

the

one

in

the

largest

font

size,

and

if

it

is

still

difficult

to

determine

which

is

the

title,

select

the

first

candidate.

(The

specification

covers

all

the

cases

we

have

encountered

in

data

annotation.)

Figures

and

show

examples

of

Office

documents

from

which

we

conduct

title

extraction.

In

Figure

‘Differences

in

API

Implementations

among

Windows

Operating

Systems"

is

the

title

of

the

Word

document.

‘Microsoft

Windows"

on

the

top

of

this

page

is

a

picture

and

thus

is

ignored.

In

Figure

‘Building

Competitive

Advantages

through

an

Agile

Infrastructure"

is

the

title

of

the

PowerPoint

document.

We

have

developed

a

tool

for

annotation

of

titles

by

human

annotators.

Figure

shows

a

snapshot

of

the

tool.

Figure

Title

annotation

tool.

TITLE

EXTRACTION

METHOD

Outline

Title

extraction

based

on

machine

learning

consists

of

training

and

extraction.

The

same

pre-processing

step

occurs

before

training

and

extraction.

During

pre-processing,

from

the

top

region

of

the

first

page

of

a

Word

document

or

the

first

slide

of

a

PowerPoint

document

a

number

of

units

for

processing

are

extracted.

If

a

line

(lines

are

separated

by

‘return"

symbols)

only

has

a

single

format,

then

the

line

will

become

a

unit.

If

a

line

has

several

parts

and

each

of

them

has

its

own

format,

then

each

part

will

become

a

unit.

Each

unit

will

be

treated

as

an

instance

in

learning.

A

unit

contains

not

only

content

information

(linguistic

information)

but

also

formatting

information.

The

input

to

pre-processing

is

a

document

and

the

output

of

pre-processing

is

a

sequence

of

units

(instances).

Figure

shows

the

units

obtained

from

the

document

in

Figure

Figure

Example

of

units.

In

learning,

the

input

is

sequences

of

units

where

each

sequence

corresponds

to

a

document.

We

take

labeled

units

(labeled

as

or

other)

in

the

sequences

as

training

data

and

construct

models

for

identifying

whether

a

unit

is

or

other.

We

employ

four

types

of

models:

Perceptron,

Maximum

Entropy

(ME),

Perceptron

Markov

Model

(PMM),

and

Maximum

Entropy

Markov

Model

(MEMM).

In

extraction,

the

input

is

a

sequence

of

units

from

one

document.

We

employ

one

type

of

model

to

identify

whether

a

unit

is

or

other.

We

then

extract

units

from

the

unit

labeled

with

to

the

unit

labeled

with

The

result

is

the

extracted

title

of

the

document.

The

unique

characteristic

of

our

approach

is

that

we

mainly

utilize

formatting

information

for

title

extraction.

Our

assumption

is

that

although

general

documents

vary

in

styles,

their

formats

have

certain

patterns

and

we

can

learn

and

utilize

the

patterns

for

title

extraction.

This

is

in

contrast

to

the

work

by

Han

et

al.,

in

which

only

linguistic

features

are

used

for

extraction

from

research

papers.

Models

The

four

models

actually

can

be

considered

in

the

same

metadata

extraction

framework.

That

is

why

we

apply

them

together

to

our

current

problem.

Each

input

is

a

sequence

of

instances

kxxx

together

with

a

sequence

of

labels

kyyy

ix

and

iy

represents

an

instance

and

its

label,

respectively

ki

L=

Recall

that

an

instance

here

represents

a

unit.

A

label

represents

or

other.

Here,

k

is

the

number

of

units

in

a

document.

In

learning,

we

train

a

model

which

can

be

generally

denoted

as

a

conditional

probability

distribution

kk

XXYYP

LL

where

iX

and

iY

denote

random

variables

taking

instance

ix

and

label

iy

as

values,

respectively

ki

L=

Learning

Tool

Extraction

Tool

nknnknn

kk

kk

yyyxxx

yyyxxx

yyyxxx

LL

LL

LL

LL

)|(maxarg

mkmmkm

xxyyP

LL

kk

XXYYP

LL

Conditional

Distribution

mkmm

xxx

Figure

Metadata

extraction

model.

We

can

make

assumptions

about

the

general

model

in

order

to

make

it

simple

enough

for

training.

For

example,

we

can

assume

that

kYY

L

are

independent

of

each

other

given

kXX

L

Thus,

we

have

kk

kk

XYPXYP

XXYYP

L

LL

In

this

way,

we

decompose

the

model

into

a

number

of

classifiers.

We

train

the

classifiers

locally

using

the

labeled

data.

As

the

classifier,

we

employ

the

Perceptron

or

Maximum

Entropy

model.

We

can

also

assume

that

the

first

order

Markov

property

holds

for

kYY

L

given

kXX

L

Thus,

we

have

kkk

kk

XYYPXYP

XXYYP

L

LL

Again,

we

obtain

a

number

of

classifiers.

However,

the

classifiers

are

conditioned

on

the

previous

label.

When

we

employ

the

Percepton

or

Maximum

Entropy

model

as

a

classifier,

the

models

become

a

Percepton

Markov

Model

or

Maximum

Entropy

Markov

Model,

respectively.

That

is

to

say,

the

two

models

are

more

precise.

In

extraction,

given

a

new

sequence

of

instances,

we

resort

to

one

of

the

constructed

models

to

assign

a

sequence

of

labels

to

the

sequence

of

instances,

i.e.,

perform

extraction.

For

Perceptron

and

ME,

we

assign

labels

locally

and

combine

the

results

globally

later

using

heuristics.

Specifically,

we

first

identify

the

most

likely

Then

we

find

the

most

likely

within

three

units

after

the

Finally,

we

extract

as

a

title

the

units

between

the

and

the

For

PMM

and

MEMM,

we

employ

the

Viterbi

algorithm

to

find

the

globally

optimal

label

sequence.

In

this

paper,

for

Perceptron,

we

actually

employ

an

improved

variant

of

it,

called

Perceptron

with

Uneven

Margin

This

version

of

Perceptron

can

work

well

especially

when

the

number

of

positive

instances

and

the

number

of

negative

instances

differ

greatly,

which

is

exactly

the

case

in

our

problem.

We

also

employ

an

improved

version

of

Perceptron

Markov

Model

in

which

the

Perceptron

model

is

the

so-called

Voted

Perceptron

In

addition,

in

training,

the

parameters

of

the

model

are

updated

globally

rather

than

locally.

Features

There

are

two

types

of

features:

format

features

and

linguistic

features.

We

mainly

use

the

former.

The

features

are

used

for

both

the

title-begin

and

the

title-end

classifiers.

Format

Features

Font

Size:

There

are

four

binary

features

that

represent

the

normalized

font

size

of

the

unit

(recall

that

a

unit

has

only

one

type

of

font).

If

the

font

size

of

the

unit

is

the

largest

in

the

document,

then

the

first

feature

will

be

otherwise

If

the

font

size

is

the

smallest

in

the

document,

then

the

fourth

feature

will

be

otherwise

If

the

font

size

is

above

the

average

font

size

and

not

the

largest

in

the

document,

then

the

second

feature

will

be

otherwise

If

the

font

size

is

below

the

average

font

size

and

not

the

smallest,

the

third

feature

will

be

otherwise

It

is

necessary

to

conduct

normalization

on

font

sizes.

For

example,

in

one

document

the

largest

font

size

might

be

while

in

another

the

smallest

one

might

be

Boldface:

This

binary

feature

represents

whether

or

not

the

current

unit

is

in

boldface.

Alignment:

There

are

four

binary

features

that

respectively

represent

the

location

of

the

current

unit:

‘left",

‘center",

‘right",

and

‘unknown

alignment".

The

following

format

features

with

respect

to

‘context"

play

an

important

role

in

title

extraction.

Empty

Neighboring

Unit:

There

are

two

binary

features

that

represent,

respectively,

whether

or

not

the

previous

unit

and

the

current

unit

are

blank

lines.

Font

Size

Change:

There

are

two

binary

features

that

represent,

respectively,

whether

or

not

the

font

size

of

the

previous

unit

and

the

font

size

of

the

next

unit

differ

from

that

of

the

current

unit.

Alignment

Change:

There

are

two

binary

features

that

represent,

respectively,

whether

or

not

the

alignment

of

the

previous

unit

and

the

alignment

of

the

next

unit

differ

from

that

of

the

current

one.

Same

Paragraph:

There

are

two

binary

features

that

represent,

respectively,

whether

or

not

the

previous

unit

and

the

next

unit

are

in

the

same

paragraph

as

the

current

unit.

Linguistic

Features

The

linguistic

features

are

based

on

key

words.

Positive

Word:

This

binary

feature

represents

whether

or

not

the

current

unit

begins

with

one

of

the

positive

words.

The

positive

words

include

‘title:",

‘subject:",

‘subject

line:"

For

example,

in

some

documents

the

lines

of

titles

and

authors

have

the

same

formats.

However,

if

lines

begin

with

one

of

the

positive

words,

then

it

is

likely

that

they

are

title

lines.

Negative

Word:

This

binary

feature

represents

whether

or

not

the

current

unit

begins

with

one

of

the

negative

words.

The

negative

words

include

‘To",

‘By",

‘created

by",

‘updated

by",

etc.

There

are

more

negative

words

than

positive

words.

The

above

linguistic

features

are

language

dependent.

Word

Count:

A

title

should

not

be

too

long.

We

heuristically

create

four

intervals:

and

and

define

one

feature

for

each

interval.

If

the

number

of

words

in

a

title

falls

into

an

interval,

then

the

corresponding

feature

will

be

otherwise

Ending

Character:

This

feature

represents

whether

the

unit

ends

with

or

other

special

characters.

A

title

usually

does

not

end

with

such

a

character.

DOCUMENT

RETRIEVAL

METHOD

We

describe

our

method

of

document

retrieval

using

extracted

titles.

Typically,

in

information

retrieval

a

document

is

split

into

a

number

of

fields

including

body,

title,

and

anchor

text.

A

ranking

function

in

search

can

use

different

weights

for

different

fields

of

the

document.

Also,

titles

are

typically

assigned

high

weights,

indicating

that

they

are

important

for

document

retrieval.

As

explained

previously,

our

experiment

has

shown

that

a

significant

number

of

documents

actually

have

incorrect

titles

in

the

file

properties,

and

thus

in

addition

of

using

them

we

use

the

extracted

titles

as

one

more

field

of

the

document.

By

doing

this,

we

attempt

to

improve

the

overall

precision.

In

this

paper,

we

employ

a

modification

of

that

allows

field

weighting

As

fields,

we

make

use

of

body,

title,

extracted

title

and

anchor.

First,

for

each

term

in

the

query

we

count

the

term

frequency

in

each

field

of

the

document;

each

field

frequency

is

then

weighted

according

to

the

corresponding

weight

parameter:

f

tfft

tfwwtf

Similarly,

we

compute

the

document

length

as

a

weighted

sum

of

lengths

of

each

field.

Average

document

length

in

the

corpus

becomes

the

average

of

all

weighted

document

lengths.

f

ff

dlwwdl

In

our

experiments

we

used

bk

Weight

for

content

was

title

was

anchor

was

and

extracted

title

was

EXPERIMENTAL

RESULTS

Data

Sets

and

Evaluation

Measures

We

used

two

data

sets

in

our

experiments.

First,

we

downloaded

and

randomly

selected

Word

documents

and

PowerPoint

documents

from

an

intranet

of

Microsoft.

We

call

it

MS

hereafter.

Second,

we

downloaded

and

randomly

selected

Word

and

PowerPoint

documents

from

the

DotGov

and

DotCom

domains

on

the

internet,

respectively.

Figure

shows

the

distributions

of

the

genres

of

the

documents.

We

see

that

the

documents

are

indeed

‘general

documents"

as

we

define

them.

Figure

Distributions

of

document

genres.

Third,

a

data

set

in

Chinese

was

also

downloaded

from

the

internet.

It

includes

Word

documents

and

PowerPoint

documents

in

Chinese.

We

manually

labeled

the

titles

of

all

the

documents,

on

the

basis

of

our

specification.

Not

all

the

documents

in

the

two

data

sets

have

titles.

Table

shows

the

percentages

of

the

documents

having

titles.

We

see

that

DotCom

and

DotGov

have

more

PowerPoint

documents

with

titles

than

MS.

This

might

be

because

PowerPoint

documents

published

on

the

internet

are

more

formal

than

those

on

the

intranet.

Table

The

portion

of

documents

with

titles

Domain

Type

MS

DotCom

DotGov

Word

PowerPoint

In

our

experiments,

we

conducted

evaluations

on

title

extraction

in

terms

of

precision,

recall,

and

F-measure.

The

evaluation

measures

are

defined

as

follows:

Precision:

P

A

A

B

Recall:

R

A

A

C

F-measure:

P

R

Here,

A,

B,

C,

and

D

are

numbers

of

documents

as

those

defined

in

Table

Table

Contingence

table

with

regard

to

title

extraction

Is

title

Is

not

title

Extracted

A

B

Not

extracted

C

D

Baselines

We

test

the

accuracies

of

the

two

baselines

described

in

section

They

are

denoted

as

‘largest

font

size"

and

‘first

line"

respectively.

Accuracy

of

Titles

in

File

Properties

We

investigate

how

many

titles

in

the

file

properties

of

the

documents

are

reliable.

We

view

the

titles

annotated

by

humans

as

true

titles

and

test

how

many

titles

in

the

file

properties

can

approximately

match

with

the

true

titles.

We

use

Edit

Distance

to

conduct

the

approximate

match.

(Approximate

match

is

only

used

in

this

evaluation).

This

is

because

sometimes

human

annotated

titles

can

be

slightly

different

from

the

titles

in

file

properties

on

the

surface,

e.g.,

contain

extra

spaces).

Given

string

A

and

string

B:

if

(D

or

D

La

Lb

θ

then

string

A

string

B

D:

Edit

Distance

between

string

A

and

string

B

La:

length

of

string

A

Lb:

length

of

string

B

θ:

t

t

n

N

wtf

avwdl

wdl

bbk

kwtf

FBM

)log(

Table

Accuracies

of

titles

in

file

properties

File

Type

Domain

Precision

Recall

MS

DotCom

DotGov

MS

DotCom

DotGov

Comparison

with

Baselines

We

conducted

title

extraction

from

the

first

data

set

(Word

and

PowerPoint

in

MS).

As

the

model,

we

used

Perceptron.

We

conduct

cross

validation.

Thus,

all

the

results

reported

here

are

those

averaged

over

trials.

Tables

and

show

the

results.

We

see

that

Perceptron

significantly

outperforms

the

baselines.

In

the

evaluation,

we

use

exact

matching

between

the

true

titles

annotated

by

humans

and

the

extracted

titles.

Table

Accuracies

of

title

extraction

with

Word

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

Precision

Recall

Model

Perceptron

Largest

font

size

Baselines

First

line

We

see

that

the

machine

learning

approach

can

achieve

good

performance

in

title

extraction.

For

Word

documents

both

precision

and

recall

of

the

approach

are

percent

higher

than

those

of

the

baselines.

For

PowerPoint

both

precision

and

recall

of

the

approach

are

percent

higher

than

those

of

the

baselines.

We

conduct

significance

tests.

The

results

are

shown

in

Table

Here,

‘Largest"

denotes

the

baseline

of

using

the

largest

font

size,

‘First"

denotes

the

baseline

of

using

the

first

line.

The

results

indicate

that

the

improvements

of

machine

learning

over

baselines

are

statistically

significant

(in

the

sense

p-value

Table

Sign

test

results

Documents

Type

Sign

test

between

p-value

Perceptron

vs.

Largest

Word

Perceptron

vs.

First

Perceptron

vs.

Largest

PowerPoint

Perceptron

vs.

First

We

see,

from

the

results,

that

the

two

baselines

can

work

well

for

title

extraction,

suggesting

that

font

size

and

position

information

are

most

useful

features

for

title

extraction.

However,

it

is

also

obvious

that

using

only

these

two

features

is

not

enough.

There

are

cases

in

which

all

the

lines

have

the

same

font

size

(i.e.,

the

largest

font

size),

or

cases

in

which

the

lines

with

the

largest

font

size

only

contain

general

descriptions

like

‘Confidential",

‘White

paper",

etc.

For

those

cases,

the

‘largest

font

size"

method

cannot

work

well.

For

similar

reasons,

the

‘first

line"

method

alone

cannot

work

well,

either.

With

the

combination

of

different

features

(evidence

in

title

judgment),

Perceptron

can

outperform

Largest

and

First.

We

investigate

the

performance

of

solely

using

linguistic

features.

We

found

that

it

does

not

work

well.

It

seems

that

the

format

features

play

important

roles

and

the

linguistic

features

are

supplements..

Figure

An

example

Word

document.

Figure

An

example

PowerPoint

document.

We

conducted

an

error

analysis

on

the

results

of

Perceptron.

We

found

that

the

errors

fell

into

three

categories.

About

one

third

of

the

errors

were

related

to

‘hard

cases".

In

these

documents,

the

layouts

of

the

first

pages

were

difficult

to

understand,

even

for

humans.

Figure

and

shows

examples.

Nearly

one

fourth

of

the

errors

were

from

the

documents

which

do

not

have

true

titles

but

only

contain

bullets.

Since

we

conduct

extraction

from

the

top

regions,

it

is

difficult

to

get

rid

of

these

errors

with

the

current

approach.

Confusions

between

main

titles

and

subtitles

were

another

type

of

error.

Since

we

only

labeled

the

main

titles

as

titles,

the

extractions

of

both

titles

were

considered

incorrect.

This

type

of

error

does

little

harm

to

document

processing

like

search,

however.

Comparison

between

Models

To

compare

the

performance

of

different

machine

learning

models,

we

conducted

another

experiment.

Again,

we

perform

cross

validation

on

the

first

data

set

(MS).

Table

shows

the

results

of

all

the

four

models.

It

turns

out

that

Perceptron

and

PMM

perform

the

best,

followed

by

MEMM,

and

ME

performs

the

worst.

In

general,

the

Markovian

models

perform

better

than

or

as

well

as

their

classifier

counterparts.

This

seems

to

be

because

the

Markovian

models

are

trained

globally,

while

the

classifiers

are

trained

locally.

The

Perceptron

based

models

perform

better

than

the

ME

based

counterparts.

This

seems

to

be

because

the

Perceptron

based

models

are

created

to

make

better

classifications,

while

ME

models

are

constructed

for

better

prediction.

Table

Comparison

between

different

learning

models

for

title

extraction

with

Word

Model

Precision

Recall

Perceptron

MEMM

PMM

ME

Table

Comparison

between

different

learning

models

for

title

extraction

with

PowerPoint

Model

Precision

Recall

Perceptron

MEMM

PMM

ME

Domain

Adaptation

We

apply

the

model

trained

with

the

first

data

set

(MS)

to

the

second

data

set

(DotCom

and

DotGov).

Tables

show

the

results.

Table

Accuracies

of

title

extraction

with

Word

in

DotGov

Precision

Recall

Model

Perceptron

Largest

font

size

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

in

DotGov

Precision

Recall

Model

Perceptron

Largest

font

size

First

line

Table

Accuracies

of

title

extraction

with

Word

in

DotCom

Precisio

n

Recall

Model

Perceptron

Largest

font

size

First

line

Table

Performance

of

PowerPoint

document

title

extraction

in

DotCom

Precisio

n

Recall

Model

Perceptron

Largest

font

size

First

line

From

the

results,

we

see

that

the

models

can

be

adapted

to

different

domains

well.

There

is

almost

no

drop

in

accuracy.

The

results

indicate

that

the

patterns

of

title

formats

exist

across

different

domains,

and

it

is

possible

to

construct

a

domain

independent

model

by

mainly

using

formatting

information.

Language

Adaptation

We

apply

the

model

trained

with

the

data

in

English

(MS)

to

the

data

set

in

Chinese.

Tables

show

the

results.

Table

Accuracies

of

title

extraction

with

Word

in

Chinese

Precision

Recall

Model

Perceptron

Largest

font

size

First

line

Table

Accuracies

of

title

extraction

with

PowerPoint

in

Chinese

Precision

Recall

Model

Perceptron

Largest

font

size

First

line

We

see

that

the

models

can

be

adapted

to

a

different

language.

There

are

only

small

drops

in

accuracy.

Obviously,

the

linguistic

features

do

not

work

for

Chinese,

but

the

effect

of

not

using

them

is

negligible.

The

results

indicate

that

the

patterns

of

title

formats

exist

across

different

languages.

From

the

domain

adaptation

and

language

adaptation

results,

we

conclude

that

the

use

of

formatting

information

is

the

key

to

a

successful

extraction

from

general

documents.

Search

with

Extracted

Titles

We

performed

experiments

on

using

title

extraction

for

document

retrieval.

As

a

baseline,

we

employed

without

using

extracted

titles.

The

ranking

mechanism

was

as

described

in

Section

The

weights

were

heuristically

set.

We

did

not

conduct

optimization

on

the

weights.

The

evaluation

was

conducted

on

a

corpus

of

M

documents

crawled

from

the

intranet

of

Microsoft

using

evaluation

queries

obtained

from

this

intranet"s

search

engine

query

logs.

queries

were

from

the

most

popular

set,

while

queries

other

were

chosen

randomly.

Users

were

asked

to

provide

judgments

of

the

degree

of

document

relevance

from

a

scale

of

meaning

detrimental,

bad,

fair,

good

and

excellent).

Figure

shows

the

results.

In

the

chart

two

sets

of

precision

results

were

obtained

by

either

considering

good

or

excellent

documents

as

relevant

(left

bars

with

relevance

threshold

or

by

considering

only

excellent

documents

as

relevant

(right

bars

with

relevance

threshold

Reciprocal

Reciprocal

Anchor,

Title,

Body

Anchor,

Title,

Body,

ExtractedTitle

Name

All

RelevanceThreshold

Data

Description

Figure

Search

ranking

results.

Figure

shows

different

document

retrieval

results

with

different

ranking

functions

in

terms

of

precision

precision

and

reciprocal

rank:

Blue

bar

including

the

fields

body,

title

(file

property),

and

anchor

text.

Purple

bar

including

the

fields

body,

title

(file

property),

anchor

text,

and

extracted

title.

With

the

additional

field

of

extracted

title

included

in

the

precision

increased

from

to

or

by

Thus,

it

is

safe

to

say

that

the

use

of

extracted

title

can

indeed

improve

the

precision

of

document

retrieval.

CONCLUSION

In

this

paper,

we

have

investigated

the

problem

of

automatically

extracting

titles

from

general

documents.

We

have

tried

using

a

machine

learning

approach

to

address

the

problem.

Previous

work

showed

that

the

machine

learning

approach

can

work

well

for

metadata

extraction

from

research

papers.

In

this

paper,

we

showed

that

the

approach

can

work

for

extraction

from

general

documents

as

well.

Our

experimental

results

indicated

that

the

machine

learning

approach

can

work

significantly

better

than

the

baselines

in

title

extraction

from

Office

documents.

Previous

work

on

metadata

extraction

mainly

used

linguistic

features

in

documents,

while

we

mainly

used

formatting

information.

It

appeared

that

using

formatting

information

is

a

key

for

successfully

conducting

title

extraction

from

general

documents.

We

tried

different

machine

learning

models

including

Perceptron,

Maximum

Entropy,

Maximum

Entropy

Markov

Model,

and

Voted

Perceptron.

We

found

that

the

performance

of

the

Perceptorn

models

was

the

best.

We

applied

models

constructed

in

one

domain

to

another

domain

and

applied

models

trained

in

one

language

to

another

language.

We

found

that

the

accuracies

did

not

drop

substantially

across

different

domains

and

across

different

languages,

indicating

that

the

models

were

generic.

We

also

attempted

to

use

the

extracted

titles

in

document

retrieval.

We

observed

a

significant

improvement

in

document

ranking

performance

for

search

when

using

extracted

title

information.

All

the

above

investigations

were

not

conducted

in

previous

work,

and

through

our

investigations

we

verified

the

generality

and

the

significance

of

the

title

extraction

approach.

ACKNOWLEDGEMENTS

We

thank

Chunyu

Wei

and

Bojuan

Zhao

for

their

work

on

data

annotation.

We

acknowledge

Jinzhu

Li

for

his

assistance

in

conducting

the

experiments.

We

thank

Ming

Zhou,

John

Chen,

Jun

Xu,

and

the

anonymous

reviewers

of

for

their

valuable

comments

on

this

paper.

REFERENCES

Berger,

A.

L.,

Della

Pietra,

S.

A.,

and

Della

Pietra,

V.

J.

A

maximum

entropy

approach

to

natural

language

processing.

Computational

Linguistics,

Collins,

M.

Discriminative

training

methods

for

hidden

markov

models:

theory

and

experiments

with

perceptron

algorithms.

In

Proceedings

of

Conference

on

Empirical

Methods

in

Natural

Language

Processing,

Cortes,

C.

and

Vapnik,

V.

Support-vector

networks.

Machine

Learning,

Chieu,

H.

L.

and

Ng,

H.

T.

A

maximum

entropy

approach

to

information

extraction

from

semi-structured

and

free

text.

In

Proceedings

of

the

Eighteenth

National

Conference

on

Artificial

Intelligence,

Evans,

D.

K.,

Klavans,

J.

L.,

and

McKeown,

K.

R.

Columbia

newsblaster:

multilingual

news

summarization

on

the

Web.

In

Proceedings

of

Human

Language

Technology

conference

North

American

chapter

of

the

Association

for

Computational

Linguistics

annual

meeting,

Ghahramani,

Z.

and

Jordan,

M.

I.

Factorial

hidden

markov

models.

Machine

Learning,

Gheel,

J.

and

Anderson,

T.

Data

and

metadata

for

finding

and

reminding,

In

Proceedings

of

the

International

Conference

on

Information

Visualization,

Giles,

C.

L.,

Petinot,

Y.,

Teregowda

P.

B.,

Han,

H.,

Lawrence,

S.,

Rangaswamy,

A.,

and

Pal,

N.

eBizSearch:

a

niche

search

engine

for

e-Business.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

Giuffrida,

G.,

Shek,

E.

C.,

and

Yang,

J.

Knowledge-based

metadata

extraction

from

PostScript

files.

In

Proceedings

of

the

Fifth

ACM

Conference

on

Digital

Libraries,

Han,

H.,

Giles,

C.

L.,

Manavoglu,

E.,

Zha,

H.,

Zhang,

Z.,

and

Fox,

E.

A.

Automatic

document

metadata

extraction

using

support

vector

machines.

In

Proceedings

of

the

Third

ACM/IEEE-CS

Joint

Conference

on

Digital

Libraries,

Kobayashi,

M.,

and

Takeda,

K.

Information

retrieval

on

the

Web.

ACM

Computing

Surveys,

Lafferty,

J.,

McCallum,

A.,

and

Pereira,

F.

Conditional

random

fields:

probabilistic

models

for

segmenting

and

labeling

sequence

data.

In

Proceedings

of

the

Eighteenth

International

Conference

on

Machine

Learning,

Li,

Y.,

Zaragoza,

H.,

Herbrich,

R.,

Shawe-Taylor

J.,

and

Kandola,

J.

S.

The

perceptron

algorithm

with

uneven

margins.

In

Proceedings

of

the

Nineteenth

International

Conference

on

Machine

Learning,

Liddy,

E.

D.,

Sutton,

S.,

Allen,

E.,

Harwell,

S.,

Corieri,

S.,

Yilmazel,

O.,

Ozgencil,

N.

E.,

Diekema,

A.,

McCracken,

N.,

and

Silverstein,

J.

Automatic

Metadata

generation

evaluation.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

Littlefield,

A.

Effective

enterprise

information

retrieval

across

new

content

formats.

In

Proceedings

of

the

Seventh

Search

Engine

Conference,

Mao,

S.,

Kim,

J.

W.,

and

Thoma,

G.

R.

A

dynamic

feature

generation

system

for

automated

metadata

extraction

in

preservation

of

digital

materials.

In

Proceedings

of

the

First

International

Workshop

on

Document

Image

Analysis

for

Libraries,

McCallum,

A.,

Freitag,

D.,

and

Pereira,

F.

Maximum

entropy

markov

models

for

information

extraction

and

segmentation.

In

Proceedings

of

the

Seventeenth

International

Conference

on

Machine

Learning,

Murphy,

L.

D.

Digital

document

metadata

in

organizations:

roles,

analytical

approaches,

and

future

research

directions.

In

Proceedings

of

the

Thirty-First

Annual

Hawaii

International

Conference

on

System

Sciences,

Pinto,

D.,

McCallum,

A.,

Wei,

X.,

and

Croft,

W.

B.

Table

extraction

using

conditional

random

fields.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

Ratnaparkhi,

A.

Unsupervised

statistical

models

for

prepositional

phrase

attachment.

In

Proceedings

of

the

Seventeenth

International

Conference

on

Computational

Linguistics.

Robertson,

S.,

Zaragoza,

H.,

and

Taylor,

M.

Simple

extension

to

multiple

weighted

fields,

In

Proceedings

of

ACM

Thirteenth

Conference

on

Information

and

Knowledge

Management,

Yi,

J.

and

Sundaresan,

N.

Metadata

based

Web

mining

for

relevance,

In

Proceedings

of

the

International

Symposium

on

Database

Engineering

Applications,

Yilmazel,

O.,

Finneran,

C.

M.,

and

Liddy,

E.

D.

MetaExtract:

An

NLP

system

to

automatically

assign

metadata.

In

Proceedings

of

the

Joint

ACM/IEEE

Conference

on

Digital

Libraries,

Zhang,

J.

and

Dimitroff,

A.

Internet

search

engines'

response

to

metadata

Dublin

Core

implementation.

Journal

of

Information

Science,

Zhang,

L.,

Pan,

Y.,

and

Zhang,

T.

Recognising

and

using

named

entities:

focused

named

entity

recognition

using

machine

learning.

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval,

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

language independence
metada of document
linguistic feature
comparison between model
model generality
usefulness of extracted title
extracted title usefulness
genre
automatic title extraction
metada extraction
machine learn

title extraction
formatting information
generality of model
classifier
document retrieval
information extraction
search