Combining

Content

and

Link

for

Classification

using

Matrix

Factorization

Shenghuo

Zhu

Kai

Yu

Yun

Chi

Yihong

Gong

{zsh,kyu,ychi,ygong}@sv.nec-labs.com

NEC

Laboratories

America,

Inc.

North

Wolfe

Road

Cupertino,

CA

USA

ABSTRACT

The

world

wide

web

contains

rich

textual

contents

that

are

interconnected

via

complex

hyperlinks.

This

huge

database

violates

the

assumption

held

by

most

of

conventional

statistical

methods

that

each

web

page

is

considered

as

an

independent

and

identical

sample.

It

is

thus

difficult

to

apply

traditional

mining

or

learning

methods

for

solving

web

mining

problems,

e.g.,

web

page

classification,

by

exploiting

both

the

content

and

the

link

structure.

The

research

in

this

direction

has

recently

received

considerable

attention

but

are

still

in

an

early

stage.

Though

a

few

methods

exploit

both

the

link

structure

or

the

content

information,

some

of

them

combine

the

only

authority

information

with

the

content

information,

and

the

others

first

decompose

the

link

structure

into

hub

and

authority

features,

then

apply

them

as

additional

document

features.

Being

practically

attractive

for

its

great

simplicity,

this

paper

aims

to

design

an

algorithm

that

exploits

both

the

content

and

linkage

information,

by

carrying

out

a

joint

factorization

on

both

the

linkage

adjacency

matrix

and

the

document-term

matrix,

and

derives

a

new

representation

for

web

pages

in

a

low-dimensional

factor

space,

without

explicitly

separating

them

as

content,

hub

or

authority

factors.

Further

analysis

can

be

performed

based

on

the

compact

representation

of

web

pages.

In

the

experiments,

the

proposed

method

is

compared

with

state-of-the-art

methods

and

demonstrates

an

excellent

accuracy

in

hypertext

classification

on

the

WebKB

and

Cora

benchmarks.

Categories

and

Subject

Descriptors:

[Information

Systems]:

Information

Search

and

Retrieval

General

Terms:

Algorithms,

Experimentation

INTRODUCTION

With

the

advance

of

the

World

Wide

Web,

more

and

more

hypertext

documents

become

available

on

the

Web.

Some

examples

of

such

data

include

organizational

and

personal

web

pages

(e.g,

the

WebKB

benchmark

data

set,

which

contains

university

web

pages),

research

papers

(e.g.,

data

in

CiteSeer),

online

news

articles,

and

customer-generated

media

(e.g.,

blogs).

Comparing

to

data

in

traditional

information

management,

in

addition

to

content,

these

data

on

the

Web

also

contain

links:

e.g.,

hyperlinks

from

a

student"s

homepage

pointing

to

the

homepage

of

her

advisor,

paper

citations,

sources

of

a

news

article,

comments

of

one

blogger

on

posts

from

another

blogger,

and

so

on.

Performing

information

management

tasks

on

such

structured

data

raises

many

new

research

challenges.

In

the

following

discussion,

we

use

the

task

of

web

page

classification

as

an

illustrating

example,

while

the

techniques

we

develop

in

later

sections

are

applicable

equally

well

to

many

other

tasks

in

information

retrieval

and

data

mining.

For

the

classification

problem

of

web

pages,

a

simple

approach

is

to

treat

web

pages

as

independent

documents.

The

advantage

of

this

approach

is

that

many

off-the-shelf

classification

tools

can

be

directly

applied

to

the

problem.

However,

this

approach

relies

only

on

the

content

of

web

pages

and

ignores

the

structure

of

links

among

them.

Link

structures

provide

invaluable

information

about

properties

of

the

documents

as

well

as

relationships

among

them.

For

example,

in

the

WebKB

dataset,

the

link

structure

provides

additional

insights

about

the

relationship

among

documents

(e.g.,

links

often

pointing

from

a

student

to

her

advisor

or

from

a

faculty

member

to

his

projects).

Since

some

links

among

these

documents

imply

the

inter-dependence

among

the

documents,

the

usual

i.i.d.

(independent

and

identical

distributed)

assumption

of

documents

does

not

hold

any

more.

From

this

point

of

view,

the

traditional

classification

methods

that

ignore

the

link

structure

may

not

be

suitable.

On

the

other

hand,

a

few

studies,

for

example

rely

solely

on

link

structures.

It

is

however

a

very

rare

case

that

content

information

can

be

ignorable.

For

example,

in

the

Cora

dataset,

the

content

of

a

research

article

abstract

largely

determines

the

category

of

the

article.

To

improve

the

performance

of

web

page

classification,

therefore,

both

link

structure

and

content

information

should

be

taken

into

consideration.

To

achieve

this

goal,

a

simple

approach

is

to

convert

one

type

of

information

to

the

other.

For

example,

in

spam

blog

classification,

Kolari

et

al.

concatenate

outlink

features

with

the

content

features

of

the

blog.

In

document

classification,

Kurland

and

Lee

convert

content

similarity

among

documents

into

weights

of

links.

However,

link

and

content

information

have

different

properties.

For

example,

a

link

is

an

actual

piece

of

evidence

that

represents

an

asymmetric

relationship

whereas

the

content

similarity

is

usually

defined

conceptually

for

every

pair

of

documents

in

a

symmetric

way.

Therefore,

directly

converting

one

type

of

information

to

the

other

usually

degrades

the

quality

of

information.

On

the

other

hand,

there

exist

some

studies,

as

we

will

discuss

in

detail

in

related

work,

that

consider

link

information

and

content

information

separately

and

then

combine

them.

We

argue

that

such

an

approach

ignores

the

inherent

consistency

between

link

and

content

information

and

therefore

fails

to

combine

the

two

seamlessly.

Some

work,

such

as

incorporates

link

information

using

cocitation

similarity,

but

this

may

not

fully

capture

the

global

link

structure.

In

Figure

for

example,

web

pages

and

co-cite

web

page

implying

that

and

are

similar

to

each

other.

In

turns,

and

should

be

similar

to

each

other,

since

and

cite

similar

web

pages

and

respectively.

But

using

cocitation

similarity,

the

similarity

between

and

is

zero

without

considering

other

information.

Figure

An

example

of

link

structure

In

this

paper,

we

propose

a

simple

technique

for

analyzing

inter-connected

documents,

such

as

web

pages,

using

factor

In

the

proposed

technique,

both

content

information

and

link

structures

are

seamlessly

combined

through

a

single

set

of

latent

factors.

Our

model

contains

two

components.

The

first

component

captures

the

content

information.

This

component

has

a

form

similar

to

that

of

the

latent

topics

in

the

Latent

Semantic

Indexing

(LSI)

in

traditional

information

retrieval.

That

is,

documents

are

decomposed

into

latent

topics/factors,

which

in

turn

are

represented

as

term

vectors.

The

second

component

captures

the

information

contained

in

the

underlying

link

structure,

such

as

links

from

homepages

of

students

to

those

of

faculty

members.

A

factor

can

be

loosely

considered

as

a

type

of

documents

(e.g.,

those

homepages

belonging

to

students).

It

is

worth

noting

that

we

do

not

explicitly

define

the

semantic

of

a

factor

a

priori.

Instead,

similar

to

LSI,

the

factors

are

learned

from

the

data.

Traditional

factor

analysis

models

the

variables

associated

with

entities

through

the

factors.

However,

in

analysis

of

link

structures,

we

need

to

model

the

relationship

of

two

ends

of

links,

i.e.,

edges

between

vertex

pairs.

Therefore,

the

model

should

involve

factors

of

both

vertices

of

the

edge.

This

is

a

key

difference

between

traditional

factor

analysis

and

our

model.

In

our

model,

we

connect

two

components

through

a

set

of

shared

factors,

that

is,

the

latent

factors

in

the

second

component

(for

contents)

are

tied

to

the

factors

in

the

first

component

(for

links).

By

doing

this,

we

search

for

a

unified

set

of

latent

factors

that

best

explains

both

content

and

link

structures

simultaneously

and

seamlessly.

In

the

formulation,

we

perform

factor

analysis

based

on

matrix

factorization:

solution

to

the

first

component

is

based

on

factorizing

the

term-document

matrix

derived

from

content

features;

solution

to

the

second

component

is

based

on

factorizing

the

adjacency

matrix

derived

from

links.

Because

the

two

factorizations

share

a

common

base,

the

discovered

bases

(latent

factors)

explain

both

content

information

and

link

structures,

and

are

then

used

in

further

information

management

tasks

such

as

classification.

This

paper

is

organized

as

follows.

Section

reviews

related

work.

Section

presents

the

proposed

approach

to

analyze

the

web

page

based

on

the

combined

information

of

links

and

content.

Section

extends

the

basic

framework

and

a

few

variants

for

fine

tune.

Section

shows

the

experiment

results.

Section

discusses

the

details

of

this

approach

and

Section

concludes.

RELATED

WORK

In

the

content

analysis

part,

our

approach

is

closely

related

to

Latent

Semantic

Indexing

(LSI)

LSI

maps

documents

into

a

lower

dimensional

latent

space.

The

latent

space

implicitly

captures

a

large

portion

of

information

of

documents,

therefore

it

is

called

the

latent

semantic

space.

The

similarity

between

documents

could

be

defined

by

the

dot

products

of

the

corresponding

vectors

of

documents

in

the

latent

space.

Analysis

tasks,

such

as

classification,

could

be

performed

on

the

latent

space.

The

commonly

used

singular

value

decomposition

(SVD)

method

ensures

that

the

data

points

in

the

latent

space

can

optimally

reconstruct

the

original

documents.

Though

our

approach

also

uses

latent

space

to

represent

web

pages

(documents),

we

consider

the

link

structure

as

well

as

the

content

of

web

pages.

In

the

link

analysis

approach,

the

framework

of

hubs

and

authorities

(HITS)

puts

web

page

into

two

categories,

hubs

and

authorities.

Using

recursive

notion,

a

hub

is

a

web

page

with

many

outgoing

links

to

authorities,

while

an

authority

is

a

web

page

with

many

incoming

links

from

hubs.

Instead

of

using

two

categories,

PageRank

uses

a

single

category

for

the

recursive

notion,

an

authority

is

a

web

page

with

many

incoming

links

from

authorities.

He

et

al.

propose

a

clustering

algorithm

for

web

document

clustering.

The

algorithm

incorporates

link

structure

and

the

co-citation

patterns.

In

the

algorithm,

all

links

are

treated

as

undirected

edge

of

the

link

graph.

The

content

information

is

only

used

for

weighing

the

links

by

the

textual

similarity

of

both

ends

of

the

links.

Zhang

et

al.

uses

the

undirected

graph

regularization

framework

for

document

classification.

Achlioptas

et

decompose

the

web

into

hub

and

authority

attributes

then

combine

them

with

content.

Zhou

et

al.

and

propose

a

directed

graph

regularization

framework

for

semi-supervised

learning.

The

framework

combines

the

hub

and

authority

information

of

web

pages.

But

it

is

difficult

to

combine

the

content

information

into

that

framework.

Our

approach

consider

the

content

and

the

directed

linkage

between

topics

of

source

and

destination

web

pages

in

one

step,

which

implies

the

topic

combines

the

information

of

web

page

as

authorities

and

as

hubs

in

a

single

set

of

factors.

Cohn

and

Hofmann

construct

the

latent

space

from

both

content

and

link

information,

using

content

analysis

based

on

probabilistic

LSI

(PLSI)

and

link

analysis

based

on

PHITS

The

major

difference

between

the

approach

of

(PLSI+PHITS)

and

our

approach

is

in

the

part

of

link

analysis.

In

PLSI+PHITS,

the

link

is

constructed

with

the

linkage

from

the

topic

of

the

source

web

page

to

the

destination

web

page.

In

the

model,

the

outgoing

links

of

the

destination

web

page

have

no

effect

on

the

source

web

page.

In

other

words,

the

overall

link

structure

is

not

utilized

in

PHITS.

In

our

approach,

the

link

is

constructed

with

the

linkage

between

the

factor

of

the

source

web

page

and

the

factor

of

the

destination

web

page,

instead

of

the

destination

web

page

itself.

The

factor

of

the

destination

web

page

contains

information

of

its

outgoing

links.

In

turn,

such

information

is

passed

to

the

factor

of

the

source

web

page.

As

the

result

of

matrix

factorization,

the

factor

forms

a

factor

graph,

a

miniature

of

the

original

graph,

preserving

the

major

structure

of

the

original

graph.

Taskar

et

al.

propose

relational

Markov

networks

(RMNs)

for

entity

classification,

by

describing

a

conditional

distribution

of

entity

classes

given

entity

attributes

and

relationships.

The

model

was

applied

to

web

page

classification,

where

web

pages

are

entities

and

hyperlinks

are

treated

as

relationships.

RMNs

apply

conditional

random

fields

to

define

a

set

of

potential

functions

on

cliques

of

random

variables,

where

the

link

structure

provides

hints

to

form

the

cliques.

However

the

model

does

not

give

an

off-the-shelf

solution,

because

the

success

highly

depends

on

the

arts

of

designing

the

potential

functions.

On

the

other

hand,

the

inference

for

RMNs

is

intractable

and

requires

belief

propagation.

The

following

are

some

work

on

combining

documents

and

links,

but

the

methods

are

loosely

related

to

our

approach.

The

experiments

of

show

that

using

terms

from

the

linked

document

improves

the

classification

accuracy.

Chakrabarti

et

use

co-citation

information

in

their

classification

model.

Joachims

et

combine

text

kernels

and

co-citation

kernels

for

classification.

Oh

et

al

use

the

Naive

Bayesian

frame

to

combine

link

information

with

content.

OUR

APPROACH

In

this

section

we

will

first

introduce

a

novel

matrix

factorization

method,

which

is

more

suitable

than

conventional

matrix

factorization

methods

for

link

analysis.

Then

we

will

introduce

our

approach

that

jointly

factorizes

the

document-term

matrix

and

link

matrix

and

obtains

compact

and

highly

indicative

factors

for

representing

documents

or

web

pages.

Link

Matrix

Factorization

Suppose

we

have

a

directed

graph

G

(V,

E),

where

the

vertex

set

V

{vi}n

represents

the

web

pages

and

the

edge

set

E

represents

the

hyperlinks

between

web

pages.

Let

A

{asd}

denotes

the

n×n

adjacency

matrix

of

G,

which

is

also

called

the

link

matrix

in

this

paper.

For

a

pair

of

vertices,

vs

and

vd,

let

asd

when

there

is

an

edge

from

vs

to

vd,

and

asd

otherwise.

Note

that

A

is

an

asymmetric

matrix,

because

hyperlinks

are

directed.

Most

machine

learning

algorithms

assume

a

feature-vector

representation

of

instances.

For

web

page

classification,

however,

the

link

graph

does

not

readily

give

such

a

vector

representation

for

web

pages.

If

one

directly

uses

each

row

or

column

of

A

for

the

job,

she

will

suffer

a

very

high

computational

cost

because

the

dimensionality

equals

to

the

number

of

web

pages.

On

the

other

hand,

it

will

produces

a

poor

classification

accuracy

(see

our

experiments

in

Section

because

A

is

extremely

The

idea

of

link

matrix

factorization

is

to

derive

a

high-quality

feature

representation

Z

of

web

pages

based

on

analyzing

the

link

matrix

A,

where

Z

is

an

n

l

matrix,

with

each

row

being

the

ldimensional

feature

vector

of

a

web

page.

The

new

representation

of

web

pages

captures

the

principal

factors

of

the

link

structure

and

makes

further

processing

more

efficient.

One

may

use

a

method

similar

to

LSI,

to

apply

the

well-known

principal

component

analysis

(PCA)

for

deriving

Z

from

A.

The

corresponding

optimization

problem

is

min

Z,U

A

ZU

F

γ

U

F

where

γ

is

a

small

positive

number,

U

is

an

l

×n

matrix,

and

F

is

the

Frobenius

norm.

The

optimization

aims

to

approximate

A

by

ZU

a

product

of

two

low-rank

matrices,

with

a

regularization

on

U.

In

the

end,

the

i-th

row

vector

of

Z

can

be

thought

as

the

hub

feature

vector

of

vertex

vi,

and

the

row

vector

of

U

can

be

thought

as

the

authority

features.

A

link

generation

model

proposed

in

is

similar

to

the

PCA

approach.

Since

A

is

a

nonnegative

matrix

here,

one

can

also

consider

to

put

nonnegative

constraints

on

U

and

Z,

which

produces

an

algorithm

similar

to

PLSA

and

NMF

Due

to

the

sparsity

of

A,

links

from

two

similar

pages

may

not

share

any

common

target

pages,

which

makes

them

to

appear

dissimilar.

However

the

two

pages

may

be

indirectly

linked

to

many

common

pages

via

their

neighbors.

Another

equivalent

form

is

minZ,U

A

ZU

F

s.

t.

U

U

I.

The

solution

Z

is

identical

subject

to

a

scaling

factor.

However,

despite

its

popularity

in

matrix

analysis,

PCA

(or

other

similar

methods

like

PLSA)

is

restrictive

for

link

matrix

factorization.

The

major

problem

is

that,

PCA

ignores

the

fact

that

the

rows

and

columns

of

A

are

indexed

by

exactly

the

same

set

of

objects

(i.e.,

web

pages).

The

approximating

matrix

˜A

ZU

shows

no

evidence

that

links

are

within

the

same

set

of

objects.

To

see

the

drawback,

let"s

consider

a

link

transitivity

situation

vi

vs

vj,

where

page

i

is

linked

to

page

s

which

itself

is

linked

to

page

j.

Since

˜A

ZU

treats

A

as

links

from

web

pages

{vi}

to

a

different

set

of

objects,

let

it

be

denoted

by

{oi},

˜A

ZU

actually

splits

an

linked

object

os

from

vs

and

breaks

down

the

link

path

into

two

parts

vi

os

and

vs

oj.

This

is

obviously

a

miss

interpretation

to

the

original

link

path.

To

overcome

the

problem

of

PCA,

in

this

paper

we

suggest

to

use

a

different

factorization:

min

Z,U

A

ZUZ

F

γ

U

F

where

U

is

an

l

l

full

matrix.

Note

that

U

is

not

symmetric,

thus

ZUZ

produces

an

asymmetric

matrix,

which

is

the

case

of

A.

Again,

each

row

vector

of

Z

corresponds

to

a

feature

vector

of

a

web

pages.

The

new

approximating

form

˜A

ZUZ

puts

a

clear

meaning

that

the

links

are

between

the

same

set

of

objects,

represented

by

features

Z.

The

factor

model

actually

maps

each

vertex,

vi,

into

a

vector

zi

{zi,k;

k

l}

in

the

Rl

space.

We

call

the

Rl

space

the

factor

space.

Then,

{zi}

encodes

the

information

of

incoming

and

outgoing

connectivity

of

vertices

{vi}.

The

factor

loadings,

U,

explain

how

these

observed

connections

happened

based

on

{zi}.

Once

we

have

the

vector

zi,

we

can

use

many

traditional

classification

methods

(such

as

SVMs)

or

clustering

tools

(such

as

K-Means)

to

perform

the

analysis.

Illustration

Based

on

a

Synthetic

Problem

To

further

illustrate

the

advantages

of

the

proposed

link

matrix

factorization

Eq.

let

us

consider

the

graph

in

Figure

Given

Figure

Summarize

Figure

with

a

factor

graph

these

observations,

we

can

summarize

the

graph

by

grouping

as

factor

graph

depicted

in

Figure

In

the

next

we

preform

the

two

factorization

methods

Eq.

and

Eq.

on

this

link

matrix.

A

good

low-rank

representation

should

reveal

the

structure

of

the

factor

graph.

First

we

try

PCA-like

decomposition,

solving

Eq.

and

obtaining

Z

U

We

can

see

that

the

row

vectors

of

and

are

the

same

in

Z,

indicating

that

and

have

the

same

hub

attributes.

The

row

vectors

of

and

are

the

same

in

U,

indicating

that

and

have

the

same

authority

attributes.

It

is

not

clear

to

see

the

similarity

between

and

because

their

inlinks

(and

outlinks)

are

different.

Then,

we

factorize

A

by

ZUZ

via

solving

Eq.

and

obtain

the

results

Z

U

The

resultant

Z

is

very

consistent

with

the

clustering

structure

of

vertices:

the

row

vectors

of

and

are

the

same,

those

of

and

are

the

same,

those

of

and

are

the

same.

Even

interestingly,

if

we

add

constraints

to

ensure

Z

and

U

be

nonnegative,

we

have

Z

U

which

clearly

tells

the

assignment

of

vertices

to

clusters

from

Z

and

the

links

of

factor

graph

from

U.

When

the

interpretability

is

not

critical

in

some

tasks,

for

example,

classification,

we

found

that

it

achieves

better

accuracies

without

the

nonnegative

constraints.

Given

our

above

analysis,

it

is

clear

that

the

factorization

ZUZ

is

more

expressive

than

ZU

in

representing

the

link

matrix

A.

Content

Matrix

Factorization

Now

let

us

consider

the

content

information

on

the

vertices.

To

combine

the

link

information

and

content

information,

we

want

to

use

the

same

latent

space

to

approximate

the

content

as

the

latent

space

for

the

links.

Using

the

bag-of-words

approach,

we

denote

the

content

of

web

pages

by

an

n×m

matrix

C,

each

of

whose

rows

represents

a

document,

each

column

represents

a

keyword,

where

m

is

the

number

of

keywords.

Like

the

latent

semantic

indexing

(LSI)

the

l-dimensional

latent

space

for

words

is

denoted

by

an

m

l

matrix

V

Therefore,

we

use

ZV

to

approximate

matrix

C,

min

V,Z

C

ZV

F

β

V

F

where

β

is

a

small

positive

number,

β

V

F

serves

as

a

regularization

term

to

improve

the

robustness.

Joint

Link-Content

Matrix

Factorization

There

are

many

ways

to

employ

both

the

content

and

link

information

for

web

page

classification.

Our

idea

in

this

paper

is

not

to

simply

combine

them,

but

rather

to

fuse

them

into

a

single,

consistent,

and

compact

feature

representation.

To

achieve

this

goal,

we

solve

the

following

problem,

min

U,V,Z

n

J

(U,

V,

Z)

def

A

ZUZ

F

α

C

ZV

F

γ

U

F

β

V

F

o

Eq.

is

the

joined

matrix

factorization

of

A

and

C

with

regularization.

The

new

representation

Z

is

ensured

to

capture

both

the

structures

of

the

link

matrix

A

and

the

content

matrix

C.

Once

we

find

the

optimal

Z,

we

can

apply

the

traditional

classification

or

clustering

methods

on

vectorial

data

Z.

The

relationship

among

these

matrices

can

be

depicted

as

Figure

A

Y

C

U

Z

V

Figure

Relationship

among

the

matrices.

Node

Y

is

the

target

of

classification.

Eq.

can

be

solved

using

gradient

methods,

such

as

the

conjugate

gradient

method

and

quasi-Newton

methods.

Then

main

computation

of

gradient

methods

is

evaluating

the

object

function

J

and

its

gradients

against

variables,

∂J

∂U

Z

ZUZ

Z

Z

AZ

γU,

∂J

∂V

=α

V

Z

Z

C

Z

βV,

∂J

∂Z

ZU

Z

ZU

ZUZ

ZU

A

ZU

AZU

α

ZV

V

CV

Because

of

the

sparsity

of

A,

the

computational

complexity

of

multiplication

of

A

and

Z

is

O(µAl),

where

µA

is

the

number

of

nonzero

entries

in

A.

Similarly,

the

computational

complexity

of

C

Z

and

CV

is

O(µC

l),

where

µC

is

the

number

of

nonzero

entries

in

C.

The

computational

complexity

of

the

rest

multiplications

in

the

gradient

computation

is

Therefore,

the

total

computational

complexity

in

one

iteration

is

O(µAl

µC

l

The

number

of

links

and

the

number

of

words

in

a

web

page

are

relatively

small

comparing

to

the

number

of

web

pages,

and

are

almost

constant

as

the

number

of

web

pages/documents

increases,

i.e.

µA

O(n)

and

µC

O(n).

Therefore,

theoretically

the

computation

time

is

almost

linear

to

the

number

of

web

pages/documents,

n.

SUPERVISED

MATRIX

FACTORIZATION

Consider

a

web

page

classification

problem.

We

can

solve

Eq.

to

obtain

Z

as

Section

then

use

a

traditional

classifier

to

perform

classification.

However,

this

approach

does

not

take

data

labels

into

account

in

the

first

step.

Believing

that

using

data

labels

improves

the

accuracy

by

obtaining

a

better

Z

for

the

classification,

we

consider

to

use

the

data

labels

to

guide

the

matrix

factorization,

called

supervised

matrix

factorization

Because

some

data

used

in

the

matrix

factorization

have

no

label

information,

the

supervised

matrix

factorization

falls

into

the

category

of

semi-supervised

learning.

Let

C

be

the

set

of

classes.

For

simplicity,

we

first

consider

binary

class

problem,

i.e.

C

Assume

we

know

the

labels

{yi}

for

vertices

in

T

V.

We

want

to

find

a

hypothesis

h

V

R,

such

that

we

assign

vi

to

when

h(vi)

−1

otherwise.

We

assume

a

transform

from

the

latent

space

to

R

is

linear,

i.e.

h(vi)

w

φ(vi)

b

w

zi

b,

School

course

dept.

faculty

other

project

staff

student

total

Cornell

Texas

Washington

Wisconsin

Table

Dataset

of

WebKB

where

w

and

b

are

parameters

to

estimate.

Here,

w

is

the

norm

of

the

decision

boundary.

Similar

to

Support

Vector

Machines

(SVMs)

we

can

use

the

hinge

loss

to

measure

the

loss,

X

i:vi∈T

yih(vi)]+

where

[x]+

is

x

if

x

if

x

However,

the

hinge

loss

is

not

smooth

at

the

hinge

point,

which

makes

it

difficult

to

apply

gradient

methods

on

the

problem.

To

overcome

the

difficulty,

we

use

a

smoothed

version

of

hinge

loss

for

each

data

point,

g(yih(vi)),

where

g(x)

when

x

x

when

x

(x

when

x

We

reduce

a

multiclass

problem

into

multiple

binary

ones.

One

simple

scheme

of

reduction

is

the

one-against-rest

coding

scheme.

In

the

one-against-rest

scheme,

we

assign

a

label

vector

for

each

class

label.

The

element

of

a

label

vector

is

if

the

data

point

belongs

the

corresponding

class,

if

the

data

point

does

not

belong

the

corresponding

class,

if

the

data

point

is

not

labeled.

Let

Y

be

the

label

matrix,

each

column

of

which

is

a

label

vector.

Therefore,

Y

is

a

matrix

of

n

c,

where

c

is

the

number

of

classes,

|C|.

Then

the

values

of

Eq.

form

a

matrix

H

ZW

where

is

a

vector

of

size

n,

whose

elements

are

all

one,

W

is

a

c

l

parameter

matrix,

and

b

is

a

parameter

vector

of

size

c.

The

total

loss

is

proportional

to

the

sum

of

Eq.

over

all

labeled

data

points

and

the

classes,

LY

(W,

b,

Z)

λ

X

i:vi∈T

,j∈C

g(YijHij),

where

λ

is

the

parameter

to

scale

the

term.

To

derive

a

robust

solution,

we

also

use

Tikhonov

regularization

for

W,

ΩW

(W)

ν

W

F

where

ν

is

the

parameter

to

scale

the

term.

Then

the

supervised

matrix

factorization

problem

becomes

min

U,V,Z,W,b

Js(U,

V,

Z,

W,

b)

where

Js(U,

V,

Z,

W,

b)

J

(U,

V,

Z)

LY

(W,

b,

Z)

ΩW

(W).

We

can

also

use

gradient

methods

to

solve

the

problem

of

Eq.

The

gradients

are

∂Js

∂U

∂J

∂U

∂Js

∂V

∂J

∂V

∂Js

∂Z

∂J

∂Z

λGW,

∂Js

∂W

=λG

Z

νW,

∂Js

∂b

=λG

where

G

is

an

n×c

matrix,

whose

ik-th

element

is

Yikg

(YikHik),

and

g

(x)

when

x

−1

when

x

(x

when

x

Once

we

obtain

w,

b,

and

Z,

we

can

apply

h

on

the

vertices

with

unknown

class

labels,

or

apply

traditional

classification

algorithms

on

Z

to

get

the

classification

results.

EXPERIMENTS

Data

Description

In

this

section,

we

perform

classification

on

two

datasets,

to

demonstrate

the

our

approach.

The

two

datasets

are

the

WebKB

data

and

the

Cora

data

set

The

WebKB

data

set

consists

of

about

web

pages

from

computer

science

departments

of

four

schools

(Cornell,

Texas,

Washington,

and

Wisconsin).

The

web

pages

are

classified

into

seven

categories.

The

numbers

of

pages

in

each

category

are

shown

in

Table

The

Cora

data

set

consists

of

the

abstracts

and

references

of

about

computer

science

research

papers.

We

use

part

of

them

to

categorize

into

one

of

subfields

of

data

structure

(DS),

hardware

and

architecture

(HA),

machine

learning

(ML),

and

programing

language

(PL).

We

remove

those

articles

without

reference

to

other

articles

in

the

set.

The

number

of

papers

and

the

number

of

subfields

in

each

area

are

shown

in

Table

area

of

papers

of

subfields

Data

structure

(DS)

Hardware

and

architecture

(HA)

Machine

learning

(ML)

Programing

language

(PL)

Table

Dataset

of

Cora

Methods

The

task

of

the

experiments

is

to

classify

the

data

based

on

their

content

information

and/or

link

structure.

We

use

the

following

methods:

WisconsinWashingtonTexasCornell

accuracy(%)

dataset

SVM

on

content

SVM

on

link

SVM

on

link-content

Directed

graph

reg.

PLSI+PHITS

link-content

MF

link-content

sup.

MF

method

Cornell

Texas

Washington

Wisconsin

SVM

on

content

SVM

on

links

SVM

on

link-content

Directed

graph

regularization

PLSI+PHITS

link-content

MF

link-content

sup.

MF

Table

Classification

accuracy

(mean

std-err

on

WebKB

data

set

SVM

on

content

We

apply

support

vector

machines

(SVM)

on

the

content

of

documents.

The

features

are

the

bag-ofwords

and

all

word

are

stemmed.

This

method

ignores

link

structure

in

the

data.

Linear

SVM

is

used.

The

regularization

parameter

of

SVM

is

selected

using

the

cross-validation

method.

The

implementation

of

SVM

used

in

the

experiments

is

SVM

on

links

We

treat

links

as

the

features

of

each

document,

i.e.

the

i-th

feature

is

link-to-pagei.

We

apply

SVM

on

link

features.

This

method

uses

link

information,

but

not

the

link

structure.

SVM

on

link-content

We

combine

the

features

of

the

above

two

methods.

We

use

different

weights

for

these

two

set

of

features.

The

weights

are

also

selected

using

crossvalidation.

Directed

graph

regularization

This

method

is

described

in

and

This

method

is

solely

based

on

link

structure.

PLSI+PHITS

This

method

is

described

in

This

method

combines

text

content

information

and

link

structure

for

analysis.

The

PHITS

algorithm

is

in

spirit

similar

to

with

an

additional

nonnegative

constraint.

It

models

the

outgoing

and

in-coming

structures

separately.

Link-content

MF

This

is

our

approach

of

matrix

factorization

described

in

Section

We

use

latent

factors

for

Z.

After

we

compute

Z,

we

train

a

linear

SVM

using

Z

as

the

feature

vectors,

then

apply

SVM

on

testing

portion

of

Z

to

obtain

the

final

result,

because

of

the

multiclass

output.

Link-content

sup.

MF

This

method

is

our

approach

of

the

supervised

matrix

factorization

in

Section

We

use

latent

factors

for

Z.

After

we

compute

Z,

we

train

a

linear

SVM

on

the

training

portion

of

Z,

then

apply

SVM

on

testing

portion

of

Z

to

obtain

the

final

result,

because

of

the

multiclass

output.

We

randomly

split

data

into

five

folds

and

repeat

the

experiment

for

five

times,

for

each

time

we

use

one

fold

for

test,

four

other

folds

for

training.

During

the

training

process,

we

use

the

crossvalidation

to

select

all

model

parameters.

We

measure

the

results

by

the

classification

accuracy,

i.e.,

the

percentage

of

the

number

of

correct

classified

documents

in

the

entire

data

set.

The

results

are

shown

as

the

average

classification

accuracies

and

it

standard

deviation

over

the

five

repeats.

Results

The

average

classification

accuracies

for

the

WebKB

data

set

are

shown

in

Table

For

this

task,

the

accuracies

of

SVM

on

links

are

worse

than

that

of

SVM

on

content.

But

the

directed

graph

regularization,

which

is

also

based

on

link

alone,

achieves

a

much

higher

accuracy.

This

implies

that

the

link

structure

plays

an

important

role

in

the

classification

of

this

dataset,

but

individual

links

in

a

web

page

give

little

information.

The

combination

of

link

and

content

using

SVM

achieves

similar

accuracy

as

that

of

SVM

on

content

alone,

which

confirms

individual

links

in

a

web

page

give

little

information.

Since

our

approach

consider

the

link

structure

as

well

as

the

content

information,

our

two

methods

give

results

a

highest

accuracies

among

these

approaches.

The

difference

between

the

results

of

our

two

methods

is

not

significant.

However

in

the

experiments

below,

we

show

the

difference

between

them.

The

classification

accuracies

for

the

Cora

data

set

are

shown

in

Table

In

this

experiment,

the

accuracies

of

SVM

on

the

combination

of

links

and

content

are

higher

than

either

SVM

on

content

or

SVM

on

links.

This

indicates

both

content

and

links

are

PLMLHADS

accuracy(%)

dataset

SVM

on

content

SVM

on

link

SVM

on

link-content

Directed

graph

reg.

PLSI+PHITS

link-content

MF

link-content

sup.

MF

method

DS

HA

ML

PL

SVM

on

content

SVM

on

links

SVM

on

link-content

Directed

graph

regularization

PLSI+PHITS

link-content

MF

link-content

sup.

MF

Table

Classification

accuracy

(mean

std-err

on

Cora

data

set

mative

for

classifying

the

articles

into

subfields.

The

method

of

directed

graph

regularization

does

not

perform

as

good

as

SVM

on

link-content,

which

confirms

the

importance

of

the

article

content

in

this

task.

Though

our

method

of

link-content

matrix

factorization

perform

slightly

better

than

other

methods,

our

method

of

linkcontent

supervised

matrix

factorization

outperform

significantly.

The

Number

of

Factors

As

we

discussed

in

Section

the

computational

complexity

of

each

iteration

for

solving

the

optimization

problem

is

quadratic

to

the

number

of

factors.

We

perform

experiments

to

study

how

the

number

of

factors

affects

the

accuracy

of

predication.

We

use

different

numbers

of

factors

for

the

Cornell

data

of

WebKB

data

set

and

the

machine

learning

(ML)

data

of

Cora

data

set.

The

result

shown

in

Figure

and

The

figures

show

that

the

accuracy

accuracy(%)

number

of

factors

link-content

sup.

MF

link-content

MF

(a)

Cornell

data

accuracy(%)

number

of

factors

link-content

sup.

MF

link-content

MF

(b)

ML

data

Figure

Accuracy

vs

number

of

factors

increases

as

the

number

of

factors

increases.

It

is

a

different

concept

from

choosing

the

optimal

number

of

clusters

in

clustering

application.

It

is

how

much

information

to

represent

in

the

latent

variables.

We

have

considered

the

regularization

over

the

factors,

which

avoids

the

overfit

problem

for

a

large

number

of

factors.

To

choose

of

the

number

of

factors,

we

need

to

consider

the

trade-off

between

the

accuracy

and

the

computation

time,

which

is

quadratic

to

the

number

of

factors.

The

difference

between

the

method

of

matrix

factorization

and

that

of

supervised

one

decreases

as

the

number

of

factors

increases.

This

indicates

that

the

usefulness

of

supervised

matrix

factorization

at

lower

number

of

factors.

DISCUSSIONS

The

loss

functions

LA

in

Eq.

and

LC

in

Eq.

use

squared

loss

due

to

computationally

convenience.

Actually,

squared

loss

does

not

precisely

describe

the

underlying

noise

model,

because

the

weights

of

adjacency

matrix

can

only

take

nonnegative

values,

in

our

case,

zero

or

one

only,

and

the

components

of

content

matrix

C

can

only

take

nonnegative

integers.

Therefore,

we

can

apply

other

types

of

loss,

such

as

hinge

loss

or

smoothed

hinge

loss,

e.g.

LA(U,

Z)

µh(A,

ZUZ

where

h(A,

B)

=P

i,j

AijBij]+

In

our

paper,

we

mainly

discuss

the

application

of

classification.

A

entry

of

matrix

Z

means

the

relationship

of

a

web

page

and

a

factor.

The

values

of

the

entries

are

the

weights

of

linear

model,

instead

of

the

probabilities

of

web

pages

belonging

to

latent

topics.

Therefore,

we

allow

the

components

take

any

possible

real

values.

When

we

come

to

the

clustering

application,

we

can

use

this

model

to

find

Z,

then

apply

K-means

to

partition

the

web

pages

into

clusters.

Actually,

we

can

use

the

idea

of

nonnegative

matrix

factorization

for

clustering

to

directly

cluster

web

pages.

As

the

example

with

nonnegative

constraints

shown

in

Section

we

represent

each

cluster

by

a

latent

topic,

i.e.

the

dimensionality

of

the

latent

space

is

set

to

the

number

of

clusters

we

want.

Then

the

problem

of

Eq.

becomes

min

U,V,Z

J

(U,

V,

Z),

s.t.Z

Solving

Eq.

we

can

obtain

more

interpretable

results,

which

could

be

used

for

clustering.

CONCLUSIONS

In

this

paper,

we

study

the

problem

of

how

to

combine

the

information

of

content

and

links

for

web

page

analysis,

mainly

on

classification

application.

We

propose

a

simple

approach

using

factors

to

model

the

text

content

and

link

structure

of

web

pages/documents.

The

directed

links

are

generated

from

the

linear

combination

of

linkage

of

between

source

and

destination

factors.

By

sharing

factors

between

text

content

and

link

structure,

it

is

easy

to

combine

both

the

content

information

and

link

structure.

Our

experiments

show

our

approach

is

effective

for

classification.

We

also

discuss

an

extension

for

clustering

application.

Acknowledgment

We

would

like

to

thank

Dr.

Dengyong

Zhou

for

sharing

his

code

of

his

algorithm.

Also,

thanks

to

the

reviewers

for

constructive

comments.

REFERENCES

CMU

world

wide

knowledge

base

(WebKB)

project.

Available

at

D.

Achlioptas,

A.

Fiat,

A.

R.

Karlin,

and

F.

McSherry.

Web

search

via

hub

synthesis.

In

IEEE

Symposium

on

Foundations

of

Computer

Science,

pages

S.

Chakrabarti,

B.

E.

Dom,

and

P.

Indyk.

Enhanced

hypertext

categorization

using

hyperlinks.

In

L.

M.

Haas

and

A.

Tiwary,

editors,

Proceedings

of

ACM

International

Conference

on

Management

of

Data,

pages

Seattle,

US,

ACM

Press,

New

York,

US.

C.-C.

Chang

and

C.-J.

Lin.

LIBSVM:

a

library

for

support

vector

machines,

Software

available

at

D.

Cohn

and

H.

Chang.

Learning

to

probabilistically

identify

authoritative

documents.

Proc.

ICML

D.

Cohn

and

T.

Hofmann.

The

missing

link

a

probabilistic

model

of

document

content

and

hypertext

connectivity.

In

T.

K.

Leen,

T.

G.

Dietterich,

and

V.

Tresp,

editors,

Advances

in

Neural

Information

Processing

Systems

pages

MIT

Press,

C.

Cortes

and

V.

Vapnik.

Support-vector

networks.

Machine

Learning,

S.

C.

Deerwester,

S.

T.

Dumais,

T.

K.

Landauer,

G.

W.

Furnas,

and

R.

A.

Harshman.

Indexing

by

latent

semantic

analysis.

Journal

of

the

American

Society

of

Information

Science,

X.

He,

H.

Zha,

C.

Ding,

and

H.

Simon.

Web

document

clustering

using

hyperlink

structures.

Computational

Statistics

and

Data

Analysis,

T.

Hofmann.

Probabilistic

latent

semantic

indexing.

In

Proceedings

of

the

Twenty-Second

Annual

International

SIGIR

Conference,

T.

Joachims,

N.

Cristianini,

and

J.

Shawe-Taylor.

Composite

kernels

for

hypertext

categorisation.

In

C.

Brodley

and

A.

Danyluk,

editors,

Proceedings

of

International

Conference

on

Machine

Learning,

pages

Williams

College,

US,

Morgan

Kaufmann

Publishers,

San

Francisco,

US.

J.

M.

Kleinberg.

Authoritative

sources

in

a

hyperlinked

environment.

J.

ACM,

P.

Kolari,

T.

Finin,

and

A.

Joshi.

SVMs

for

the

Blogosphere:

Blog

Identification

and

Splog

Detection.

In

AAAI

Spring

Symposium

on

Computational

Approaches

to

Analysing

Weblogs,

March

O.

Kurland

and

L.

Lee.

Pagerank

without

hyperlinks:

structural

re-ranking

using

links

induced

by

language

models.

In

SIGIR

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

information

retrieval,

pages

New

York,

NY,

USA,

ACM

Press.

A.

McCallum,

K.

Nigam,

J.

Rennie,

and

K.

Seymore.

Automating

the

contruction

of

internet

portals

with

machine

learning.

Information

Retrieval

Journal,

H.-J.

Oh,

S.

H.

Myaeng,

and

M.-H.

Lee.

A

practical

hypertext

catergorization

method

using

links

and

incrementally

available

class

information.

In

SIGIR

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

information

retrieval,

pages

New

York,

NY,

USA,

ACM

Press.

L.

Page,

S.

Brin,

R.

Motowani,

and

T.

Winograd.

PageRank

citation

ranking:

bring

order

to

the

web.

Stanford

Digital

Library

working

paper

C.

Spearman.

General

Intelligence,

objectively

determined

and

measured.

The

American

Journal

of

Psychology,

Apr

B.

Taskar,

P.

Abbeel,

and

D.

Koller.

Discriminative

probabilistic

models

for

relational

data.

In

Proceedings

of

International

UAI

Conference,

W.

Xu,

X.

Liu,

and

Y.

Gong.

Document

clustering

based

on

non-negative

matrix

factorization.

In

SIGIR

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

informaion

retrieval,

pages

ACM

Press,

Y.

Yang,

S.

Slattery,

and

R.

Ghani.

A

study

of

approaches

to

hypertext

categorization.

Journal

of

Intelligent

Information

Systems,

K.

Yu,

S.

Yu,

and

V.

Tresp.

Multi-label

informed

latent

semantic

indexing.

In

SIGIR

Proceedings

of

the

annual

international

ACM

SIGIR

conference

on

Research

and

development

in

information

retrieval,

pages

New

York,

NY,

USA,

ACM

Press.

T.

Zhang,

A.

Popescul,

and

B.

Dom.

Linear

prediction

models

with

graph

regularization

for

web-page

categorization.

In

KDD

Proceedings

of

the

ACM

SIGKDD

international

conference

on

Knowledge

discovery

and

data

mining,

pages

New

York,

NY,

USA,

ACM

Press.

D.

Zhou,

J.

Huang,

and

B.

Sch¨olkopf.

Learning

from

labeled

and

unlabeled

data

on

a

directed

graph.

In

Proceedings

of

the

International

Conference

on

Machine

Learning,

Bonn,

Germany,

D.

Zhou,

B.

Sch¨olkopf,

and

T.

Hofmann.

Semi-supervised

learning

on

directed

graphs.

Proc.

Neural

Info.

Processing

Systems,

B I I I O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O B I O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O B I O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O B I O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

web mining problem
document-term matrix
low-dimensional factor space
webkb and cora benchmark
asymmetric relationship

matrix factorization
combining content and link
classification
link structure
content information
authority information
joint factorization
linkage adjacency matrix
relationship
cocitation similarity
text content
factor analysis