A

Holistic

Approach

to

High-Performance

Computing:

Xgrid

Experience

David

Przybyla

Ringling

School

of

Art

and

Design

North

Tamiami

Trail

Sarasota,

Florida

dprzybyl@ringling.edu

Karissa

Miller

Ringling

School

of

Art

and

Design

North

Tamiami

Trail

Sarasota,

Florida

kmiller@ringling.edu

Mahmoud

Pegah

Ringling

School

of

Art

and

Design

North

Tamiami

Trail

Sarasota,

Florida

mpegah@ringling.edu

ABSTRACT

The

Ringling

School

of

Art

and

Design

is

a

fully

accredited

fouryear

college

of

visual

arts

and

design.

With

a

student

to

computer

ratio

of

better

than

the

Ringling

School

has

achieved

national

recognition

for

its

large-scale

integration

of

technology

into

collegiate

visual

art

and

design

education.

We

have

found

that

Mac

OS

X

is

the

best

operating

system

to

train

future

artists

and

designers.

Moreover,

we

can

now

buy

Macs

to

run

high-end

graphics,

nonlinear

video

editing,

animation,

multimedia,

web

production,

and

digital

video

applications

rather

than

expensive

UNIX

workstations.

As

visual

artists

cross

from

paint

on

canvas

to

creating

in

the

digital

realm,

the

demand

for

a

highperformance

computing

environment

grows.

In

our

public

computer

laboratories,

students

use

the

computers

most

often

during

the

workday;

at

night

and

on

weekends

the

computers

see

only

light

use.

In

order

to

harness

the

lost

processing

time

for

tasks

such

as

video

rendering,

we

are

testing

Xgrid,

a

suite

of

Mac

OS

X

applications

recently

developed

by

Apple

for

parallel

and

distributed

high-performance

computing.

As

with

any

new

technology

deployment,

IT

managers

need

to

consider

a

number

of

factors

as

they

assess,

plan,

and

implement

Xgrid.

Therefore,

we

would

like

to

share

valuable

information

we

learned

from

our

implementation

of

an

Xgrid

environment

with

our

colleagues.

In

our

report,

we

will

address

issues

such

as

assessing

the

needs

for

grid

computing,

potential

applications,

management

tools,

security,

authentication,

integration

into

existing

infrastructure,

application

support,

user

training,

and

user

support.

Furthermore,

we

will

discuss

the

issues

that

arose

and

the

lessons

learned

during

and

after

the

implementation

process.

Categories

and

Subject

Descriptors

[Computer-Communication

Networks]:

Distributed

Systemsdistributed

applications.

General

Terms

Management,

Documentation,

Performance,

Design,

Economics,

Reliability,

Experimentation.

INTRODUCTION

Grid

computing

does

not

have

a

single,

universally

accepted

definition.

The

technology

behind

grid

computing

model

is

not

new.

Its

roots

lie

in

early

distributed

computing

models

that

date

back

to

early

where

scientists

harnessed

the

computing

power

of

idle

workstations

to

let

compute

intensive

applications

to

run

on

multiple

workstations,

which

dramatically

shortening

processing

times.

Although

numerous

distributed

computing

models

were

available

for

discipline-specific

scientific

applications,

only

recently

have

the

tools

became

available

to

use

general-purpose

applications

on

a

grid.

Consequently,

the

grid

computing

model

is

gaining

popularity

and

has

become

a

show

piece

of

"utility

computing".

Since

in

the

IT

industry,

various

computing

models

are

used

interchangeably

with

grid

computing,

we

first

sort

out

the

similarities

and

difference

between

these

computing

models

so

that

grid

computing

can

be

placed

in

perspective.

Clustering

A

cluster

is

a

group

of

machines

in

a

fixed

configuration

united

to

operate

and

be

managed

as

a

single

entity

to

increase

robustness

and

performance.

The

cluster

appears

as

a

single

high-speed

system

or

a

single

highly

available

system.

In

this

model,

resources

can

not

enter

and

leave

the

group

as

necessary.

There

are

at

least

two

types

of

clusters:

parallel

clusters

and

highavailability

clusters.

Clustered

machines

are

generally

in

spatial

proximity,

such

as

in

the

same

server

room,

and

dedicated

solely

to

their

task.

In

a

high-availability

cluster,

each

machine

provides

the

same

service.

If

one

machine

fails,

another

seamlessly

takes

over

its

workload.

For

example,

each

computer

could

be

a

web

server

for

a

web

site.

Should

one

web

server

"die,"

another

provides

the

service,

so

that

the

web

site

rarely,

if

ever,

goes

down.

A

parallel

cluster

is

a

type

of

supercomputer.

Problems

are

split

into

many

parts,

and

individual

cluster

members

are

given

part

of

the

problem

to

solve.

An

example

of

a

parallel

cluster

is

composed

of

Apple

Power

Mac

computers

at

Virginia

Tech

University

Distributed

Computing

Distributed

computing

spatially

expands

network

services

so

that

the

components

providing

the

services

are

separated.

The

major

objective

of

this

computing

model

is

to

consolidate

processing

power

over

a

network.

A

simple

example

is

spreading

services

such

as

file

and

print

serving,

web

serving,

and

data

storage

across

multiple

machines

rather

than

a

single

machine

handling

all

the

tasks.

Distributed

computing

can

also

be

more

fine-grained,

where

even

a

single

application

is

broken

into

parts

and

each

part

located

on

different

machines:

a

word

processor

on

one

server,

a

spell

checker

on

a

second

server,

etc.

Utility

Computing

Literally,

utility

computing

resembles

common

utilities

such

as

telephone

or

electric

service.

A

service

provider

makes

computing

resources

and

infrastructure

management

available

to

a

customer

as

needed,

and

charges

for

usage

rather

than

a

flat

rate.

The

important

thing

to

note

is

that

resources

are

only

used

as

needed,

and

not

dedicated

to

a

single

customer.

Grid

Computing

Grid

computing

contains

aspects

of

clusters,

distributed

computing,

and

utility

computing.

In

the

most

basic

sense,

grid

turns

a

group

of

heterogeneous

systems

into

a

centrally

managed

but

flexible

computing

environment

that

can

work

on

tasks

too

time

intensive

for

the

individual

systems.

The

grid

members

are

not

necessarily

in

proximity,

but

must

merely

be

accessible

over

a

network;

the

grid

can

access

computers

on

a

LAN,

WAN,

or

anywhere

in

the

world

via

the

Internet.

In

addition,

the

computers

comprising

the

grid

need

not

be

dedicated

to

the

grid;

rather,

they

can

function

as

normal

workstations,

and

then

advertise

their

availability

to

the

grid

when

not

in

use.

The

last

characteristic

is

the

most

fundamental

to

the

grid

described

in

this

paper.

A

well-known

example

of

such

an

ad

hoc

grid

is

the

SETI@home

project

of

the

University

of

California

at

Berkeley,

which

allows

any

person

in

the

world

with

a

computer

and

an

Internet

connection

to

donate

unused

processor

time

for

analyzing

radio

telescope

data.

Comparing

the

Grid

and

Cluster

A

computer

grid

expands

the

capabilities

of

the

cluster

by

loosing

its

spatial

bounds,

so

that

any

computer

accessible

through

the

network

gains

the

potential

to

augment

the

grid.

A

fundamental

grid

feature

is

that

it

scales

well.

The

processing

power

of

any

machine

added

to

the

grid

is

immediately

availably

for

solving

problems.

In

addition,

the

machines

on

the

grid

can

be

generalpurpose

workstations,

which

keep

down

the

cost

of

expanding

the

grid.

ASSESSING

THE

NEED

FOR

GRID

COMPUTING

Effective

use

of

a

grid

requires

a

computation

that

can

be

divided

into

independent

(i.e.,

parallel)

tasks.

The

results

of

each

task

cannot

depend

on

the

results

of

any

other

task,

and

so

the

members

of

the

grid

can

solve

the

tasks

in

parallel.

Once

the

tasks

have

been

completed,

the

results

can

be

assembled

into

the

solution.

Examples

of

parallelizable

computations

are

the

Mandelbrot

set

of

fractals,

the

Monte

Carlo

calculations

used

in

disciplines

such

as

Solid

State

Physics,

and

the

individual

frames

of

a

rendered

animation.

This

paper

is

concerned

with

the

last

example.

Applications

Appropriate

for

Grid

Computing

The

applications

used

in

grid

computing

must

either

be

specifically

designed

for

grid

use,

or

scriptable

in

such

a

way

that

they

can

receive

data

from

the

grid,

process

the

data,

and

then

return

results.

In

other

words,

the

best

candidates

for

grid

computing

are

applications

that

run

the

same

or

very

similar

computations

on

a

large

number

of

pieces

of

data

without

any

dependencies

on

the

previous

calculated

results.

Applications

heavily

dependent

on

data

handling

rather

than

processing

power

are

generally

more

suitable

to

run

on

a

traditional

environment

than

on

a

grid

platform.

Of

course,

the

applications

must

also

run

on

the

computing

platform

that

hosts

the

grid.

Our

interest

is

in

using

the

Alias

Maya

application

with

Apple"s

Xgrid

on

Mac

OS

X.

Commercial

applications

usually

have

strict

license

requirements.

This

is

an

important

concern

if

we

install

a

commercial

application

such

as

Maya

on

all

members

of

our

grid.

By

its

nature,

the

size

of

the

grid

may

change

as

the

number

of

idle

computers

changes.

How

many

licenses

will

be

required?

Our

resolution

of

this

issue

will

be

discussed

in

a

later

section.

Integration

into

the

Existing

Infrastructure

The

grid

requires

a

controller

that

recognizes

when

grid

members

are

available,

and

parses

out

job

to

available

members.

The

controller

must

be

able

to

see

members

on

the

network.

This

does

not

require

that

members

be

on

the

same

subnet

as

the

controller,

but

if

they

are

not,

any

intervening

firewalls

and

routers

must

be

configured

to

allow

grid

traffic.

XGRID

Xgrid

is

Apple"s

grid

implementation.

It

was

inspired

by

Zilla,

a

desktop

clustering

application

developed

by

NeXT

and

acquired

by

Apple.

In

this

report

we

describe

the

Xgrid

Technology

Preview

a

free

download

that

requires

Mac

OS

X

or

later

and

a

minimum

MB

RAM

Xgrid,

leverages

Apple"s

traditional

ease

of

use

and

configuration.

If

the

grid

members

are

on

the

same

subnet,

by

default

Xgrid

automatically

discovers

available

resources

through

Rendezvous

Tasks

are

submitted

to

the

grid

through

a

GUI

interface

or

by

the

command

line.

A

System

Preference

Pane

controls

when

each

computer

is

available

to

the

grid.

It

may

be

best

to

view

Xgrid

as

a

facilitator.

The

Xgrid

architecture

handles

software

and

data

distribution,

job

execution,

and

result

aggregation.

However,

Xgrid

does

not

perform

the

actual

calculations.

Xgrid

Components

Xgrid

has

three

major

components:

the

client,

controller,

and

the

agent.

Each

component

is

included

in

the

default

installation,

and

any

computer

can

easily

be

configured

to

assume

any

role.

In

fact,

for

testing

purposes,

a

computer

can

simultaneously

assume

all

roles

in

local

mode.

The

more

typical

production

use

is

called

cluster

mode.

The

client

submits

jobs

to

the

controller

through

the

Xgrid

GUI

or

command

line.

The

client

defines

how

the

job

will

be

broken

into

tasks

for

the

grid.

If

any

files

or

executables

must

be

sent

as

part

of

a

job,

they

must

reside

on

the

client

or

at

a

location

accessible

to

the

client.

When

a

job

is

complete,

the

client

can

retrieve

the

results

from

the

controller.

A

client

can

only

connect

to

a

single

controller

at

a

time.

The

controller

runs

the

GridServer

process.

It

queues

tasks

received

from

clients,

distributes

those

tasks

to

the

agents,

and

handles

failover

if

an

agent

cannot

complete

a

task.

In

Xgrid

Technology

Preview

a

controller

can

handle

a

maximum

of

agent

connections.

Only

one

controller

can

exist

per

logical

grid.

The

agents

run

the

GridAgent

process.

When

the

GridAgent

process

starts,

it

registers

with

a

controller;

an

agent

can

only

be

connected

to

one

controller

at

a

time.

Agents

receive

tasks

from

their

controller,

perform

the

specified

computations,

and

then

send

the

results

back

to

the

controller.

An

agent

can

be

configured

to

always

accept

tasks,

or

to

just

accept

them

when

the

computer

is

not

otherwise

busy.

Security

and

Authentication

By

default,

Xgrid

requires

two

passwords.

First,

a

client

needs

a

password

to

access

a

controller.

Second,

the

controller

needs

a

password

to

access

an

agent.

Either

password

requirement

can

be

disabled.

Xgrid

uses

two-way-random

mutual

authentication

protocol

with

hashes.

At

this

time,

data

encryption

is

only

used

for

passwords.

As

mentioned

earlier,

an

agent

registers

with

a

controller

when

the

GridAgent

process

starts.

There

is

no

native

method

for

the

controller

to

reject

agents,

and

so

it

must

accept

any

agent

that

registers.

This

means

that

any

agent

could

submit

a

job

that

consumes

excessive

processor

and

disk

space

on

the

agents.

Of

course,

since

Mac

OS

X

is

a

BSD-based

operating

system,

the

controller

could

employ

Unix

methods

of

restricting

network

connections

from

agents.

The

Xgrid

daemons

run

as

the

user

nobody,

which

means

the

daemons

can

read,

write,

or

execute

any

file

according

to

world

permissions.

Thus,

Xgrid

jobs

can

execute

many

commands

and

write

to

/tmp

and

/Volumes.

In

general,

this

is

not

a

major

security

risk,

but

is

does

require

a

level

of

trust

between

all

members

of

the

grid.

Using

Xgrid

Installation

Basic

Xgrid

installation

and

configuration

is

described

both

in

Apple

documentation

and

online

at

the

University

of

Utah

web

site

The

installation

is

straightforward

and

offers

no

options

for

customization.

This

means

that

every

computer

on

which

Xgrid

is

installed

has

the

potential

to

be

a

client,

controller,

or

agent.

Agent

and

Controller

Configuration

The

agents

and

controllers

can

be

configured

through

the

Xgrid

Preference

Pane

in

the

System

Preferences

or

XML

files

in

Here

the

GridServer

and

GridAgent

processes

are

started,

passwords

set,

and

the

controller

discovery

method

used

by

agents

is

selected.

By

default,

agents

use

Rendezvous

to

find

a

controller,

although

the

agents

can

also

be

configured

to

look

for

a

specific

host.

The

Xgrid

Preference

Pane

also

sets

whether

the

Agents

will

always

accept

jobs,

or

only

accept

jobs

when

idle.

In

Xgrid

terms,

idle

either

means

that

the

Xgrid

screen

saver

has

activated,

or

the

mouse

and

keyboard

have

not

been

used

for

more

than

minutes.

Even

if

the

agent

is

configured

to

always

accept

tasks,

if

the

computer

is

being

used

these

tasks

will

run

in

the

background

at

a

low

priority.

However,

if

an

agent

only

accepts

jobs

when

idle,

any

unfinished

task

being

performed

when

the

computer

ceases

being

idle

are

immediately

stopped

and

any

intermediary

results

lost.

Then

the

controller

assigns

the

task

to

another

available

member

of

the

grid.

Advertising

the

controller

via

Rendezvous

can

be

disabled

by

editing

This,

however,

will

not

prevent

an

agent

from

connecting

to

the

controller

by

hostname.

Sending

Jobs

from

an

Xgrid

Client

The

client

sends

jobs

to

the

controller

either

through

the

Xgrid

GUI

or

the

command

line.

The

Xgrid

GUI

submits

jobs

via

small

applications

called

plug-ins.

Sample

plug-ins

are

provided

by

Apple,

but

they

are

only

useful

as

simple

testing

or

as

examples

of

how

to

create

a

custom

plug-in.

If

we

are

to

employ

Xgrid

for

useful

work,

we

will

require

a

custom

plug-in.

James

Reynolds

details

the

creation

of

custom

plug-ins

on

the

University

of

Utah

Mac

OS

web

site

Xgrid

stores

plug-ins

in

or

depending

on

whether

the

plug-in

was

installed

with

Xgrid

or

created

by

a

user.

The

core

plug-in

parameter

is

the

command,

which

includes

the

executable

the

agents

will

run.

Another

important

parameter

is

the

working

directory.

This

directory

contains

necessary

files

that

are

not

installed

on

the

agents

or

available

to

them

over

a

network.

The

working

directory

will

always

be

copied

to

each

agent,

so

it

is

best

to

keep

this

directory

small.

If

the

files

are

installed

on

the

agents

or

available

over

a

network,

the

working

directory

parameter

is

not

needed.

The

command

line

allows

the

options

available

with

the

GUI

plug-in,

but

it

can

be

slightly

more

cumbersome.

However,

the

command

line

probably

will

be

the

method

of

choice

for

serious

work.

The

command

arguments

must

be

included

in

a

script

unless

they

are

very

basic.

This

can

be

a

shell,

perl,

or

python

script,

as

long

as

the

agent

can

interpret

it.

Running

the

Xgrid

Job

When

the

Xgrid

job

is

started,

the

command

tells

the

controller

how

to

break

the

job

into

tasks

for

the

agents.

Then

the

command

is

tarred

and

gzipped

and

sent

to

each

agent;

if

there

is

a

working

directory,

this

is

also

tarred

and

gzipped

and

sent

to

the

agents.

The

agents

extract

these

files

into

/tmp

and

run

the

task.

Recall

that

since

the

GridAgent

process

runs

as

the

user

nobody,

everything

associated

with

the

command

must

be

available

to

nobody.

Executables

called

by

the

command

should

be

installed

on

the

agents

unless

they

are

very

simple.

If

the

executable

depends

on

libraries

or

other

files,

it

may

not

function

properly

if

transferred,

even

if

the

dependent

files

are

referenced

in

the

working

directory.

When

the

task

is

complete,

the

results

are

available

to

the

client.

In

principle,

the

results

are

sent

to

the

client,

but

whether

this

actually

happens

depends

on

the

command.

If

the

results

are

not

sent

to

the

client,

they

will

be

in

/tmp

on

each

agent.

When

available,

a

better

solution

is

to

direct

the

results

to

a

network

volume

accessible

to

the

client.

Limitations

and

Idiosyncrasies

Since

Xgrid

is

only

in

its

second

preview

release,

there

are

some

rough

edges

and

limitations.

Apple

acknowledges

some

limitations

For

example,

the

controller

cannot

determine

whether

an

agent

is

trustworthy

and

the

controller

always

copies

the

command

and

working

directory

to

the

agent

without

checking

to

see

if

these

exist

on

the

agent.

Other

limitations

are

likely

just

a

by-product

of

an

unfinished

work.

Neither

the

client

nor

controller

can

specify

which

agents

will

receive

the

tasks,

which

is

particularly

important

if

the

agents

contain

a

variety

of

processor

types

and

speeds

and

the

user

wants

to

optimize

the

calculations.

At

this

time,

the

best

solution

to

this

problem

may

be

to

divide

the

computers

into

multiple

logical

grids.

There

is

also

no

standard

way

to

monitor

the

progress

of

a

running

job

on

each

agent.

The

Xgrid

GUI

and

command

line

indicate

which

agents

are

working

on

tasks,

but

gives

no

indication

of

progress.

Finally,

at

this

time

only

Mac

OS

X

clients

can

submit

jobs

to

the

grid.

The

framework

exists

to

allow

third

parties

to

write

plug-ins

for

other

Unix

flavors,

but

Apple

has

not

created

them.

XGRID

IMPLEMENTATION

Our

goal

is

an

Xgrid

render

farm

for

Alias

Maya.

The

Ringling

School

has

about

Apple

Power

Mac

and

in

computer

labs.

The

computers

range

from

MHz

singleprocessor

and

MHz

and

GHz

dual-processor

to

GHz

dual-processor

All

of

these

computers

are

lightly

used

in

the

evening

and

on

weekends

and

represent

an

enormous

processing

resource

for

our

student

rendering

projects.

Software

Installation

During

our

Xgrid

testing,

we

loaded

software

on

each

computer

multiple

times,

including

the

operating

systems.

We

saved

time

by

facilitating

our

installations

with

the

remote

administration

daemon

(radmind)

software

developed

at

the

University

of

Michigan

Everything

we

installed

for

testing

was

first

created

as

a

radmind

base

load

or

overload.

Thus,

Mac

OS

X,

Mac

OS

X

Developer

Tools,

Xgrid,

POV-Ray

and

Alias

Maya

were

stored

on

a

radmind

server

and

then

installed

on

our

test

computers

when

needed.

Initial

Testing

We

used

six

GHz

dual-processor

Apple

Power

Mac

for

our

Xgrid

tests.

Each

computer

ran

Mac

OS

X

and

contained

GB

RAM.

As

shown

in

Figure

one

computer

served

as

both

client

and

controller,

while

the

other

five

acted

as

agents.

Before

attempting

Maya

rendering

with

Xgrid,

we

performed

basic

calculations

to

cement

our

understanding

of

Xgrid.

Apple"s

Xgrid

documentation

is

sparse,

so

finding

helpful

web

sites

facilitated

our

learning.

We

first

ran

the

Mandelbrot

set

plug-in

provided

by

Apple,

which

allowed

us

to

test

the

basic

functionality

of

our

grid.

Then

we

performed

benchmark

rendering

with

the

Open

Source

Application

POV-Ray,

as

described

by

Daniel

Côté

and

James

Reynolds

Our

results

showed

that

one

dual-processor

rendering

the

benchmark

POV-Ray

image

took

minutes.

Breaking

the

image

into

three

equal

parts

and

using

Xgrid

to

send

the

parts

to

three

agents

required

minutes.

However,

two

agents

finished

their

rendering

in

minutes,

while

the

third

agent

used

minutes;

the

entire

render

was

only

as

fast

as

the

slowest

agent.

These

results

gave

us

two

important

pieces

of

information.

First,

the

much

longer

rendering

time

for

one

of

the

tasks

indicated

that

we

should

be

careful

how

we

split

jobs

into

tasks

for

the

agents.

All

portions

of

the

rendering

will

not

take

equal

amounts

of

time,

even

if

the

pixel

size

is

the

same.

Second,

since

POV-Ray

cannot

take

advantage

of

both

processors

in

a

neither

can

an

Xgrid

task

running

POV-Ray.

Alias

Maya

does

not

have

this

limitation.

Rendering

with

Alias

Maya

We

first

installed

Alias

Maya

for

Mac

OS

X

on

the

client/controller

and

each

agent.

Maya

requires

licenses

for

use

as

a

workstation

application.

However,

if

it

is

just

used

for

rendering

from

the

command

line

or

a

script,

no

license

is

needed.

We

thus

created

a

minimal

installation

of

Maya

as

a

radmind

overload.

The

application

was

installed

in

a

hidden

directory

inside

/Applications.

This

was

done

so

that

normal

users

of

the

workstations

would

not

find

and

attempt

to

run

Maya,

which

would

fail

because

these

installations

are

not

licensed

for

such

use.

In

addition,

Maya

requires

the

existence

of

a

directory

ending

in

the

path

/maya.

The

directory

must

be

readable

and

writable

by

the

Maya

user.

For

a

user

running

Maya

on

a

Mac

OS

X

workstation,

the

path

would

usually

be

Unless

otherwise

specified,

this

directory

will

be

the

default

location

for

Maya

data

and

output

files.

If

the

directory

does

not

Figure

Xgrid

test

grid.

Client/

Controller

Agent

Agent

Agent

Agent

Agent

Network

Volume

Jobs

Data

Data

exist,

Maya

will

try

to

create

it,

even

if

the

user

specifies

that

the

data

and

output

files

exist

in

other

locations.

However,

Xgrid

runs

as

the

user

nobody,

which

does

not

have

a

home

directory.

Maya

is

unable

to

create

the

needed

directory,

and

looks

instead

for

This

directory

also

does

not

exist,

and

the

user

nobody

has

insufficient

rights

to

create

it.

Our

solution

was

to

manually

create

and

give

the

user

nobody

read

and

write

permissions.

We

also

created

a

network

volume

for

storage

of

both

the

rendering

data

and

the

resulting

rendered

frames.

This

avoided

sending

the

Maya

files

and

associated

textures

to

each

agent

as

part

of

a

working

directory.

Such

a

solution

worked

well

for

us

because

our

computers

are

geographically

close

on

a

LAN;

if

greater

distance

had

separated

the

agents

from

the

client/controller,

specifying

a

working

directory

may

have

been

a

better

solution.

Finally,

we

created

a

custom

GUI

plug-in

for

Xgrid.

The

plug-in

command

calls

a

Perl

script

with

three

arguments.

Two

arguments

specify

the

beginning

and

end

frames

of

the

render

and

the

third

argument

the

number

of

frames

in

each

job

(which

we

call

the

cluster

size).

The

script

then

calculates

the

total

number

of

jobs

and

parses

them

out

to

the

agents.

For

example,

if

we

begin

at

frame

and

end

at

frame

with

frames

for

each

job,

the

plug-in

will

create

jobs

and

send

them

out

to

the

agents.

Once

the

jobs

are

sent

to

the

agents,

the

script

executes

the

command

on

each

agent

with

the

parameters

appropriate

for

the

particular

job.

The

results

are

sent

to

the

network

volume.

With

the

setup

described,

we

were

able

to

render

with

Alias

Maya

on

our

test

grid.

Rendering

speed

was

not

important

at

this

time;

our

first

goal

was

to

implement

the

grid,

and

in

that

we

succeeded.

Pseudo

Code

for

Perl

Script

in

Custom

Xgrid

Plug-in

In

this

section

we

summarize

in

simplified

pseudo

code

format

the

Perl

script

used

in

our

Xgrig

plug-in.

Read

beginning

frame,

end

frame,

and

cluster

size

of

render.

Check

whether

the

render

can

be

divided

into

an

integer

number

of

jobs

based

on

the

cluster

size.

If

there

are

not

an

integer

number

of

jobs,

reduce

the

cluster

size

of

the

last

job

and

set

its

last

frame

to

the

end

frame

of

the

render.

Determine

the

start

frame

and

end

frame

for

each

job.

Execute

the

Render

command.

Lessons

Learned

Rendering

with

Maya

from

the

Xgrid

GUI

was

not

trivial.

The

lack

of

Xgrid

documentation

and

the

requirements

of

Maya

combined

into

a

confusing

picture,

where

it

was

difficult

to

decide

the

true

cause

of

the

problems

we

encountered.

Trial

and

error

was

required

to

determine

the

best

way

to

set

up

our

grid.

The

first

hurdle

was

creating

the

directory

with

read

and

write

permissions

for

the

user

nobody.

The

second

hurdle

was

learning

that

we

got

the

best

performance

by

storing

the

rendering

data

on

a

network

volume.

The

last

major

hurdle

was

retrieving

our

results

from

the

agents.

Unlike

the

POV-Ray

rendering

tests,

our

initial

Maya

results

were

never

returned

to

the

client;

instead,

Maya

stored

the

results

in

/tmp

on

each

agent.

Specifying

in

the

plug-in

where

to

send

the

results

would

not

change

this

behavior.

We

decided

this

was

likely

a

Maya

issue

rather

than

an

Xgrid

issue,

and

the

solution

was

to

send

the

results

to

the

network

volume

via

the

Perl

script.

FUTURE

PLANS

Maya

on

Xgrid

is

not

yet

ready

to

be

used

by

the

students

of

Ringling

School.

In

order

to

do

this,

we

must

address

at

least

the

following

concerns.

Continue

our

rendering

tests

through

the

command

line

rather

than

the

GUI

plug-in.

This

will

be

essential

for

the

following

step.

Develop

an

appropriate

interface

for

users

to

send

jobs

to

the

Xgrid

controller.

This

will

probably

be

an

extension

to

the

web

interface

of

our

existing

render

farm,

where

the

student

specifies

parameters

that

are

placed

in

a

script

that

issues

the

Render

command.

Perform

timed

Maya

rendering

tests

with

Xgrid.

Part

of

this

should

compare

the

rendering

times

for

Power

Mac

and

CONCLUSION

Grid

computing

continues

to

advance.

Recently,

the

IT

industry

has

witnessed

the

emergence

of

numerous

types

of

contemporary

grid

applications

in

addition

to

the

traditional

grid

framework

for

compute

intensive

applications.

For

instance,

peer-to-peer

applications

such

as

Kazaa,

are

based

on

storage

grids

that

do

not

share

processing

power

but

instead

an

elegant

protocol

to

swap

files

between

systems.

Although

in

our

campuses

we

discourage

students

from

utilizing

peer-to-peer

applications

from

music

sharing,

the

same

protocol

can

be

utilized

on

applications

such

as

decision

support

and

data

mining.

The

National

Virtual

Collaboratory

grid

project

will

link

earthquake

researchers

across

the

U.S.

with

computing

resources,

allowing

them

to

share

extremely

large

data

sets,

research

equipment,

and

work

together

as

virtual

teams

over

the

Internet.

There

is

an

assortment

of

new

grid

players

in

the

IT

world

expanding

the

grid

computing

model

and

advancing

the

grid

technology

to

the

next

level.

SAP

is

piloting

a

project

to

grid-enable

SAP

ERP

applications,

Dell

has

partnered

with

Platform

Computing

to

consolidate

computing

resources

and

provide

grid-enabled

systems

for

compute

intensive

applications,

Oracle

has

integrated

support

for

grid

computing

in

their

release

United

Devices

offers

hosting

service

for

gridon-demand,

and

Sun

Microsystems

continues

their

research

and

development

of

Sun"s

Grid

engine

which

combines

grid

and

clustering

platforms.

Simply,

the

grid

computing

is

up

and

coming.

The

potential

benefits

of

grid

computing

are

colossal

in

higher

education

learning

while

the

implementation

costs

are

low.

Today,

it

would

be

difficult

to

identify

an

application

with

as

high

a

return

on

investment

as

grid

computing

in

information

technology

divisions

in

higher

education

institutions.

It

is

a

mistake

to

overlook

this

technology

with

such

a

high

payback.

ACKNOWLEDGMENTS

The

authors

would

like

to

thank

Scott

Hanselman

of

the

IT

team

at

the

Ringling

School

of

Art

and

Design

for

providing

valuable

input

in

the

planning

of

our

Xgrid

testing.

We

would

also

like

to

thank

the

posters

of

the

Xgrid

Mailing

List

for

providing

insight

into

many

areas

of

Xgrid.

REFERENCES

Apple

Academic

Research,

SETI@home:

Search

for

Extraterrestrial

Intelligence

at

home.

Alias,

Apple

Computer,

Xgrid,

Xgrid

Guide,

Apple

Mac

OS

X

Features,

Xgrid

Manual

Page,

James

Reynolds,

Xgrid

Presentation,

University

of

Utah,

Research

Systems

Unix

Group,

Radmind,

University

of

Michigan,

the

Radmind

Command

Line

Tools

to

Maintain

Multiple

Mac

OS

X

Machines,

Côté,

Xgrid

example:

Parallel

graphics

rendering

in

POVray,

Computing,

Devices,

Inc.,

Grid

Engine

software/gridware/index.html/.

Users

Mailing

List,

O O O O B I B O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O B I I O O O B I O O O O O O O O O O O O O O O O B I I B B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O B O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I B I I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O B I I O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

large-scale integration of technology
design education
high-end graphic
digital video application
macintosh os x

visual art
design
mac os x
operating system
nonlinear video editing
animation
multimedia
web production
xgrid environment
xgrid
grid computing
cluster
highperformance computing
rendezvous
render


A Framework for Architecting Peer-to-Peer Receiver-driven Overlays Reza Rejaie Department of Computer Science University of Oregon reza@cs.uoregon.edu Shad Stafford Department of Computer Science University of Oregon staffors@cs.uoregon.edu ABSTRACT This paper presents a simple and scalable framework for architecting peer-to-peer overlays called Peer-to-peer Receiverdriven Overlay (or PRO). PRO is designed for non-interactive streaming applications and its primary design goal is to maximize delivered bandwidth (and thus delivered quality) to peers with heterogeneous and asymmetric bandwidth. To achieve this goal, PRO adopts a receiver-driven approach where each receiver (or participating peer) (i) independently discovers other peers in the overlay through gossiping, and (ii) selfishly determines the best subset of parent peers through which to connect to the overlay to maximize its own delivered bandwidth. Participating peers form an unstructured overlay which is inherently robust to high churn rate. Furthermore, each receiver leverages congestion controlled bandwidth from its parents as implicit signal to detect and react to long-term changes in network or overlay condition without any explicit coordination with other participating peers. Independent parent selection by individual peers dynamically converge to an efficient overlay structure. Categories and Subject Descriptors: C.2.4 [ComputerCommunication Networks]: Distributed Systems General Terms: Design, Measurement 1. INTRODUCTION Limited deployment of IP multicast has motivated a new distribution paradigm over the Internet based on overlay networks where a group of participating end-systems (or peers) form an overlay structure and actively participate in distribution of content without any special support from the network (e.g., [7]). Since overlay structures are layered over the best-effort Internet, any approach for constructing overlay should address the following fundamental challenges: (i) Scalability with the number of participating peers, (ii) Robustness to dynamics of peer participation, (iii) Adaptation to variations of network bandwidth, and (iv) Accommodating heterogeneity and asymmetry of bandwidth connectivity among participating peers[19]. Coping with bandwidth variations, heterogeneity and asymmetry are particularly important in design of peer-to-peer overlay for streaming applications because delivered quality to each peer is directly determined by its bandwidth connectivity to (other peer(s) on) the overlay. This paper presents a simple framework for architecting Peer-to-peer Receiver-driven Overlay, called PRO. PRO can accommodate a spectrum of non-interactive streaming applications ranging from playback to lecture-mode live sessions. The main design philosophy in PRO is that each peer should be allowed to independently and selfishly determine the best way to connect to the overlay in order to maximize its own delivered quality. Toward this end, each peer can connect to the overlay topology at multiple points (i.e., receive content through multiple parent peers). Therefore, participating peers form an unstructured overlay that can gracefully cope with high churn rate[5]. Furthermore, having multiple parent peers accommodates bandwidth heterogeneity and asymmetry while improves resiliency against dynamics of peer participation. PRO consists of two key components: (i) Gossip-based Peer Discovery: Each peer periodically exchanges message (i.e., gossips) with other known peers to progressively learn about a subset of participating peers in the overlay that are likely to be good parents. Gossiping provides a scalable and efficient approach to peer discovery in unstructured peer-to-peer networks that can be customized to guide direction of discovery towards peers with desired properties (e.g., peers with shorter distance or higher bandwidth). (ii) Receiver-driven Parent Selection: Given the collected information about other participating peers by gossiping mechanism, each peer (or receiver) gradually improves its own delivered quality by dynamically selecting a proper subset of parent peers that collectively maximize provided bandwidth to the receiver. Since the available bandwidth from different participating peers to a receiver (and possible correlation among them) can be measured only at that receiver, a receiver-driven approach is the natural solution to maximize available bandwidth to heterogeneous peers. Furthermore, the available bandwidth from parent peers serves as an implicit signal for a receiver to detect and react to changes in network or overlay condition without any explicit coordination with other participating peers. Independent parent selection by individual peers leads to an efficient overlay that maximizes delivered quality to each peer. PRO incorporates 42 several damping functions to ensure stability of the overlay despite uncoordinated actions by different peers. PRO is part of a larger architecture that we have developed for peer-to-peer streaming. In our earlier work, we developed a mechanism called PALS [18] that enables a receiver to stream layered structured content from a given set of congestion controlled senders. Thus, PRO and PALS are both receiver-driven but complement each other. More specifically, PRO determines a proper subset of parent peers that collectively maximize delivered bandwidth to each receiver whereas PALS coordinates in-time streaming of different segments of multimedia content from these parents despite unpredictable variations in their available bandwidth. This division of functionality provides a great deal of flexibility because it decouples overlay construction from delivery mechanism. In this paper, we primarily focus on the overlay construction mechanism, or PRO. The rest of this paper is organized as follows: In Section 2, we revisit the problem of overlay construction for peerto-peer streaming and identify its two key components and explore their design space. We illustrate the differences between PRO and previous solutions, and justify our design choices. We present our proposed framework in Section 3. In Sections 4 and 5, the key components of our framework are described in further detail. Finally, Section 6 concludes the paper and presents our future plans. 2. REVISITING THE PROBLEM Constructing a peer-to-peer overlay for streaming applications should not only accommodate global design goals such as scalability and resilience but also satisfy the local design goal of maximizing delivered quality to individual peers 1 . More specifically, delivered quality of streaming content to each peer should be proportional to its incoming access link bandwidth. Achieving these goals is particularly challenging because participating peers often exhibit heterogeneity and asymmetry in their bandwidth connectivity. Solutions for constructing peer-to-peer overlays often require two key mechanisms to be implemented at each peer: Peer Discovery (PD) and Parent Selection (PS). The PD mechanism enables each peer to learn about other participating peers in the overlay. Information about other peers are used by the PS mechanism at each peer to determine proper parent peers through which it should connect to the overlay. The collective behavior of PD and PS mechanisms at all participating peers leads to an overlay structure that achieves the above design goals. There has been a wealth of previous research that explored design space of the PD and PS mechanisms as follows: Peer Discovery: In structured peer-to-peer networks, the existing structure enables each peer to find other participating peers in a scalable fashion (e.g., [4]). However, structured peer-to-peer networks may not be robust against high churn rate [5]. In contrast, unstructured peer-to-peer networks can gracefully accommodate high churn rate [5] but require a separate peer discovery mechanism. Meshfirst approaches (e.g., [7, 6]) that require each peer to know about all other participating peers as well as centralized approaches (e.g., [16]) to peer discovery exhibit limited scalability. NICE [2] leverages a hierarchal structure to achieve 1 It is worth clarifying that our design goal is different from common goals in building application-level multicast trees [7] (i.e., minimizing stretch and stress). scalability but each peer only knows about a group of closeby peers who may not be good parents (i.e., may not provide sufficient bandwidth). Parent Selection: We examine two key aspects of parent selections: (i) Selection Criteria: There are two main criteria for parent selections: relative delay and available bandwidth between two peers. Relative delay between any two peers can be estimated in a scalable fashion with one of the existing landmark-based solutions such as Global Network Positioning (GNP) [15]. However, estimating available bandwidth between two peers requires end-to-end measurement. Using available bandwidth as criteria for parent selection does not scale for two reasons: First, to cope with dynamics of bandwidth variations, each peer requires to periodically estimate the available bandwidth from all other peers through measurement (e.g., [6]). Second, the probability of interference among different measurements grows with the number of peers in an overlay (similar to joint experiment in RLM [13]). Most of the previous solutions adopted the idea of application level multicast and used delay as the main selection criteria. Participating peers cooperatively run a distributed algorithm to organize themselves into a source-rooted tree structure in order to minimize either overall delay across all branches of the tree (e.g., [7]), or delay between source and each receiver peer (e.g., [20]). While these parent selection strategies minimize associated network load, they may not provide sufficient bandwidth to individual peers because delay is often not a good indicator for available bandwidth between two peers [12, 14]. The key issue is that minimizing overall delay (global design goal) and maximizing delivered bandwidth to each peer (local design goal) could easily be in conflict. More specifically, parent peers with longer relative distance may provide higher bandwidth than close-by parents. This suggests that there might exist a tradeoff between maximizing provided bandwidth to each peer and minimizing overall delay across the overlay. (ii) Single vs Multiple Parents: A single tree structure for the overlay (where each peer has a single parent) is inherently unable to accommodate peers with heterogeneous and asymmetric bandwidth. A common approach to accommodating bandwidth heterogeneity is to use layer structured content (either layered or multiple description encodings) and allow each receiver to have multiple parents. This approach could accommodate heterogeneity but it introduces several new challenges. First, parent selection strategy should be determined based on location of a bottleneck. If the bottleneck is at the (outgoing) access links of parent peers 2 , then a receiver should simply look for more parents. However, when the bottleneck is else where in the network, a receiver should select parents with a diverse set of paths (i.e., utilize different network paths). In practice, a combination of these cases might simultaneously exist among participating peers [1]. Second, streaming a single content from multiple senders is challenging for two reasons: 1) This requires tight coordination among senders to determine overall delivered quality (e.g., number of layers) and decide which sender is responsible for delivery of each segment. 2) Delivered segments from different senders should arrive before their playout times despite uncorrelated vari2 if bottleneck is at the receiver"s access link, then provided bandwidth to the receiver is already maximized. 43 ations in (congestion controlled) bandwidth from different senders. This also implies that those solutions that build multi-parent overlay structure but do not explicitly ensure in-time delivery of individual segments (e.g., [3, 11]) may not be able to support streaming applications. One approach to build a multi-parent overlay is to organize participating peers into different trees where each layer of the stream is sent to a separate tree (e.g., [4, 16]). Each peer can maximize its quality by participating in a proper number of trees. This approach raises several issues: 1) the provided bandwidth to peers in each tree is limited by minimum uplink bandwidth among upstream peers on that tree. In the presence of bandwidth asymmetry, this could easily limit delivered bandwidth on each tree below the required bandwidth for a single layer, 2) it is not feasible to build separate trees that are all optimal for a single selection criteria (e.g., overall delay),. 3) connections across different trees are likely to compete for available bandwidth on a single bottleneck3 . We conclude that a practical solution for peer-topeer streaming applications should incorporate the following design properties: (i) it should use an unstructured, multiparent peer-to-peer overlay, (ii) it should provide a scalable peer discovery mechanism that enables each peer to find its good parents efficiently, (iii) it should detect (and possibly avoid) any shared bottleneck among different connections in the overlay, and (iv) it should deploy congestion controlled connections but ensure in-time arrival of delivered segments to each receiver. In the next section, we explain how PRO incorporates all the above design properties. 3. P2P RECEIVER-DRIVEN OVERLAY Assumptions: We assume that each peer can estimate the relative distance between any two peers using the GNP mechanism [15]. Furthermore, each peer knows the incoming and outgoing bandwidth of its access link. Each peer uses the PALS mechanism to stream content from multiple parent peers. All connections are congestion controlled by senders (e.g., [17]). To accommodate peer bandwidth heterogeneity, we assume that the content has a layered representation. In other words, with proper adjustment, the framework should work with both layered and multipledescription encodings. Participating peers have heterogeneous and asymmetric bandwidth connectivity. Furthermore, peers may join and leave in an arbitrary fashion. Overview: In PRO, each peer (or receiver) progressively searches for a subset of parents that collectively maximize delivered bandwidth and minimize overall delay from all parents to the receiver. Such a subset of parents may change over time as some parents join (or leave) the overlay, or available bandwidth from current parents significantly changes. Note that each peer can be both receiver and parent at the same time 4 . Each receiver periodically exchanges messages (i.e., gossips) with other peers in the overlay to learn about those participating peers that are potentially good parents. Potentially good parents for a receiver are identified based on their relative utility for the receiver. The utility of a parent peer pi for a receiver pj is a function of their relative network distance (delij) and the outgoing access link bandwidth of the parent (outbwi), (i.e., U(pi, pj) 3 These multi-tree approaches often do not use congestion control for each connection. 4 Throughout this paper we use receiver and parent as short form for receiver peer and parent peer. = f(delij, outbwi)). Using parents" access link bandwidth instead of available bandwidth has several advantages: (i) outgoing bandwidth is an upper bound for available bandwidth from a parent. Therefore, it enables the receiver to roughly classify different parents. (ii) estimating available bandwidth requires end-to-end measurement and such a solution does not scale with the number of peers, and more importantly, (iii) given a utility function, this approach enables any peer in the overlay to estimate relative utility of any other two peers. Each receiver only maintains information about a fixed (and relatively small) number of promising parent peers in its local image. The local image at each receiver is dynamically updated with new gossip messages as other peers join/leave the overlay. Each peer selects a new parent in a demand-driven fashion in order to minimize the number of end-to-end bandwidth measurements, and thus improve scalability. When a receiver needs a new parent, its PS mechanism randomly selects a peer from its local image where probability of selecting a peer directly depends on its utility. Then, the actual properties (i.e., available bandwidth and delay) of the selected parent are verified through passive measurement. Toward this end, the selected parent is added to the parent list which triggers PALS to request content from this parent. Figure 1 depicts the interactions between PD and PS mechanisms. In PRO, each receiver leverages congestion controlled bandwidth from its parents as an implicit signal to detect two events: (i) any measurable shared bottleneck among connections from different parents, and (ii) any change in network or overlay conditions (e.g., departure or arrival of other close-by peers). Figure 2 shows part of an overlay to illustrate this feature. Each receiver continuously monitors available bandwidth from all its parents. Receiver p0 initially has only p1 as a parent. When p0 adds a new parent (p2), the receiver examines the smoothed available bandwidth from p1 and p2 and any measurable correlation between them. If the available bandwidth from p1 decreases after p2 is added, the receiver can conclude that these two parents are behind the same bottleneck (i.e., link L0). We note that paths from two parents might have some overlap that does not include any bottleneck. Assume another receiver p3 selects p1 as a parent and thus competes with receiver p0 for available bandwidth on link L1. Suppose that L1 becomes a bottleneck and the connection between p1 to p3 obtains a significantly higher share of L1"s bandwidth than connection between p1 to p0. This change in available bandwidth from p1 serves as a signal for p0. Whenever a receiver detects such a drop in bandwidth, it waits for a random period of time (proportional to the available bandwidth) and then drops Source Peer Disc. Peer Selec. gossip Exam. a New Parent Criteriafor PeerDiscovery Update LocalImage oftheOverlay Unknown peers in the overlay Known peers in the overlay Select Internal Components of Receiver Peer Receiver Peer Figure 1: Interactions between PD and PS mechanisms through local image 44 P1 P3 P0 L0 L1 L3 Overlay connection Network Path P2 L2 P1 P3 P0 L0 L1 L3 P2 L2 Initial Overlay Reshaped Overlay Figure 2: Using congestion controlled bandwidth as signal to reshape the overlay the corresponding parent if its bandwidth remains low [8]. Therefore, the receiver with a higher bandwidth connectivity (p3) is more likely to keep p1 as parent whereas p0 may examine other parents with higher bandwidth including p3. The congestion controlled bandwidth signals the receiver to properly reshape the overlay. We present a summary of key features and limitations of PRO in the next two sections. Table 1 summarizes our notation throughout this paper. Main Features: Gossiping provides a scalable approach to peer discovery because each peer does not require global knowledge about all group members, and its generated traffic can be controlled. The PD mechanism actively participates in peer selection by identifying peers for the local image which limits the possible choices of parents for the PS mechanism. PRO constructs a multi-parent, unstructured overlay. But PRO does not have the same limitations that exist in multi-tree approaches because it allows each receiver to independently micro-manage its parents to maximize its overall bandwidth based on local information. PRO conducts passive measurement not only to determine available bandwidth from a parent but also to detect any shared bottleneck between paths from different parents. Furthermore, by selecting a new parent from the local image, PRO increases the probability of finding a good parent in each selection, and thus significantly decreases number of required measurements which in turn improves scalability. PRO can gracefully accommodate bandwidth heterogeneity and asymmetry among peers since PALS is able to manage delivery of content from a group of parents with different bandwidth. Limitations and Challenges: The main hypothesis in our framework is that the best subset of parents for each receiver are likely to be part of its local image i.e., PD mechanism can find the best parents. Whenever this condition is not satisfied, either a receiver may not be able to maximize its overall bandwidth or resulting overlay may not be efficient. Table 1: Notation used throughout the paper Symbol Definition. pi Peer i inbwi Incoming access link BW for pi outbwi Outgoing access link BW for pi min nopi Min. No of parents for pi max nopi Max. No of parents for pi nopi(t) No of active parents for pi at time t img sz Size of local image at each peer sgm Size of gossip message delij Estimated delay between pi and pj Clearly, properties of the selected utility function as well as accuracy of estimated parameters (in particular using outgoing bandwidth instead of available bandwidth) determine properties of the local image at each peer which in turn affects performance of the framework in some scenarios. In these cases, the utility value may not effectively guide the search process in identifying good parents which increases the average convergence time until each peer finds a good subset of parents. Similar to many other adaptive mechanisms (e.g., [13]), the parent selection mechanism should address the fundamental tradeoff between responsiveness and stability. Finally, the congestion controlled bandwidth from parent peers may not provide a measurable signal to detect a shared bottleneck when level of multiplexing is high at the bottleneck link. However, this is not a major limitation since the negative impact of a shared bottleneck in these cases is minimal. All the above limitations are in part due to the simplicity of our framework and would adversely affect its performance. However, we believe that this is a reasonable design tradeoff since simplicity is one of our key design goals. In the following sections, we describe the two key components of our framework in further details. 4. GOSSIP-BASED PEER DISCOVERY Peer discovery at each receiver is basically a search among all participating peers in the overlay for a certain number (img sz) of peers with the highest relative utility. PRO adopts a gossip-like [10] approach to peer discovery. Gossiping (or rumor spreading) has been frequently used as a scalable alternative to flooding that gradually spreads information among a group of peers.However, we use gossiping as a search mechanism [9] for finding promising parents since it has two appealing properties (i) the volume of exchanged messages can be controlled, and (ii) the gossip-based information exchange can be customized to leverage relative utility values to improve search efficiency. The gossip mechanism works as follow: each peer maintains a local image that contains up to img sz records where each record represents the following information for a previously discovered peer pi in the overlay: 1) IP address, 2) GNP coordinates, 3) number of received layers, 4) timestamp when the record was last generated by a peer, 5) outbwi and 6) inbwi. To bootstrap the discovery process, a new receiver needs to learn about a handful of other participating peers in the overlay. This information can be obtained from the original server (or a well-known rendezvous point). The server should implement a strategy for selecting the initial peers that are provided to each new receiver. We call this the initial parent selection mechanism. Once the initial set of peers are known, each peer pi periodically invokes a target selection mechanism to determine a target peer (pj) from its local image for gossip. Given a utility function, peer pi uses a content selection strategy to select sgm records (or smaller number when sgm records are not available) from its local image that are most useful for pj and send those records to pj. In response, pj follows the same steps and replies with a gossip message that includes sgm records from its local image that are most useful for pi, i.e., bidirectional gossip. When a gossip message arrives at each peer, an image maintenance scheme integrates new records into the current local image and discards excess records such that certain property of the local image is improved (e.g., increase overall utility of peers in the image) Aggregate performance of 45 a gossip mechanism can be presented by two average metrics and their distribution among peers: (i) Average Convergence Time: average number of gossip messages until all peers in an overlay reach their final images, and (ii) Average Efficiency Ratio: average ratio of unique records to the total number of received records by each peer. We have been exploring the design space of four key components of the gossip mechanism. Frequency and size of gossip messages determine average freshness of local images. Currently, the server randomly selects the initial parents from its local image for each new peer. Target Selection: Target selection randomly picks a peer from the current image to evenly obtain information from different areas of the overlay and speed up discovery. Content Selection: peer pk determines relative utility of all the peers (pj) in its local image for target peer pi, and then randomly selects sgm peers to prepare a gossip message for pi. However, probability of selecting a peer directly depends on its utility. This approach is biased towards peers with higher utility but its randomness tend to reduce number of duplicate records in different gossip message from one peer (i.e., improves efficiency). A potential drawback of this approach is the increase in convergence time. We plan to examine more efficient information sharing schemes such as bloom filters [3] in our future work. PRO uses joint-ranking [15] to determine relative utility of a parent for a receiver. Given a collection of peers in a local image of pk, the jointranking scheme ranks all the peers once based on their outgoing bandwidth, and then based on their estimated delay from a target peer pi. The utility of peer pj (U(pj, pi)) is inversely proportional to the sum of pj"s ranks in both rankings. Values for each property (i.e., bandwidth and delay) of various peers are divided into multiple ranges (i.e., bins) where all peers within each range are assumed to have the same value for that property. This binning scheme minimizes the sensitivity to minor differences in delay or bandwidth among different peers. Image maintenance: Image maintenance mechanism evicts extra records (beyond img sz) that satisfy one of the following conditions: (i) represent peers with the lower utility, (ii) represent peers that were already dropped by the PS mechanism due to poor performance and (iii) have a timestamp older than a threshold. This approach attempts to balance image quality (in terms of overall utility of existing peers) and its freshness. Note that the gossip mechanism can discover any peer in the overlay as long as reachability is provided through overlap among local images at different peers. The higher the amount of overlap, the higher the efficiency of discovery, and the higher the robustness of the overlay to dynamics of peer participations. The amount of overlap among images depends on both the size and shape of the local images at each peer. The shape of the local image is a function of the deployed utility function. Joint-ranking utility gives the same weight to delay and bandwidth. Delay tends to bias selection towards near-by peers whereas outgoing bandwidth introduces some degree of randomness in location of selected peers. Therefore, the resulting local images should exhibit a sufficient degree of overlap. 5. PARENT SELECTION The PS mechanism at each peer is essentially a progressive search within the local image for a subset of parent peers such that the following design goals are achieved: (i) maximizing delivered bandwidth 5 , (ii) minimizing the total delay from all parents to the receiver, and (iii) maximizing diversity of paths from parents (whenever it is feasible). Whenever these goals are in conflict, a receiver optimizes the goal with the highest priority. Currently, our framework does not directly consider diversity of paths from different parents as a criteria for parent selection. However, the indirect effect of shared path among parents is addressed because of its potential impact on available bandwidth from a parent when two or more parents are behind the same bottleneck. The number of active parents (nopi(t)) for each receiver should be within a configured range [min nop, max nop]. Each receiver tries to maximize its delivered bandwidth with the minimum number of parents. If this goal can not be achieved after evaluation of a certain number of new parents, the receiver will gradually increase its number of parents. This flexibility is important in order to utilize available bandwidth from low bandwidth parents, i.e., cope with bandwidth heterogeneity. min nop determines minimum degree of resilience to parent departure, and minimum level of path diversity (whenever diverse paths are available). The number of children for each peer should not be limited. Instead, each peer only limits maximum outgoing bandwidth that it is able (or willing) to provide to its children. This allows child peers to compete for congestion controlled bandwidth from a parent which motivates child peers with poor bandwidth connectivity to look for other parents (i.e., properly reshape the overlay). Design of a PS mechanism should address three main questions as follows: 1) When should a new parent be selected? There is a fundamental tradeoff between responsiveness of a receiver to changes in network conditions (or convergence time after a change) and stability of the overlay. PRO adopts a conservative approach where each peer selects a new parent in a demand-driven fashion. This should significantly reduce number of new parent selections, which in turn improves scalability (by minimizing the interference caused by new connections) and stability of the overlay structure. A new parent is selected in the following scenarios: (i) Initial Phase: when a new peer joins the overlay, it periodically adds a new parent until it has min nop parents. (ii) Replacing a Poorly-Performing Parent: when available bandwidth from an existing parent is significantly reduced for a long time or a parent leaves the session, the receiver can select another peer after a random delay. Each receiver selects a random delay proportional to its available bandwidth from the parent peer [8]. This approach dampens potential oscillation in the overlay while increasing the chance for receivers with higher bandwidth connectivity to keep a parent (i.e., properly reshapes the overlay). (iii) Improvement in Performance: when it is likely that a new parent would significantly improve a non-optimized aspect of performance (increase the bandwidth or decrease the delay). This strategy allows gradual improvement of the parent subset as new peers are discovered (or joined) the overlay. The available information for each peer in the image is used as a heuristic to predict performance of a new peer. Such an improvement should be examined infrequently. A hysteresis mechanism 5 The target bandwidth is the lower value between maximum stream bandwidth and receiver"s incoming bandwidth. 46 is implemented in scenario (ii) and (iii) to dampen any potential oscillation in the overlay. 2) Which peer should be selected as a new parent? At any point of time, peers in the local image are the best known candidate peers to serve as parent. In PRO, each receiver randomly selects a parent from its current image where the probability of selecting a parent is proportional to its utility. Deploying this selection strategy by all peers lead to proportional utilization of outgoing bandwidth of all peers without making the selection heavily biased towards high bandwidth peers. This approach (similar to [5]) leverages heterogeneity among peers since number of children for each peer is proportional to its outgoing bandwidth. 3) How should a new parent be examined? Each receiver continuously monitors available bandwidth from all parents and potential correlation between bandwidth of two or more connections as signal for shared bottleneck. The degree of such correlation also reveals the level of multiplexing at the bottleneck link, and could serve as an indicator for separating remote bottlenecks from a local one. Such a monitoring should use average bandwidth of each flow over a relatively long time scale (e.g., hundreds of RTT) to filter out any transient variations in bandwidth. To avoid selecting a poorly-performing parent in the near future, the receiver associates a timer to each parent and exponentially backs off the timer after each failed experience [13]. After the initial phase, each receiver maintains a fixed number of parents at any point of time (nopi(t)). Thus, a new parent should replace one of the active parents. However, to ensure monotonic improvement in overall performance of active parents, a new parent is always added before one of the existing parents is dropped (i.e., a receiver can temporarily have one extra parent). Given the available bandwidth from all parents (including the new one) and possible correlation among them, a receiver can use one of the following criteria to drop a parent: (i) to maximize the bandwidth, the receiver can drop the parent that contributes minimum bandwidth, (ii) to maximize path diversity among connections from parents, the receiver should drop the parent that is located behind the same bottleneck with the largest number of active parents and contributes minimum bandwidth among them. Finally, if the aggregate bandwidth from all parents remains below the required bandwidth after examining certain number of new parents (and nopi(t) < max nop), the receiver can increase the total number of parents by one. 6. CONCLUSIONS AND FUTURE WORK In this paper, we presented a simple receiver-driven framework for architecting peer-to-pee overlay structures called PRO. PRO allows each peer to selfishly and independently determine the best way to connect to the overlay to maximize its performance. Therefore, PRO should be able to maximize delivered quality to peers with heterogeneous and asymmetric bandwidth connectivity. Both peer discovery and peer selection in this framework are scalable. Furthermore, PRO uses congestion controlled bandwidth as an implicit signal to detect shared bottleneck among existing parents as well as changes in network or overlay conditions to properly reshape the structure. We described the basic framework and its key components, and sketched our strawman solutions. This is a starting point for our work on PRO. We are currently evaluating various aspects of this framework via simulation, and exploring the design space of key components. We are also prototyping this framework to conduct real-world experiments on the Planet-Lab in a near future. 7. REFERENCES [1] A. Akella, S. Seshan, and A. Shaikh. An empirical evaluation of wide-area internet bottlenecks. In Internet Measurement Conference, 2003. [2] S. Banerjee, B. Bhattacharjee, and C. Kommareddy. Scalable application layer multicast. In ACM SIGCOMM, 2002. [3] J. Byers, J. Considine, M. Mitzenmacher, and S. Rost. Informed Content Delivery Across Adaptive Overlay Networks. In ACM SIGCOMM, 2002. [4] M. Castro, P. Druschel, A.-M. Kermarrec, A. R. A. Nandi, and A. Singh. SplitStream: High-bandwidth content distribution in a cooperative environment. In ACM SOSP, 2003. [5] Y. Chawathe, S. Ratnasamy, L. Breslau, N. Lanham, and S. Shenker. Making gnutella-like p2p systems scalable. In ACM SIGCOMM, 2003. [6] Y. Chu, S. G. Rao, S. Seshan, and H. Zhang. Enabling conferencing applications on the internet using an overlay multicast architecture. In ACM SIGCOMM, 2001. [7] Y. Chu, S. G. Rao, and H. Zhang. A case for end-system multicast. In ACM SIGMETRICS, 2000. [8] S. Floyd, V. Jacobson, C. Liu, S. McCanne, and L. Zhang. Reliable Multicast Framework for Light-Weight Sessions and Application Level Framing. ACM/IEEE Transactions on Networking, 1997. [9] M. Harchol-Balter, F. T. Leighton, and D. Lewin. Resource discovery in distributed networks. In Symposium on Principles of Distributed Computing, pages 229-237, 1999. [10] S. Hedetniemi, S. Hedetniemi, and A. Liestman. A Survey of Gossiping and Broadcasting in Communication Networks. In Networks, 1988. [11] D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat. Bullet: High bandwidth data dissemination using an overlay mesh. In SOSP, 2003. [12] K. Lakshminarayanan and V. N. Padmanabhan. Some findings on the network performance of broadband hosts. In Internet Measurement Conference, 2003. [13] S. McCanne, V. Jacobson, and M. Vettereli. Receiver-driven layered multicast. In ACM SIGCOMM, 1996. [14] T. S. E. Ng, Y. Chu, S. G. Rao, K. Sripanidkulchai, and H. Zhang. Measurement-based optimization techniques for bandwidth-demanding peer-to-peer systems. In IEEE INFOCOM, 2003. [15] T. S. E. Ng and H. Zhang. Predicting internet network distance with coordinates-based approaches. In IEEE INFOCOM, 2002. [16] V. N. Padmanabhan, H. J. Wang, and P. A. Chou. Resilient peer-to-peer streaming. In IEEE ICNP, 2003. [17] R. Rejaie, M. Handley, and D. Estrin. RAP: An end-to-end rate-based congestion control mechanism for realtime streams in the internet. In IEEE INFOCOM, 1999. [18] R. Rejaie and A. Ortega. PALS: Peer-to-Peer Adaptive Layered Streaming. In NOSSDAV, 2003. [19] S. Saroiu, P. K. Gummadi, and S. D. Gribble. Measurement study of peer-to-peer file system sharing. In SPIE MMCN, 2002. [20] D. A. Tran, K. A. Hua, and T. Do. Zigzag: An efficient peer-to-peer scheme for media streaming. In IEEE INFOCOM, 2003. 47

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

peer-to-peer streaming
receiver-driven approach
receiver-driven overlay
distributed system
proper subset of parent peer
gossip-based peer discovery
receiver-driven parent selection
peer-to-peer stream

congestion control
design
measurement
efficient overlay structure
pro