David Muir Sharnoff > OOPS > OOPS



Annotate this POD


New  2
Open  0
View/Report Bugs
Module Version: 0.2005   Source  


 OOPS - Object Oriented Persistent Store


 use OOPS;

 transaction(sub {
        $oops = new OOPS 
                dbi_dsn => $DBI_DSN, 
                username => $username, 
                password => $password,
                table_prefix => "MY",
                readonly => 0;





        my $dbh = $oops->dbh();








OOPS maps all perl objects to the same RDBMS schema. No advance schema definition is required on the part of the user of OOPS. The name of the package (OOPS) comes from the realization that perl's data model is much more complicated than I initially understood. Internally, the RDBMS schema uses four tables: a table of objects, a table of attributes (keys and values), a table of large attributes that are too big to fit in normal (VARCHAR) columns, and a table of counters.


 create OOPS instance

 access some objects

 modify some objects






 use OOPS;

 transaction(sub {
        my $oops = new OOPS 
                username => 'MY-USERNAME', 
                password => 'MY-PASSWORD',
                table_prefix => "MY-TABLE-PREFIX";








Large HASHes are supported by only loading keys as tehy are accessed.

HASHes, array elements, and REFs are implemented with tie(). ARRAYs are not currently tie()d because of bugs in perl. Multiple references to the same scalar are supported. References to array elements and hash values are supported. Persistent data is reference counted and cycles must be manually broken to assure de-allocation. A persistent data garbage collector OOPS::GC is supplied if you cannot guarantee to break your own cycles.


Support for the following RDBMSs are is included:


OOPS 0.2005 has been tested with PostgreSQL version 9.1.3 on Linux. OOPS 0.2002 has been tested with PostgreSQL version 8.1.8 on Linux. OOPS 0.1004 was tested with PostgreSQL version 8.1.4 on Linux. OOPS 0.1003 was tested with 7.4.2 and 7.3.5 on Linux and 7.3.5 on pre-release DragonflyBSD. Somewhere between 7.4.2 and 8.1.4 PostgreSQL got more strict about TEXT versus BYTEA types. OOPS 1.003 was using TEXT where it should have been using BYTEA. Be sure to upgrade OOPS to 0.1004 or later *before* upgrading PostgreSQL to 8.x or later.

Biggest issue: PostgreSQL runs the test suite more slowly than mysql or SQLite.

On rare occaision PostgresSQL will fail with an ERROR: duplicate key violates unique constraint error. If this happens inside a call to transaction() it will automatically restart. You only need to worry about this if you are not using transaction().


OOPS 0.2005 has been tested with mysql 5.1.46 on Linux. OOPS 0.2003 has been tested with mysql 5.0.45 on Linux. OOPS 0.2002 has been tested with mysql 5.0.22 on Linux. OOPS 0.1004 has been tested with mysql 4.1.15 on Linux. OOPS 0.1003 was tested with mysql 4.0.16 and 4.0.18 using InnoDB tables.


Mysql locking doesn't work correctly. To get correct behavior, access to the database must be exclusive http://bugs.mysql.com/?id=28029. This problem is fixed in 5.0.45.


DBD::SQLite versions 0.x use SQLite 2.x. DBD::SQLite2 is the continuation of DBD::SQLite version 0.x codebase and uses SQLite 2.x.

DBD::SQLite2 is not 8-bit clean with respect to \0. OOPS uses a DBD::SQLite2 feature to translate binary nulls. A side-effect is that backslash will be doubled \ -> \\ in DBD::SQLite2 (the extras will be removed transparently by DBD::SQLite2.

OOPS does not use the counters table with DBD::SQLite2.



DBD::SQLite versions 1.x leak file descriptors and will eventually die if you create use and destroy too many OOPS instances. DBD::SQLite versions 1.x cannot pass the regression suite and may not be reliable in production use. Use DBD::SQLite2 instead.

OOPS does not use the counters table with DBD::SQLite.

SQLite is not 8-bit clean with respect to \0. OOPS uses a DBD::SQLite feature to translate binary nulls. A side-effect is that backslash will be doubled \ -> \\ in DBD::SQLite (the extras will be removed transparently by DBD::SQLite. As of 0.2002, OOPS doesn't make use of the newer (version 1.x) DBD::SQLite features for handling nulls.

To work around the problem with the leaked file descriptors, on systems that support /proc, file descriptors in excess of $OOPS::sqlite_v3::enable_fd_leak_workaround (500 by default) that match the sqlite database file will be closed when new OOPS instances are created. This sometimes helps. Set $OOPS::sqlite_v3::enable_fd_leak_workaround to 0 to disable.

Related bug reports: http://rt.cpan.org/Ticket/Display.html?id=20286, http://rt.cpan.org/Ticket/Display.html?id=20288.


OOPS 0.2002 and 0.1004 have been tested on Linux 2.6.15 (Ubuntu 6.06LTS - Dapper Drake). OOPS 0.1003 was tested on Linux 2.4.23 (Debian unstable as of April '04); on FreeBSD 4.9; and on DragonflyBSD prerelease.

As far as performance goes, mysql and SQLite are both about twice as fast as PostgreSQL for applications that only have one transaction at at time. SQLite and mysql are particularly slow when there are multiple transactions as their lock granularity is the entire database.

Each RDBMS uses a somewhat different schema.


transaction($funcref, @args)

transaction() is a wrapper for a complete transaction. Transactions that fail due to deadlock with other processes will be re-run automatically.

The first parameter is a reference to a function. Any additional parameters will be passed as parameters to that function. The return value of transaction() is the return value of &$funcref().

It is not necessary to use the transaction() method. Beware that nearly any operation on persistent date (even read operations) can cause deadlock.

Any use of persistent data can trigger a deadlock. The transaction() function catches this and retries automatically up to $OOPS::transaction_maxtries times (15 times unless you change it). If you don't use transaction() you might want to catch the exceptions that transaction() catches. To do this, you can regex match $@ against $OOPS::transfailrx.

Basically, transaction is a slightly fancier version of the following:

 sub transaction
        my ($sub, @args) = @_;
        for (0..15) {
                eval {
                last unless $@;
                die $@;


The following package variables are available during transactions:


The number of attempts at this transaction. Starts with 1.


The maximum number of attempts that will be made.


The minimum amount of time to sleep() if the transaction fails.


The maximum amount of time to sleep() if the transaction fails.


A list of functions to call in the event of a transaction rollback (code refs).

getref(%hash, $key)

References to tied hash keys are buggy in all perls through 5.8.7 (and beyond?). Use getref(%hash, $key) to create your reference to a tied hash key. See: http://rt.perl.org/rt3/Ticket/Display.html?id=27555 and http://rt.perl.org/rt3/Ticket/Display.html?id=29224.

 $ref = getref(%hash, $key);

Alternatively, use $oops->workaround27555($ref).


walk_hash(%hash, $stride, [ $previous_last_key ])

If you have a large virtual hash, you may want to be able to iterate over the keys in the hash as a bunch of transactions rather than as one transaction.

Walk_hash() can be called on any HASH. If it isn't an OOPS virtual hash, it will return keys in batches just like if it had been called on an OOPS virtual hash.

Here is an example of how to use it:

 my $stride = 20;
 my $done = 0;
 my $lastkey;
 while (not $done) {
        transaction(sub {
                my $oops = OOPS->new(...);
                my @things = walk_hash(%{$oops->{my_table}}, $stride, $lastkey);
                for my $thing (@things) {
                        # do stuff
                $lastkey = $things[$#things];
                $done = 1 unless @things == $stride;

Here is another example of how to use it (with OOPS::TxHash):

 use OOPS::TxHash;

 my $stride = 20;

 my %real_todo = ( stuff => undef );
 while (%real_todo) {
        transaction(sub {
                my $th = tie my %todo, 'OOPS::TxHash', \%real_todo or die;

                if (exists $todo{stuff}) {
                        my @keys = walk_hash(%{$oops->{stuff}}, $stride, $todo{stuff});

                        for my $k (@keys) {
                                # work on $oops->{stuff}{$k}

                        $todo{stuff} = $keys[$#keys];
                        delete $todo{stuff} unless @keys == $stride;


Here is yet another example (with a wrapper function):

        sub { OOPS->new(...) },
        sub { my $oops = shift; $oops->{my_table} },
        sub { 
                my ($oops, $thing) = @_;
                # do stuff
        allatonce => 0

 sub walk_eval
        my ($stride, $get_oops, $get_hash, $agent, %opts) = @_;
        my $done = 0;
        my $last = undef;
        $stride ||= 100;
        while (! $done) {
                transaction(sub {
                        my $oops = &$get_oops();
                        my $hash = &$get_hash($oops);
                        my @items = walk_hash(%$hash, $stride, $last);
                        if ($opts{allatonce}) {
                                &$agent($oops, @items);
                        } else {
                                for my $item (@items) {
                                        &$agent($oops, $item);
                        $last = $items[$#items];
                        $done = 1 unless @items == $stride;




The %parameters are documented in the next section.



The %parameters are documented in the next section.


The new() and initial_setup() methods take a hash specification for their behavior. Here are the parameters allowed:





password / $ENV{OOPS_PASS} / $ENV{DBI_PASS}



If true, commit() will be disabled and the transaction isolation level will be set to READ COMMITTED instead of SERIALIZABLE for the databases where this matters (mysql, PostgreSQL). This should be used if you plan to scan across large amounts of data because otherwise everything you read will end up locked. This feature is still experimental (not covered by the test suite).


If true, iterating over the keys of a virtual hash (see virtual_object) will not cache the key values in memory. If your virtual hash is too big to fit in memory, this is a very good idea. Alternatively, you can use the clear_cache method (below). This feature is still experimental (not covered by the test suite).

table_prefix / $ENV{OOPS_PREFIX}

OOPS allows a prefix to be supplied for it's internal table names. If you set a prefix of FOO_ then it will use a FOO_object table instead of an object table. This can be set as an argument to new() or it can be set with the environment variable $OOPS_PREFIX. This allows multiple separate object spaces to exist within the same backend SQL database. It's intended use is to support testing vs. production environments but it could also be used to segregate object spaces that don't link to each other.

The current tables and indexes are:


Each row is an object.


Each row is an attribute value in an object.


Each row is a large attribute value.


Counters for things like object ids.


A temporary table.


Object grouping index.


Lookup by attribute value.

auto_upgrade / $ENV{OOPS_UPGRADE}


auto_initialize / $ENV{OOPS_INIT}

The DBMS must be initialized before it can be used: tables created, a few rows inserted, etc. This can be handled by calling initial_setup() or it can be handled automatically by new() if auto_initialize or $OOPS_AUTO_INIT is set.

default_synchronous / $ENV{OOPS_SYNC}

With SQLite, an additional parameter to OOPS->new() is recognized: default_synchronous. Possible values are:


Sync() all transactions to disk before returning.




Don't sync() at all and go really fast.






This may be handled automatically in the future.


References to tied hash keys are buggy in all perls through 5.12.1 (and beyond?). Use workaround27555($reference) to register your new tied hash key references so that they can be transformed into references that actually work correctly.

 $ref = \%hash{$key};

workaround27555() is harmless if called on other sorts of references so it is safe to use indiscriminately. See http://rt.perl.org/rt3/Ticket/Display.html?id=27555.

Alternatively, use getref(%hash, $key).


This returns the main DBI database handle used by OOPS. This function is provided for those who want to hand-write queries. Please note: no changes are written to the DBMS by OOPS until $oops-commit()>.




This will clear out some of the cache that oops maintains. It is useful if you're iterating over the keys of a large virtual object and don't want to overflow memory. This feature is still experimental (not covered by the test suite).


Lock an object or a scalar. While this is never needed for transactional consistentcy within the OOPS data, it can be useful for transactions that have external side-effects like sending an email. In theory, if you pre-lock everything you might change, your transaction will not need to be rolled back due to deadlock. When you lock an object, it is the object record that is locked, not the contents. Locking \%hash won't stop another process from modifying $hash->{key}. Lock \$hash->{key} if you want that.


OOPS has been thoroughly tested. The regression suite is very well developed: there is twice as much code in the test suite as there is in the module itself. The suite does over 1.5million tests. I have so much confidence in my test suite, I'm offering a bounty on bugs!


OOPS caches in memory all data that is pulled in during a transaction. For virtual_object hash'es only, there are ways to prevent absolutely everything from being cached (the less_caching contructor agrument and the clear_cache method) but for other objects you need to have enough memory to hold everything you touch.

You can only call ->commit once per OOPS instance. The cache is emptied when the OOPS instance is destroyed. This means that OOPS will be loading up its cache over and over.

Known bugs in OOPS

memory leaks


delayed DESTROY


other magic

Other perl magic attributes are not currently stored persistently. Many probably could be supported, but many could not. For example, taint does not work on tied hashes: http://rt.perl.org/rt3/Ticket/Display.html?id=6758.

unreferenced blessed scalars


 my $x = 'foobar';
 my $y = \$x;
 bless $y, 'baz';
 $y = 7;
 $y = \$x;
 my $z = ref($y);

At the current time, OOPS does not store such blessings. OOPS does not remember blessings when there isn't a reference.

re-blessing the OOPS object

Reblessing the OOPS object is not supported.


DBD::Pg does not easily support ASCII NULL. OOPS has only partial support for ASCII NULL with PostgreSQL. Everthing works, but don't have ASCII NULL in your class names.

Circular references sometimes break

OOPS mostly allows you to make circular self-references:

 my $x;
 $x = \$x;

In some (rare) cases circular self-references sometimes break. Since I can't think on any good reason to make a circular self-reference, I'm not going to fix this bug.

Long hash keys

The database schema for OOPS does not support huge hash keys.

Bugs in perl that effect OOPS

References to hash keys


All of the above can either be avoided or you can workaround them by either calling workaround27555($YOUR_REFERENCE) whenever you create a tied hash key reference or by using getref(%hash, $key) to create your reference.

The perl bugs are documented in: http://rt.perl.org/rt3/Ticket/Display.html?id=27555 and http://rt.perl.org/rt3/Ticket/Display.html?id=29224.

local and tie

local(%some_tied_hash) doesn't work right. Thus local(%some_persistent_hash) won't work right either: http://rt.perl.org/rt3/Ticket/Display.html?id=6017.


Tied scalar(%hash) support was added in perl 5.8.3 and does not exist in 5.8.2.

Tied arrays don't work right

There are a couple of bugs with tied arrays that prevent OOPS from using them: http://rt.perl.org/rt3/Ticket/Display.html?id=22570 and http://rt.perl.org/rt3/Ticket/Display.html?id=22571. OOPS fully loads arrays into memory to work-around this problem. This isn't a big deal unless you've got big arrays.

This was fixed in perl 5.10, but OOPS does not yet take advantage of tied arrays.

SQLite and perl's malloc().

If SQLite is used with a perl that has been compiled to use perl's malloc(), it will report LOTS of Bad free() ignored (PERL_CORE) errors. It is not currently known if these errors are harmful beyond generating lots of output to STDERR. The default perl configuration on FreeBSD uses perl's malloc().


As OOPS is developed, the database schema that OOPS uses changes.




Like Perl, OOPS uses reference counting. If you have circular data structures and you delete all you references to the circular data structure, you'll leak persistent memory.

Use OOPS::GC to mop up the leak.

Previous versions of OOPS have had bugs that caused the reference counts on the persistent data to become inaccurate. Use OOPS::Fsck to fix such problems. Do this before once, before using OOPS::GC.



fix the bugs


code cleanup and general performance enhancements


perl-syntax SQL query translator
 SELECT Employee WHERE $Employee->{salary} > 5000


better grouping




weak references

Support for persistent weak references is possible.

external references to objects





support for 'base' & accessor methods


schema enforcement


RDBMS -> object mapping

Map existing RDBMS schemas into objects.

data viewer


support for tied data structures


support for other base types.


on-line data migration




Each perl HASH, REF/SCALAR, or ARRAY has a row in the object table and multiple rows in the attribute and big tables.

Here are the columns you'll care about:


There is one row per perl object.


The object id.


The blessed class name (limited to 255 characters).


The type of object:










The object id.


The hash key or array index.


he hash value or array value. Limited to 255 characters.


Flags the type of the value. Possible values are:


A normal value. Numeric or string.


An big value. pval will be a copy of the start of the value for the first N characters. The end of pval will be a MD5 checksum of the full big value.


A reference to another object.




The object id.


The hash key or array index.








 id             class           otype

 1              OOPS::NamedObj  H
 383            SCALAR          R
 384            SCALAR          R
 385            SCALAR          R
 386            REF             R
 400            HASH            H
 500            ARRAY           A

 id             pkey            pval            ptype

 1              A500            500             R
 1              H400            400             R
 1              R383            383             R
 1              R384            384             R
 1              R385            385             R
 1              R386            386             R

 383            400             'a-key'         0

 384            384             'nopkey'        0
 384            'nopkey'        'a-value'       0

 385            384             'nopkey'        0

 386            386             'nopkey'        0
 386            'nopkey'        500             R

 400            'a-key'         'a-value        0
 400            'another-key'   'another-value' 0
 400            'A500'          500             R

 500            0               'a-value'       0
 500            1               'another-value' 0

HASH 1 is %$oops.





HASH 400 is a normal hash.

ARRAY 500 is a normal hash.


 my $oops = new OOPS 
        username => 'MY-USERNAME', 
        password => 'MY-PASSWORD';


 $oops->{H400} = { 







 SELECT object.id
 FROM   object, attribute
 WHERE  object.class = 'Foobar'
 AND    object.id = attribute.id
 AND    object.otype = 'H'
 AND    attribute.pkey = 'xyz'
 AND    attribute.pval = 'abc'
 AND    attribute.ptype = '0'


 SELECT ohash.object
 FROM   object AS ohash,
        attribute AS ahash,
        object AS oref,
        attribute AS aref,
        attribute AS target
 WHERE  ohash.class = 'Foobar'
 AND    ohash.otype = 'H'
 AND    ahash.id = ohash.id
 AND    ahash.pkey = 'xyz'
 AND    ahash.ptype = 'R'
 AND    oref.id = ahash.pval
 AND    oref.otype = 'S'                # this is the outer ref
 AND    oref.id = aref.id               
 AND    aref.pval = target.pkey         # here's the reference indirection
 AND    target.pval = 'abc'
 AND    target.ptype = '0'

If you construct a query like these examples that return object id's, then use $object = $oops->load_object($id) to load them into memory.






Beware mysql logging. On Debian unstable, the default configuration for mysql used to log every SQL statement. Running the test suite to completion will generate several gigabytes of log file. Running out of disk space will cause the tests to fail. On DragonflyBSD the default mysql configuration includes making replication master logs.



Other object persistence solutions either store data in blobs that cannot be querried or they they're a mapping from a relation database into perl data that does not allow arbitrary data to be stored.

http://poop.sourceforge.net/ has an overview of options.


The following applications use OOPS: Qpsmtpd::Plugin::Quarantine


Copyright(C) 2004-2006 David Muir Sharnoff <muir@idiom.org> Copyright(C) 2012 Google, Inc

OOPS may be used, modifified and redistributed on the same terms as Perl itself.

syntax highlighting: