DBCacheContrib

Reusable code that treats TWiki forms as if they were table rows in a database.

<--

   PLEASE DO NOT EDIT THIS TOPIC

   It is automatically generated from the subversion repository, and any
   changes you make will simply be overwritten the next time a release is
   generated.

   Instead, you could check your fix in, raise a bug in the Bugs web, or
   mail the author.
-->

WARNING: TWiki-4 only. If you want to use this extension with an earlier version of TWiki, please see an earlier version of this topic

Summary of Contents

This module supports structured queries over a database built on the fly from the forms in TWiki topics. It does not support any tags, as it is provided as a service for other plugins that want to treat a TWiki web as a simple database; for example, the TWiki:Plugins/FormQueryPlugin, which supports the display of query results.

The plugin encapsulates code that was formerly in the "engine room" of the FormQueryPlugin^?. It has been abstracted out in the belief that it will be useful to anyone who wants to do simple search operations from a plugin.

Features

• Perform complex queries on the TWiki database
• Cache TWiki topics for rapid queries

How the database gets built

Each topic in the web automatically gets a number of standard fields, generated by reading the metadata from the topic (see TWikiMetaData)

• name - name of the topic
• parent - name of parent topic
• _up - reference to the Map of the parent topic, if it exists
• attachments - array of Maps, each of which contains:
  • _up - reference to the Map for the topic
  • name - attachment name
  • attr - e.g hidden
  • comment - attachment comment
  • path - client path used to upload attachment
  • size - size in Kb
  • user - who uploaded the attachment
  • version - e.g. 1.3
• info - Map containing:
  • _up - reference to the Map for the topic
  • author - most recent author
  • date - date of last change
  • format - topic format version
  • version - topic version number
• moved - Map containing:
  • _up - reference to the Map for the topic
  • by - who moved it
  • date - when they moved it
  • from - where they moved it from
  • to - where they moved it to
• form - form type
• form name - e.g. if a "MyForm" is attached, this will be MyForm. This is a reference to a Map containing a key for each field in the form. Each key maps to the value in the form data for that key. The Map will also have an _up reference to the Map for the topic.
• text - raw text of the topic)

The sub-Maps created for info, form name, moved, and each row in attachments also have a reference back to the topic Map, called _up.

Other fields may be added by subclasses. Refer to the documentation for the plugin that is using the DBCache for more details.

The cache

Extending or customising

The module is shipped with a perl build file, which should be used for installation and testing. Testing is done using CPAN:Test::Unit, and is invoked using the 'test' build target. Writing tests is a useful way of feeding back bugs as well. I can't encourage you enough to maintain and extend the tests!

Detailed Documentation

class DBCacheContrib

General purpose cache that treats TWiki topics as hashes. Useful for rapid read and search of the database. Only works on one web.

Typical usage:

  use TWiki::Contrib::DBCacheContrib;

  $db = new TWiki::Contrib::DBCacheContrib( $web ); # always done
  $db->load(); # may be always done, or only on demand when a tag is parsed that needs it

  # the DB is a hash of topics keyed on their name
  foreach my $topic ($db->getKeys()) {
     my $attachments = $db->get($topic)->get("attachments");
     # attachments is an array
     foreach my $val ($attachments->getValues()) {
       my $aname = $attachments->get("name");
       my $acomment = $attachments->get("comment");
       my $adate = $attachments->get("date");
       ...
     }
  }

new($dataDir, $web)

• $dataDir location of cache file
• $web name of web to create the object for.

readTopicLine($topic, $meta, $line, $fh) --> text

• $topic - name of the topic being read
• $meta - reference to the hash object for this topic
• line - the line being read
• $fh - the file handle of the file
• return text to insert in place of line in the text field of the topic

onReload($topics)

• $topics - perl array of topic names that have just been loaded (or reloaded)

load()

Load the web into the database. Returns a string containing 3 numbers that give the number of topics read from the cache, the number read from file, and the number of previously cached topics that have been removed.

write($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to

read($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from

class Search

Warning single and double quotes are not allowed in values!

The following operators are available:

Dates for EARLIER_THAN, LATER_THAN and WITHIN_DAYS must be dates in the format expected by Time::ParseDate (like the ActionTrackerPlugin^?). WITHIN_DAYS works out the number of working days (i.e. excluding Saturday and Sunday). Apologies in advance if your weekend is offset ± a day!

A search object implements the "matches" method as its general contract with the rest of the world.

Example

  $db = new TWiki::Contrib::DBCacheContrib::DBCache( $web ); # always done
  $db->load();
  my $search = new TWiki::Contrib::DBCacheContrib::Search("date EARLIER_THAN '1st January 2000'");

  foreach my $topic ($db->getKeys()) {
     my $attachments = $topic->get("attachments");
     foreach my $val ($attachments->getValues()) {
       if ($search->matches($val)) {
          print $val->get("name") . "\n";
       }
     }
  }

new($string)

• $string - string containing an expression to parse

matches($object) -> boolean

• $object - object to test; must implement get

toString() -> string

class FileTime^?

Object that handles a file/time tuple for use in Storable and TWiki::Contrib::DBCacheContrib::Archive.

new($file)

• $file - filename

uptodate() -< boolean

toString() -> string

write()

read()

class Array

Generic array object. This is required because perl arrays are not objects, and cannot be subclassed e.g. for serialisation. To avoid lots of horrid code to handle special cases of the different perl data structures, we use this array object instead.

new()

add($object)

• $object any perl data type

find($object) -> integer

• $object datum of the same type as the content of the array

remove($index)

• $index - integer index

get($key, $root) -> datum

• $k - key
• $root - what # refers to

• get("9", $r) where $n is a number will get the 9th entry in the array
• get("[9]", $r) will also get the 9th entry
• get(".9", $r) will also get the 9th entry
• get(".X", $r) will return the sum of the subfield X of each entry
• get("[?search]", $r) will perform the given search over the entries in the array. Always returns an array result, even when there is only one result. For example: [?name='Sam'] will return an array of all the entries that have their subfield name set to Sam.
• # means "reset to root". So get("#[3]", $r) will return the 4th entry of $r (assuming $r is an array!).
• get("[*X]", $r) will get a new array made from subfield X of each entry in this array.

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,

get("parent.UserTable[?SubTopic='ThisTopic'].UserName", $web);

See also TWiki::Contrib::DBCacheContrib::Map for syntax that applies to maps.

size() -> integer

sum($field) -> number

• $field - name of a field in the class of objects stored by this array

search($search) -> search result

• $search - TWiki::Contrib::DBCacheContrib::Search object to use in the search

getValues() -> perl array

Get a "perl" array of the values in the array, suitable for use with foreach

toString($limit, $level, $strung) -> string

• $limit - recursion limit for expansion of elements
• $level - currentl recursion level

write($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to

read($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from

class Map

new($string)

• $string - optional attribute string in standard TWiki syntax

fastget($k) -> datum

• $k - key

get($k, $root) -> datum

• $k - key
• $root what # refers to

Subfield syntax

• get("X",$r) will get the subfield named X.
• get("X.Y",$r) will get the subfield Y of the subfield named X.
• get("[X]",$r) = will get the subfield named =X (so X[Y] and X.Y are synonymous)..
• # means "reset to root". So get("#.Y", $r) will return the subfield =Y of $r (assuming $r is a map!), as will get("#[Y]".

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,

get("UserTable[0].Surname", $web);

See also TWiki::Contrib::DBCacheContrib::Array for syntax that applies to arrays.

set($k, $v)

• $k - key
• $v - value

size() -> integer

remove($index) -> old value

• $index - integer index

getKeys() -> perl array

Get a "perl" array of the keys in the map, suitable for use with foreach

getValues() -> perl array

Get a "perl" array of the values in the Map, suitable for use with foreach

search($search) -> search result

• $search - TWiki::Contrib::DBCacheContrib::Search object to use in the search

toString($limit, $level, $strung) -> string

• $limit - recursion limit for expansion of elements
• $level - currentl recursion level

write($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being written to

read($archive)

• $archive - the TWiki::Contrib::DBCacheContrib::Archive being read from

class Archive

This module is only used if Storable isn't available. Storable is much faster, because it is implemented in C.

new($file, $rw)

• $file - archive file path
• $rw - mode "r" or "w"

close()

writeByte($b)

• $b - byte to write

writeString($s)

• $s - string to write

writeInt($i)

• $i integer to write

writeObject()

readByte() -> byte

readString() -> string

readInt() -> integer

readObject() -> object

<-- 

 Set STUB = TWiki::Contrib::DBCacheContrib
 
 Set SHORTDESCRIPTION = Reusable code that treats TWiki forms as if they were table rows in a database
 
-->

Installation Instructions

Like many other TWiki extensions, this module is shipped with a fully automatic installer script written using the BuildContrib.

• If you have TWiki 4.1 or later, and Perl 5.8, you can install from the configure interface (Go to Plugins->Find More Extensions)
• If you have a permanent connection to the internet (and Perl 5.8), you are recommended to use the automatic installer script
  • Just download the DBCacheContrib_installer perl script and run it.
  • If the $TWIKI_PACKAGES environment variable is set to point to a directory, the installer will try to get archives from there. Otherwise it will try to download from twiki.org or cpan.org, as appropriate.
  • (Developers only: the script will look for twikiplugins/DBCacheContrib/DBCacheContrib.tgz before downloading from TWiki.org)
• If you don't have a permanent connection, you can still use the automatic installer, by downloading all required TWiki archives to a local directory.
  • Point the environment variable $TWIKI_PACKAGES to this directory, and the installer script will look there first for required TWiki packages.
    • $TWIKI_PACKAGES is actually a path; you can list several directories separated by :
  • If you are behind a firewall that blocks access to CPAN, you can build a local CPAN mini-mirror, as described at http://twiki.org/cgi-bin/view/Codev/BuildingDakar#CPAN_local_minimirror
• The installer script will:
  • Automatically resolve dependencies,
  • Copy files into the right places in your local install even if you have renamed data directories,
  • check in new versions of any installed files that have existing RCS histories files in your existing install.
• If you don't want to use the script, or have problems on your platform (e.g. you don't have Perl 5.8), then you can still install manually:
  1. Download and unpack one of the .zip or .tgz archives to a temporary directory.
  2. Manually copy the contents across to the relevant places in your TWiki installation.
  3. Check in any installed files that have existing ,v files in your existing install (take care not to lock the files when you check in)
  4. Manually edit LocalSite.cfg to set any configuration variables.
  5. Run configure and enable the module, if it is a plugin.
  6. Repeat from step 1 for any missing dependencies.

Contrib Info