tiddlyweb Package

tiddlyweb Package

For more complete information also see:

TiddlyWeb is a web service and library for managing and manipulating resources useful in the creation of dynamic wiki-like collections of content and functionality. The model of the data was originally designed for creating custom TiddlyWiki implementations, where the content of the TiddlyWiki can be saved to the server, and shared among multiple users.

TiddlyWeb presents an HTTP API for resource management. The API follows, as possible, RESTful principles to keep the API flexible and scalable. The URLs for this interface are kept in a file called urls.map found in the tiddlyweb package. urls.map dispatches web requests at specific URLs to specific functions in modules in the tiddlyweb.web.handler package. urls.map may be located in another place by changing the urls_map key in tiddlywebconfig.py. There are also mechanisms for overriding storage (see tiddlyweb.store), serialization (see tiddlyweb.serializer) and authentication (see tiddlyweb.web.challenger and tiddlyweb.web.extractor) systems. There are also system_plugins and twanager_plugins for further extensibility.

The primary resources presented by the server are Recipes, Bags and Tiddlers. See the tiddlyweb.model package.

TiddlyWeb includes twanager, a command line tool for doing a variety of TiddlyWeb activities. Run twanager without arguments for a list of commands.

See the documentation for other modules and packages within tiddlyweb for additional details.

config Module

The configuration of a particular instance of TiddlyWeb, carried around as a dict in the WSGI environ as tiddlyweb.config.

If there is a tiddlywebconfig.py file in the working directory where twanager or the web server is started, its values will override the defaults established in this module.

The server administrator may add additional keys to the config via extensions.

Config Keys

system_plugins
A list of Python module names that act as plugins for the running server. At server startup time they are found, compiled, and the function init(config) is called on them, where config is a reference to the current config. Use this to add functionality to the server that cannot be accomplished from the defaults, such as adding additional web handlers, storage hooks or overriding existing behaviors.
twanager_plugins
A list of Python module names that act as plugins for twanager, adding command line functionality. As with system_plugins init(config) is called.
server_store
A list containing a module name and a configuration dictionary. The named module is an implementation of tiddlyweb.stores.StorageInterface (first looked up in the tiddlyweb.stores package space, then in sys.path). The configuration is an arbitrary dictionary of information to be passed to the store (e.g. database username and password).
server_request_filters
A list of WSGI applications which, in order, process the incoming requests made to the server. This can extract, add, or filter information as necessary. The defaults provide query string processing, content negotiation and establish environ settings.
server_response_filters
A list of WSGI applications which, in order, process the outgoing response from the server. This can transform, log, or handle exceptions as necessary.
server_host
The hostname of this server, usually set from whatever starts the server. This is a dictionary with keys: scheme, host, port.
server_prefix
A URL path portion which is a prefix to every URL the system uses and produces. Use this to host TiddlyWeb in a subdirectory (e.g. /wiki). Default is ''.
extension_types
A dictionary that pairs extension strings used in URLs as human controlled content-negotiation with the MIME types they represent. Add to this if you add to serializers.
serializers
Incoming request Accept headers, or extension MIME types paired with a tiddlyweb.serializations.Serializer implementation and an outgoing MIME type for that type of serialization.
extractors
An extractor is a credential extractor (see tiddlyweb.web.extractors.ExtractorInterface) that looks in an incoming request to attempt to extract information from it that indicates a potential user in the system. This config item is an ordered list of extractors, tried in succession until one returns tiddlyweb.usersign information or there are no more left.
auth_systems
A list of challengers available to the system when it needs to ask for a user. (See tiddlyweb.web.challengers.ChallengerInterface) If there is more than one challenger the user is presented with a list of those available. If there is only one, the user is automatically directed to just that one. A challenger needs to work with the extractors system so that the challenger provides something for future requests that the extractor can extract.
secret
A string used to encrypt the cookie installed by some of the challengers and used by the cookie extractor. NOTE: EVERY INSTALLATION SHOULD CHANGE THIS IN ITS OWN CONFIGURATION.
urls_map
The file location of the text file that maps URL paths to Python code, doing method dispatch. Usually it is better to use plugins to change the available URLs and handlers.
bag_create_policy
A policy statement on who or what kind of user can create new bags on the system through the web API. ANY means any authenticated user can. ADMIN means any user with role ADMIN can. '' means anyone can.
recipe_create_policy
A policy statement on who or what kind of user can create new recipes on the system through the web API. See bag_create_policy.
log_file
Path and filename of the TiddlyWeb log file.
log_level
String of loglevel to log. Pick one of CRITICAL, DEBUG, ERROR, INFO, WARNING.
css_uri
A URI of a css file that can be used to style the HTML output of the server. See tiddlyweb.serializations.html for the classes and ids used.
wikitext.default_renderer
The default module for rendering tiddler.text to HTML when tiddler.type is None.
wikitext.type_render_map
A dictionary mapping tiddler.type MIME types to modules with a render() function for turning that type into HTML.
wsgi_server
The name of a module that provides a start_server method which starts a server to run this TiddlyWeb instance. Used by the twanager server command only.
special_bag_detectors
A list of functions that take an environ and bag name and return a tuple of two functions: the first returns the tiddlers in that bag, the second returns one tiddler from that bag.
collections.use_memory
If True Tiddler Collections are kept in memory during a single request. Defaults to False to save memory.

control Module

control provides routines which integrate the basic model classes with the rest of the system. The model classes are intentionally simple. The methods here act as controllers on those classes.

These are primarily related to handling recipes.

tiddlyweb.control.determine_bag_for_tiddler(recipe, tiddler, environ=None)

Return the bag which this tiddler would be in if we were to save it to the named recipe rather than to a bag.

This is done reversing the recipe list and seeing if the tiddler passes the constraint of the bag and its associated filter. If bag+filter is true, return that bag.

tiddlyweb.control.determine_bag_from_recipe(recipe, tiddler, environ=None)

Given a recipe and a tiddler determine the bag in which this tiddler can be found. This is different from determine_bag_for_tiddler(). That one finds the bag the tiddler could be in. This is the bag the tiddler is in.

This is done by reversing the recipe’s list, and filtering each bag according to any filters present. The resulting tiddlers are checked.

If an indexer is configured use the index to determine if a tiddler exists in a bag.

tiddlyweb.control.filter_tiddlers(tiddlers, filters, environ=None)

Return a generator of tiddlers resulting from filtering the provided iterator of tiddlers by the provided filters.

If filters is a string, it will be parsed for filters.

tiddlyweb.control.get_tiddlers_from_bag(bag)

Yield the individual tiddlers that are in a bag.

The tiddlers return are empty objects that have not been loaded from the store.

Rarely used, see tiddlyweb.store.Store.list_bag_tiddlers().

tiddlyweb.control.get_tiddlers_from_recipe(recipe, environ=None)

Return the list of tiddlers that result from processing the recipe.

This list of tiddlers is unique by title with tiddlers later in the recipe taking precedence over those earlier in the recipe.

The tiddlers returned are empty objects (i.e. not loaded from the store).

tiddlyweb.control.readable_tiddlers_by_bag(store, tiddlers, usersign)

Yield those tiddlers which are readable by the current usersign. This means, depending on the read constraint on the tiddler's bag's policy, yield or not.

tiddlyweb.control.recipe_template(environ)

Provide a means to specify custom {{ key }} values in recipes which are then replaced with the value specified in environ['tiddlyweb.recipe_template'].

This allows recipes to be dynamic in the face of conditions in the current request.

manage Module

manage provides the workings for the twanager command line tool. twanager calls handle(), making available all commands that have been put into the COMMANDS dictionary by the make_command() decorator. See tiddlyweb.commands for examples.

Plugins which add commands must be added to the twanager_plugins config setting so they are imported at the proper time.

tiddlyweb.manage.handle(args)

Dispatch to the proper function for the command given in args[1].

tiddlyweb.manage.make_command()

A decorator that marks the decorated method as a member of the commands dictionary, with associated help.

The pydoc of the method is used in automatically generated :py:func:usage information.

tiddlyweb.manage.usage(args)

List this help

serializer Module

Serialize TiddlyWeb entities for the sake of taking input and sending output.

This module provides the facade for accessing the possibly many modules which act as serializations. It is asked by calling code to provide a serialization for a given MIME type. Plugins may override what MIME types are handled and by what modules. See tiddlyweb.config for related configuration settings.

exception tiddlyweb.serializer.BagFormatError

Bases: exceptions.Exception

The provided input is insufficient to form a valid Bag.

exception tiddlyweb.serializer.NoSerializationError

Bases: exceptions.Exception

There is a NoSerialization of this type for the entity.

exception tiddlyweb.serializer.RecipeFormatError

Bases: exceptions.Exception

The provided input is insufficient to form a valid Recipe.

class tiddlyweb.serializer.Serializer(engine, environ=None)

Bases: object

A Serializer is a facade to a Serialization which implements the tiddlyweb.serializations.SerializationInterface to turn a TiddlyWeb entity into a particular representation or vice versa.

A Serializer can also list collections of entities in a particular representation.

A single Serializer is a reusable tool which can serialize more than one object. You must set serializer.object after initialization and then again for each subsequent object being serialized.

The following example turns the tiddler into JSON and vice-versa:

tiddler = Tiddler('cow', 'bag')
tiddler.text = 'moo'
serializer = Serializer('json', environ)
serializer.object = tiddler
json_string = serializer.to_string()
assert '"text": "moo"' in json_string
new_string = json_string.replace('moo', 'meow')
serializer.from_string(new_string)
assert tiddler.text == 'meow'

Note that to_string() and from_string() operate on the Serializer which dispatches to a method in the SerializationInterface implementation based on the class of the object being serialized.

from_string(input_string)

Turn the provided input_string into a TiddlyWeb entity object of the type of self.object. That is: populate self.object based on input_string.

list_bags(bags)

Provide a (usually unicode) string representation of the provided bags in the current serialization.

list_recipes(recipes)

Provide a (usually unicode) string representation of the provided recipes in the current serialization.

list_tiddlers(tiddlers)

Provide a (usually unicode) string representation of the tiddlers in the provided Tiddlers collection.

to_string()

Provide a (usually unicode) string representation of the bag, recipe or tiddler at self.object.

exception tiddlyweb.serializer.TiddlerFormatError

Bases: exceptions.Exception

The provided input is insufficient to form a valid Tiddler.

specialbag Module

Special bags are a feature implemented in plugins that allow non-standard collections of data to be represented as a bag of tiddlers. An example is remotebag.

If config['special_bag_detectors'] is set, it is a list of functions that take two arguments: a WSGI environ and a string and return either:

  • two functions
  • None

The first function yields tiddlers, like tiddlyweb.store.list_bag_tiddlers(). It’s arguments are a WSGI environ and a string.

The second function returns a single tiddler. It’s arguments are a WSGI environ and a tiddler object (with at least title and bag set).

exception tiddlyweb.specialbag.SpecialBagError

Bases: exceptions.Exception

A generic exception to be raised by special bag implementations.

tiddlyweb.specialbag.get_bag_retriever(environ, bag)

When loading bag or tiddlers within it from the store, this method is used to inspect config['special_bag_detectors'] to determine if there is a special handler. If there is, the handler is returned and used for retrieval, otherwise None is returned and the store is used as normal.

store Module

Store TiddlyWeb entities to a configured persistence layer.

This module provides the facade for accessing one of many possible modules which provide storage for entities. It provides a general interface to get, put, delete or list entities.

Each of the single entity methods can be augmented with hooks provided by plugins. This allows actions to be performed based on data in the store being retrieved or updated, such as updating an index.

exception tiddlyweb.store.NoBagError

Bases: tiddlyweb.store.StoreError

No tiddlyweb.model.bag.Bag was found.

exception tiddlyweb.store.NoRecipeError

Bases: tiddlyweb.store.StoreError

No tiddlyweb.model.recipe.Recipe was found.

exception tiddlyweb.store.NoTiddlerError

Bases: tiddlyweb.store.StoreError

No tiddlyweb.model.tiddler.Tiddler was found.

exception tiddlyweb.store.NoUserError

Bases: tiddlyweb.store.StoreError

No tiddlyweb.model.user.User was found.

class tiddlyweb.store.Store(engine, config=None, environ=None)

Bases: object

A Store is a facade to an implementation of tiddlyweb.stores.StorageInterface to handle the storage and retrieval of all entities in the TiddlyWeb system.

Because of the facade system it is relatively straightforward to create diverse storage systems for all sorts of or multiple media. In addition stores can be layered to provide robust caching and reliability.

The Store distinguishes between single entities and collections. With single entities, an entity is passed to the store and the store is asked to get(), put() or delete() it. When get() is used the provided object is updated in place in operation that could be described as population. Dispatch is based on the class of the provided entity.

After any of those operations optional HOOKS are called.

With collections there are specific list methods:

Finally a store may optionally provide a search(). How search works and what it even means is up to the implementation.

delete(thing)

Delete a thing: recipe, bag, tiddler or user.

get(thing)

Get a thing: recipe, bag, tiddler or user.

list_bag_tiddlers(bag)

List all the tiddlers in the bag.

list_bags()

List all the available bags in the system.

list_recipes()

List all the available recipes in the system.

list_tiddler_revisions(tiddler)

List the revision ids of the revisions of the indicated tiddler in reverse chronological older (newest first).

list_users()

List all the available users in the system.

put(thing)

Put a thing, recipe, bag, tiddler or user.

search(search_query)

Search in the store, using a search algorithm specific to the tiddlyweb.stores.StorageInterface implementation.

exception tiddlyweb.store.StoreEncodingError

Bases: tiddlyweb.store.StoreError

Something about an entity made it impossible to be encoded to the form required by the store.

exception tiddlyweb.store.StoreError

Bases: exceptions.IOError

Base Exception for Store Exceptions.

exception tiddlyweb.store.StoreLockError

Bases: tiddlyweb.store.StoreError

This process was unable to get a lock on the store.

exception tiddlyweb.store.StoreMethodNotImplemented

Bases: tiddlyweb.store.StoreError

A tiddlyweb.stores.StorageInterface does not implement this method.

tiddlyweb.store.get_entity(entity, store)

Load the provided entity from the store if it has not already been loaded. If it can’t be found, still return the same entity, just keep it empty.

This works for tiddlers, bags and recipes. Not users!

util Module

This module provides a centralized collection of miscellaneous utility functions used throughout TiddlyWeb and plugins.

Web specific utilities are in tiddlyweb.web.util.

exception tiddlyweb.util.LockError

Bases: exceptions.IOError

This process was unable to get a lock.

tiddlyweb.util.binary_tiddler(tiddler)

Test if a tiddler represents binary content (e.g. an image).

Return True if this Tiddler has a type which suggests the content of the tiddler is non-textual.

tiddlyweb.util.initialize_logging(config, server=False)

Initialize the system’s logging.

If this code is reached from twanager when there is no sub_command logging is not started. This avoids spurious tiddlyweb.log files popping up all over the place.

tiddlyweb.util.merge_config(global_config, additional_config, reconfig=True)

Update the global_config with the additional data provided in the dict additional_config. If reconfig is True, then reread and merge tiddlywebconfig.py so its overrides continue to operate.

Note that if the value of a existing key is a dict, then it is updated (merged) with the value from additional_config. Otherwise the value is replaced.

Warning: Please ensure (via tests) when using this that it will give the desired results.

tiddlyweb.util.pseudo_binary(content_type)

Test if a tiddler represents textual content that should be treated as a pseudo-binary.

A pseudo binary is defined as textual content for which (this) TiddlyWeb (instance) has no serialization or is not treated as wikitext. It is identified by a MIME type that looks like text, json, xml or javascript.

TiddlyWeb requires that such content be uploaded encoded as UTF-8.

tiddlyweb.util.read_config(global_config)

Read in a local configuration override, named tiddlywebconfig.py, from the current working directory. If the file exists but can’t be imported as valid Python an exception will be thrown, preventing unexpected results.

What is expected in the override file is a dict with the name config.

global_config is a reference to the currently operational main TiddlyWeb config. The read configuration data is merged into it.

tiddlyweb.util.read_utf8_file(filename)

Read the UTF-8 encoded file at filename and return unicode.

Allow any exceptions to raise.

tiddlyweb.util.renderable(tiddler, environ=None)

Return True if the provided tiddler's type is one that can be rendered to HTML by the wikitext rendering subsystem.

tiddlyweb.util.sha(data='')

Create a sha1 digest of the data.

tiddlyweb.util.std_error_message(message)

Display message on the stderr console.

tiddlyweb.util.superclass_name(instance)

Given an instance return the lowerclass name of the penultimate class in the hierarchy (the last is object). This is used to do dynamic method lookups in adaptor classes via serializer.py and store.py while still allowing model entities to be subclassed. Those subclasses must insure that their __mro__ results in Bag, User, Recipe or Tiddler in the penultimate slot.

tiddlyweb.util.write_lock(filename)

Create an advisory lock file based on filename.

This is primarily used by the text store.

tiddlyweb.util.write_unlock(filename)

Unlock the write lock associated with filename.

tiddlyweb.util.write_utf8_file(filename, content)

Write the unicode string in content to a UTF-8 encoded file named filename.

Allow any exceptions to raise.