Package API Documentation

Package configmix

A library for helping with configuration files.

Author:

Franz Glasner

Copyright:

(c) 2015–2023, Franz Glasner. All rights reserved.

License:

BSD 3-Clause “New” or “Revised” License. See LICENSE.txt for details.

ID:

@(#) $Header$

configmix.load(*files, **kwargs)

Load the given configuration files, merge them in the given order and return the resulting configuration dictionary.

Parameters:

• files – the filenames of the configuration files to read and merge; if a filename starts with <dir> then the name is interpreted as directory and all files are loaded in sorted order (non-resursively, ignoring unknown filetypes)

• defaults (dict-alike or None) – optional configuration dictionary with some default settings where the settings from files are merged into

• extras (dict-alike or None) –

  optional configuration dictionary that will applied last

  Use this for example to overwrite configuration file settings from commandline arguments.

• strict (bool) – enable strict parsing mode for parsers that support it (e.g. to prevent duplicate keys)

• merge_lists (str or None) – When None then lists will be overwritten by the merge process. When extend then lists will be extended instead. This parameter is passed to merge().

Returns:

the configuration

Return type:

Configuration

configmix.safe_load(*files, **kwargs)

Analogous to load() but do merging with safe_merge() instead of merge()

configmix.try_determine_filemode(filename)

Try to determine an explicitely given filemode from an Emacs-compatible mode declaration (e.g. mode=python).

Parameters:

filename (str) –

Returns:

the found mode string or None

Return type:

str or None

Only the first two lines are searched for.

Conveniently to be used in calls to set_assoc() to determine the file-mode by content instead of filename extension.

configmix.DEFAULT_MODE_LOADERS = {'-*- ignore -*-': <function _load_ignore>, '-*-ignore-*-': <function _load_ignore>, 'conf': <function _load_ini>, 'conf-toml': <function _load_toml>, 'conf-windows': <function _load_ini>, 'ini': <function _load_ini>, 'javascript': <function _load_json>, 'json': <function _load_json>, 'python': <function _load_py>, 'toml': <function _load_toml>, 'yaml': <function _load_yaml>}

Default associations between file modes and loader functions

configmix.DEFAULT_ASSOC = [('*.yml', 'yaml'), ('*.yaml', 'yaml'), ('*.json', 'json'), ('*.py', 'python'), ('*.ini', 'conf'), ('*.toml', 'toml')]

The builtin default associations of filename extensions with file modes – in that order.

The “mode” part may be a string or a callable with a filename parameter that returns the mode string for the file or None if it can not determined.

configmix.USE_DEFAULT_ASSOC = <object object>

Marker for the default association for an extension.

To be used in set_assoc().

configmix.get_default_assoc(pattern)

Return the default file-mode association for the fnmatch pattern pattern.

Raises:

KeyError if the pattern is not found.

configmix.mode_loaders = {'-*- ignore -*-': <function _load_ignore>, '-*-ignore-*-': <function _load_ignore>, 'conf': <function _load_ini>, 'conf-toml': <function _load_toml>, 'conf-windows': <function _load_ini>, 'ini': <function _load_ini>, 'javascript': <function _load_json>, 'json': <function _load_json>, 'python': <function _load_py>, 'toml': <function _load_toml>, 'yaml': <function _load_yaml>}

All configured associations between file modes and loader functions.

See DEFAULT_MODE_LOADERS.

configmix.clear_assoc()

Remove all configured loader associations.

The DEFAULT_ASSOC are not changed.

configmix.get_assoc(pattern)

Return the default loader for the fnmatch pattern pattern.

Raises:

KeyError if the pattern is not found.

configmix.set_assoc(fnpattern, mode, append=False)

Associate a fnmatch style pattern fnpattern with a file-mode mode that determines what will be called when load() encounters a file argument that matches fnpattern.

Parameters:

• fnpattern (str) – the fnmatch pattern to associate a loader with

• mode (str or callable) – a mode string or a callable that accepts a filename argument and returns a file-mode for the given file (or None)

• append (bool) – If False (which is the default) then this function inserts the given pattern at the head position of the currently defined associations, if True the pattern will be appended

The OS specific case-sensitivity behaviour of fnmatch.fnmatch() applies (i.e. os.path.normpath() will be called for both arguments).

If loader is USE_DEFAULT_ASSOC then the default association from DEFAULT_ASSOC will be used – if any.

configmix.del_assoc(fnpattern)

Remove all associations for fnpattern.

Parameters:

fnpattern (str) – the fnmatch pattern to associate a loader with

configmix.merge(user, default, filter_comments=True, merge_lists=None)

Logically merge the configuration in user into default.

Parameters:

• user (Configuration) – the new configuration that will be logically merged into default

• default (Configuration) – the base configuration where user is logically merged into

• filter_comments (bool) – flag whether to filter comment keys that start with any of the items in COMMENTS

• merge_lists (str or None) – When None then lists will be overwritten by the merge process. When extend then lists will be extended instead.

Returns:

user with the necessary amendments from default. If user is None then default is returned.

Note

The configuration in user is augmented/changed inplace.

The configuration in default will be changed inplace when filtering out comments (which is the default).

If a value in user is equal to constants.DEL_VALUE ({{::DEL::}}) the corresponding key will be deleted from the merged output.

From http://stackoverflow.com/questions/823196/yaml-merge-in-python

configmix.safe_merge(user, default, filter_comments=True, merge_lists=None)

A more safe version of merge() that makes deep copies of the returned container objects.

Contrary to merge() no given argument is ever changed inplace. Every object from default is decoupled from the result – so changing the default configuration later does not propagate into a merged configuration later.

Module configmix.compat

Some minimal compatibility shim between Python2 and Python3

configmix.compat.text_to_native_os_str(s, encoding=None)

configmix.compat.native_os_str_to_text(s, encoding=None)

configmix.compat.u(s, encoding='utf-8')

configmix.compat.u2fs(s, force=False)

Convert a text (Unicode) string to the filesystem encoding.

Note

The filesystem encoding on Python 3 is a Unicode text string. The function is a noop when called on Python 3.

Note

If s is already a byte string be permissive and return s unchanged.

configmix.compat.uchr(n)

configmix.compat.n(s, encoding='utf-8')

configmix.compat.str_and_u(v)

Convert the value in v of any type to a native string and then to text (Unicode)

Module configmix.config

The unified configuration dictionary with attribute support or variable substitution.

class configmix.config.Configuration(*args, **kwds)

The configuration dictionary with attribute support and variable interpolation/substitution.

Note

When retrieving by attribute names variables will be interpolated.

getvar()

For documentation and the signature see py_getvar().

This method is an alias of py_getvar() or fast_getvar() – depending on the availability of the configmix._speedups module.

py_getvar(varname, default=<object object>)

Get a variable of the form [ns:][[key1.]key2.]name - including variables from other namespaces.

No variable interpolation is done and no filters are applied.

Special characters (e.g. : and .) must be quoted when using the default namespace.

See also quote().

Pure-Python implementation.

fast_getvar()

Implemented in C in configmix._speedups.

getvar_s()

For documentation and the signature see py_getvar_s().

This method is an alias of py_getvar_s() or fast_getvar_s() – depending on the availability of the configmix._speedups module.

py_getvar_s(varname, default=<object object>)

Get a variable - including variables from other namespaces.

varname is interpreted as in getvar(). But variables will be interpolated recursively within the variable values and filters are applied.

For more details see chapter Variable Interpolation.

Pure-Python implementation.

fast_getvar_s()

Implemented in C in configmix._speedups.

getvarl()

For documentation and the signature see py_getvarl().

This method is an alias of py_getvarl() or fast_getvarl() – depending on the availability of the configmix._speedups module.

py_getvarl(*path, **kwds)

Get a variable where the hierarchy is given in path as sequence and the namespace is given in the namespace keyword argument.

No variable interpolation is done and no filters are applied.

Quoting of path and namespace is not needed and wrong.

Pure-Python implementation.

fast_getvarl()

Implemented in C in configmix._speedups.

getvarl_s()

For documentation and the signature see py_getvarl_s().

This method is an alias of py_getvarl_s() or fast_getvarl_s() – depending on the availability of the configmix._speedups module.

py_getvarl_s(*path, **kwds)

Get a variable - including variables from other namespaces.

path and namespace are interpreted as in getvarl(). But variables will be interpolated recursively within the variable values and filters are applied.

For more details see chapter Variable Interpolation.

Pure-Python implementation.

fast_getvarl_s()

Implemented in C in configmix._speedups.

interpolate_variables()

For documentation and the signature see py_interpolate_variables().

This method is an alias of py_interpolate_variables() or fast_interpolate_variables() – depending on the availability of the configmix._speedups module.

py_interpolate_variables(s)

Expand all variables in the single string s

Pure-Python implementation.

fast_interpolate_variables()

Implemented in C in configmix._speedups.

is_jail = False

Flag to show that this is not a jail for another configuration

clear_cache()

Clear the internal lookup cache and the interpolation cache

disable_cache()

Disable the internal lookup cache and the interpolation cache

enable_cache()

Enable the internal lookup cache and the interpolation cache.

The caches are empty after enabling.

__getitem__(key)

Mapping and list interface that forwards to getvarl_s()

get(key, default=None)

Mapping interface that forwards to getvarl_s()

__contains__(key)

Containment test

getitem_ns(key)

Just forward to the original dict.__getitem__().

No variable interpolation and key path access.

items()

Items without interpolation

values()

Values without interpolation

getkeysl(*path, **kwds)

Yield the keys of a variable value.

Return type:

A generator

Raises:

KeyError –

Note

Dictionary keys are not subject to interpolation.

getfirstvarl(*paths, **kwds)

A variant of getvarl() that returns the first found variable in the paths list.

Every item in paths is either a tuple or list or a dict. If the path item is a dict then it must have two keys “namespace” and “path”. If the path item is a list or tuple then the namespace is assumed to be None.

Note that a caller that wants to use variables from a non-default namespace must use a sequence of dicts.

No variable interpolation is done and no filters are applied.

Quoting of anything in paths is not needed and wrong.

getkeys(varname)

Yield all the keys of a variable value.

Return type:

A generator

Raises:

KeyError –

Note

Dictionary keys are not subject to interpolation.

getfirstvar(*varnames, **kwds)

A variant of getvar() that returns the first found variable in the list of given variables in varnames.

getfirstvarl_s(*paths, **kwds)

A variant of getfirstvarl() that does variable interpolation.

paths and kwds are interpreted as in getfirstvarl(). But variables will be interpolated recursively within the variable values and filters are applied.

For more details see chapter Variable Interpolation.

getfirstvar_s(*varnames, **kwds)

A variant of getvar_s() that returns the first found variable in the list of given variables in varnames.

expand_if_reference(v)

Check whether v is a configuration reference and – if true – then expand it.

v must match the pattern {{ref:<REFERENCE>}}

All non-matching texttypes and all non-texttypes are returned unchanged.

Raises:

KeyError – If the reference cannot found

expand_ref_uri(uri)

Raises:

KeyError – If the reference URI is not found

try_get_reference_uri(v)

Check whether v is a configuration reference and – if true – return the configuration path where the reference points to.

If v is not a text type or not a reference return None.

Does not check whether the referenced configuration object exists.

Return type:

None or str

substitute_variables_in_obj(obj)

Recursively expand variables in the object tree obj.

jailed(rootpath=None, root=None, bind_root=True)

Return a “jailed” configuration of the current configuration.

Parameters:

• rootpath (list or tuple) – a sequence of strings (or objects) that shall emcompass the chroot-like jail of the returned configuration

• root (str) – a string path expression that shall encompass the chroot-like jail of the returned configuration

• bind_root (bool) – if you do a rebind() just after creation of a jailed config you can set bind_root to False; otherwise use the default

Returns:

a jailed (aka restricted) configuration

Return type:

_JailedConfiguration

Exactly one of rootpath or root must be given.

iter_jailed(rootpath=None, root=None)

Iterator that yields properly jailed configurations.

rootpath or root must refer to a list or dict container.

extract_new_config(*path, **kwds)

Get the value at path and make a new Configuration from it.

The new configuration is a deepcopy and completely independent of the source configuration.

copy_new_config_without(*path)

Copy the current configuration but leave out the value at key path.

The new configuration is a deepcopy and completely independent of the source configuration.

Note

Currently only a “simple” path with length 1 is supported. References are not supported.

class configmix.config.CoercingMethodsMixin

Mixin to provide some common implementations for retrieval methods that convert return values to a fixed type (int, bool, float).

Both Configuration and _JailedConfiguration use this mixin.

getintvarl_s(*path, **kwds)

Get a (possibly substituted) variable and coerce text to a number.

getfirstintvarl_s(*paths, **kwds)

Get a (possibly substituted) variable and coerce text to a number.

getintvar_s(varname, default=<object object>)

Get a (possibly substituted) variable and coerce text to a number.

getfirstintvar_s(*varnames, **kwds)

A variant of getintvar_s() that returns the first found variable in the list of given variables in varnames.

getboolvarl_s(*path, **kwds)

Get a (possibly substituted) variable and convert text to a boolean

getfirstboolvarl_s(*paths, **kwds)

Get a (possibly substituted) variable and convert text to a boolean

getboolvar_s(varname, default=<object object>)

Get a (possibly substituted) variable and convert text to a boolean

getfirstboolvar_s(*varnames, **kwds)

A variant of getboolvar_s() that returns the first found variable in the list of given variables in varnames.

getfloatvarl_s(*path, **kwds)

Get a (possibly substituted) variable and convert text to a float

getfirstfloatvarl_s(*path, **kwds)

Get a (possibly substituted) variable and convert text to a float

getfloatvar_s(varname, default=<object object>)

Get a (possibly substituted) variable and convert text to a float

getfirstfloatvar_s(varname, default=<object object>)

Get a (possibly substituted) variable and convert text to a float

While not instantiable directly, but only by Configuration.jailed(), the API use will want to know its interface:

class configmix.config._JailedConfiguration(*path)

A jailed and restricted variant of Configuration.

Restriction is two-fold:

• The access to configuration variables in config is restricted to the configuration sub-tree that is configured in path.

• Not all access-methods of Configuration are implemented yet.

See also

Jailed Configurations

Note

There is no namespace support.

Note

Do not call the constructor directly. Instantiate a jailed configuration from the parent configuration’s jailed() factory method.

is_jail = True

Flag to show that this is a jail for another configuration

property base

Ask for the base (aka parent) configuration”.

This configuration is always unjailed.

rebind(new_base)

Bind the jail to a new unjailed configuration new_base.

The new configuration base also must have an existing path to the root.

Parameters:

new_base (Configuration) – the new base

__getattr__(name)

Attribute-style access.

Result values are interpolated (i.e. forwarded to getvarl_s())

__getitem__(key)

Mapping and list interface that forwards to getvarl_s()

get(key, default=None)

__contains__(key)

Containment support for containers

getvarl(*path, **kwds)

getkeysl(*path, **kwds)

getfirstvarl(*paths, **kwds)

getvarl_s(*path, **kwds)

getfirstvarl_s(*paths, **kwds)

getvar(varname, **kwds)

getkeys(varname)

getfirstvar(*varnames, **kwds)

getvar_s(varname, **kwds)

getfirstvar_s(*varnames, **kwds)

__iter__()

Iteration support for containers

__len__()

Length support for containers

iter_jailed()

Iteration support for containers which yields properly jailed sub-jails.

Only supported for type list or type dict jails.

__bool__()

Map- and list-style evaluation in boolean context

jailed(rootpath=None, root=None, bind_root=True)

Return a “jailed” configuration that effectively is a subjail of the current jail

For a more complete description see Configuration.jailed().

Some public helper functions:

configmix.config.quote()

For documentation and the signature see py_quote().

This function is an alias of py_quote() or fast_quote() – depending on the availability of the configmix._speedups module.

configmix.config.fast_quote()

Implemented in C in configmix._speedups.

configmix.config.py_quote(s)

Replace important special characters in string s by replacing them with %xNN where NN are the two hexadecimal digits of the character’s unicode codepoint value.

Handled are the important special chars: %, ., :, #; ', ", |, {, }, [ and ].

See also the Quoting section.

Pure-Python implementation.

configmix.config.unquote()

For documentation and the signature see py_unquote().

This function is an alias of py_unquote() or fast_unquote() – depending on the availability of the configmix._speedups module.

configmix.config.fast_unquote()

Implemented in C in configmix._speedups.

configmix.config.py_unquote(s)

Unquote the content of s: handle all patterns %xNN, %uNNNN or %UNNNNNNNN.

This is the inverse of quote().

Pure-Python implementation.

configmix.config.pathstr2path()

For documentation and the signature see py_pathstr2path().

This function is an alias of py_pathstr2path() or fast_pathstr2path() – depending on the availability of the configmix._speedups module.

configmix.config.fast_pathstr2path()

Implemented in C in configmix._speedups.

configmix.config.py_pathstr2path(varname)

Parse a dot-separated path string varname into a tuple of unquoted path items

Parameters:

varname (str) – The quoted and dot-separated path string

Returns:

The unquoted and parsed path items

Return type:

tuple

Used e.g. by getvar(), getvar_s() and jailed().

The returned value is suitable as input for getvarl(), getvarl_s() and friends.

An empty varname returns an empty tuple.

Pure-Python implementation.

Module configmix.constants

Some important public contants

configmix.constants.COMMENTS = ['__comment', '__doc']

Prefixes for comment configuration keys that are to be handled as comments

configmix.constants.DIR_PREFIX = '<dir>'

Prefix for configuration values to read other configuration files from given directory

configmix.constants.DEL_VALUE = '{{::DEL::}}'

Value for configuration items to signal that the corresponding key-value is to be deleted when configurations are merged.

Despite having “interpolation” syntax this value will never be substituted.

configmix.constants.REF_NAMESPACE = 'ref'

Special internal namespace used for implementation of tree references

configmix.constants.NONE_FILTER = 'None'

The public name of the special None filter

configmix.constants.EMPTY_FILTER = 'Empty'

The public name of the special None filter

Module configmix.ini

Read INI-style configuration files.

class configmix.ini.INIConfigParser(filename, executable=None, encoding=None)

A case sensitive config parser that returns all-unicode string values.

read(filenames)

Not implemented. Use read_file() instead.

readfp(fp, filename)

Compatibility for older Python versions.

Use read_file() instead.

read_file(fp, filename)

Read from a file-like object fp.

The fp argument must be iterable (Python 3.2+) or have a readline() method (Python 2, <3.2).

getx(section, option)

Extended get() with some automatic type conversion support.

Default: Fetch as string (like get()).

If annotated with :bool: fetch as bool, if annotated with :int: fetch as int, if annotated with :float: fetch as float.

itemsx(section, options)

Get all the options given in options of section section. Fetch them with getx() in the order given.

Return a list of (name, value) pairs for each option in options in the given section.

items_as_dictx(section, options)

Similar to itemsx() but return a (possibly ordered) dict instead of a list of key-value pairs.

configmix.ini.load(filename, extract=['config'], encoding='utf-8')

Load a single INI file and read/interpolate the sections given in extract.

Flattens the given sections into the resulting dictionary.

Then build a tree out of sections which start with any of the extract content value and a dot ..

The encoding of the file is given in encoding.

Module configmix.json

Read JSON-style configuration files.

configmix.json.load(filename, encoding='utf-8')

Load a single JSON file with name filename and encoding encoding.

Module configmix.py

Read configuration settings from Python files.

configmix.py.load(filename, extract=None)

Load Python-style configuration files.

Files are loaded and executed with exec() using an empty dict as global and local context.#

Parameters:

• filename – the path to the configuration file

• extract – an optional list of variable names (keys) to extract into the resulting configuration dictionary

Returns:

the configuration as dictionary

Return type:

collections.OrderedDict or ordereddict.OrderedDict or dict

Module configmix.toml

Read TOML style configuration files.

configmix.toml.load(filename, encoding='utf-8')

Load a single TOML file with name filename and encoding encoding.

Note

The TOML standard requires that all TOML files are UTF-8 encoded.

Module configmix.variables

Variable interpolation: implementation of namespaces and filters

configmix.variables.add_varns(name, fn)

Register a new variable namespace name and it’s implementing function fn

..note:: This function checks that name is not the special

namespace REF_NAMESPACE.

configmix.variables.lookup_varns(name)

Lookup the variable namespace name and return it’s implementing function

Parameters:

name (str) – the namespace name

Returns:

the implementing function

Raises:

KeyError – if the namespace name doesn’t exist

..note:: This function checks that name is not the special

namespace REF_NAMESPACE.

configmix.variables.add_filter(name, fn)

Register a variable filter function with name name and implementation fn.

name may not contain , and | characters.

Raises:

ValueError – If an invalid name is given

configmix.variables.lookup_filter(name)

Lookup a variable filter with name name and return it’s implementation function

Parameters:

name (str) – the logical filter name

Returns:

the implementing filter function

Raises:

KeyError – if the filter cannot be found

configmix.variables.filter(name)

Decorator for a filter function.

Example usage:

@filter("myfilter")
def myfilter_impl(appconfig, variable_value):
    filtered_value = ...
    return filtered_value

configmix.variables.urlquote(config, v)

Filter function to replace all special characters in string v using the %xx escape

configmix.variables.urlquote_plus(config, v)

Filter function to replace all special characters (including spaces) in string v using the %xx escape

configmix.variables.saslprep(config, v)

Filter function to perform a SASLprep according to RFC 4013 on v.

This is a Stringprep Profile for usernames and passwords

configmix.variables.normpath_impl(config, v)

Implementation of the normpath filter function

configmix.variables.abspath_impl(config, v)

Implementation of the abspath filter function

configmix.variables.posixpath_impl(config, v)

Implementation of the posixpath filter function

configmix.variables.lower_impl(config, v)

Implementation of the lower filter function

configmix.variables.upper_impl(config, v)

Implementation of the upper filter function

configmix.variables.None_filter_impl(config, v)

Identity.

The None filter is just a marker to not throw KeyError but return None. It is a no-op within the filter-chain itself.

configmix.variables.Empty_filter_impl(config, v)

Identity.

The Empty filter is just a marker to not throw KeyError but return the empty string. It is a no-op within the filter-chain itself.

Module configmix.yaml

Simple wrapper for yaml to support all-unicode strings when loading configuration files.

class configmix.yaml.ConfigLoader(*args, **kwds)

A YAML loader, which makes all !!str strings to Unicode. Standard PyYAML does this only in the non-ASCII case.

If an OrderedDict implementation is available then all “map” and “omap” nodes are constructed as OrderedDict. This is against YAML specs but within configuration files it seems more natural.

Initialize the scanner.

class configmix.yaml.ConfigSafeLoader(*args, **kwds)

A safe YAML loader, which makes all !!str strings to Unicode. Standard PyYAML does this only in the non-ASCII case.

If an OrderedDict implementation is available then all “map” and “omap” nodes are constructed as OrderedDict. This is against YAML specs but within configuration files it seems more natural.

Initialize the scanner.

configmix.yaml.load(stream, Loader=None, strict=False)

Parse the given stream and return a Python object constructed from for the first document in the stream.

If strict is True then duplicate mapping keys within a YAML document are detected and prevented. If a Loader is given then strict does not apply.

configmix.yaml.load_all(stream, Loader=None, strict=False)

Parse the given stream and return a sequence of Python objects corresponding to the documents in the stream.

If strict is True then duplicate mapping keys within a YAML document are detected and prevented. If a Loader is given then strict does not apply.

configmix.yaml.safe_load(stream, strict=False)

Parse the given stream and return a Python object constructed from for the first document in the stream.

Recognizes only standard YAML tags and cannot construct an arbitrary Python object.

If strict is True then duplicate mapping keys within a YAML document are detected and prevented.

configmix.yaml.safe_load_all(stream, strict=False)

Return the list of all decoded YAML documents in the file stream.

Recognizes only standard YAML tags and cannot construct an arbitrary Python object.

If strict is True then duplicate mapping keys within a YAML document are detected and prevented.

Module configmix._speedups

This module is implemented in C using Python’s C-API. It contains alternate implementations for some heavily used functions and/or methods. The module functions are not supposed to be called directly. Their signatures may or may not match their pure-Python equivalents because they may be called by appropriate tiny Python wrappers.

This module is optional.

The module is only available for CPython >= 3.7 and uses its stable API.