Python/apps/py-cutils: cutils/util/__init_

annotate cutils/util/init.py @ 177:089c40240061

Add an alternate implementation for generating directory tree digests: - Do not use something like os.walk() but use os.scandir() directly. - Recursively generate the subdirectory digests only when needed and in the right order. This fixes that the order of subdirectories in the output did not match the application order of its directory digests. The new implementation also should make filtering (that will be implemented later) easier. NOTE: The tree digests of the old and the new implementation are identical.

author	Franz Glasner <fzglas.hg@dom66.de>
date	Sat, 11 Jan 2025 17:41:28 +0100
parents	e081b6ee5570
children	6154b8e4ba94

rev	line source
115 e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	1 # -- coding: utf-8 --
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	2 # :-
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	3 # :Copyright: (c) 2020-2025 Franz Glasner
115 e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	4 # :License: BSD-3-Clause
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	5 # :-
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	6 r"""Utility package.
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	10 __all__ = ["PY2",
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	11 "PY35",
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	12 "normalize_filename",
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	13 "argv2algo",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	14 "algotag2algotype",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	15 "get_blake2b",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	16 "get_blake2b_256",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	17 "get_blake2s",
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	18 "default_algotag",
163 fa7dd54e9715 FIX: Add "fsencode" to public functions in cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 124 diff changeset	19 "fsencode",
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	20 ]
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	21
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	22
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	23 import argparse
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	24 import hashlib
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	25 import os
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	26 import sys
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	27
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	28
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 163 diff changeset	29 PY2 = sys.version_info[0] < 3
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	30 PY35 = sys.version_info[:2] >= (3, 5)
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	31
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	32
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	33 def default_algotag():
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	34 """Determine the "best" default algorithm.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	35
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	36 Depend on availability in :mod:`hashlib`.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	37
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	38 Prefer BLAKE2b-256, SHA256 or SHA1 -- in this order.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	39
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	40 Does not consider :mod:`pyblake2` if it is available eventually.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	41
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	42 """
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	43 if "blake2b" in hashlib.algorithms_available:
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	44 return "BLAKE2b-256"
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	45 if "sha256" in hashlib.algorithms_available:
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	46 return "SHA256"
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	47 return "SHA1"
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	48
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	49
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	50 def get_blake2b():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	51 """Get the factory for blake2b"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	52 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	53 return hashlib.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	54 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	55 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	56 return pyblake2.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	57
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	58
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	59 def get_blake2b_256():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	60 """Get the factory for blake2b-256"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	61
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	62 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	63 hashlib.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	64 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	65 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	66
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	67 def _get_blake():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	68 return pyblake2.blake2b(digest_size=32)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	69
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	70 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	71
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	72 def _get_blake():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	73 return hashlib.blake2b(digest_size=32)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	74
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	75 return _get_blake
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	76
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	77
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	78 def get_blake2s():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	79 """Get the factory for blake2s"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	80 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	81 return hashlib.blake2s
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	82 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	83 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	84 return pyblake2.blake2s
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	85
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	86
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	87 def argv2algo(s):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	88 """Convert a command line algorithm specifier into a tuple with the
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	89 type/factory of the digest and the algorithms tag for output purposes.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	90
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	91 :param str s: the specifier from the command line; should include all
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 166 diff changeset	92 algorithm tags also (for proper round-tripping)
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	93 :return: the internal digest specification
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	94 :rtype: a tuple (digest_type_or_factory, name_in_output)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	95 :raises argparse.ArgumentTypeError: for unrecognized algorithms or names
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	96
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	97 String comparisons are done case-insensitively.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	98
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	99 """
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	100 s = s.lower()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	101 if s in ("1", "sha1"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	102 return (hashlib.sha1, "SHA1")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	103 elif s in ("224", "sha224"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	104 return (hashlib.sha224, "SHA224")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	105 elif s in ("256", "sha256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	106 return (hashlib.sha256, "SHA256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	107 elif s in ("384", "sha384"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	108 return (hashlib.sha384, "SHA384")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	109 elif s in ("512", "sha512"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	110 return (hashlib.sha512, "SHA512")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	111 elif s in ("3-224", "sha3-224"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	112 return (hashlib.sha3_224, "SHA3-224")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	113 elif s in ("3-256", "sha3-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	114 return (hashlib.sha3_256, "SHA3-256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	115 elif s in ("3-384", "sha3-384"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	116 return (hashlib.sha3_384, "SHA3-384")
120 a548783381b6 More accurate description of --algorithm. Franz Glasner <fzglas.hg@dom66.de> parents: 118 diff changeset	117 elif s in ("3", "3-512", "sha3-512"):
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	118 return (hashlib.sha3_512, "SHA3-512")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	119 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	120 return (get_blake2b(), "BLAKE2b")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	121 elif s in ("blake2s", "blake2s-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	122 return (get_blake2s(), "BLAKE2s")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	123 elif s in ("blake2-256", "blake2b-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	124 return (get_blake2b_256(), "BLAKE2b-256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	125 elif s == "md5":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	126 return (hashlib.md5, "MD5")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	127 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	128 raise argparse.ArgumentTypeError(
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	129 "`{}' is not a recognized algorithm".format(s))
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	130
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	131
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	132 def algotag2algotype(s):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	133 """Convert the algorithm specifier in a BSD-style digest file to the
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	134 type/factory of the corresponding algorithm.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	135
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	136 :param str s: the tag (i.e. normalized name) or the algorithm
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	137 :return: the digest type or factory for `s`
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	138 :raises ValueError: on unknown and/or unhandled algorithms
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	139
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	140 All string comparisons are case-sensitive.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	141
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	142 """
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	143 if s == "SHA1":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	144 return hashlib.sha1
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	145 elif s == "SHA224":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	146 return hashlib.sha224
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	147 elif s == "SHA256":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	148 return hashlib.sha256
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	149 elif s == "SHA384":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	150 return hashlib.sha384
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	151 elif s == "SHA512":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	152 return hashlib.sha512
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	153 elif s == "SHA3-224":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	154 return hashlib.sha3_224
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	155 elif s == "SHA3-256":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	156 return hashlib.sha3_256
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	157 elif s == "SHA3-384":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	158 return hashlib.sha3_384
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	159 elif s == "SHA3-512":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	160 return hashlib.sha3_512
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	161 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	162 return get_blake2b()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	163 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	164 return get_blake2s()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	165 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	166 return get_blake2b_256()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	167 elif s == "MD5":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	168 return hashlib.md5
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	169 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	170 raise ValueError("unknown algorithm: {}".format(s))
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	171
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	172
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	173 def normalize_filename(filename, strip_leading_dot_slash=False):
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	174 if isinstance(filename, bytes):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	175 filename = filename.replace(b"\\", b"/")
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	176 if strip_leading_dot_slash:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	177 while filename.startswith(b"./"):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	178 filename = filename[2:]
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	179 else:
166 ed45abb4940f FIX: in util.normalize_filename(): use the 'u"' prefix in the else part for non-byte strings Franz Glasner <fzglas.hg@dom66.de> parents: 164 diff changeset	180 filename = filename.replace(u"\\", u"/")
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	181 if strip_leading_dot_slash:
166 ed45abb4940f FIX: in util.normalize_filename(): use the 'u"' prefix in the else part for non-byte strings Franz Glasner <fzglas.hg@dom66.de> parents: 164 diff changeset	182 while filename.startswith(u"./"):
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	183 filename = filename[2:]
118 12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases) Franz Glasner <fzglas.hg@dom66.de> parents: 115 diff changeset	184 return filename
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	185
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	186
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	187 def fsencode(what):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	188 """A somewhat compatibility function for :func:`os.fsencode`.
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	189
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	190 If `what` is of type :class:`bytes` no :func:`os.fsencode` is required.
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	191
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	192 """
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	193 if isinstance(what, bytes):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	194 return what
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: 120 diff changeset	195 return os.fsencode(what)
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	196
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	197
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	198 def interpolate_bytes(formatstr, *values):
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	199 """Interpolate byte strings also on Python 3.4.
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	200
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	201 :param bytes formatstr:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	202 :param values: params for interpolation: may not contain Unicode strings
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	203 :rvalue: the formatted octet
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	204 :rtype: bytes
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	205
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	206 """
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	207 assert isinstance(formatstr, bytes)
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	208 # Python 3.5+ or Python2 know how to interpolate byte strings
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	209 if PY35 or PY2:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	210 return formatstr % values
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	211 # Workaround with a Latin-1 dance
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	212 tformatstr = formatstr.decode("latin1")
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	213 tvalues = []
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	214 for v in values:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	215 if PY2:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	216 if isinstance(v, unicode): # noqa: F821 undefined name 'unicode'
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	217 assert False
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	218 else:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	219 if isinstance(v, str):
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	220 assert False
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	221 if isinstance(v, bytes):
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	222 tvalues.append(v.decode("latin1"))
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	223 else:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	224 tvalues.append(v)
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	225 return (tformatstr % tuple(tvalues)).encode("latin1")

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/__init__.py @ 177:089c40240061

annotate cutils/util/init.py @ 177:089c40240061