Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/__init__.py @ 379:6d7659a709f2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 12 May 2025 12:12:44 +0200
parents 54a6d4534ef4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
115
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
323
48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/.
Franz Glasner <fzglas.hg@dom66.de>
parents: 307
diff changeset
3 # SPDX-FileCopyrightText: © 2020-2025 Franz Glasner
48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/.
Franz Glasner <fzglas.hg@dom66.de>
parents: 307
diff changeset
4 # SPDX-License-Identifier: BSD-3-Clause
115
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Utility package.
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
e15b3d1ff0d9 New subpackage with a "contextlib.nullcontext" for older Python versions
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
196
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 188
diff changeset
10 from __future__ import print_function, absolute_import
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 188
diff changeset
11
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 188
diff changeset
12
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
13 __all__ = ["PY2",
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
14 "PY35",
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
15 "n", "b", "u",
379
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
16 "parse_grouped_decimal_number",
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
17 "normalize_filename",
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
18 "escape_for_output",
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
19 "argv2algo",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
20 "algotag2algotype",
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
21 "algotag2digest_size",
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
22 "get_blake2b",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
23 "get_blake2b_256",
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
24 "get_blake2s",
172
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
25 "default_algotag",
163
fa7dd54e9715 FIX: Add "fsencode" to public functions in cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 124
diff changeset
26 "fsencode",
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
27 "interpolate_bytes",
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
28 ]
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
29
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
30
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
31 import argparse
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
32 import hashlib
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
33 import os
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
34 import sys
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
35
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
36
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 163
diff changeset
37 PY2 = sys.version_info[0] < 3
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
38 PY35 = sys.version_info[:2] >= (3, 5)
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
39
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
40
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
41 if PY2:
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
42
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
43 def n(s, encoding="ascii"):
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
44 """Convert `s` to the native string implementation"""
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
45 if isinstance(s, unicode): # noqa: F821 undefined name 'unicode'
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
46 return s.encode(encoding)
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
47 return s
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
48
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
49 def b(s, encoding="ascii", errors="strict"):
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
50 """Convert `s` to bytes"""
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
51 if isinstance(s, unicode): # noqa: F821 undefined name 'unicode'
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
52 return s.encode(encoding, errors)
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
53 return s
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
54
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
55 def u(s, encoding="ascii"):
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
56 """Convert `s` to a unicode string"""
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
57 if isinstance(s, str):
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
58 return s.decode(encoding)
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
59 return s
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
60
379
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
61 def parse_grouped_decimal_number(s):
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
62 return int(n(s).translate(None, "., '_"), 10)
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
63
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
64 else:
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
65
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
66 def n(s, encoding="ascii"):
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
67 """Convert `s` to the native string implementation"""
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
68 if isinstance(s, (bytes, bytearray)):
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
69 return s.decode(encoding)
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
70 return s
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
71
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
72 def b(s, encoding="ascii", errors="strict"):
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
73 """Convert `s` to bytes"""
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
74 if isinstance(s, str):
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
75 return s.encode(encoding, errors)
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
76 return s
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
77
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
78 u = n
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
79
379
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
80 def parse_grouped_decimal_number(s):
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
81 return int(n(s).translate(NORMALIZATION_DELETE_CHARS), 10)
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
82
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
83 NORMALIZATION_DELETE_CHARS = {
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
84 ord('.'): None,
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
85 ord(','): None,
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
86 ord(' '): None,
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
87 ord("'"): None,
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
88 ord('_'): None
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
89 }
6d7659a709f2 FIX: treesum: str.translate() behaves differently in Python3: make it PY3-compatible
Franz Glasner <fzglas.hg@dom66.de>
parents: 376
diff changeset
90
188
2784fdcc99e5 Implement basic parsing of treesum output.
Franz Glasner <fzglas.hg@dom66.de>
parents: 185
diff changeset
91
372
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
92 def escape_for_output(what):
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
93 """Escape `what` in such a way that the output can be safely written into
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
94 a line and/or column-oriented output file
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
95
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
96 """
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
97 if isinstance(what, bytes):
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
98 return (what.replace(b'\\', b"\\\\")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
99 .replace(b'\n', b"\\x0a")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
100 .replace(b'\r', b"\\x0d")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
101 .replace(b'\t', b"\\x09"))
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
102 else:
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
103 return (what.replace(u'\\', u"\\\\")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
104 .replace(u'\n', u"\\x0a")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
105 .replace(u'\r', u"\\x0d")
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
106 .replace(u'\t', u"\\x09"))
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
107
bfe1160fbfd3 treesum: Make ERROR outputs more consistent: use native paths where possible
Franz Glasner <fzglas.hg@dom66.de>
parents: 365
diff changeset
108
172
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
109 def default_algotag():
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
110 """Determine the "best" default algorithm.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
111
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
112 Depend on availability in :mod:`hashlib`.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
113
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
114 Prefer BLAKE2b-256, SHA256 or SHA1 -- in this order.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
115
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
116 Does not consider :mod:`pyblake2` if it is available eventually.
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
117
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
118 """
365
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
119 # Python <2.7.9 has no algorithms_available: fall back to algorithms
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
120 try:
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
121 algos = hashlib.algorithms_available
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
122 except AttributeError:
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
123 try:
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
124 algos = hashlib.algorithms
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
125 except AttributeError:
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
126 algos = []
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
127 if "blake2b" in algos:
172
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
128 return "BLAKE2b-256"
365
d5c920ace3cb treesum: FIX: Python<2.7.9 has no hashlib.algorithms_available: fall back to hashlib.algorithms in this case
Franz Glasner <fzglas.hg@dom66.de>
parents: 323
diff changeset
129 if "sha256" in algos:
172
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
130 return "SHA256"
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
131 return "SHA1"
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
132
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
133
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
134 def get_blake2b():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
135 """Get the factory for blake2b"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
136 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
137 return hashlib.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
138 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
139 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
140 return pyblake2.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
141
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
142
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
143 def get_blake2b_256():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
144 """Get the factory for blake2b-256"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
145
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
146 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
147 hashlib.blake2b
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
148 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
149 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
150
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
151 def _get_blake():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
152 return pyblake2.blake2b(digest_size=32)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
153
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
154 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
155
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
156 def _get_blake():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
157 return hashlib.blake2b(digest_size=32)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
158
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
159 return _get_blake
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
160
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
161
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
162 def get_blake2s():
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
163 """Get the factory for blake2s"""
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
164 try:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
165 return hashlib.blake2s
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
166 except AttributeError:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
167 import pyblake2
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
168 return pyblake2.blake2s
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
169
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
170
184
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
171 def get_crc(name):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
172 """Get the factory for a CRC"""
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
173
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
174 from ..crcmod.predefined import PredefinedCrc
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
175
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
176 def _crc_type():
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
177 return PredefinedCrc(name)
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
178
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
179 return _crc_type
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
180
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
181
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
182 def argv2algo(s):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
183 """Convert a command line algorithm specifier into a tuple with the
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
184 type/factory of the digest and the algorithms tag for output purposes.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
185
172
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
186 :param str s: the specifier from the command line; should include all
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib
Franz Glasner <fzglas.hg@dom66.de>
parents: 166
diff changeset
187 algorithm tags also (for proper round-tripping)
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
188 :return: the internal digest specification
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
189 :rtype: a tuple (digest_type_or_factory, name_in_output)
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
190 :raises argparse.ArgumentTypeError: for unrecognized algorithms or names
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
191
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
192 String comparisons are done case-insensitively.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
193
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
194 """
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
195 s = s.lower()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
196 if s in ("1", "sha1"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
197 return (hashlib.sha1, "SHA1")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
198 elif s in ("224", "sha224"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
199 return (hashlib.sha224, "SHA224")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
200 elif s in ("256", "sha256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
201 return (hashlib.sha256, "SHA256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
202 elif s in ("384", "sha384"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
203 return (hashlib.sha384, "SHA384")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
204 elif s in ("512", "sha512"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
205 return (hashlib.sha512, "SHA512")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
206 elif s in ("3-224", "sha3-224"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
207 return (hashlib.sha3_224, "SHA3-224")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
208 elif s in ("3-256", "sha3-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
209 return (hashlib.sha3_256, "SHA3-256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
210 elif s in ("3-384", "sha3-384"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
211 return (hashlib.sha3_384, "SHA3-384")
120
a548783381b6 More accurate description of --algorithm.
Franz Glasner <fzglas.hg@dom66.de>
parents: 118
diff changeset
212 elif s in ("3", "3-512", "sha3-512"):
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
213 return (hashlib.sha3_512, "SHA3-512")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
214 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
215 return (get_blake2b(), "BLAKE2b")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
216 elif s in ("blake2s", "blake2s-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
217 return (get_blake2s(), "BLAKE2s")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
218 elif s in ("blake2-256", "blake2b-256"):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
219 return (get_blake2b_256(), "BLAKE2b-256")
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
220 elif s == "md5":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
221 return (hashlib.md5, "MD5")
184
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
222 elif s in ("crc24", "crc-24",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
223 "crc24-openpgp", "crc-24-openpgp"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
224 return (get_crc("crc-24"), "CRC-24")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
225 elif s in ("crc32", "crc-32",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
226 "crc32-pkzip", "crc-32-pkzip",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
227 "crc32-iso", "crc-32-iso",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
228 "crc32-iso-hdlc", "crc-32-iso-hdlc"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
229 return (get_crc("crc-32"), "CRC-32-ISO")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
230 elif s in ("crc32-posix", "crc-32-posix",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
231 "crc32-cksum", "crc-32-cksum",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
232 "posix"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
233 return (get_crc("posix"), "CRC-32-POSIX")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
234 elif s in ("crc64", "crc-64",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
235 "crc64-iso", "crc-64-iso"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
236 return (get_crc("crc-64"), "CRC-64-ISO")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
237 elif s in ("crc64-2", "crc-64-2",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
238 "crc64-iso-2", "crc-64-iso-2",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
239 "crc64-mcrc64", "crc-64-mcrc64"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
240 return (get_crc("crc-64-2"), "CRC-64-ISO-2")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
241 elif s in ("crc64-ecma", "crc-64-ecma"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
242 return (get_crc("crc-64-ecma"), "CRC-64-ECMA")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
243 elif s in ("crc64-xz", "crc-64-xz",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
244 "crc64-go-ecma", "crc-64-go-ecma"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
245 return (get_crc("crc-64-xz"), "CRC-64-XZ")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
246 elif s in ("crc64-go", "crc-64-go",
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
247 "crc64-go-iso", "crc-64-go-iso"):
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
248 return (get_crc("crc-64-go"), "CRC-64-GO-ISO")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
249 elif s in ("crc64-redis", "crc-64-redis"):
185
f04d4b1c14b3 FIX: "digest" had overwritteh the digest module
Franz Glasner <fzglas.hg@dom66.de>
parents: 184
diff changeset
250 return (get_crc("crc-64-redis"), "CRC-64-REDIS")
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
251 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
252 raise argparse.ArgumentTypeError(
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
253 "`{}' is not a recognized algorithm".format(s))
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
254
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
255
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
256 def algotag2algotype(s):
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
257 """Convert the algorithm specifier in a BSD-style digest file to the
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
258 type/factory of the corresponding algorithm.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
259
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
260 :param str s: the tag (i.e. normalized name) or the algorithm
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
261 :return: the digest type or factory for `s`
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
262 :raises ValueError: on unknown and/or unhandled algorithms
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
263 :raises ImportError: if a module that is required to handle given
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
264 specifier `s` is not available (e.g. BLAKE2b on
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
265 Python 2)
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
266
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
267 All string comparisons are case-sensitive.
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
268
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
269 """
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
270 # Standard in Python2.7
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
271 if s == "MD5":
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
272 return hashlib.md5
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
273 elif s == "SHA1":
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
274 return hashlib.sha1
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
275 elif s == "SHA224":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
276 return hashlib.sha224
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
277 elif s == "SHA256":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
278 return hashlib.sha256
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
279 elif s == "SHA384":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
280 return hashlib.sha384
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
281 elif s == "SHA512":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
282 return hashlib.sha512
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
283 # Available in Python 3.6+
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
284 elif s == "SHA3-224":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
285 return hashlib.sha3_224
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
286 elif s == "SHA3-256":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
287 return hashlib.sha3_256
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
288 elif s == "SHA3-384":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
289 return hashlib.sha3_384
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
290 elif s == "SHA3-512":
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
291 return hashlib.sha3_512
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
292 # Available in Python 3.6+ or if pyblake2 is installed
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
293 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
294 return get_blake2b()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
295 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
296 return get_blake2s()
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
297 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
298 return get_blake2b_256()
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
299 # Vendored in cutils.crcmod
184
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
300 elif s == "CRC-24":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
301 return get_crc("crc-24")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
302 elif s == "CRC-32-ISO":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
303 return get_crc("crc-32")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
304 elif s == "CRC-32-POSIX":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
305 return get_crc("posix")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
306 elif s == "CRC-64-ISO":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
307 return get_crc("crc-64")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
308 elif s == "CRC-64-ISO-2":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
309 return get_crc("crc-64-2")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
310 elif s == "CRC-64-ECMA":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
311 return get_crc("crc-64-ecma")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
312 elif s == "CRC-64-XZ":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
313 return get_crc("crc-64-xz")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
314 elif s == "CRC-64-GO-ISO":
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
315 return get_crc("crc-64-go")
6154b8e4ba94 Include some new algorithms: CRC
Franz Glasner <fzglas.hg@dom66.de>
parents: 173
diff changeset
316 elif s == "CRC-64-REDIS":
185
f04d4b1c14b3 FIX: "digest" had overwritteh the digest module
Franz Glasner <fzglas.hg@dom66.de>
parents: 184
diff changeset
317 return get_crc("crc-64-redis")
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
318 else:
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
319 raise ValueError("unknown algorithm: {}".format(s))
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
320
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
321
307
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
322 def algotag2digest_size(s):
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
323 """Get the `digest_size` in bytes from given algorithm specifier `s`.
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
324
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
325 Contains a small static database of digest sizes for algorithms that
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
326 are not available by default in older Python versions.
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
327
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
328 :raises ValueError: on unknown and/or unhandled algorithms
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
329 :raises ImportError: if a module that is required to handle given
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
330 specifier `s` is not available (e.g. BLAKE2b on
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
331 Python 2)
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
332
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
333 All string comparisons are case-sensitive.
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
334
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
335 """
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
336 try:
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
337 dgst = algotag2algotype(s)()
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
338 return dgst.digest_size
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
339 except ImportError:
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
340 sz = {
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
341 "SHA3-224": 28,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
342 "SHA3-256": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
343 "SHA3-384": 48,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
344 "SHA3-512": 64,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
345 "BLAKE2b": 64,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
346 "BLAKE2b-512": 64,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
347 "BLAKE2b512": 64,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
348 "BLAKE2b-256": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
349 "BLAKE2b256": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
350 "BLAKE2s": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
351 "BLAKE2s-256": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
352 "BLAKE2s256": 32,
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
353 }.get(s, None)
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
354 if not sz:
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
355 raise
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
356 return sz
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
357
64df94bf4659 treesum: Build a little static database of digest sizes.
Franz Glasner <fzglas.hg@dom66.de>
parents: 279
diff changeset
358
278
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
359 def normalize_filename(filename, strip_dot_slashes=False):
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
360 if isinstance(filename, bytes):
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
361 filename = filename.replace(b"\\", b"/")
278
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
362 if strip_dot_slashes:
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
363 while filename.startswith(b"./"):
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
364 filename = filename[2:]
278
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
365 # This also handles adjacent /./ cases
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
366 while b"/./" in filename:
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
367 filename = filename.replace(b"/./", b"/", 1)
279
44e62e36cad4 treesum: when normalizing filenames also remove trailing "/."
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
368 while filename.endswith(b"/."):
44e62e36cad4 treesum: when normalizing filenames also remove trailing "/."
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
369 filename = filename[:-2]
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
370 else:
166
ed45abb4940f FIX: in util.normalize_filename(): use the 'u"' prefix in the else part for non-byte strings
Franz Glasner <fzglas.hg@dom66.de>
parents: 164
diff changeset
371 filename = filename.replace(u"\\", u"/")
278
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
372 if strip_dot_slashes:
166
ed45abb4940f FIX: in util.normalize_filename(): use the 'u"' prefix in the else part for non-byte strings
Franz Glasner <fzglas.hg@dom66.de>
parents: 164
diff changeset
373 while filename.startswith(u"./"):
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
374 filename = filename[2:]
278
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
375 # This also handles adjacent /./ cases
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
376 while u"/./" in filename:
822cf3a1da22 treesum: FIX: Removal of backslashes in output: do this only where really needed.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
377 filename = filename.replace(u"/./", u"/", 1)
279
44e62e36cad4 treesum: when normalizing filenames also remove trailing "/."
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
378 while filename.endswith(u"/."):
44e62e36cad4 treesum: when normalizing filenames also remove trailing "/."
Franz Glasner <fzglas.hg@dom66.de>
parents: 278
diff changeset
379 filename = filename[:-2]
118
12339ac2148d Move some functions into cutils.util (i.e. algorithms and their aliases)
Franz Glasner <fzglas.hg@dom66.de>
parents: 115
diff changeset
380 return filename
124
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
381
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
382
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
383 def fsencode(what):
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
384 """A somewhat compatibility function for :func:`os.fsencode`.
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
385
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
386 If `what` is of type :class:`bytes` no :func:`os.fsencode` is required.
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
387
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
388 """
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
389 if isinstance(what, bytes):
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
390 return what
3bd3f32b5e60 A first version of "treesum" is working
Franz Glasner <fzglas.hg@dom66.de>
parents: 120
diff changeset
391 return os.fsencode(what)
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
392
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
393
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
394 def interpolate_bytes(formatstr, *values):
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
395 """Interpolate byte strings also on Python 3.4.
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
396
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
397 :param bytes formatstr:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
398 :param values: params for interpolation: may *not* contain Unicode strings
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
399 :rvalue: the formatted octet
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
400 :rtype: bytes
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
401
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
402 """
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
403 assert isinstance(formatstr, bytes)
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
404 # Python 3.5+ or Python2 know how to interpolate byte strings
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
405 if PY35 or PY2:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
406 return formatstr % values
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
407 # Workaround with a Latin-1 dance
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
408 tformatstr = formatstr.decode("latin1")
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
409 tvalues = []
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
410 for v in values:
376
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
411 # Do not automatically convert text (unicode) string values into bytes.
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
412 if PY2:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
413 if isinstance(v, unicode): # noqa: F821 undefined name 'unicode'
376
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
414 raise TypeError(
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
415 "unicode values not supported when interpolating"
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
416 " into bytes")
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
417 else:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
418 if isinstance(v, str):
376
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
419 raise TypeError(
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
420 "unicode (native string) values not supported when"
54a6d4534ef4 Convert some further value checks from plain Python assertions into real checks
Franz Glasner <fzglas.hg@dom66.de>
parents: 372
diff changeset
421 " interpolating into bytes")
173
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
422 if isinstance(v, bytes):
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
423 tvalues.append(v.decode("latin1"))
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
424 else:
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
425 tvalues.append(v)
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting.
Franz Glasner <fzglas.hg@dom66.de>
parents: 172
diff changeset
426 return (tformatstr % tuple(tvalues)).encode("latin1")