Python/apps/py-cutils: cutils/shasum.py annotate

annotate cutils/shasum.py @ 89:72684020f2f3

By default use mmap only for files up to 8MiB in size. This follows the FreeBSD cp(1) implementation.

author	Franz Glasner <fzglas.hg@dom66.de>
date	Thu, 21 Apr 2022 01:20:35 +0200
parents	f69353f26937
children	42419f57eda9

rev	line source
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	1 # -- coding: utf-8 --
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	6 r"""Pure Python implementation of `shasum`
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
72 ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	10 from __future__ import print_function, absolute_import
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	11
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12 from . import (__version__, __revision__)
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	13
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	14
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	15 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	16
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	17
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	18 import argparse
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	19 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	20 import binascii
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	21 import errno
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	22 import hashlib
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	23 import io
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	24 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	25 import mmap
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	26 except ImportError:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	27 mmap = None
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	28 import os
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	29 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	30 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	31 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	32 pathlib = None
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	33 import re
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	34 import stat
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	35 import sys
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	36
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	39
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	47
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	52
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 fromfile_prefix_chars='@')
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	57 aparser.add_argument(
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	58 "--algorithm", "-a", action="store", type=argv2algo,
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	60 aparser.add_argument(
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	61 "--base64", action="store_true",
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	63 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	65 help="Read in binary mode (default)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	66 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	69 aparser.add_argument(
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	70 "--check", "-c", action="store_true",
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	77 If no file is given, stdin is used.""")
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	78 aparser.add_argument(
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	82 generate an error.""")
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	88 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	89 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	90 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	91 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	92 "--tag", action="store_true", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	93 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	94 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	95 "--text", "-t", action="store_true", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	96 help="Read in text mode (not supported)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	97 aparser.add_argument(
27 c29bd96dcd4f Put the HG revision into the version output also Franz Glasner <fzglas.hg@dom66.de> parents: 26 diff changeset	98 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	99 aparser.add_argument(
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	100 "files", nargs="*", metavar="FILE")
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	101
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	102 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	103
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	104 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	105 print("ERROR: text mode not supported", file=sys.stderr)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	106 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	107
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	108 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	109 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	110 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	111 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	112
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	113 if not opts.algorithm:
8 048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum Franz Glasner <fzglas.hg@dom66.de> parents: 7 diff changeset	114 opts.algorithm = argv2algo("1")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	115
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	116 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	117
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	118 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	119
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	120
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	121 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	122 checklist=False, check=False, dest=None, base64=False):
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	123 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	124 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	125 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	126 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	127 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	128 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	129 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	130 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	131 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	132 check=check,
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	133 text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	134 dest=dest,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	135 base64=base64)
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	136 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	137
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	138
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	139 def shasum(opts):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	140 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	141 return verify_digests_from_files(opts)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	142 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	143 return verify_digests_with_checklist(opts)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	144 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	145 return generate_digests(opts)
10 77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	146
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	147
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	148 def generate_digests(opts):
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	149 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	150 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	151 else:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	152 out = out_std
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	153 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	154 if PY2:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	155 if sys.platform == "win32":
11 15c3416d3677 FIX: "msvcrt" is a top-level module on Windows Franz Glasner <fzglas.hg@dom66.de> parents: 10 diff changeset	156 import os, msvcrt # noqa: E401
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	157 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	158 source = sys.stdin
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	159 else:
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	160 source = sys.stdin.buffer
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	161 out(sys.stdout,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	162 compute_digest_stream(opts.algorithm[0], source),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	163 None,
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	164 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	165 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	166 opts.base64)
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	167 else:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	168 for fn in opts.files:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	169 out(opts.dest or sys.stdout,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	170 compute_digest_file(opts.algorithm[0], fn),
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	171 fn,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	172 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	173 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	174 opts.base64)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	175 return 0
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	176
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	177
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	178 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	179 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	180 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	181
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	182 :param bytes given_digest:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	183 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	184 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	185 :type expected_digest: str or bytes or bytearray
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	186 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	187 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	188 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	189
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	190 """
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	191 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	192 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	193 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	194 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	195 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	196 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	197 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	198 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	199 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	200 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	201 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	202 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	203 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	204 else:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	205 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	206 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	207 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	208 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	209 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	210 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	211 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	212 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	213 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	214 return False
75 a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	215 return given_digest == exd
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	216
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	217
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	218 def verify_digests_with_checklist(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	219 dest = opts.dest or sys.stdout
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	220 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	221 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	222 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	223 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	224 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	225 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	226 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	227 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	228 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	229 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	230 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	231 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	232 print("-: MISSING", file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	233 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	234 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	235 computed_digest = compute_digest_stream(algo, source)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	236 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	237 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	238 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	239 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	240 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	241 print("{}: {}: {}".format(tag, "-", res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	242 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	243 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	244 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	245 if pl is None:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	246 print("{}: MISSING".format(fn), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	247 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	248 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	249 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	250 computed_digest = compute_digest_file(algo, fn)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	251 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	252 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	253 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	254 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	255 res = "FAILED"
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	256 print("{}: {}: {}".format(tag, fn, res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	257 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	258
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	259
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	260 def verify_digests_from_files(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	261 dest = opts.dest or sys.stdout
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	262 exit_code = 0
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	263 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	264 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	265 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	266 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	267 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	268 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	269 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	270 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	271 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	272 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	273 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	274 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	275 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	276 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	277 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	278 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	279 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	280 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	281 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	282 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	283 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	284 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	285 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	286 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	287 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	288
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	289
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	290 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	291 """
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	292 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	293 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	294 :rtype: tuple(str, str, str)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	295
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	296 """
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	297 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	298 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	299 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	300 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	301 tag, algo, fn, digest = parts
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	302 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	303 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	304 return (None, None, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	305 try:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	306 d = compute_digest_file(algo, fn)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	307 if compare_digests_equal(d, digest, algo):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	308 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	309 else:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	310 return ("failed", fn, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	311 except EnvironmentError:
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	312 return ("missing", fn, tag)
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	313
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	314
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	315 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	316 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	317 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	318 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	319 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	320 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	321 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	322 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	323 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	324 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	325 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	326 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	327 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	328 "improperly formatted digest line: {}".format(checkline))
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	329 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	330 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	331 continue
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	332 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	333 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	334 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	335 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	336 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	337
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	338
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	339 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	340 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	341
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	342 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	343 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	344 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	345
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	346 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	347 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	348 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	349 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	350
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	351 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	352
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	353 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	354 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	355 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	356 mo = re.search(r"\A(\S+)\s\((.)\)\s=\s(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	357 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	358 # (tag, algorithm, filename, digest)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	359 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	360 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	361 return ("SIZE", None, None, mo.group(3))
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	362 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	363 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	364 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	365 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	366 else:
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	367 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	368 mo = re.search(r"\ATIMESTAMP\s=\s([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	369 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	370 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	371
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	372 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	373 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	374 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	375 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	376 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	377 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	378 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	379 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	380 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	381 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	382
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	383
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	384 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	385 """Get the factory for blake2b"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	386 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	387 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	388 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	389 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	390 return pyblake2.blake2b
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	391
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	392
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	393 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	394 """Get the factory for blake2s"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	395 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	396 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	397 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	398 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	399 return pyblake2.blake2s
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	400
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	401
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	402 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	403 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	404
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	405 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	406 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	407 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	408 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	409
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	410 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	411 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	412
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	413 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	414
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	415 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	416 return hashlib.blake2b(digest_size=32)
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	417
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	418 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	419
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	420
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	421 def argv2algo(s):
42 e5f6af8364db Typo in docs Franz Glasner <f.glasner@feldmann-mg.com> parents: 30 diff changeset	422 """Convert a command line algorithm specifier into a tuple with the
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	423 type/factory of the digest and the algorithms tag for output purposes.
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	424
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	425 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	426 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	427 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	428
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	429 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	430
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	431 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	432 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	433 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	434 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	435 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	436 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	437 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	438 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	439 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	440 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	441 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	442 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	443 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	444 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	445 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	446 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	447 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	448 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	449 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	450 return (hashlib.sha3_512, "SHA3-512")
86 fd1cfd1b0f9d Make "blake2" an alias of "blake2b" Franz Glasner <fzglas.hg@dom66.de> parents: 85 diff changeset	451 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	452 return (get_blake2b(), "BLAKE2b")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	453 elif s in ("blake2s", "blake2s-256"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	454 return (get_blake2s(), "BLAKE2s")
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	455 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	456 return (get_blake2_256(), "BLAKE2b-256")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	457 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	458 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	459 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	460 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	461 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	462
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	463
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	464 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	465 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	466 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	467
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	468 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	469 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	470
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	471 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	472
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	473 """
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	474 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	475 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	476 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	477 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	478 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	479 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	480 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	481 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	482 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	483 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	484 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	485 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	486 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	487 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	488 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	489 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	490 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	491 return hashlib.sha3_512
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	492 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	493 return get_blake2b()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	494 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	495 return get_blake2s()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	496 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	497 return get_blake2_256()
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	498 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	499 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	500 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	501 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	502
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	503
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	504 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	505 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	506 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	507
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	508 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	509 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	510 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	511 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	512 digest = binascii.hexlify(digest).decode("ascii")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	513 if filename is None:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	514 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	515 else:
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	516 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	517 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	518 digest),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	519 file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	520
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	521
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	522 def out_std(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	523 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	524
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	525 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	526 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	527 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	528 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	529 digest = binascii.hexlify(digest).decode("ascii")
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	530 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	531 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	532 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	533 '-' if filename is None else normalize_filename(filename)),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	534 file=dest)
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	535
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	536
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	537 def compute_digest_file(hashobj, path, use_mmap=None):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	538 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	539 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	540 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	541 binary mode (also a socket or pipe)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	542 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	543 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	544 :type use_mmap: bool or None
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	545 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	546 :rtype: bytes
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	547
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	548 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	549
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	550 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	551 h = hashobj()
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	552 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	553 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	554 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	555 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	556 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	557 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	558 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	559 own_fd = False
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	560 try:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	561 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	562 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	563 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	564 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	565 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	566 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	567 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	568 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	569 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	570 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	571 filesize = st[stat.ST_SIZE]
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	572 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	573 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	574 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	575 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	576 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	577 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	578 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	579 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	580 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	581 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	582 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	583 use_mmap = False
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	584 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	585 use_mmap = False
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	586 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	587 use_mmap = True
67 19893b4f42a5 Flag to disable the use of mmap Franz Glasner <fzglas.hg@dom66.de> parents: 66 diff changeset	588 if mmap is None or not use_mmap:
84 163de6dd6e05 Enhance comment and -- while being there -- fix a typo Franz Glasner <f.glasner@feldmann-mg.com> parents: 83 diff changeset	589 # No mmap available or wanted -> use traditional low-level file IO
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	590 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	591 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	592 buf = os.read(fd, READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	593 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	594 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK,
8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	595 errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	596 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	597 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	598 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	599 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	600 h.update(buf)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	601 else:
58 ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	602 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	603 # Use mmap
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	604 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	605 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	606 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	607 #
61 c9f9401abc0c Use getattr when trying to get mmap.madvise() Franz Glasner <fzglas.hg@dom66.de> parents: 60 diff changeset	608 madvise = getattr(mmap.mmap, "madvise", None)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	609 if filesize <= MAP_WINDOW_SIZE:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	610 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	611 else:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	612 mapsize = MAP_WINDOW_SIZE
57 0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap Franz Glasner <f.glasner@feldmann-mg.com> parents: 56 diff changeset	613 mapoffset = 0
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	614 rest = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	615 while rest > 0:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	616 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	617 mapsize,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	618 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	619 offset=mapoffset)
60 21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	620 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	621 madvise(m, mmap.MADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	622 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	623 h.update(m)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	624 finally:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	625 m.close()
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	626 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	627 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	628 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	629 mapsize = rest
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	630 finally:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	631 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	632 os.close(fd)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	633 return h.digest()
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	634
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	635
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	636 def compute_digest_stream(hashobj, instream):
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	637 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	638
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	639 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	640 :param instream: a bytes input stream to read the data to be hashed from
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	641 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	642 :rtype: bytes
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	643
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	644 """
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	645 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	646 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	647 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	648 buf = instream.read(READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	649 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	650 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	651 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	652 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	653 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	654 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	655 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	656 h.update(buf)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	657 return h.digest()
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	658
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	659
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	660 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	661 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	662 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	663 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	664 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	665 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	666
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	667
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	668 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	669 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 89:72684020f2f3