Python/apps/py-cutils: cutils/shasum.py annotate

annotate cutils/shasum.py @ 95:fc2dd6afd594

Style: canonical global variable and import ordering

author	Franz Glasner <f.glasner@feldmann-mg.com>
date	Thu, 21 Apr 2022 18:52:53 +0200
parents	8352da172a3e
children	59253a1e6ef2

rev	line source
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	1 # -- coding: utf-8 --
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	6 r"""Pure Python implementation of `shasum`
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
72 ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	10 from __future__ import print_function, absolute_import
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	11
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	13 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	14
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	15
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import argparse
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	17 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	18 import binascii
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	19 import errno
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import hashlib
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	21 import io
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	22 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	23 import mmap
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	24 except ImportError:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	25 mmap = None
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	26 import os
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	27 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	28 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	29 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	30 pathlib = None
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	31 import re
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	32 import stat
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 import sys
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34
95 fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	35 from . import (__version__, __revision__)
fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	36
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	39
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	47
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	52
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 fromfile_prefix_chars='@')
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	57 aparser.add_argument(
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	58 "--algorithm", "-a", action="store", type=argv2algo,
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	60 aparser.add_argument(
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	61 "--base64", action="store_true",
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	63 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	65 help="Read in binary mode (default)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	66 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	69 aparser.add_argument(
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	70 "--check", "-c", action="store_true",
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	77 If no file is given, stdin is used.""")
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	78 aparser.add_argument(
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	82 generate an error.""")
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	88 aparser.add_argument(
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	89 "--mmap", action="store_true", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	90 help="""Use mmap if available. Default is to determine automatically
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	91 from the filesize.""")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	92 aparser.add_argument(
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	93 "--no-mmap", action="store_false", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	94 help="Dont use mmap.")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	95
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	96 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	97 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	98 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	99 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	100 "--tag", action="store_true", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	101 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	102 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	103 "--text", "-t", action="store_true", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	104 help="Read in text mode (not supported)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	105 aparser.add_argument(
27 c29bd96dcd4f Put the HG revision into the version output also Franz Glasner <fzglas.hg@dom66.de> parents: 26 diff changeset	106 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	107 aparser.add_argument(
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	108 "files", nargs="*", metavar="FILE")
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	109
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	110 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	111
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	112 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	113 print("ERROR: text mode not supported", file=sys.stderr)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	114 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	115
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	116 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	117 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	118 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	119 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	120
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	121 if not opts.algorithm:
8 048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum Franz Glasner <fzglas.hg@dom66.de> parents: 7 diff changeset	122 opts.algorithm = argv2algo("1")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	123
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	124 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	125
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	126 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	127
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	128
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	129 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	130 checklist=False, check=False, dest=None, base64=False):
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	131 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	132 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	133 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	134 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	135 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	136 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	137 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	138 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	139 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	140 check=check,
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	141 text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	142 dest=dest,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	143 base64=base64)
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	144 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	145
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	146
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	147 def shasum(opts):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	148 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	149 return verify_digests_from_files(opts)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	150 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	151 return verify_digests_with_checklist(opts)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	152 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	153 return generate_digests(opts)
10 77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	154
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	155
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	156 def generate_digests(opts):
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	157 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	158 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	159 else:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	160 out = out_std
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	161 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	162 if PY2:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	163 if sys.platform == "win32":
11 15c3416d3677 FIX: "msvcrt" is a top-level module on Windows Franz Glasner <fzglas.hg@dom66.de> parents: 10 diff changeset	164 import os, msvcrt # noqa: E401
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	165 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	166 source = sys.stdin
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	167 else:
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	168 source = sys.stdin.buffer
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	169 out(sys.stdout,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	170 compute_digest_stream(opts.algorithm[0], source),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	171 None,
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	172 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	173 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	174 opts.base64)
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	175 else:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	176 for fn in opts.files:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	177 out(opts.dest or sys.stdout,
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	178 compute_digest_file(opts.algorithm[0], fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	179 use_mmap=opts.mmap),
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	180 fn,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	181 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	182 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	183 opts.base64)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	184 return 0
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	185
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	186
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	187 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	188 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	189 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	190
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	191 :param bytes given_digest:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	192 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	193 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	194 :type expected_digest: str or bytes or bytearray
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	195 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	196 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	197 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	198
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	199 """
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	200 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	201 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	202 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	203 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	204 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	205 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	206 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	207 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	208 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	209 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	210 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	211 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	212 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	213 else:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	214 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	215 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	216 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	217 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	218 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	219 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	220 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	221 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	222 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	223 return False
75 a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	224 return given_digest == exd
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	225
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	226
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	227 def verify_digests_with_checklist(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	228 dest = opts.dest or sys.stdout
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	229 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	230 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	231 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	232 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	233 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	234 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	235 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	236 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	237 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	238 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	239 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	240 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	241 print("-: MISSING", file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	242 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	243 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	244 computed_digest = compute_digest_stream(algo, source)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	245 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	246 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	247 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	248 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	249 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	250 print("{}: {}: {}".format(tag, "-", res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	251 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	252 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	253 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	254 if pl is None:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	255 print("{}: MISSING".format(fn), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	256 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	257 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	258 tag, algo, cl_filename, cl_digest = pl
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	259 computed_digest = compute_digest_file(algo, fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	260 use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	261 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	262 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	263 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	264 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	265 res = "FAILED"
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	266 print("{}: {}: {}".format(tag, fn, res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	267 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	268
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	269
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	270 def verify_digests_from_files(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	271 dest = opts.dest or sys.stdout
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	272 exit_code = 0
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	273 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	274 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	275 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	276 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	277 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	278 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	279 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	280 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	281 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	282 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	283 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	284 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	285 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	286 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	287 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	288 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	289 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	290 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	291 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	292 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	293 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	294 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	295 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	296 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	297 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	298
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	299
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	300 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	301 """
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	302 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	303 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	304 :rtype: tuple(str, str, str)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	305
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	306 """
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	307 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	308 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	309 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	310 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	311 tag, algo, fn, digest = parts
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	312 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	313 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	314 return (None, None, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	315 try:
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	316 d = compute_digest_file(algo, fn, use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	317 if compare_digests_equal(d, digest, algo):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	318 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	319 else:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	320 return ("failed", fn, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	321 except EnvironmentError:
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	322 return ("missing", fn, tag)
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	323
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	324
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	325 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	326 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	327 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	328 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	329 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	330 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	331 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	332 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	333 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	334 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	335 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	336 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	337 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	338 "improperly formatted digest line: {}".format(checkline))
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	339 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	340 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	341 continue
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	342 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	343 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	344 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	345 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	346 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	347
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	348
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	349 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	350 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	351
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	352 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	353 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	354 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	355
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	356 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	357 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	358 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	359 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	360
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	361 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	362
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	363 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	364 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	365 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	366 mo = re.search(r"\A(\S+)\s\((.)\)\s=\s(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	367 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	368 # (tag, algorithm, filename, digest)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	369 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	370 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	371 return ("SIZE", None, None, mo.group(3))
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	372 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	373 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	374 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	375 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	376 else:
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	377 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	378 mo = re.search(r"\ATIMESTAMP\s=\s([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	379 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	380 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	381
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	382 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	383 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	384 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	385 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	386 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	387 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	388 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	389 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	390 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	391 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	392
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	393
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	394 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	395 """Get the factory for blake2b"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	396 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	397 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	398 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	399 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	400 return pyblake2.blake2b
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	401
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	402
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	403 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	404 """Get the factory for blake2s"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	405 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	406 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	407 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	408 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	409 return pyblake2.blake2s
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	410
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	411
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	412 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	413 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	414
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	415 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	416 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	417 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	418 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	419
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	420 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	421 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	422
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	423 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	424
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	425 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	426 return hashlib.blake2b(digest_size=32)
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	427
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	428 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	429
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	430
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	431 def argv2algo(s):
42 e5f6af8364db Typo in docs Franz Glasner <f.glasner@feldmann-mg.com> parents: 30 diff changeset	432 """Convert a command line algorithm specifier into a tuple with the
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	433 type/factory of the digest and the algorithms tag for output purposes.
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	434
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	435 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	436 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	437 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	438
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	439 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	440
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	441 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	442 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	443 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	444 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	445 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	446 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	447 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	448 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	449 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	450 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	451 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	452 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	453 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	454 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	455 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	456 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	457 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	458 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	459 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	460 return (hashlib.sha3_512, "SHA3-512")
86 fd1cfd1b0f9d Make "blake2" an alias of "blake2b" Franz Glasner <fzglas.hg@dom66.de> parents: 85 diff changeset	461 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	462 return (get_blake2b(), "BLAKE2b")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	463 elif s in ("blake2s", "blake2s-256"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	464 return (get_blake2s(), "BLAKE2s")
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	465 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	466 return (get_blake2_256(), "BLAKE2b-256")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	467 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	468 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	469 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	470 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	471 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	472
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	473
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	474 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	475 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	476 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	477
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	478 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	479 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	480
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	481 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	482
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	483 """
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	484 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	485 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	486 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	487 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	488 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	489 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	490 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	491 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	492 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	493 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	494 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	495 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	496 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	497 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	498 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	499 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	500 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	501 return hashlib.sha3_512
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	502 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	503 return get_blake2b()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	504 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	505 return get_blake2s()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	506 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	507 return get_blake2_256()
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	508 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	509 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	510 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	511 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	513
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	514 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	515 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	516 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	517
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	518 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	519 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	520 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	521 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	522 digest = binascii.hexlify(digest).decode("ascii")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	523 if filename is None:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	524 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	525 else:
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	526 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	527 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	528 digest),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	529 file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	530
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	531
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	532 def out_std(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	533 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	534
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	535 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	536 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	537 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	538 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	539 digest = binascii.hexlify(digest).decode("ascii")
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	540 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	541 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	542 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	543 '-' if filename is None else normalize_filename(filename)),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	544 file=dest)
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	545
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	546
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	547 def compute_digest_file(hashobj, path, use_mmap=None):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	548 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	549 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	550 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	551 binary mode (also a socket or pipe)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	552 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	553 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	554 :type use_mmap: bool or None
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	555 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	556 :rtype: bytes
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	557
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	558 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	559
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	560 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	561 h = hashobj()
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	562 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	563 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	564 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	565 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	566 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	567 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	568 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	569 own_fd = False
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	570 try:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	571 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	572 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	573 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	574 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	575 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	576 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	577 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	578 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	579 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	580 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	581 filesize = st[stat.ST_SIZE]
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	582 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	583 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	584 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	585 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	586 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	587 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	588 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	589 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	590 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	591 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	592 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	593 use_mmap = False
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	594 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	595 use_mmap = False
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	596 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	597 use_mmap = True
67 19893b4f42a5 Flag to disable the use of mmap Franz Glasner <fzglas.hg@dom66.de> parents: 66 diff changeset	598 if mmap is None or not use_mmap:
84 163de6dd6e05 Enhance comment and -- while being there -- fix a typo Franz Glasner <f.glasner@feldmann-mg.com> parents: 83 diff changeset	599 # No mmap available or wanted -> use traditional low-level file IO
94 8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	600 fadvise = getattr(os, "posix_fadvise", None)
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	601 if fadvise:
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	602 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	603 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	604 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	605 buf = os.read(fd, READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	606 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	607 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK,
8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	608 errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	609 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	610 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	611 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	612 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	613 h.update(buf)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	614 else:
58 ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	615 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	616 # Use mmap
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	617 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	618 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	619 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	620 #
61 c9f9401abc0c Use getattr when trying to get mmap.madvise() Franz Glasner <fzglas.hg@dom66.de> parents: 60 diff changeset	621 madvise = getattr(mmap.mmap, "madvise", None)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	622 if filesize <= MAP_WINDOW_SIZE:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	623 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	624 else:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	625 mapsize = MAP_WINDOW_SIZE
57 0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap Franz Glasner <f.glasner@feldmann-mg.com> parents: 56 diff changeset	626 mapoffset = 0
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	627 rest = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	628 while rest > 0:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	629 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	630 mapsize,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	631 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	632 offset=mapoffset)
60 21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	633 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	634 madvise(m, mmap.MADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	635 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	636 h.update(m)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	637 finally:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	638 m.close()
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	639 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	640 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	641 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	642 mapsize = rest
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	643 finally:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	644 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	645 os.close(fd)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	646 return h.digest()
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	647
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	648
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	649 def compute_digest_stream(hashobj, instream):
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	650 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	651
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	652 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	653 :param instream: a bytes input stream to read the data to be hashed from
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	654 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	655 :rtype: bytes
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	656
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	657 """
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	658 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	659 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	660 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	661 buf = instream.read(READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	662 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	663 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	664 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	665 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	666 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	667 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	668 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	669 h.update(buf)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	670 return h.digest()
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	671
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	672
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	673 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	674 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	675 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	676 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	677 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	678 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	679
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	680
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	681 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	682 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 95:fc2dd6afd594