Python/apps/py-cutils: cutils/shasum.py annotate

annotate cutils/shasum.py @ 100:f95918115c6b

FIX: Implement some new commandline flags in "gen_opts()" also

author	Franz Glasner <fzglas.hg@dom66.de>
date	Fri, 29 Apr 2022 04:57:08 +0200
parents	59253a1e6ef2
children	08fd0609fdd4

rev	line source
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	1 # -- coding: utf-8 --
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	6 r"""Pure Python implementation of `shasum`
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
72 ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	10 from __future__ import print_function, absolute_import
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	11
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	13 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	14
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	15
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import argparse
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	17 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	18 import binascii
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	19 import errno
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import hashlib
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	21 import io
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	22 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	23 import mmap
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	24 except ImportError:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	25 mmap = None
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	26 import os
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	27 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	28 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	29 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	30 pathlib = None
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	31 import re
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	32 import stat
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 import sys
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34
95 fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	35 from . import (__version__, __revision__)
fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	36
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	39
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	47
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	52
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 fromfile_prefix_chars='@')
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	57 aparser.add_argument(
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	58 "--algorithm", "-a", action="store", type=argv2algo,
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	60 aparser.add_argument(
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	61 "--base64", action="store_true",
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	63 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	65 help="Read in binary mode (default)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	66 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	69 aparser.add_argument(
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	70 "--check", "-c", action="store_true",
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	77 If no file is given, stdin is used.""")
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	78 aparser.add_argument(
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	82 generate an error.""")
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	88 aparser.add_argument(
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	89 "--mmap", action="store_true", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	90 help="""Use mmap if available. Default is to determine automatically
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	91 from the filesize.""")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	92 aparser.add_argument(
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	93 "--no-mmap", action="store_false", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	94 help="Dont use mmap.")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	95
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	96 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	97 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	98 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	99 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	100 "--tag", action="store_true", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	101 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	102 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	103 "--text", "-t", action="store_true", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	104 help="Read in text mode (not supported)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	105 aparser.add_argument(
27 c29bd96dcd4f Put the HG revision into the version output also Franz Glasner <fzglas.hg@dom66.de> parents: 26 diff changeset	106 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	107 aparser.add_argument(
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	108 "files", nargs="*", metavar="FILE")
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	109
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	110 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	111
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	112 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	113 print("ERROR: text mode not supported", file=sys.stderr)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	114 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	115
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	116 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	117 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	118 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	119 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	120
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	121 if not opts.algorithm:
8 048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum Franz Glasner <fzglas.hg@dom66.de> parents: 7 diff changeset	122 opts.algorithm = argv2algo("1")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	123
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	124 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	125
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	126 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	127
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	128
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	129 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	130 checklist=False, check=False, dest=None, base64=False,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	131 allow_distinfo=False, mmap=None):
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	132 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	133 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	134 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	135 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	136 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	137 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	138 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	139 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	140 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	141 check=check,
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	142 text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	143 dest=dest,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	144 base64=base64,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	145 allow_distinfo=allow_distinfo,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	146 mmap=mmap)
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	147 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	148
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	149
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	150 def shasum(opts):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	151 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	152 return verify_digests_from_files(opts)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	153 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	154 return verify_digests_with_checklist(opts)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	155 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	156 return generate_digests(opts)
10 77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	157
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	158
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	159 def generate_digests(opts):
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	160 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	161 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	162 else:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	163 out = out_std
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	164 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	165 if PY2:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	166 if sys.platform == "win32":
11 15c3416d3677 FIX: "msvcrt" is a top-level module on Windows Franz Glasner <fzglas.hg@dom66.de> parents: 10 diff changeset	167 import os, msvcrt # noqa: E401
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	168 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	169 source = sys.stdin
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	170 else:
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	171 source = sys.stdin.buffer
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	172 out(sys.stdout,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	173 compute_digest_stream(opts.algorithm[0], source),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	174 None,
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	175 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	176 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	177 opts.base64)
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	178 else:
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	179 for fn in opts.files:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	180 out(opts.dest or sys.stdout,
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	181 compute_digest_file(opts.algorithm[0], fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	182 use_mmap=opts.mmap),
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	183 fn,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	184 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	185 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	186 opts.base64)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	187 return 0
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	188
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	189
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	190 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	191 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	192 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	193
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	194 :param bytes given_digest:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	195 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	196 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	197 :type expected_digest: str or bytes or bytearray
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	198 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	199 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	200 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	201
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	202 """
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	203 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	204 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	205 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	206 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	207 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	208 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	209 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	210 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	211 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	212 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	213 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	214 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	215 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	216 else:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	217 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	218 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	219 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	220 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	221 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	222 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	223 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	224 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	225 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	226 return False
75 a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	227 return given_digest == exd
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	228
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	229
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	230 def verify_digests_with_checklist(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	231 dest = opts.dest or sys.stdout
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	232 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	233 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	234 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	235 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	236 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	237 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	238 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	239 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	240 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	241 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	242 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	243 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	244 print("-: MISSING", file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	245 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	246 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	247 computed_digest = compute_digest_stream(algo, source)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	248 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	249 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	250 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	251 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	252 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	253 print("{}: {}: {}".format(tag, "-", res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	254 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	255 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	256 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	257 if pl is None:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	258 print("{}: MISSING".format(fn), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	259 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	260 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	261 tag, algo, cl_filename, cl_digest = pl
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	262 computed_digest = compute_digest_file(algo, fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	263 use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	264 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	265 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	266 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	267 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	268 res = "FAILED"
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	269 print("{}: {}: {}".format(tag, fn, res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	270 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	271
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	272
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	273 def verify_digests_from_files(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	274 dest = opts.dest or sys.stdout
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	275 exit_code = 0
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	276 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	277 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	278 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	279 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	280 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	281 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	282 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	283 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	284 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	285 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	286 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	287 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	288 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	289 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	290 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	291 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	292 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	293 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	294 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	295 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	296 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	297 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	298 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	299 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	300 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	301
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	302
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	303 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	304 """
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	305 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	306 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	307 :rtype: tuple(str, str, str)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	308
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	309 """
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	310 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	311 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	312 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	313 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	314 tag, algo, fn, digest = parts
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	315 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	316 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	317 return (None, None, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	318 try:
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	319 d = compute_digest_file(algo, fn, use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	320 if compare_digests_equal(d, digest, algo):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	321 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	322 else:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	323 return ("failed", fn, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	324 except EnvironmentError:
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	325 return ("missing", fn, tag)
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	326
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	327
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	328 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	329 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	330 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	331 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	332 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	333 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	334 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	335 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	336 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	337 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	338 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	339 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	340 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	341 "improperly formatted digest line: {}".format(checkline))
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	342 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	343 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	344 continue
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	345 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	346 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	347 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	348 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	349 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	350
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	351
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	352 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	353 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	354
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	355 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	356 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	357 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	358
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	359 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	360 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	361 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	362 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	363
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	364 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	365
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	366 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	367 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	368 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	369 mo = re.search(r"\A(\S+)\s\((.)\)\s=\s(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	370 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	371 # (tag, algorithm, filename, digest)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	372 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	373 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	374 return ("SIZE", None, None, mo.group(3))
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	375 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	376 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	377 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	378 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	379 else:
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	380 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	381 mo = re.search(r"\ATIMESTAMP\s=\s([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	382 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	383 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	384
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	385 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	386 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	387 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	388 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	389 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	390 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	391 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	392 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	393 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	394 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	395
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	396
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	397 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	398 """Get the factory for blake2b"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	399 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	400 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	401 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	402 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	403 return pyblake2.blake2b
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	404
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	405
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	406 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	407 """Get the factory for blake2s"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	408 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	409 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	410 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	411 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	412 return pyblake2.blake2s
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	413
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	414
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	415 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	416 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	417
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	418 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	419 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	420 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	421 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	422
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	423 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	424 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	425
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	426 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	427
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	428 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	429 return hashlib.blake2b(digest_size=32)
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	430
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	431 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	432
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	433
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	434 def argv2algo(s):
42 e5f6af8364db Typo in docs Franz Glasner <f.glasner@feldmann-mg.com> parents: 30 diff changeset	435 """Convert a command line algorithm specifier into a tuple with the
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	436 type/factory of the digest and the algorithms tag for output purposes.
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	437
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	438 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	439 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	440 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	441
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	442 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	443
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	444 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	445 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	446 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	447 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	448 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	449 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	450 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	451 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	452 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	453 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	454 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	455 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	456 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	457 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	458 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	459 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	460 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	461 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	462 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	463 return (hashlib.sha3_512, "SHA3-512")
86 fd1cfd1b0f9d Make "blake2" an alias of "blake2b" Franz Glasner <fzglas.hg@dom66.de> parents: 85 diff changeset	464 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	465 return (get_blake2b(), "BLAKE2b")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	466 elif s in ("blake2s", "blake2s-256"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	467 return (get_blake2s(), "BLAKE2s")
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	468 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	469 return (get_blake2_256(), "BLAKE2b-256")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	470 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	471 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	472 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	473 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	474 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	475
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	476
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	477 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	478 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	479 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	480
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	481 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	482 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	483
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	484 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	485
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	486 """
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	487 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	488 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	489 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	490 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	491 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	492 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	493 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	494 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	495 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	496 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	497 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	498 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	499 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	500 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	501 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	502 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	503 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	504 return hashlib.sha3_512
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	505 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	506 return get_blake2b()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	507 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	508 return get_blake2s()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	509 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	510 return get_blake2_256()
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	511 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	512 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	513 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	514 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	515
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	516
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	517 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	518 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	519 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	520
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	521 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	522 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	523 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	524 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	525 digest = binascii.hexlify(digest).decode("ascii")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	526 if filename is None:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	527 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	528 else:
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	529 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	530 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	531 digest),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	532 file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	533
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	534
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	535 def out_std(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	536 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	537
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	538 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	539 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	540 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	541 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	542 digest = binascii.hexlify(digest).decode("ascii")
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	543 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	544 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	545 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	546 '-' if filename is None else normalize_filename(filename)),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	547 file=dest)
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	548
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	549
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	550 def compute_digest_file(hashobj, path, use_mmap=None):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	551 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	552 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	553 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	554 binary mode (also a socket or pipe)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	555 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	556 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	557 :type use_mmap: bool or None
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	558 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	559 :rtype: bytes
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	560
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	561 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	562
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	563 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	564 h = hashobj()
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	565 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	566 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	567 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	568 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	569 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	570 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	571 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	572 own_fd = False
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	573 try:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	574 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	575 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	576 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	577 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	578 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	579 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	580 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	581 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	582 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	583 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	584 filesize = st[stat.ST_SIZE]
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	585 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	586 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	587 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	588 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	589 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	590 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	591 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	592 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	593 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	594 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	595 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	596 use_mmap = False
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	597 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	598 use_mmap = False
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	599 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	600 use_mmap = True
67 19893b4f42a5 Flag to disable the use of mmap Franz Glasner <fzglas.hg@dom66.de> parents: 66 diff changeset	601 if mmap is None or not use_mmap:
84 163de6dd6e05 Enhance comment and -- while being there -- fix a typo Franz Glasner <f.glasner@feldmann-mg.com> parents: 83 diff changeset	602 # No mmap available or wanted -> use traditional low-level file IO
94 8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	603 fadvise = getattr(os, "posix_fadvise", None)
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	604 if fadvise:
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	605 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
96 59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	606 if not PY2:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	607 fileobj = io.FileIO(fd, mode="r", closefd=False)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	608 buf = bytearray(READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	609 with memoryview(buf) as full_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	610 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	611 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	612 n = fileobj.readinto(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	613 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	614 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	615 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	616 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	617 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	618 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	619 if n == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	620 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	621 if n == READ_CHUNK_SIZE:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	622 h.update(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	623 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	624 with full_view[:n] as partial_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	625 h.update(partial_view)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	626 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	627 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	628 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	629 buf = os.read(fd, READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	630 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	631 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	632 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	633 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	634 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	635 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	636 if len(buf) == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	637 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	638 h.update(buf)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	639 else:
58 ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	640 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	641 # Use mmap
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	642 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	643 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	644 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	645 #
61 c9f9401abc0c Use getattr when trying to get mmap.madvise() Franz Glasner <fzglas.hg@dom66.de> parents: 60 diff changeset	646 madvise = getattr(mmap.mmap, "madvise", None)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	647 if filesize <= MAP_WINDOW_SIZE:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	648 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	649 else:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	650 mapsize = MAP_WINDOW_SIZE
57 0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap Franz Glasner <f.glasner@feldmann-mg.com> parents: 56 diff changeset	651 mapoffset = 0
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	652 rest = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	653 while rest > 0:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	654 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	655 mapsize,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	656 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	657 offset=mapoffset)
60 21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	658 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	659 madvise(m, mmap.MADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	660 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	661 h.update(m)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	662 finally:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	663 m.close()
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	664 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	665 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	666 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	667 mapsize = rest
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	668 finally:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	669 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	670 os.close(fd)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	671 return h.digest()
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	672
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	673
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	674 def compute_digest_stream(hashobj, instream):
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	675 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	676
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	677 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	678 :param instream: a bytes input stream to read the data to be hashed from
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	679 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	680 :rtype: bytes
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	681
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	682 """
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	683 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	684 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	685 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	686 buf = instream.read(READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	687 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	688 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	689 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	690 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	691 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	692 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	693 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	694 h.update(buf)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	695 return h.digest()
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	696
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	697
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	698 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	699 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	700 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	701 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	702 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	703 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	704
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	705
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	706 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	707 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 100:f95918115c6b