Python/apps/py-cutils: cutils/shasum.py annotate

annotate cutils/shasum.py @ 104:08fd0609fdd4

Implement "--recurse" for shasum: recurse into directories

author	Franz Glasner <fzglas.hg@dom66.de>
date	Thu, 26 May 2022 23:14:38 +0200
parents	f95918115c6b
children	b0631f320efd

rev	line source
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	1 # -- coding: utf-8 --
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	6 r"""Pure Python implementation of `shasum`
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
72 ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	10 from __future__ import print_function, absolute_import
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	11
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	13 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	14
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	15
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import argparse
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	17 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	18 import binascii
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	19 import errno
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import hashlib
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	21 import io
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	22 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	23 import mmap
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	24 except ImportError:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	25 mmap = None
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	26 import os
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	27 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	28 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	29 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	30 pathlib = None
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	31 import re
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	32 import stat
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 import sys
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34
95 fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	35 from . import (__version__, __revision__)
fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	36
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	39
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	47
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	52
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 fromfile_prefix_chars='@')
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	57 aparser.add_argument(
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	58 "--algorithm", "-a", action="store", type=argv2algo,
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	60 aparser.add_argument(
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	61 "--base64", action="store_true",
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	63 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	65 help="Read in binary mode (default)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	66 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	69 aparser.add_argument(
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	70 "--check", "-c", action="store_true",
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	77 If no file is given, stdin is used.""")
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	78 aparser.add_argument(
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	82 generate an error.""")
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	88 aparser.add_argument(
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	89 "--mmap", action="store_true", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	90 help="""Use mmap if available. Default is to determine automatically
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	91 from the filesize.""")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	92 aparser.add_argument(
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	93 "--no-mmap", action="store_false", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	94 help="Dont use mmap.")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	95
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	96 aparser.add_argument(
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	97 "--recurse", action="store_true",
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	98 help="Recurse into sub-directories while interpreting every FILE as a directory")
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	99
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	100 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	101 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	102 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	103 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	104 "--tag", action="store_true", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	105 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	106 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	107 "--text", "-t", action="store_true", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	108 help="Read in text mode (not supported)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	109 aparser.add_argument(
27 c29bd96dcd4f Put the HG revision into the version output also Franz Glasner <fzglas.hg@dom66.de> parents: 26 diff changeset	110 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	111 aparser.add_argument(
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	112 "files", nargs="*", metavar="FILE")
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	113
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	114 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	115
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	116 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	117 print("ERROR: text mode not supported", file=sys.stderr)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	118 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	119
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	120 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	121 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	122 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	123 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	124
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	125 if not opts.algorithm:
8 048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum Franz Glasner <fzglas.hg@dom66.de> parents: 7 diff changeset	126 opts.algorithm = argv2algo("1")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	127
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	128 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	129
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	130 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	131
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	132
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	133 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	134 checklist=False, check=False, dest=None, base64=False,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	135 allow_distinfo=False, mmap=None):
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	136 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	137 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	138 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	139 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	140 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	141 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	142 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	143 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	144 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	145 check=check,
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	146 text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	147 dest=dest,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	148 base64=base64,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	149 allow_distinfo=allow_distinfo,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	150 mmap=mmap)
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	151 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	152
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	153
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	154 def shasum(opts):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	155 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	156 return verify_digests_from_files(opts)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	157 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	158 return verify_digests_with_checklist(opts)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	159 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	160 return generate_digests(opts)
10 77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	161
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	162
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	163 def generate_digests(opts):
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	164 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	165 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	166 else:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	167 out = out_std
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	168 if opts.recurse:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	169 if not opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	170 opts.files.append(".")
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	171 for dn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	172 if not os.path.isdir(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	173 if os.path.exists(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	174 raise OSError(errno.ENOTDIR, "not a directory", dn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	175 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	176 raise OSError(errno.ENOENT, "directory does not exist", dn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	177 for dirpath, dirnames, dirfiles in os.walk(dn, followlinks=True):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	178 for fn in dirfiles:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	179 path = os.path.join(dirpath, fn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	180 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	181 compute_digest_file(opts.algorithm[0], path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	182 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	183 path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	184 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	185 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	186 opts.base64)
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	187 else:
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	188 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	189 if PY2:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	190 if sys.platform == "win32":
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	191 import msvcrt # noqa: E401
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	192 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	193 source = sys.stdin
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	194 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	195 source = sys.stdin.buffer
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	196 out(sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	197 compute_digest_stream(opts.algorithm[0], source),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	198 None,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	199 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	200 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	201 opts.base64)
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	202 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	203 for fn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	204 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	205 compute_digest_file(opts.algorithm[0], fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	206 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	207 fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	208 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	209 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	210 opts.base64)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	211 return 0
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	212
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	213
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	214 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	215 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	216 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	217
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	218 :param bytes given_digest:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	219 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	220 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	221 :type expected_digest: str or bytes or bytearray
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	222 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	223 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	224 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	225
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	226 """
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	227 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	228 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	229 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	230 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	231 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	232 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	233 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	234 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	235 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	236 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	237 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	238 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	239 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	240 else:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	241 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	242 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	243 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	244 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	245 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	246 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	247 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	248 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	249 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	250 return False
75 a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	251 return given_digest == exd
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	252
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	253
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	254 def verify_digests_with_checklist(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	255 dest = opts.dest or sys.stdout
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	256 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	257 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	258 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	259 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	260 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	261 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	262 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	263 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	264 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	265 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	266 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	267 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	268 print("-: MISSING", file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	269 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	270 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	271 computed_digest = compute_digest_stream(algo, source)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	272 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	273 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	274 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	275 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	276 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	277 print("{}: {}: {}".format(tag, "-", res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	278 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	279 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	280 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	281 if pl is None:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	282 print("{}: MISSING".format(fn), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	283 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	284 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	285 tag, algo, cl_filename, cl_digest = pl
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	286 computed_digest = compute_digest_file(algo, fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	287 use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	288 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	289 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	290 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	291 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	292 res = "FAILED"
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	293 print("{}: {}: {}".format(tag, fn, res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	294 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	295
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	296
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	297 def verify_digests_from_files(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	298 dest = opts.dest or sys.stdout
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	299 exit_code = 0
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	300 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	301 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	302 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	303 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	304 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	305 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	306 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	307 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	308 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	309 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	310 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	311 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	312 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	313 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	314 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	315 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	316 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	317 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	318 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	319 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	320 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	321 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	322 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	323 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	324 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	325
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	326
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	327 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	328 """
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	329 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	330 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	331 :rtype: tuple(str, str, str)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	332
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	333 """
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	334 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	335 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	336 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	337 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	338 tag, algo, fn, digest = parts
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	339 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	340 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	341 return (None, None, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	342 try:
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	343 d = compute_digest_file(algo, fn, use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	344 if compare_digests_equal(d, digest, algo):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	345 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	346 else:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	347 return ("failed", fn, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	348 except EnvironmentError:
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	349 return ("missing", fn, tag)
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	350
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	351
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	352 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	353 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	354 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	355 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	356 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	357 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	358 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	359 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	360 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	361 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	362 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	363 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	364 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	365 "improperly formatted digest line: {}".format(checkline))
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	366 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	367 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	368 continue
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	369 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	370 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	371 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	372 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	373 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	374
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	375
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	376 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	377 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	378
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	379 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	380 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	381 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	382
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	383 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	384 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	385 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	386 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	387
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	388 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	389
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	390 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	391 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	392 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	393 mo = re.search(r"\A(\S+)\s\((.)\)\s=\s(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	394 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	395 # (tag, algorithm, filename, digest)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	396 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	397 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	398 return ("SIZE", None, None, mo.group(3))
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	399 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	400 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	401 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	402 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	403 else:
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	404 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	405 mo = re.search(r"\ATIMESTAMP\s=\s([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	406 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	407 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	408
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	409 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	410 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	411 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	412 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	413 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	414 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	415 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	416 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	417 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	418 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	419
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	420
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	421 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	422 """Get the factory for blake2b"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	423 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	424 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	425 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	426 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	427 return pyblake2.blake2b
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	428
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	429
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	430 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	431 """Get the factory for blake2s"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	432 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	433 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	434 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	435 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	436 return pyblake2.blake2s
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	437
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	438
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	439 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	440 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	441
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	442 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	443 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	444 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	445 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	446
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	447 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	448 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	449
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	450 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	451
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	452 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	453 return hashlib.blake2b(digest_size=32)
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	454
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	455 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	456
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	457
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	458 def argv2algo(s):
42 e5f6af8364db Typo in docs Franz Glasner <f.glasner@feldmann-mg.com> parents: 30 diff changeset	459 """Convert a command line algorithm specifier into a tuple with the
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	460 type/factory of the digest and the algorithms tag for output purposes.
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	461
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	462 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	463 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	464 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	465
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	466 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	467
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	468 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	469 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	470 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	471 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	472 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	473 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	474 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	475 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	476 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	477 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	478 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	479 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	480 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	481 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	482 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	483 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	484 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	485 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	486 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	487 return (hashlib.sha3_512, "SHA3-512")
86 fd1cfd1b0f9d Make "blake2" an alias of "blake2b" Franz Glasner <fzglas.hg@dom66.de> parents: 85 diff changeset	488 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	489 return (get_blake2b(), "BLAKE2b")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	490 elif s in ("blake2s", "blake2s-256"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	491 return (get_blake2s(), "BLAKE2s")
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	492 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	493 return (get_blake2_256(), "BLAKE2b-256")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	494 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	495 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	496 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	497 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	498 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	499
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	500
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	501 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	502 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	503 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	504
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	505 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	506 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	507
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	508 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	509
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	510 """
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	511 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	512 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	513 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	514 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	515 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	516 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	517 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	518 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	519 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	520 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	521 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	522 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	523 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	524 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	525 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	526 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	527 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	528 return hashlib.sha3_512
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	529 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	530 return get_blake2b()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	531 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	532 return get_blake2s()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	533 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	534 return get_blake2_256()
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	535 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	536 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	537 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	538 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	539
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	540
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	541 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	542 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	543 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	544
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	545 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	546 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	547 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	548 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	549 digest = binascii.hexlify(digest).decode("ascii")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	550 if filename is None:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	551 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	552 else:
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	553 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	554 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	555 digest),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	556 file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	557
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	558
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	559 def out_std(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	560 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	561
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	562 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	563 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	564 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	565 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	566 digest = binascii.hexlify(digest).decode("ascii")
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	567 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	568 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	569 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	570 '-' if filename is None else normalize_filename(filename)),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	571 file=dest)
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	572
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	573
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	574 def compute_digest_file(hashobj, path, use_mmap=None):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	575 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	576 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	577 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	578 binary mode (also a socket or pipe)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	579 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	580 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	581 :type use_mmap: bool or None
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	582 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	583 :rtype: bytes
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	584
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	585 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	586
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	587 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	588 h = hashobj()
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	589 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	590 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	591 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	592 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	593 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	594 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	595 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	596 own_fd = False
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	597 try:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	598 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	599 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	600 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	601 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	602 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	603 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	604 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	605 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	606 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	607 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	608 filesize = st[stat.ST_SIZE]
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	609 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	610 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	611 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	612 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	613 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	614 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	615 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	616 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	617 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	618 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	619 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	620 use_mmap = False
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	621 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	622 use_mmap = False
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	623 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	624 use_mmap = True
67 19893b4f42a5 Flag to disable the use of mmap Franz Glasner <fzglas.hg@dom66.de> parents: 66 diff changeset	625 if mmap is None or not use_mmap:
84 163de6dd6e05 Enhance comment and -- while being there -- fix a typo Franz Glasner <f.glasner@feldmann-mg.com> parents: 83 diff changeset	626 # No mmap available or wanted -> use traditional low-level file IO
94 8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	627 fadvise = getattr(os, "posix_fadvise", None)
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	628 if fadvise:
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	629 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
96 59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	630 if not PY2:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	631 fileobj = io.FileIO(fd, mode="r", closefd=False)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	632 buf = bytearray(READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	633 with memoryview(buf) as full_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	634 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	635 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	636 n = fileobj.readinto(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	637 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	638 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	639 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	640 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	641 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	642 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	643 if n == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	644 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	645 if n == READ_CHUNK_SIZE:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	646 h.update(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	647 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	648 with full_view[:n] as partial_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	649 h.update(partial_view)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	650 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	651 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	652 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	653 buf = os.read(fd, READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	654 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	655 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	656 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	657 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	658 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	659 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	660 if len(buf) == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	661 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	662 h.update(buf)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	663 else:
58 ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	664 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	665 # Use mmap
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	666 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	667 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	668 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	669 #
61 c9f9401abc0c Use getattr when trying to get mmap.madvise() Franz Glasner <fzglas.hg@dom66.de> parents: 60 diff changeset	670 madvise = getattr(mmap.mmap, "madvise", None)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	671 if filesize <= MAP_WINDOW_SIZE:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	672 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	673 else:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	674 mapsize = MAP_WINDOW_SIZE
57 0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap Franz Glasner <f.glasner@feldmann-mg.com> parents: 56 diff changeset	675 mapoffset = 0
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	676 rest = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	677 while rest > 0:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	678 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	679 mapsize,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	680 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	681 offset=mapoffset)
60 21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	682 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	683 madvise(m, mmap.MADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	684 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	685 h.update(m)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	686 finally:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	687 m.close()
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	688 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	689 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	690 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	691 mapsize = rest
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	692 finally:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	693 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	694 os.close(fd)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	695 return h.digest()
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	696
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	697
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	698 def compute_digest_stream(hashobj, instream):
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	699 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	700
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	701 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	702 :param instream: a bytes input stream to read the data to be hashed from
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	703 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	704 :rtype: bytes
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	705
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	706 """
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	707 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	708 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	709 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	710 buf = instream.read(READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	711 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	712 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	713 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	714 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	715 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	716 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	717 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	718 h.update(buf)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	719 return h.digest()
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	720
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	721
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	722 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	723 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	724 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	725 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	726 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	727 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	728
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	729
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	730 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	731 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 104:08fd0609fdd4