Python/apps/py-cutils: cutils/shasum.py annotate

annotate cutils/shasum.py @ 105:b0631f320efd

Implement "--follow-symlinks" to allow to control whether to follow directory symlinks

author	Franz Glasner <fzglas.hg@dom66.de>
date	Sun, 29 May 2022 01:06:25 +0200
parents	08fd0609fdd4
children	5fe6f63f0be7

rev	line source
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	1 # -- coding: utf-8 --
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	6 r"""Pure Python implementation of `shasum`
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
72 ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	10 from __future__ import print_function, absolute_import
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	11
73 c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	13 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	14
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	15
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import argparse
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	17 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	18 import binascii
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	19 import errno
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import hashlib
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	21 import io
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	22 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	23 import mmap
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	24 except ImportError:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	25 mmap = None
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	26 import os
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	27 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	28 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	29 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	30 pathlib = None
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	31 import re
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	32 import stat
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 import sys
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34
95 fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	35 from . import (__version__, __revision__)
fc2dd6afd594 Style: canonical global variable and import ordering Franz Glasner <f.glasner@feldmann-mg.com> parents: 94 diff changeset	36
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	39
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	47
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	52
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 fromfile_prefix_chars='@')
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	57 aparser.add_argument(
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	58 "--algorithm", "-a", action="store", type=argv2algo,
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	60 aparser.add_argument(
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	61 "--base64", action="store_true",
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	63 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	65 help="Read in binary mode (default)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	66 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	69 aparser.add_argument(
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	70 "--check", "-c", action="store_true",
17 184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages Franz Glasner <fzglas.hg@dom66.de> parents: 14 diff changeset	77 If no file is given, stdin is used.""")
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	78 aparser.add_argument(
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	82 generate an error.""")
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	88 aparser.add_argument(
105 b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	89 "--follow-symlinks", action="store_true", dest="follow_symlinks",
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	90 help="""Also follow symlinks that resolve to directories. Only effective if `--recurse` is activated.""")
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	91
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	92 aparser.add_argument(
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	93 "--mmap", action="store_true", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	94 help="""Use mmap if available. Default is to determine automatically
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	95 from the filesize.""")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	96 aparser.add_argument(
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	97 "--no-mmap", action="store_false", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	98 help="Dont use mmap.")
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	99
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	100 aparser.add_argument(
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	101 "--recurse", action="store_true",
105 b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	102 help="Recurse into sub-directories while interpreting every FILE as a directory.")
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	103
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	104 aparser.add_argument(
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	105 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	106 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	107 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	108 "--tag", action="store_true", dest="bsd", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	109 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	110 aparser.add_argument(
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	111 "--text", "-t", action="store_true", dest="text_mode", default=False,
69 a23371a8780f Writing style: Begin all help messages with an uppercase letter Franz Glasner <fzglas.hg@dom66.de> parents: 68 diff changeset	112 help="Read in text mode (not supported)")
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	113 aparser.add_argument(
27 c29bd96dcd4f Put the HG revision into the version output also Franz Glasner <fzglas.hg@dom66.de> parents: 26 diff changeset	114 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13 db64e282b049 Implement a version option Franz Glasner <fzglas.hg@dom66.de> parents: 12 diff changeset	115 aparser.add_argument(
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	116 "files", nargs="*", metavar="FILE")
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	117
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	118 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	119
3 5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	120 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode Franz Glasner <fzglas.hg@dom66.de> parents: 2 diff changeset	121 print("ERROR: text mode not supported", file=sys.stderr)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	122 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	123
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	124 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	125 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	126 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	127 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	128
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	129 if not opts.algorithm:
8 048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum Franz Glasner <fzglas.hg@dom66.de> parents: 7 diff changeset	130 opts.algorithm = argv2algo("1")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	131
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	132 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	133
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	134 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	135
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	136
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	137 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	138 checklist=False, check=False, dest=None, base64=False,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	139 allow_distinfo=False, mmap=None):
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	140 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	141 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	142 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	143 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	144 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	145 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	146 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	147 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	148 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	149 check=check,
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	150 text_mode=False,
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	151 dest=dest,
100 f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	152 base64=base64,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	153 allow_distinfo=allow_distinfo,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also Franz Glasner <fzglas.hg@dom66.de> parents: 96 diff changeset	154 mmap=mmap)
45 b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	155 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	156
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	157
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	158 def shasum(opts):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	159 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	160 return verify_digests_from_files(opts)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	161 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	162 return verify_digests_with_checklist(opts)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	163 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	164 return generate_digests(opts)
10 77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	165
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	166
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()". Franz Glasner <fzglas.hg@dom66.de> parents: 9 diff changeset	167 def generate_digests(opts):
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	168 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	169 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	170 else:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	171 out = out_std
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	172 if opts.recurse:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	173 if not opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	174 opts.files.append(".")
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	175 for dn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	176 if not os.path.isdir(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	177 if os.path.exists(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	178 raise OSError(errno.ENOTDIR, "not a directory", dn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	179 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	180 raise OSError(errno.ENOENT, "directory does not exist", dn)
105 b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	181 for dirpath, dirnames, dirfiles in os.walk(
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks Franz Glasner <fzglas.hg@dom66.de> parents: 104 diff changeset	182 dn, followlinks=opts.follow_symlinks):
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	183 for fn in dirfiles:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	184 path = os.path.join(dirpath, fn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	185 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	186 compute_digest_file(opts.algorithm[0], path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	187 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	188 path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	189 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	190 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	191 opts.base64)
4 67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum Franz Glasner <fzglas.hg@dom66.de> parents: 3 diff changeset	192 else:
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	193 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	194 if PY2:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	195 if sys.platform == "win32":
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	196 import msvcrt # noqa: E401
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	197 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	198 source = sys.stdin
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	199 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	200 source = sys.stdin.buffer
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	201 out(sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	202 compute_digest_stream(opts.algorithm[0], source),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	203 None,
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	204 opts.algorithm[1],
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	205 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	206 opts.base64)
104 08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	207 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	208 for fn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	209 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	210 compute_digest_file(opts.algorithm[0], fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	211 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	212 fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	213 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	214 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories Franz Glasner <fzglas.hg@dom66.de> parents: 100 diff changeset	215 opts.base64)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	216 return 0
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	217
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	218
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	219 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	220 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	221 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	222
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	223 :param bytes given_digest:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	224 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	225 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	226 :type expected_digest: str or bytes or bytearray
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	227 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	228 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	229 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	230
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	231 """
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	232 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	233 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	234 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	235 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	236 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	237 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	238 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	239 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	240 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	241 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	242 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	243 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	244 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	245 else:
71 29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	246 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	247 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	248 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=\|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	249 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	250 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	251 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	252 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	253 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also Franz Glasner <fzglas.hg@dom66.de> parents: 70 diff changeset	254 else:
52 5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests Franz Glasner <f.glasner@feldmann-mg.com> parents: 51 diff changeset	255 return False
75 a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	256 return given_digest == exd
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	257
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	258
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	259 def verify_digests_with_checklist(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	260 dest = opts.dest or sys.stdout
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	261 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	262 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	263 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	264 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	265 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	266 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	267 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	268 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	269 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	270 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	271 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	272 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	273 print("-: MISSING", file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	274 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	275 tag, algo, cl_filename, cl_digest = pl
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	276 computed_digest = compute_digest_stream(algo, source)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	277 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	278 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	279 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	280 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	281 exit_code = 1
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	282 print("{}: {}: {}".format(tag, "-", res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	283 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	284 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	285 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	286 if pl is None:
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	287 print("{}: MISSING".format(fn), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	288 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	289 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	290 tag, algo, cl_filename, cl_digest = pl
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	291 computed_digest = compute_digest_file(algo, fn,
42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	292 use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	293 if compare_digests_equal(computed_digest, cl_digest, algo):
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	294 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	295 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	296 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	297 res = "FAILED"
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	298 print("{}: {}: {}".format(tag, fn, res), file=dest)
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	299 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	300
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	301
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	302 def verify_digests_from_files(opts):
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	303 dest = opts.dest or sys.stdout
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	304 exit_code = 0
20 8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list Franz Glasner <fzglas.hg@dom66.de> parents: 19 diff changeset	305 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	306 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	307 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	308 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	309 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	310 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	311 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	312 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	313 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	314 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	315 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	316 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	317 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	318 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	319 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	320 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	321 continue
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	322 r, fn, tag = handle_checkline(opts, checkline)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	323 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	324 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	325 continue
47 5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects Franz Glasner <fzglas.hg@dom66.de> parents: 45 diff changeset	326 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	327 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	328 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	329 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	330
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	331
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	332 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	333 """
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	334 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	335 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	336 :rtype: tuple(str, str, str)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	337
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	338 """
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	339 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	340 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	341 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	342 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	343 tag, algo, fn, digest = parts
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	344 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	345 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	346 return (None, None, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	347 try:
90 42419f57eda9 Allow to control the use of mmap from the command-line Franz Glasner <fzglas.hg@dom66.de> parents: 89 diff changeset	348 d = compute_digest_file(algo, fn, use_mmap=opts.mmap)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	349 if compare_digests_equal(d, digest, algo):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	350 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	351 else:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	352 return ("failed", fn, tag)
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	353 except EnvironmentError:
18 285848db0b52 When verifying/checking digests: also print the digest tag used Franz Glasner <fzglas.hg@dom66.de> parents: 17 diff changeset	354 return ("missing", fn, tag)
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	355
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	356
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	357 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	358 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	359 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	360 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	361 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	362 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	363 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	364 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	365 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	366 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	367 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	368 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	369 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	370 "improperly formatted digest line: {}".format(checkline))
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	371 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	372 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	373 continue
22 6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	374 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	375 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	376 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	377 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	378 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	379
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum Franz Glasner <fzglas.hg@dom66.de> parents: 21 diff changeset	380
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	381 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	382 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	383
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	384 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	385 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	386 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	387
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	388 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	389 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	390 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	391 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	392
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	393 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	394
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	395 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	396 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	397 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	398 mo = re.search(r"\A(\S+)\s\((.)\)\s=\s(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	399 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	400 # (tag, algorithm, filename, digest)
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	401 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	402 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	403 return ("SIZE", None, None, mo.group(3))
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	404 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	405 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	406 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	407 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	408 else:
83 05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	409 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	410 mo = re.search(r"\ATIMESTAMP\s=\s([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	411 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	412 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles. Franz Glasner <f.glasner@feldmann-mg.com> parents: 79 diff changeset	413
21 f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	414 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	415 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	416 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	417 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	418 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	419 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	420 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	421 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	422 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	423 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	424
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion Franz Glasner <fzglas.hg@dom66.de> parents: 20 diff changeset	425
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	426 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	427 """Get the factory for blake2b"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	428 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	429 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	430 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	431 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	432 return pyblake2.blake2b
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	433
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	434
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	435 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	436 """Get the factory for blake2s"""
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	437 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	438 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	439 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	440 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	441 return pyblake2.blake2s
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	442
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	443
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	444 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	445 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	446
88 f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	447 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	448 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	449 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	450 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	451
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	452 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	453 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	454
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	455 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	456
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	457 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib. Franz Glasner <fzglas.hg@dom66.de> parents: 87 diff changeset	458 return hashlib.blake2b(digest_size=32)
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	459
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	460 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	461
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	462
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	463 def argv2algo(s):
42 e5f6af8364db Typo in docs Franz Glasner <f.glasner@feldmann-mg.com> parents: 30 diff changeset	464 """Convert a command line algorithm specifier into a tuple with the
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	465 type/factory of the digest and the algorithms tag for output purposes.
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	466
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	467 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	468 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	469 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	470
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	471 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	472
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	473 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	474 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	475 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	476 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	477 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	478 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	479 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	480 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	481 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	482 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	483 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	484 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	485 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	486 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	487 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	488 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	489 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	490 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	491 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	492 return (hashlib.sha3_512, "SHA3-512")
86 fd1cfd1b0f9d Make "blake2" an alias of "blake2b" Franz Glasner <fzglas.hg@dom66.de> parents: 85 diff changeset	493 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	494 return (get_blake2b(), "BLAKE2b")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	495 elif s in ("blake2s", "blake2s-256"):
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	496 return (get_blake2s(), "BLAKE2s")
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	497 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	498 return (get_blake2_256(), "BLAKE2b-256")
7 47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	499 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	500 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	501 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	502 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	503 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	504
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`. Franz Glasner <fzglas.hg@dom66.de> parents: 6 diff changeset	505
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	506 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	507 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	508 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	509
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	510 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	511 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	513 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	514
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	515 """
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	516 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	517 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	518 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	519 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	520 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	521 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	522 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	523 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	524 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	525 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	526 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	527 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	528 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	529 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	530 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	531 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	532 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	533 return hashlib.sha3_512
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	534 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	535 return get_blake2b()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	536 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85 d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256 Franz Glasner <fzglas.hg@dom66.de> parents: 84 diff changeset	537 return get_blake2s()
87 b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	538 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256). Franz Glasner <fzglas.hg@dom66.de> parents: 86 diff changeset	539 return get_blake2_256()
12 5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	540 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	541 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	542 else:
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	543 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	544
5e2c9123f93f Implemented digest verification: -c or --check option Franz Glasner <fzglas.hg@dom66.de> parents: 11 diff changeset	545
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	546 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	547 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	548 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	549
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	550 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	551 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	552 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	553 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	554 digest = binascii.hexlify(digest).decode("ascii")
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	555 if filename is None:
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	556 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	557 else:
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	558 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	559 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	560 digest),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	561 file=dest)
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	562
bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	563
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	564 def out_std(dest, digest, filename, digestname, binary, use_base64):
9 81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	565 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	566
81f28bf89c26 Some more output selection options and documentation Franz Glasner <fzglas.hg@dom66.de> parents: 8 diff changeset	567 """
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	568 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	569 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	570 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	571 digest = binascii.hexlify(digest).decode("ascii")
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	572 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	573 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	574 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	575 '-' if filename is None else normalize_filename(filename)),
5 bbcb225640de Handle standard and BSD-style output formats Franz Glasner <fzglas.hg@dom66.de> parents: 4 diff changeset	576 file=dest)
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	577
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	578
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	579 def compute_digest_file(hashobj, path, use_mmap=None):
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	580 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	581 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	582 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	583 binary mode (also a socket or pipe)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	584 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	585 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	586 :type use_mmap: bool or None
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	587 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	588 :rtype: bytes
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	589
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	590 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	591
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	592 """
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	593 h = hashobj()
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	594 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	595 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	596 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	597 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	598 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	599 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	600 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	601 own_fd = False
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	602 try:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	603 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	604 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	605 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	606 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	607 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	608 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	609 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	610 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	611 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	612 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	613 filesize = st[stat.ST_SIZE]
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	614 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	615 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	616 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	617 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	618 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	619 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	620 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	621 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	622 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	623 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	624 #
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	625 use_mmap = False
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	626 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	627 use_mmap = False
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	628 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	629 use_mmap = True
67 19893b4f42a5 Flag to disable the use of mmap Franz Glasner <fzglas.hg@dom66.de> parents: 66 diff changeset	630 if mmap is None or not use_mmap:
84 163de6dd6e05 Enhance comment and -- while being there -- fix a typo Franz Glasner <f.glasner@feldmann-mg.com> parents: 83 diff changeset	631 # No mmap available or wanted -> use traditional low-level file IO
94 8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	632 fadvise = getattr(os, "posix_fadvise", None)
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	633 if fadvise:
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available Franz Glasner <f.glasner@feldmann-mg.com> parents: 90 diff changeset	634 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
96 59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	635 if not PY2:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	636 fileobj = io.FileIO(fd, mode="r", closefd=False)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	637 buf = bytearray(READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	638 with memoryview(buf) as full_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	639 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	640 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	641 n = fileobj.readinto(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	642 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	643 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	644 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	645 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	646 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	647 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	648 if n == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	649 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	650 if n == READ_CHUNK_SIZE:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	651 h.update(buf)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	652 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	653 with full_view[:n] as partial_view:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	654 h.update(partial_view)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	655 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	656 while True:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	657 try:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	658 buf = os.read(fd, READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	659 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	660 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	661 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	662 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	663 raise
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	664 else:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	665 if len(buf) == 0:
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	666 break
59253a1e6ef2 Employ .readinto() on Python 3 Franz Glasner <f.glasner@feldmann-mg.com> parents: 95 diff changeset	667 h.update(buf)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	668 else:
58 ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	669 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	670 # Use mmap
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	671 #
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	672 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	673 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments Franz Glasner <fzglas.hg@dom66.de> parents: 57 diff changeset	674 #
61 c9f9401abc0c Use getattr when trying to get mmap.madvise() Franz Glasner <fzglas.hg@dom66.de> parents: 60 diff changeset	675 madvise = getattr(mmap.mmap, "madvise", None)
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	676 if filesize <= MAP_WINDOW_SIZE:
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	677 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	678 else:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	679 mapsize = MAP_WINDOW_SIZE
57 0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap Franz Glasner <f.glasner@feldmann-mg.com> parents: 56 diff changeset	680 mapoffset = 0
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	681 rest = filesize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	682 while rest > 0:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	683 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	684 mapsize,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	685 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	686 offset=mapoffset)
60 21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	687 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file Franz Glasner <fzglas.hg@dom66.de> parents: 59 diff changeset	688 madvise(m, mmap.MADV_SEQUENTIAL)
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	689 try:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	690 h.update(m)
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	691 finally:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	692 m.close()
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	693 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	694 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	695 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	696 mapsize = rest
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	697 finally:
68 4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	698 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor. Franz Glasner <fzglas.hg@dom66.de> parents: 67 diff changeset	699 os.close(fd)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	700 return h.digest()
23 232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	701
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	702
232063b73e45 Optimized reading of files by using mmap. Franz Glasner <fzglas.hg@dom66.de> parents: 22 diff changeset	703 def compute_digest_stream(hashobj, instream):
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	704 """
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	705
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	706 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	707 :param instream: a bytes input stream to read the data to be hashed from
56 6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	708 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions Franz Glasner <f.glasner@feldmann-mg.com> parents: 52 diff changeset	709 :rtype: bytes
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	710
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	711 """
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	712 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	713 while True:
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	714 try:
89 72684020f2f3 By default use mmap only for files up to 8MiB in size. Franz Glasner <fzglas.hg@dom66.de> parents: 88 diff changeset	715 buf = instream.read(READ_CHUNK_SIZE)
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	716 except OSError as e:
79 8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK Franz Glasner <fzglas.hg@dom66.de> parents: 75 diff changeset	717 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66 c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	718 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	719 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	720 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	721 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	722 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files Franz Glasner <fzglas.hg@dom66.de> parents: 61 diff changeset	723 h.update(buf)
51 58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal Franz Glasner <f.glasner@feldmann-mg.com> parents: 47 diff changeset	724 return h.digest()
2 5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	725
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows Franz Glasner <fzglas.hg@dom66.de> parents: 1 diff changeset	726
19 2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	727 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	728 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	729 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	730 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	731 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	732 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	733
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests Franz Glasner <fzglas.hg@dom66.de> parents: 18 diff changeset	734
1 bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	735 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	736 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 105:b0631f320efd