Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/shasum.py @ 106:5fe6f63f0be7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 30 May 2022 09:15:18 +0200
parents b0631f320efd
children 3060aa4cb252
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
73
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
1 # -*- coding: utf-8 -*-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
2 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
3 # :Copyright: (c) 2020-2022 Franz Glasner
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
4 # :License: BSD-3-Clause
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
5 # :-
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
6 r"""Pure Python implementation of `shasum`
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
72
ae2df602beb4 Make shasum.py and dos2unix sub-modules to the new "cutils" package
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
10 from __future__ import print_function, absolute_import
13
db64e282b049 Implement a version option
Franz Glasner <fzglas.hg@dom66.de>
parents: 12
diff changeset
11
73
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
12
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
13 __all__ = []
c3268f4e752f Adjust all license notes to (a) more literally comply with the BSD3 templates and to the style guide
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
14
13
db64e282b049 Implement a version option
Franz Glasner <fzglas.hg@dom66.de>
parents: 12
diff changeset
15
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 import argparse
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
17 import base64
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
18 import binascii
66
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
19 import errno
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20 import hashlib
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
21 import io
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
22 try:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
23 import mmap
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
24 except ImportError:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
25 mmap = None
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
26 import os
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
27 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
28 import pathlib
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
29 except ImportError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
30 pathlib = None
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
31 import re
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
32 import stat
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 import sys
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34
95
fc2dd6afd594 Style: canonical global variable and import ordering
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 94
diff changeset
35 from . import (__version__, __revision__)
fc2dd6afd594 Style: canonical global variable and import ordering
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 94
diff changeset
36
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
38 PY2 = sys.version_info[0] < 3
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
39
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
40 if PY2:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
41 PATH_TYPES = (unicode, str) # noqa: F821 (undefined name 'unicode')
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
42 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
43 if pathlib:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
44 PATH_TYPES = (str, bytes, pathlib.Path)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
45 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
46 PATH_TYPES = (str, bytes)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
47
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
48 READ_CHUNK_SIZE = 2 * 1024 * 1024 # like BUFSIZE_MAX on FreeBSD
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
49 MAX_AUTO_MAP_SIZE = 8 * 1024 * 1024
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
50 MAP_WINDOW_SIZE = MAX_AUTO_MAP_SIZE # do not totally trash memory on big files
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
51
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
52
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 def main(argv=None):
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54 aparser = argparse.ArgumentParser(
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 description="Python implementation of shasum",
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 fromfile_prefix_chars='@')
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
57 aparser.add_argument(
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
58 "--algorithm", "-a", action="store", type=argv2algo,
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
59 help="1 (default), 224, 256, 384, 512, 3-224, 3-256, 3-384, 3-512, blake2b, blake2s, blake2, blake2-256, md5")
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
60 aparser.add_argument(
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
61 "--base64", action="store_true",
69
a23371a8780f Writing style: Begin all help messages with an uppercase letter
Franz Glasner <fzglas.hg@dom66.de>
parents: 68
diff changeset
62 help="Output checksums in base64 notation, not hexadecimal (OpenBSD).")
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
63 aparser.add_argument(
3
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
64 "--binary", "-b", action="store_false", dest="text_mode", default=False,
69
a23371a8780f Writing style: Begin all help messages with an uppercase letter
Franz Glasner <fzglas.hg@dom66.de>
parents: 68
diff changeset
65 help="Read in binary mode (default)")
3
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
66 aparser.add_argument(
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
67 "--bsd", "-B", action="store_true", dest="bsd", default=False,
17
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
68 help="Write BSD style output. This is also the default output format of :command:`openssl dgst`.")
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
69 aparser.add_argument(
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
70 "--check", "-c", action="store_true",
17
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
71 help="""Read digests from FILEs and check them.
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
72 If this option is specified, the FILE options become checklists. Each
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
73 checklist should contain hash results in a supported format, which will
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
74 be verified against the specified paths. Output consists of the digest
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
75 used, the file name, and an OK, FAILED, or MISSING for the result of
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
76 the comparison. This will validate any of the supported checksums.
184ab1da1307 Extend help messages
Franz Glasner <fzglas.hg@dom66.de>
parents: 14
diff changeset
77 If no file is given, stdin is used.""")
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
78 aparser.add_argument(
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
79 "--checklist", "-C", metavar="CHECKLIST",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
80 help="""Compare the checksum of each FILE against the checksums in
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
81 the CHECKLIST. Any specified FILE that is not listed in the CHECKLIST will
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
82 generate an error.""")
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
83 aparser.add_argument(
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
84 "--checklist-allow-distinfo", action="store_true",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
85 dest="allow_distinfo",
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
86 help='Allow FreeBSD "distinfo" formatted checklists: ignore SIZE and TIMESTAMP lines.')
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
87
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
88 aparser.add_argument(
105
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
89 "--follow-symlinks", action="store_true", dest="follow_symlinks",
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
90 help="""Also follow symlinks that resolve to directories. Only effective if `--recurse` is activated.""")
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
91
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
92 aparser.add_argument(
90
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
93 "--mmap", action="store_true", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
94 help="""Use mmap if available. Default is to determine automatically
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
95 from the filesize.""")
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
96 aparser.add_argument(
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
97 "--no-mmap", action="store_false", dest="mmap", default=None,
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
98 help="Dont use mmap.")
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
99
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
100 aparser.add_argument(
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
101 "--recurse", action="store_true",
105
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
102 help="Recurse into sub-directories while interpreting every FILE as a directory.")
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
103
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
104 aparser.add_argument(
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
105 "--reverse", "-r", action="store_false", dest="bsd", default=False,
69
a23371a8780f Writing style: Begin all help messages with an uppercase letter
Franz Glasner <fzglas.hg@dom66.de>
parents: 68
diff changeset
106 help="Explicitely select normal coreutils style output (to be option compatible with BSD style commands and :command:`openssl dgst -r`)")
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
107 aparser.add_argument(
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
108 "--tag", action="store_true", dest="bsd", default=False,
69
a23371a8780f Writing style: Begin all help messages with an uppercase letter
Franz Glasner <fzglas.hg@dom66.de>
parents: 68
diff changeset
109 help="Alias for the `--bsd' option (to be compatible with :command:`b2sum`)")
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
110 aparser.add_argument(
3
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
111 "--text", "-t", action="store_true", dest="text_mode", default=False,
69
a23371a8780f Writing style: Begin all help messages with an uppercase letter
Franz Glasner <fzglas.hg@dom66.de>
parents: 68
diff changeset
112 help="Read in text mode (not supported)")
3
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
113 aparser.add_argument(
27
c29bd96dcd4f Put the HG revision into the version output also
Franz Glasner <fzglas.hg@dom66.de>
parents: 26
diff changeset
114 "--version", "-v", action="version", version="%s (rv:%s)" % (__version__, __revision__))
13
db64e282b049 Implement a version option
Franz Glasner <fzglas.hg@dom66.de>
parents: 12
diff changeset
115 aparser.add_argument(
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
116 "files", nargs="*", metavar="FILE")
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 opts = aparser.parse_args(args=argv)
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119
3
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
120 if opts.text_mode:
5a6ed622846c Add comman line switches for reading in binary and text mode
Franz Glasner <fzglas.hg@dom66.de>
parents: 2
diff changeset
121 print("ERROR: text mode not supported", file=sys.stderr)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
122 sys.exit(78) # :manpage:`sysexits(3)` EX_CONFIG
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
123
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
124 if opts.check and opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
125 print("ERROR: only one of --check or --checklist allowed",
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
126 file=sys.stderr)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
127 sys.exit(64) # :manpage:`sysexits(3)` EX_USAGE
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
128
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
129 if not opts.algorithm:
8
048b97213a23 Change the default algorithm to SHA1 to be compatible with Perl's shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 7
diff changeset
130 opts.algorithm = argv2algo("1")
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
131
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
132 opts.dest = None
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
133
45
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
134 return shasum(opts)
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
135
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
136
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
137 def gen_opts(files=[], algorithm="SHA1", bsd=False, text_mode=False,
100
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 96
diff changeset
138 checklist=False, check=False, dest=None, base64=False,
106
5fe6f63f0be7 Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
139 allow_distinfo=False, mmap=None, recurse=False,
5fe6f63f0be7 Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
140 follow_symlinks=False):
45
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
141 if text_mode:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
142 raise ValueError("text mode not supported")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
143 if checklist and check:
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
144 raise ValueError("only one of `checklist' or `check' is allowed")
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
145 opts = argparse.Namespace(files=files,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
146 algorithm=(algotag2algotype(algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
147 algorithm),
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
148 bsd=bsd,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
149 checklist=checklist,
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
150 check=check,
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
151 text_mode=False,
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
152 dest=dest,
100
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 96
diff changeset
153 base64=base64,
f95918115c6b FIX: Implement some new commandline flags in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 96
diff changeset
154 allow_distinfo=allow_distinfo,
106
5fe6f63f0be7 Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
155 mmap=mmap,
5fe6f63f0be7 Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
156 recurse=recurse,
5fe6f63f0be7 Implement "--recurse" and "--follow-symlinks" in "gen_opts()" also
Franz Glasner <fzglas.hg@dom66.de>
parents: 105
diff changeset
157 follow_symlinks=follow_symlinks)
45
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
158 return opts
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
159
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
160
b25ef7293bf2 Enhance shasum.py to allow it to be used as Python module from within other programs more easily
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
161 def shasum(opts):
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
162 if opts.check:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
163 return verify_digests_from_files(opts)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
164 elif opts.checklist:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
165 return verify_digests_with_checklist(opts)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
166 else:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
167 return generate_digests(opts)
10
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()".
Franz Glasner <fzglas.hg@dom66.de>
parents: 9
diff changeset
168
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()".
Franz Glasner <fzglas.hg@dom66.de>
parents: 9
diff changeset
169
77446cd3ea6f Move the digest generation loop from "main()" into an own function "generate_digests()".
Franz Glasner <fzglas.hg@dom66.de>
parents: 9
diff changeset
170 def generate_digests(opts):
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
171 if opts.bsd:
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
172 out = out_bsd
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
173 else:
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
174 out = out_std
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
175 if opts.recurse:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
176 if not opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
177 opts.files.append(".")
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
178 for dn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
179 if not os.path.isdir(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
180 if os.path.exists(dn):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
181 raise OSError(errno.ENOTDIR, "not a directory", dn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
182 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
183 raise OSError(errno.ENOENT, "directory does not exist", dn)
105
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
184 for dirpath, dirnames, dirfiles in os.walk(
b0631f320efd Implement "--follow-symlinks" to allow to control whether to follow directory symlinks
Franz Glasner <fzglas.hg@dom66.de>
parents: 104
diff changeset
185 dn, followlinks=opts.follow_symlinks):
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
186 for fn in dirfiles:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
187 path = os.path.join(dirpath, fn)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
188 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
189 compute_digest_file(opts.algorithm[0], path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
190 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
191 path,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
192 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
193 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
194 opts.base64)
4
67d10529ce88 FIX: "-" filename handling now consistent with Perl shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 3
diff changeset
195 else:
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
196 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
197 if PY2:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
198 if sys.platform == "win32":
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
199 import msvcrt # noqa: E401
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
200 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
201 source = sys.stdin
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
202 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
203 source = sys.stdin.buffer
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
204 out(sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
205 compute_digest_stream(opts.algorithm[0], source),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
206 None,
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
207 opts.algorithm[1],
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
208 True,
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
209 opts.base64)
104
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
210 else:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
211 for fn in opts.files:
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
212 out(opts.dest or sys.stdout,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
213 compute_digest_file(opts.algorithm[0], fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
214 use_mmap=opts.mmap),
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
215 fn,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
216 opts.algorithm[1],
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
217 True,
08fd0609fdd4 Implement "--recurse" for shasum: recurse into directories
Franz Glasner <fzglas.hg@dom66.de>
parents: 100
diff changeset
218 opts.base64)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
219 return 0
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
220
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
221
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
222 def compare_digests_equal(given_digest, expected_digest, algo):
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
223 """Compare a newly computed binary digest `given_digest` with a digest
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
224 string (hex or base64) in `expected_digest`.
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
225
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
226 :param bytes given_digest:
71
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
227 :param expected_digest: digest (as bytes) or hexlified or base64 encoded
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
228 digest (as str)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
229 :type expected_digest: str or bytes or bytearray
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
230 :param algo: The algorithm (factory)
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
231 :return: `True` if the digests are equal, `False` if not
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
232 :rtype: bool
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
233
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
234 """
71
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
235 if isinstance(expected_digest, (bytes, bytearray)) \
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
236 and len(expected_digest) == algo().digest_size:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
237 exd = expected_digest
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
238 else:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
239 if len(expected_digest) == algo().digest_size * 2:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
240 # hex
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
241 if re.search(r"\A[a-fA-F0-9]+\Z", expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
242 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
243 exd = binascii.unhexlify(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
244 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
245 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
246 else:
52
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 51
diff changeset
247 return False
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 51
diff changeset
248 else:
71
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
249 # base64
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
250 if re.search(
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
251 r"\A(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?\Z",
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
252 expected_digest):
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
253 try:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
254 exd = base64.b64decode(expected_digest)
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
255 except TypeError:
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
256 return False
29fb33aa639a "compare_digests_equal()" now accepts binary (aka un-encoded) expected digests also
Franz Glasner <fzglas.hg@dom66.de>
parents: 70
diff changeset
257 else:
52
5935055edea6 More proper formal checks (with regexp) for valid hex and base64 encoding of digests
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 51
diff changeset
258 return False
75
a31de3c65877 Remove the use of "hmac.compare_digest()": there are no secrets to protect here
Franz Glasner <fzglas.hg@dom66.de>
parents: 73
diff changeset
259 return given_digest == exd
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
260
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
261
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
262 def verify_digests_with_checklist(opts):
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
263 dest = opts.dest or sys.stdout
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
264 exit_code = 0
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
265 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
266 if PY2:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
267 if sys.platform == "win32":
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
268 import os, msvcrt # noqa: E401
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
269 msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
270 source = sys.stdin
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
271 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
272 source = sys.stdin.buffer
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
273 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, None)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
274 if pl is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
275 exit_code = 1
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
276 print("-: MISSING", file=dest)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
277 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
278 tag, algo, cl_filename, cl_digest = pl
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
279 computed_digest = compute_digest_stream(algo, source)
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
280 if compare_digests_equal(computed_digest, cl_digest, algo):
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
281 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
282 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
283 res = "FAILED"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
284 exit_code = 1
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
285 print("{}: {}: {}".format(tag, "-", res), file=dest)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
286 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
287 for fn in opts.files:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
288 pl = get_parsed_digest_line_from_checklist(opts.checklist, opts, fn)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
289 if pl is None:
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
290 print("{}: MISSING".format(fn), file=dest)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
291 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
292 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
293 tag, algo, cl_filename, cl_digest = pl
90
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
294 computed_digest = compute_digest_file(algo, fn,
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
295 use_mmap=opts.mmap)
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
296 if compare_digests_equal(computed_digest, cl_digest, algo):
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
297 res = "OK"
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
298 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
299 exit_code = 1
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
300 res = "FAILED"
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
301 print("{}: {}: {}".format(tag, fn, res), file=dest)
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
302 return exit_code
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
303
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
304
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
305 def verify_digests_from_files(opts):
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
306 dest = opts.dest or sys.stdout
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
307 exit_code = 0
20
8f0241ed4a00 Do not append "-" to an empty FILE list any more but check explicitely for an empty list
Franz Glasner <fzglas.hg@dom66.de>
parents: 19
diff changeset
308 if not opts.files or (len(opts.files) == 1 and opts.files[0] == '-'):
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
309 for checkline in sys.stdin:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
310 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
311 continue
18
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
312 r, fn, tag = handle_checkline(opts, checkline)
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
313 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
314 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
315 continue
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
316 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
317 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
318 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
319 else:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
320 for fn in opts.files:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
321 with io.open(fn, "rt", encoding="utf-8") as checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
322 for checkline in checkfile:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
323 if not checkline:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
324 continue
18
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
325 r, fn, tag = handle_checkline(opts, checkline)
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
326 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
327 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
328 continue
47
5bec7a5d894a Allow internal output redirection: print() always to explicitely given file objects
Franz Glasner <fzglas.hg@dom66.de>
parents: 45
diff changeset
329 print("{}: {}: {}".format(tag, fn, r.upper()), file=dest)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
330 if r != "ok" and exit_code == 0:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
331 exit_code = 1
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
332 return exit_code
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
333
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
334
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
335 def handle_checkline(opts, line):
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
336 """
18
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
337 :return: a tuple with static "ok", "missing", or "failed", the filename and
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
338 the digest used
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
339 :rtype: tuple(str, str, str)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
340
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
341 """
21
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
342 parts = parse_digest_line(opts, line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
343 if not parts:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
344 raise ValueError(
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
345 "improperly formatted digest line: {}".format(line))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
346 tag, algo, fn, digest = parts
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
347 if tag in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
348 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
349 return (None, None, tag)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
350 try:
90
42419f57eda9 Allow to control the use of mmap from the command-line
Franz Glasner <fzglas.hg@dom66.de>
parents: 89
diff changeset
351 d = compute_digest_file(algo, fn, use_mmap=opts.mmap)
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
352 if compare_digests_equal(d, digest, algo):
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
353 return ("ok", fn, tag)
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
354 else:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
355 return ("failed", fn, tag)
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
356 except EnvironmentError:
18
285848db0b52 When verifying/checking digests: also print the digest tag used
Franz Glasner <fzglas.hg@dom66.de>
parents: 17
diff changeset
357 return ("missing", fn, tag)
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
358
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
359
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
360 def get_parsed_digest_line_from_checklist(checklist, opts, filename):
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
361 if filename is None:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
362 filenames = ("-", "stdin", "", )
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
363 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
364 filenames = (
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
365 normalize_filename(filename, strip_leading_dot_slash=True),)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
366 with io.open(checklist, "rt", encoding="utf-8") as clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
367 for checkline in clf:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
368 if not checkline:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
369 continue
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
370 parts = parse_digest_line(opts, checkline)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
371 if not parts:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
372 raise ValueError(
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
373 "improperly formatted digest line: {}".format(checkline))
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
374 if parts[0] in ("SIZE", "TIMESTAMP"):
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
375 assert opts.allow_distinfo
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
376 continue
22
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
377 fn = normalize_filename(parts[2], strip_leading_dot_slash=True)
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
378 if fn in filenames:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
379 return parts
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
380 else:
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
381 return None
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
382
6bdfc5ad4656 Implemented OpenBSD's -C (aka --checklist) option for shasum
Franz Glasner <fzglas.hg@dom66.de>
parents: 21
diff changeset
383
21
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
384 def parse_digest_line(opts, line):
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
385 """Parse a `line` of a digest file and return its parts.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
386
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
387 This is rather strict. But if `opts.allow_distinfo` is `True` then
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
388 some additional keywords ``SIZE`` and ``TIMESTAMP``are recignized
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
389 and returned. The caller is responsible to handle them.
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
390
21
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
391 :return: a tuple of the normalized algorithm tag, the algorithm
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
392 constructor, the filename and the hex digest;
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
393 if `line` cannot be parsed successfully `None` is returned
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
394 :rtype: tuple(str, obj, str, str) or None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
395
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
396 Handles coreutils and BSD-style file formats.
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
397
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
398 """
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
399 # determine checkfile format (BSD or coreutils)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
400 # BSD?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
401 mo = re.search(r"\A(\S+)\s*\((.*)\)\s*=\s*(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
402 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
403 # (tag, algorithm, filename, digest)
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
404 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
405 if mo.group(1) == "SIZE":
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
406 return ("SIZE", None, None, mo.group(3))
21
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
407 return (mo.group(1),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
408 algotag2algotype(mo.group(1)),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
409 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
410 mo.group(3))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
411 else:
83
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
412 if opts.allow_distinfo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
413 mo = re.search(r"\ATIMESTAMP\s*=\s*([0-9]+)\s*\n\Z", line)
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
414 if mo:
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
415 return ("TIMESTAMP", None, None, mo.group(1))
05e2bf4796fd Add an option "--checklist-allow-distinfo" to allow FreeBSD "distinfo" formatted files as checkfiles.
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 79
diff changeset
416
21
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
417 # coreutils?
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
418 mo = re.search(r"([^\ ]+) [\*\ ]?(.+)\n?\Z", line)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
419 if mo:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
420 # (tag, algorithm, filename, digest)
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
421 return (opts.algorithm[1],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
422 opts.algorithm[0],
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
423 mo.group(2),
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
424 mo.group(1))
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
425 else:
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
426 return None
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
427
f2d634270e1c Refactor: parse a line of a digest file within a dedicated funcion
Franz Glasner <fzglas.hg@dom66.de>
parents: 20
diff changeset
428
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
429 def get_blake2b():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
430 """Get the factory for blake2b"""
88
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
431 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
432 return hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
433 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
434 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
435 return pyblake2.blake2b
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
436
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
437
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
438 def get_blake2s():
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
439 """Get the factory for blake2s"""
88
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
440 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
441 return hashlib.blake2s
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
442 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
443 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
444 return pyblake2.blake2s
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
445
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
446
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
447 def get_blake2_256():
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
448 """Get the factory for blake2-256"""
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
449
88
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
450 try:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
451 hashlib.blake2b
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
452 except AttributeError:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
453 import pyblake2
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
454
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
455 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
456 return pyblake2.blake2b(digest_size=32)
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
457
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
458 else:
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
459
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
460 def _get_blake():
f69353f26937 Support for using the pyblake2 package if native support for BLAKE2 is not available in hashlib.
Franz Glasner <fzglas.hg@dom66.de>
parents: 87
diff changeset
461 return hashlib.blake2b(digest_size=32)
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
462
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
463 return _get_blake
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
464
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
465
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
466 def argv2algo(s):
42
e5f6af8364db Typo in docs
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 30
diff changeset
467 """Convert a command line algorithm specifier into a tuple with the
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
468 type/factory of the digest and the algorithms tag for output purposes.
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
469
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
470 :param str s: the specifier from the commane line
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
471 :return: the internal digest specification
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
472 :rtype: a tuple (digest_type_or_factory, name_in_output)
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
473
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
474 String comparisons are done case-insensitively.
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
475
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
476 """
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
477 s = s.lower()
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
478 if s in ("1", "sha1"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
479 return (hashlib.sha1, "SHA1")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
480 elif s in ("224", "sha224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
481 return (hashlib.sha224, "SHA224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
482 elif s in ("256", "sha256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
483 return (hashlib.sha256, "SHA256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
484 elif s in ("384", "sha384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
485 return (hashlib.sha384, "SHA384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
486 elif s in ("512", "sha512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
487 return (hashlib.sha512, "SHA512")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
488 elif s in ("3-224", "sha3-224"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
489 return (hashlib.sha3_224, "SHA3-224")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
490 elif s in ("3-256", "sha3-256"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
491 return (hashlib.sha3_256, "SHA3-256")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
492 elif s in ("3-384", "sha3-384"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
493 return (hashlib.sha3_384, "SHA3-384")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
494 elif s in ("3-512", "sha3-512"):
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
495 return (hashlib.sha3_512, "SHA3-512")
86
fd1cfd1b0f9d Make "blake2" an alias of "blake2b"
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
496 elif s in ("blake2b", "blake2b-512", "blake2", "blake2-512"):
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
497 return (get_blake2b(), "BLAKE2b")
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
498 elif s in ("blake2s", "blake2s-256"):
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
499 return (get_blake2s(), "BLAKE2s")
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
500 elif s in ("blake2-256", "blake2b-256"):
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
501 return (get_blake2_256(), "BLAKE2b-256")
7
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
502 elif s == "md5":
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
503 return (hashlib.md5, "MD5")
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
504 else:
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
505 raise argparse.ArgumentTypeError(
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
506 "`{}' is not a recognized algorithm".format(s))
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
507
47b4c98e4d40 Allow algorithm selection for all algorithms in :mod:`hashlib`.
Franz Glasner <fzglas.hg@dom66.de>
parents: 6
diff changeset
508
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
509 def algotag2algotype(s):
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
510 """Convert the algorithm specifier in a BSD-style digest file to the
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
511 type/factory of the corresponding algorithm.
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
512
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
513 :param str s: the tag (i.e. normalized name) or the algorithm
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
514 :return: the digest type or factory for `s`
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
515
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
516 All string comparisons are case-sensitive.
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
517
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
518 """
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
519 if s == "SHA1":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
520 return hashlib.sha1
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
521 elif s == "SHA224":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
522 return hashlib.sha224
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
523 elif s == "SHA256":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
524 return hashlib.sha256
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
525 elif s == "SHA384":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
526 return hashlib.sha384
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
527 elif s == "SHA512":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
528 return hashlib.sha512
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
529 elif s == "SHA3-224":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
530 return hashlib.sha3_224
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
531 elif s == "SHA3-256":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
532 return hashlib.sha3_256
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
533 elif s == "SHA3-384":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
534 return hashlib.sha3_384
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
535 elif s == "SHA3-512":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
536 return hashlib.sha3_512
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
537 elif s in ("BLAKE2b", "BLAKE2b-512", "BLAKE2b512"): # compat for openssl
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
538 return get_blake2b()
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
539 elif s in ("BLAKE2s", "BLAKE2s-256", "BLAKE2s256"): # compat for openssl
85
d445534b80bb Get the factory for blake2b and blake2s indirectly: prepare for blake2-256
Franz Glasner <fzglas.hg@dom66.de>
parents: 84
diff changeset
540 return get_blake2s()
87
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
541 elif s in ("BLAKE2b-256", "BLAKE2b256"): # also compat for openssl dgst
b46673c42894 Implement support for BLAKE2b-256 (aka BLAKE2-256).
Franz Glasner <fzglas.hg@dom66.de>
parents: 86
diff changeset
542 return get_blake2_256()
12
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
543 elif s == "MD5":
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
544 return hashlib.md5
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
545 else:
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
546 raise ValueError("unknown algorithm: {}".format(s))
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
547
5e2c9123f93f Implemented digest verification: -c or --check option
Franz Glasner <fzglas.hg@dom66.de>
parents: 11
diff changeset
548
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
549 def out_bsd(dest, digest, filename, digestname, binary, use_base64):
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
550 """BSD format output, also :command:`openssl dgst` and
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
551 :command:`b2sum --tag" format output
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
552
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
553 """
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
554 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
555 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
556 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
557 digest = binascii.hexlify(digest).decode("ascii")
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
558 if filename is None:
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
559 print(digest, file=dest)
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
560 else:
19
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
561 print("{} ({}) = {}".format(digestname,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
562 normalize_filename(filename),
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
563 digest),
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
564 file=dest)
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
565
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
566
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
567 def out_std(dest, digest, filename, digestname, binary, use_base64):
9
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
568 """Coreutils format (:command:`shasum` et al.)
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
569
81f28bf89c26 Some more output selection options and documentation
Franz Glasner <fzglas.hg@dom66.de>
parents: 8
diff changeset
570 """
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
571 if use_base64:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
572 digest = base64.b64encode(digest).decode("ascii")
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
573 else:
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
574 digest = binascii.hexlify(digest).decode("ascii")
19
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
575 print("{} {}{}".format(
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
576 digest,
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
577 '*' if binary else ' ',
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
578 '-' if filename is None else normalize_filename(filename)),
5
bbcb225640de Handle standard and BSD-style output formats
Franz Glasner <fzglas.hg@dom66.de>
parents: 4
diff changeset
579 file=dest)
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
580
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
581
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
582 def compute_digest_file(hashobj, path, use_mmap=None):
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
583 """
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
584 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
585 :param path: filename within the filesystem or a file descriptor opened in
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
586 binary mode (also a socket or pipe)
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
587 :param use_mmap: Use the :mod:`mmap` module if available.
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
588 If `None` determine automatically.
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
589 :type use_mmap: bool or None
56
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 52
diff changeset
590 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 52
diff changeset
591 :rtype: bytes
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
592
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
593 If a file descriptor is given is must support :func:`os.read`.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
594
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
595 """
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
596 h = hashobj()
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
597 if isinstance(path, PATH_TYPES):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
598 flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) \
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
599 | getattr(os, "O_SEQUENTIAL", 0) | getattr(os, "O_NOCTTY", 0)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
600 fd = os.open(path, flags)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
601 own_fd = True
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
602 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
603 fd = path
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
604 own_fd = False
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
605 try:
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
606 try:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
607 st = os.fstat(fd)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
608 except TypeError:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
609 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
610 # "fd" is most probably a Python socket object.
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
611 # (a pipe typically supports fstat)
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
612 #
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
613 use_mmap = False
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
614 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
615 if stat.S_ISREG(st[stat.ST_MODE]):
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
616 filesize = st[stat.ST_SIZE]
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
617 if (use_mmap is None) \
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
618 and (filesize > MAX_AUTO_MAP_SIZE):
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
619 #
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
620 # This is borrowed from FreeBSD's cp(1) implementation:
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
621 # Mmap and process if less than 8M (the limit is
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
622 # so we don't totally trash memory on big files.
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
623 # This is really a minor hack, but it wins some
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
624 # CPU back. Some filesystems, such as smbnetfs,
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
625 # don't support mmap, so this is a best-effort
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
626 # attempt.
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
627 #
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
628 use_mmap = False
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
629 else:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
630 use_mmap = False
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
631 if use_mmap is None:
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
632 use_mmap = True
67
19893b4f42a5 Flag to disable the use of mmap
Franz Glasner <fzglas.hg@dom66.de>
parents: 66
diff changeset
633 if mmap is None or not use_mmap:
84
163de6dd6e05 Enhance comment and -- while being there -- fix a typo
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 83
diff changeset
634 # No mmap available or wanted -> use traditional low-level file IO
94
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 90
diff changeset
635 fadvise = getattr(os, "posix_fadvise", None)
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 90
diff changeset
636 if fadvise:
8352da172a3e Use "fadvise()" with POSIX_FADV_SEQUENTIAL -- if available
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 90
diff changeset
637 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
96
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
638 if not PY2:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
639 fileobj = io.FileIO(fd, mode="r", closefd=False)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
640 buf = bytearray(READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
641 with memoryview(buf) as full_view:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
642 while True:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
643 try:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
644 n = fileobj.readinto(buf)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
645 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
646 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
647 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
648 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
649 raise
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
650 else:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
651 if n == 0:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
652 break
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
653 if n == READ_CHUNK_SIZE:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
654 h.update(buf)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
655 else:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
656 with full_view[:n] as partial_view:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
657 h.update(partial_view)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
658 else:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
659 while True:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
660 try:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
661 buf = os.read(fd, READ_CHUNK_SIZE)
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
662 except OSError as e:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
663 if e.errno not in (errno.EAGAIN,
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
664 errno.EWOULDBLOCK,
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
665 errno.EINTR):
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
666 raise
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
667 else:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
668 if len(buf) == 0:
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
669 break
59253a1e6ef2 Employ .readinto() on Python 3
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 95
diff changeset
670 h.update(buf)
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
671 else:
58
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
672 #
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
673 # Use mmap
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
674 #
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
675 # NOTE: On Windows mmapped files with length 0 are not supported.
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
676 # So ensure to not call mmap.mmap() if the file size is 0.
ae5b31c10b41 Enhance comments
Franz Glasner <fzglas.hg@dom66.de>
parents: 57
diff changeset
677 #
61
c9f9401abc0c Use getattr when trying to get mmap.madvise()
Franz Glasner <fzglas.hg@dom66.de>
parents: 60
diff changeset
678 madvise = getattr(mmap.mmap, "madvise", None)
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
679 if filesize <= MAP_WINDOW_SIZE:
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
680 mapsize = filesize
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
681 else:
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
682 mapsize = MAP_WINDOW_SIZE
57
0fa2067bedb8 Common initialization is now common when computing the hashes from a mmap
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 56
diff changeset
683 mapoffset = 0
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
684 rest = filesize
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
685 while rest > 0:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
686 m = mmap.mmap(fd,
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
687 mapsize,
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
688 access=mmap.ACCESS_READ,
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
689 offset=mapoffset)
60
21d2589c96b9 Use madvise if available to sequentially read a file
Franz Glasner <fzglas.hg@dom66.de>
parents: 59
diff changeset
690 if madvise:
21d2589c96b9 Use madvise if available to sequentially read a file
Franz Glasner <fzglas.hg@dom66.de>
parents: 59
diff changeset
691 madvise(m, mmap.MADV_SEQUENTIAL)
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
692 try:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
693 h.update(m)
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
694 finally:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
695 m.close()
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
696 rest -= mapsize
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
697 mapoffset += mapsize
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
698 if rest < mapsize:
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
699 mapsize = rest
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
700 finally:
68
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
701 if own_fd:
4c2da9c74d7c "compute_digest_file()" now also accepts an already opened file descriptor.
Franz Glasner <fzglas.hg@dom66.de>
parents: 67
diff changeset
702 os.close(fd)
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
703 return h.digest()
23
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
704
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
705
232063b73e45 Optimized reading of files by using mmap.
Franz Glasner <fzglas.hg@dom66.de>
parents: 22
diff changeset
706 def compute_digest_stream(hashobj, instream):
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
707 """
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
708
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
709 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
710 :param instream: a bytes input stream to read the data to be hashed from
56
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 52
diff changeset
711 :return: the digest in binary form
6e91c530545f FIX: Docstring: Computes hashes are now returned as bytes from the low-level functions
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 52
diff changeset
712 :rtype: bytes
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
713
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
714 """
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
715 h = hashobj()
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
716 while True:
66
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
717 try:
89
72684020f2f3 By default use mmap only for files up to 8MiB in size.
Franz Glasner <fzglas.hg@dom66.de>
parents: 88
diff changeset
718 buf = instream.read(READ_CHUNK_SIZE)
66
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
719 except OSError as e:
79
8708c34e2723 Handle EINTR also in combination with EAGAIN and EWOULDBLOCK
Franz Glasner <fzglas.hg@dom66.de>
parents: 75
diff changeset
720 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
66
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
721 raise
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
722 else:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
723 if buf is not None:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
724 if len(buf) == 0:
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
725 break
c52e5f86b0ab Handle EAGAIN and EWOULDBLOCK when reading files
Franz Glasner <fzglas.hg@dom66.de>
parents: 61
diff changeset
726 h.update(buf)
51
58d5a0b6e5b3 Implement the OpenBSD variant (with --base64) to encode digests in base64, not hexadecimal
Franz Glasner <f.glasner@feldmann-mg.com>
parents: 47
diff changeset
727 return h.digest()
2
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
728
5510a39a2d04 Basic hashing with proper binary stdin/stdout support for Py2, Py3 and Windows
Franz Glasner <fzglas.hg@dom66.de>
parents: 1
diff changeset
729
19
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
730 def normalize_filename(filename, strip_leading_dot_slash=False):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
731 filename = filename.replace("\\", "/")
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
732 if strip_leading_dot_slash:
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
733 while filename.startswith("./"):
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
734 filename = filename[2:]
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
735 return filename
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
736
2f9e702e3f7a Convert backslashes in filenames to forward slashes when creating digests
Franz Glasner <fzglas.hg@dom66.de>
parents: 18
diff changeset
737
1
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
738 if __name__ == "__main__":
bbf4e0f5b651 Begin the shasum.py script
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
739 sys.exit(main())