Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/digest.py @ 376:54a6d4534ef4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Convert some further value checks from plain Python assertions into real checks
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 05 May 2025 13:04:49 +0200
parents 48430941c18c
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
323
48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
3 # SPDX-FileCopyrightText: © 2020-2025 Franz Glasner
48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/.
Franz Glasner <fzglas.hg@dom66.de>
parents: 196
diff changeset
4 # SPDX-License-Identifier: BSD-3-Clause
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Utility sub-module to implement a file and stream digest computations.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
196
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 164
diff changeset
10 from __future__ import print_function, absolute_import
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 164
diff changeset
11
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import
Franz Glasner <fzglas.hg@dom66.de>
parents: 164
diff changeset
12
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 __all__ = ["compute_digest_file", "compute_digest_stream"]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 import errno
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17 import io
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18 import os
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20 import mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21 except ImportError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22 mmap = None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23 import stat
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 122
diff changeset
25 from . import PY2
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
26 from . import constants
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
29 def compute_digest_file(hashobj, path, use_mmap=None):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30 """Compute the digest for a file with a filename of an open fd.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
31
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 :param path: filename within the filesystem or a file descriptor opened in
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 binary mode (also a socket or pipe)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 :param use_mmap: Use the :mod:`mmap` module if available.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36 If `None` determine automatically.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37 :type use_mmap: bool or None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41 If a file descriptor is given is must support :func:`os.read`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 if isinstance(path, constants.PATH_TYPES):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 | getattr(os, "O_SEQUENTIAL", 0) | getattr(os, "O_NOCTTY", 0)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 fd = os.open(path, flags)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49 own_fd = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51 fd = path
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 own_fd = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 st = os.fstat(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 except TypeError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
57 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
58 # "fd" is most probably a Python socket object.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
59 # (a pipe typically supports fstat)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
60 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
61 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
62 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 if stat.S_ISREG(st[stat.ST_MODE]):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64 filesize = st[stat.ST_SIZE]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65 if (use_mmap is None) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 and (filesize > constants.MAX_AUTO_MAP_SIZE):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 # This is borrowed from FreeBSD's cp(1) implementation:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 # Mmap and process if less than 8M (the limit is
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 # so we don't totally trash memory on big files.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71 # This is really a minor hack, but it wins some
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 # CPU back. Some filesystems, such as smbnetfs,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73 # don't support mmap, so this is a best-effort
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
74 # attempt.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
75 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
76 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79 if use_mmap is None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 use_mmap = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81 if mmap is None or not use_mmap:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 # No mmap available or wanted -> use traditional low-level file IO
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
83 fadvise = getattr(os, "posix_fadvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
84 if fadvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
85 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 122
diff changeset
86 if not PY2:
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
87 fileobj = io.FileIO(fd, mode="r", closefd=False)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
88 buf = bytearray(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
89 with memoryview(buf) as full_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 n = fileobj.readinto(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 if n == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 if n == constants.READ_CHUNK_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
103 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
104 with full_view[:n] as partial_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
105 h.update(partial_view)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
106 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
107 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
108 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
109 buf = os.read(fd, constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
110 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
111 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
116 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
120 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
121 # Use mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
122 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
123 # NOTE: On Windows mmapped files with length 0 are not supported.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
124 # So ensure to not call mmap.mmap() if the file size is 0.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
125 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
126 madvise = getattr(mmap.mmap, "madvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
127 if filesize <= constants.MAP_WINDOW_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
128 mapsize = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
129 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
130 mapsize = constants.MAP_WINDOW_SIZE
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
131 mapoffset = 0
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
132 rest = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
133 while rest > 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
134 m = mmap.mmap(fd,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
135 mapsize,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
136 access=mmap.ACCESS_READ,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
137 offset=mapoffset)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
138 if madvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
139 madvise(m, mmap.MADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
140 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
141 h.update(m)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
142 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
143 m.close()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
144 rest -= mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
145 mapoffset += mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
146 if rest < mapsize:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
147 mapsize = rest
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
148 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
149 if own_fd:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
150 os.close(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
151 return h.digest()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
152
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
153
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
154 def compute_digest_stream(hashobj, instream):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
155 """Compute the digest for a given byte string `instream`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
156
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
157 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
158 :param instream: a bytes input stream to read the data to be hashed from
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
159 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
160 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
161
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
162 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
163 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
164 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
165 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
166 buf = instream.read(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
167 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
168 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
169 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
170 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
171 if buf is not None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
172 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
173 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
174 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
175 return h.digest()