Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/digest.py @ 165:b32b41297893

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
The scandir package is not needed any more on Python <3.5 because fs walking has been implemented using os.listdir() also
author Franz Glasner <fzglas.hg@dom66.de>
date Thu, 09 Jan 2025 13:57:03 +0100
parents a813094ae4f5
children 0f4febf646f5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # :Copyright: (c) 2020-2025 Franz Glasner
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # :License: BSD-3-Clause
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Utility sub-module to implement a file and stream digest computations.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10 __all__ = ["compute_digest_file", "compute_digest_stream"]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
11
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
12
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 import errno
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14 import io
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15 import os
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17 import mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18 except ImportError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 mmap = None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20 import stat
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 122
diff changeset
22 from . import PY2
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23 from . import constants
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
25
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
26 def compute_digest_file(hashobj, path, use_mmap=None):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27 """Compute the digest for a file with a filename of an open fd.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
29 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30 :param path: filename within the filesystem or a file descriptor opened in
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
31 binary mode (also a socket or pipe)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32 :param use_mmap: Use the :mod:`mmap` module if available.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 If `None` determine automatically.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 :type use_mmap: bool or None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38 If a file descriptor is given is must support :func:`os.read`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42 if isinstance(path, constants.PATH_TYPES):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43 flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44 | getattr(os, "O_SEQUENTIAL", 0) | getattr(os, "O_NOCTTY", 0)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 fd = os.open(path, flags)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 own_fd = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 fd = path
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49 own_fd = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 st = os.fstat(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 except TypeError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 # "fd" is most probably a Python socket object.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 # (a pipe typically supports fstat)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
57 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
58 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
59 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
60 if stat.S_ISREG(st[stat.ST_MODE]):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
61 filesize = st[stat.ST_SIZE]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
62 if (use_mmap is None) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 and (filesize > constants.MAX_AUTO_MAP_SIZE):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65 # This is borrowed from FreeBSD's cp(1) implementation:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 # Mmap and process if less than 8M (the limit is
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 # so we don't totally trash memory on big files.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 # This is really a minor hack, but it wins some
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 # CPU back. Some filesystems, such as smbnetfs,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 # don't support mmap, so this is a best-effort
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71 # attempt.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
74 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
75 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
76 if use_mmap is None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77 use_mmap = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78 if mmap is None or not use_mmap:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79 # No mmap available or wanted -> use traditional low-level file IO
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 fadvise = getattr(os, "posix_fadvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81 if fadvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
164
a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util
Franz Glasner <fzglas.hg@dom66.de>
parents: 122
diff changeset
83 if not PY2:
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
84 fileobj = io.FileIO(fd, mode="r", closefd=False)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
85 buf = bytearray(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
86 with memoryview(buf) as full_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
87 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
88 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
89 n = fileobj.readinto(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 if n == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 if n == constants.READ_CHUNK_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 with full_view[:n] as partial_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 h.update(partial_view)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
103 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
104 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
105 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
106 buf = os.read(fd, constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
107 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
108 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
109 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
110 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
111 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
116 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 # Use mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
120 # NOTE: On Windows mmapped files with length 0 are not supported.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
121 # So ensure to not call mmap.mmap() if the file size is 0.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
122 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
123 madvise = getattr(mmap.mmap, "madvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
124 if filesize <= constants.MAP_WINDOW_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
125 mapsize = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
126 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
127 mapsize = constants.MAP_WINDOW_SIZE
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
128 mapoffset = 0
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
129 rest = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
130 while rest > 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
131 m = mmap.mmap(fd,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
132 mapsize,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
133 access=mmap.ACCESS_READ,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
134 offset=mapoffset)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
135 if madvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
136 madvise(m, mmap.MADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
137 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
138 h.update(m)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
139 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
140 m.close()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
141 rest -= mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
142 mapoffset += mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
143 if rest < mapsize:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
144 mapsize = rest
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
145 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
146 if own_fd:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
147 os.close(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
148 return h.digest()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
149
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
150
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
151 def compute_digest_stream(hashobj, instream):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
152 """Compute the digest for a given byte string `instream`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
153
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
154 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
155 :param instream: a bytes input stream to read the data to be hashed from
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
156 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
157 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
158
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
159 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
160 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
161 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
162 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
163 buf = instream.read(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
164 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
165 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
166 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
167 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
168 if buf is not None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
169 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
170 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
171 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
172 return h.digest()