Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/digest.py @ 155:bf74ce3c968d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
When computing digests use the order imposed by names alone. No different loops for dirs and nondirs.
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 06 Jan 2025 13:39:12 +0100
parents 1e5127028254
children a813094ae4f5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
122
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # :Copyright: (c) 2020-2025 Franz Glasner
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # :License: BSD-3-Clause
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Utility sub-module to implement a file and stream digest computations.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10 __all__ = ["compute_digest_file", "compute_digest_stream"]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
11
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
12
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 import errno
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14 import io
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15 import os
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17 import mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18 except ImportError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 mmap = None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20 import stat
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22 from . import constants
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
25 def compute_digest_file(hashobj, path, use_mmap=None):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
26 """Compute the digest for a file with a filename of an open fd.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
29 :param path: filename within the filesystem or a file descriptor opened in
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
30 binary mode (also a socket or pipe)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
31 :param use_mmap: Use the :mod:`mmap` module if available.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32 If `None` determine automatically.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 :type use_mmap: bool or None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37 If a file descriptor is given is must support :func:`os.read`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41 if isinstance(path, constants.PATH_TYPES):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
42 flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
43 | getattr(os, "O_SEQUENTIAL", 0) | getattr(os, "O_NOCTTY", 0)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44 fd = os.open(path, flags)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 own_fd = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 fd = path
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 own_fd = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51 st = os.fstat(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 except TypeError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
54 # "fd" is most probably a Python socket object.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
55 # (a pipe typically supports fstat)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
57 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
58 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
59 if stat.S_ISREG(st[stat.ST_MODE]):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
60 filesize = st[stat.ST_SIZE]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
61 if (use_mmap is None) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
62 and (filesize > constants.MAX_AUTO_MAP_SIZE):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64 # This is borrowed from FreeBSD's cp(1) implementation:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65 # Mmap and process if less than 8M (the limit is
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 # so we don't totally trash memory on big files.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 # This is really a minor hack, but it wins some
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 # CPU back. Some filesystems, such as smbnetfs,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 # don't support mmap, so this is a best-effort
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 # attempt.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
74 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
75 if use_mmap is None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
76 use_mmap = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77 if mmap is None or not use_mmap:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78 # No mmap available or wanted -> use traditional low-level file IO
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79 fadvise = getattr(os, "posix_fadvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 if fadvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 if not constants.PY2:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
83 fileobj = io.FileIO(fd, mode="r", closefd=False)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
84 buf = bytearray(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
85 with memoryview(buf) as full_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
86 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
87 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
88 n = fileobj.readinto(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
89 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 if n == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 if n == constants.READ_CHUNK_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 with full_view[:n] as partial_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 h.update(partial_view)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
103 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
104 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
105 buf = os.read(fd, constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
106 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
107 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
108 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
109 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
110 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
111 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
116 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117 # Use mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119 # NOTE: On Windows mmapped files with length 0 are not supported.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
120 # So ensure to not call mmap.mmap() if the file size is 0.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
121 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
122 madvise = getattr(mmap.mmap, "madvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
123 if filesize <= constants.MAP_WINDOW_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
124 mapsize = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
125 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
126 mapsize = constants.MAP_WINDOW_SIZE
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
127 mapoffset = 0
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
128 rest = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
129 while rest > 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
130 m = mmap.mmap(fd,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
131 mapsize,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
132 access=mmap.ACCESS_READ,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
133 offset=mapoffset)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
134 if madvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
135 madvise(m, mmap.MADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
136 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
137 h.update(m)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
138 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
139 m.close()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
140 rest -= mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
141 mapoffset += mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
142 if rest < mapsize:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
143 mapsize = rest
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
144 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
145 if own_fd:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
146 os.close(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
147 return h.digest()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
148
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
149
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
150 def compute_digest_stream(hashobj, instream):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
151 """Compute the digest for a given byte string `instream`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
152
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
153 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
154 :param instream: a bytes input stream to read the data to be hashed from
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
155 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
156 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
157
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
158 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
159 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
160 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
161 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
162 buf = instream.read(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
163 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
164 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
165 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
166 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
167 if buf is not None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
168 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
169 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
170 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
171 return h.digest()