Python/apps/py-cutils: cutils/util/digest.py annotate

annotate cutils/util/digest.py @ 177:089c40240061

Add an alternate implementation for generating directory tree digests: - Do not use something like os.walk() but use os.scandir() directly. - Recursively generate the subdirectory digests only when needed and in the right order. This fixes that the order of subdirectories in the output did not match the application order of its directory digests. The new implementation also should make filtering (that will be implemented later) easier. NOTE: The tree digests of the old and the new implementation are identical.

author	Franz Glasner <fzglas.hg@dom66.de>
date	Sat, 11 Jan 2025 17:41:28 +0100
parents	a813094ae4f5
children	0f4febf646f5

rev	line source
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	1 # -- coding: utf-8 --
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	2 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	3 # :Copyright: (c) 2020-2025 Franz Glasner
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	4 # :License: BSD-3-Clause
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	5 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	6 r"""Utility sub-module to implement a file and stream digest computations.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	10 __all__ = ["compute_digest_file", "compute_digest_stream"]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	11
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	12
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	13 import errno
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	14 import io
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	15 import os
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	17 import mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	18 except ImportError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	19 mmap = None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import stat
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	21
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 122 diff changeset	22 from . import PY2
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	23 from . import constants
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	24
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	25
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	26 def compute_digest_file(hashobj, path, use_mmap=None):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	27 """Compute the digest for a file with a filename of an open fd.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	28
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	29 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	30 :param path: filename within the filesystem or a file descriptor opened in
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	31 binary mode (also a socket or pipe)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	32 :param use_mmap: Use the :mod:`mmap` module if available.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 If `None` determine automatically.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34 :type use_mmap: bool or None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	35 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	36 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	38 If a file descriptor is given is must support :func:`os.read`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	39
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	40 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	41 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	42 if isinstance(path, constants.PATH_TYPES):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	43 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	44 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	45 fd = os.open(path, flags)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	46 own_fd = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	47 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	48 fd = path
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	49 own_fd = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	50 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	51 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	52 st = os.fstat(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 except TypeError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 # "fd" is most probably a Python socket object.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 # (a pipe typically supports fstat)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	57 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	58 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	59 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	60 if stat.S_ISREG(st[stat.ST_MODE]):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	61 filesize = st[stat.ST_SIZE]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	62 if (use_mmap is None) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	63 and (filesize > constants.MAX_AUTO_MAP_SIZE):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	64 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	65 # This is borrowed from FreeBSD's cp(1) implementation:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	66 # Mmap and process if less than 8M (the limit is
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	67 # so we don't totally trash memory on big files.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	68 # This is really a minor hack, but it wins some
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	69 # CPU back. Some filesystems, such as smbnetfs,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	70 # don't support mmap, so this is a best-effort
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	71 # attempt.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	72 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	73 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	74 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	75 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	76 if use_mmap is None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	77 use_mmap = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	78 if mmap is None or not use_mmap:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	79 # No mmap available or wanted -> use traditional low-level file IO
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	80 fadvise = getattr(os, "posix_fadvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	81 if fadvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	82 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 122 diff changeset	83 if not PY2:
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	84 fileobj = io.FileIO(fd, mode="r", closefd=False)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	85 buf = bytearray(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	86 with memoryview(buf) as full_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	87 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	88 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	89 n = fileobj.readinto(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	90 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	91 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	92 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	93 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	94 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	95 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	96 if n == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	97 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	98 if n == constants.READ_CHUNK_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	99 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	100 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	101 with full_view[:n] as partial_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	102 h.update(partial_view)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	103 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	104 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	105 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	106 buf = os.read(fd, constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	107 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	108 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	109 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	110 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	111 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	112 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	113 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	114 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	115 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	116 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	117 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	118 # Use mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	119 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	120 # NOTE: On Windows mmapped files with length 0 are not supported.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	121 # So ensure to not call mmap.mmap() if the file size is 0.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	122 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	123 madvise = getattr(mmap.mmap, "madvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	124 if filesize <= constants.MAP_WINDOW_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	125 mapsize = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	126 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	127 mapsize = constants.MAP_WINDOW_SIZE
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	128 mapoffset = 0
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	129 rest = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	130 while rest > 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	131 m = mmap.mmap(fd,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	132 mapsize,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	133 access=mmap.ACCESS_READ,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	134 offset=mapoffset)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	135 if madvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	136 madvise(m, mmap.MADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	137 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	138 h.update(m)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	139 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	140 m.close()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	141 rest -= mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	142 mapoffset += mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	143 if rest < mapsize:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	144 mapsize = rest
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	145 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	146 if own_fd:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	147 os.close(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	148 return h.digest()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	149
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	150
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	151 def compute_digest_stream(hashobj, instream):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	152 """Compute the digest for a given byte string `instream`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	153
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	154 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	155 :param instream: a bytes input stream to read the data to be hashed from
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	156 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	157 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	158
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	159 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	160 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	161 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	162 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	163 buf = instream.read(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	164 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	165 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	166 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	167 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	168 if buf is not None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	169 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	170 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	171 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	172 return h.digest()

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/digest.py @ 177:089c40240061