Python/apps/py-cutils: cutils/util/digest.py annotate

annotate cutils/util/digest.py @ 323:48430941c18c

Adopt copyright and license wordings from https://reuse.software/faq/. While there normalize copyright years for every file to start with the file's addition to the project (i.e. with the year of file creation).

author	Franz Glasner <fzglas.hg@dom66.de>
date	Wed, 26 Mar 2025 18:42:23 +0100
parents	0f4febf646f5
children

rev	line source
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	1 # -- coding: utf-8 --
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	2 # :-
323 48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/. Franz Glasner <fzglas.hg@dom66.de> parents: 196 diff changeset	3 # SPDX-FileCopyrightText: © 2020-2025 Franz Glasner
48430941c18c Adopt copyright and license wordings from https://reuse.software/faq/. Franz Glasner <fzglas.hg@dom66.de> parents: 196 diff changeset	4 # SPDX-License-Identifier: BSD-3-Clause
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	5 # :-
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	6 r"""Utility sub-module to implement a file and stream digest computations.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
196 0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import Franz Glasner <fzglas.hg@dom66.de> parents: 164 diff changeset	10 from __future__ import print_function, absolute_import
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import Franz Glasner <fzglas.hg@dom66.de> parents: 164 diff changeset	11
0f4febf646f5 Prepare for more Python2/3 compatibility: everywhere import print_function and absolute_import Franz Glasner <fzglas.hg@dom66.de> parents: 164 diff changeset	12
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	13 __all__ = ["compute_digest_file", "compute_digest_stream"]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	14
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	15
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import errno
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	17 import io
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	18 import os
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	19 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	20 import mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	21 except ImportError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	22 mmap = None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	23 import stat
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	24
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 122 diff changeset	25 from . import PY2
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	26 from . import constants
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	27
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	28
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	29 def compute_digest_file(hashobj, path, use_mmap=None):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	30 """Compute the digest for a file with a filename of an open fd.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	31
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	32 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33 :param path: filename within the filesystem or a file descriptor opened in
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34 binary mode (also a socket or pipe)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	35 :param use_mmap: Use the :mod:`mmap` module if available.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	36 If `None` determine automatically.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	37 :type use_mmap: bool or None
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	38 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	39 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	40
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	41 If a file descriptor is given is must support :func:`os.read`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	42
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	43 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	44 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	45 if isinstance(path, constants.PATH_TYPES):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	46 flags = os.O_RDONLY \| getattr(os, "O_BINARY", 0) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	47 \| getattr(os, "O_SEQUENTIAL", 0) \| getattr(os, "O_NOCTTY", 0)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	48 fd = os.open(path, flags)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	49 own_fd = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	50 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	51 fd = path
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	52 own_fd = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	53 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	54 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	55 st = os.fstat(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	56 except TypeError:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	57 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	58 # "fd" is most probably a Python socket object.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	59 # (a pipe typically supports fstat)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	60 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	61 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	62 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	63 if stat.S_ISREG(st[stat.ST_MODE]):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	64 filesize = st[stat.ST_SIZE]
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	65 if (use_mmap is None) \
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	66 and (filesize > constants.MAX_AUTO_MAP_SIZE):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	67 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	68 # This is borrowed from FreeBSD's cp(1) implementation:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	69 # Mmap and process if less than 8M (the limit is
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	70 # so we don't totally trash memory on big files.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	71 # This is really a minor hack, but it wins some
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	72 # CPU back. Some filesystems, such as smbnetfs,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	73 # don't support mmap, so this is a best-effort
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	74 # attempt.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	75 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	76 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	77 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	78 use_mmap = False
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	79 if use_mmap is None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	80 use_mmap = True
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	81 if mmap is None or not use_mmap:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	82 # No mmap available or wanted -> use traditional low-level file IO
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	83 fadvise = getattr(os, "posix_fadvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	84 if fadvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	85 fadvise(fd, 0, 0, os.POSIX_FADV_SEQUENTIAL)
164 a813094ae4f5 Move PY2 from cutils.util.constants into cutils.util Franz Glasner <fzglas.hg@dom66.de> parents: 122 diff changeset	86 if not PY2:
122 1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	87 fileobj = io.FileIO(fd, mode="r", closefd=False)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	88 buf = bytearray(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	89 with memoryview(buf) as full_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	90 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	91 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	92 n = fileobj.readinto(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	93 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	94 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	95 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	96 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	97 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	98 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	99 if n == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	100 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	101 if n == constants.READ_CHUNK_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	102 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	103 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	104 with full_view[:n] as partial_view:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	105 h.update(partial_view)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	106 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	107 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	108 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	109 buf = os.read(fd, constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	110 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	111 if e.errno not in (errno.EAGAIN,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	112 errno.EWOULDBLOCK,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	113 errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	114 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	115 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	116 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	117 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	118 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	119 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	120 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	121 # Use mmap
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	122 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	123 # NOTE: On Windows mmapped files with length 0 are not supported.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	124 # So ensure to not call mmap.mmap() if the file size is 0.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	125 #
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	126 madvise = getattr(mmap.mmap, "madvise", None)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	127 if filesize <= constants.MAP_WINDOW_SIZE:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	128 mapsize = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	129 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	130 mapsize = constants.MAP_WINDOW_SIZE
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	131 mapoffset = 0
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	132 rest = filesize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	133 while rest > 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	134 m = mmap.mmap(fd,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	135 mapsize,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	136 access=mmap.ACCESS_READ,
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	137 offset=mapoffset)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	138 if madvise:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	139 madvise(m, mmap.MADV_SEQUENTIAL)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	140 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	141 h.update(m)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	142 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	143 m.close()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	144 rest -= mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	145 mapoffset += mapsize
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	146 if rest < mapsize:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	147 mapsize = rest
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	148 finally:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	149 if own_fd:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	150 os.close(fd)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	151 return h.digest()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	152
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	153
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	154 def compute_digest_stream(hashobj, instream):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	155 """Compute the digest for a given byte string `instream`.
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	156
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	157 :param hashobj: a :mod:`hashlib` compatible hash algorithm type or factory
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	158 :param instream: a bytes input stream to read the data to be hashed from
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	159 :return: the digest in binary form
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	160 :rtype: bytes
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	161
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	162 """
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	163 h = hashobj()
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	164 while True:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	165 try:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	166 buf = instream.read(constants.READ_CHUNK_SIZE)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	167 except OSError as e:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	168 if e.errno not in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	169 raise
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	170 else:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	171 if buf is not None:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	172 if len(buf) == 0:
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	173 break
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	174 h.update(buf)
1e5127028254 Move the real computation of digests from files and streams into dedicated submodule cutils.util.digest Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	175 return h.digest()

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/util/digest.py @ 323:48430941c18c