Python/apps/py-cutils: cutils/treesum.py annotate

annotate cutils/treesum.py @ 197:48e2610978e5

Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. On Python3 it is always unsigned.

author	Franz Glasner <fzglas.hg@dom66.de>
date	Thu, 16 Jan 2025 20:40:24 +0100
parents	fb36e71f6ba8
children	c1e875ba4bdc

rev	line source
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	1 # -- coding: utf-8 --
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	2 # :-
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	3 # :Copyright: (c) 2020-2025 Franz Glasner
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	4 # :License: BSD-3-Clause
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	5 # :-
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	6 r"""Generate and verify checksums for directory trees.
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	8 """
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	10 from __future__ import print_function, absolute_import
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	11
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	12
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	13 __all__ = []
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	14
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	15
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16 import argparse
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	17 import base64
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	18 import binascii
128 7c646921a479 Add TIMESTAMP and ISOTIMESTAMP to the output Franz Glasner <fzglas.hg@dom66.de> parents: 127 diff changeset	19 import datetime
176 7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	20 import logging
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	21 import os
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	22 import re
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	23 import stat
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	24 import sys
128 7c646921a479 Add TIMESTAMP and ISOTIMESTAMP to the output Franz Glasner <fzglas.hg@dom66.de> parents: 127 diff changeset	25 import time
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	26 import zlib
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	27
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	28 from . import (__version__, __revision__)
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	29 from . import util
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	30 from .util import cm
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	31 from .util import digest
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	32 from .util import walk
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	33
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	34
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	35 def main(argv=None):
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	36
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	37 def _populate_generate_arguments(gp):
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	38 """Use to populate command aliases.
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	39
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	40 This is because :class:`argparse.ArgumentParser` does not
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	41 support them for all supported Python versions.
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	42
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	43 """
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	44 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	45 "--algorithm", "-a", action="store", type=util.argv2algo,
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	46 help="1 (aka sha1), 224, 256 (aka sha256), 384, 512 (aka sha512), "
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	47 "3 (alias for sha3-512), 3-224, 3-256, 3-384, 3-512, "
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	48 "blake2b, blake2b-256, blake2s, "
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	49 "blake2 (alias for blake2b), "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	50 "blake2-256 (alias for blake2b-256), "
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	51 "md5. "
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	52 "The default depends on availability in hashlib: "
804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	53 "blake2b-256, sha256 or sha1.")
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	54 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	55 "--append-output", action="store_true", dest="append_output",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	56 help="Append to the output file instead of overwriting it.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	57 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	58 "--base64", action="store_true",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	59 help="Output checksums in base64 notation, not hexadecimal "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	60 "(OpenBSD).")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	61 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	62 "--comment", action="append", default=[],
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	63 help="Put given comment COMMENT into the output as \"COMMENT\". "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	64 "Can be given more than once.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	65 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	66 "--follow-directory-symlinks", "-l", action="store_true",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	67 dest="follow_directory_symlinks",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	68 help="Follow symbolic links to directories when walking a "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	69 "directory tree. Note that this is different from using "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	70 "\"--logical\" or \"--physical\" for arguments given "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	71 "directly on the command line")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	72 gp.add_argument(
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	73 "--full-mode", action="store_true", dest="metadata_full_mode",
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	74 help="Consider all mode bits as returned from stat(2) when "
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	75 "computing directory digests. "
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	76 "Note that mode bits on symbolic links itself are not "
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	77 "considered.")
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	78 gp.add_argument(
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	79 "--logical", "-L", dest="logical", action="store_true",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	80 default=None,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	81 help="Follow symbolic links given on command line arguments."
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	82 " Note that this is a different setting as to follow symbolic"
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	83 " links to directories when traversing a directory tree.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	84 gp.add_argument(
150 f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	85 "--minimal", nargs="?", const="", default=None, metavar="TAG",
f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	86 help="Produce minimal output only. If a TAG is given and not "
f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	87 "empty use it as the leading \"ROOT (<TAG>)\" output.")
f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	88 gp.add_argument(
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	89 "--mmap", action="store_true", dest="mmap", default=None,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	90 help="Use mmap if available. Default is to determine "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	91 "automatically from the filesize.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	92 gp.add_argument(
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	93 "--mode", action="store_true", dest="metadata_mode",
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	94 help="Consider the permission bits of stat(2) using S_IMODE (i.e. "
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	95 "all bits without the filetype bits) when "
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	96 "computing directory digests. Note that mode bits on "
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	97 "symbolic links itself are not considered.")
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	98 gp.add_argument(
151 b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	99 "--mtime", action="store_true", dest="metadata_mtime",
b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	100 help="Consider the mtime of files (non-directories) when "
b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	101 "generating digests for directories. Digests for files are "
b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	102 "not affected.")
b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	103 gp.add_argument(
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	104 "--no-mmap", action="store_false", dest="mmap", default=None,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	105 help="Dont use mmap.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	106 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	107 "--output", "-o", action="store", metavar="OUTPUT",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	108 help="Put the checksum into given file. "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	109 "If not given or if it is given as `-' then stdout is used.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	110 gp.add_argument(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	111 "--physical", "-P", dest="logical", action="store_false",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	112 default=None,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	113 help="Do not follow symbolic links given on comment line "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	114 "arguments. This is the default.")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	115 gp.add_argument(
168 bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	116 "--print-size", action="store_true",
bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	117 help="""Print the size of a file or the accumulated sizes of
bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	118 directory content into the output also.
bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	119 The size is not considered when computing digests. For symbolic links
bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	120 the size is not printed also.""")
bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	121 gp.add_argument(
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	122 "--size-only", action="store_true",
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	123 help="""Print only the size of files and for each directory its
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	124 accumulated directory size. Digests are not computed.""")
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	125 gp.add_argument(
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	126 "directories", nargs="*", metavar="DIRECTORY")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	127
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	128 def _populate_info_arguments(ip):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	129 ip.add_argument(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	130 "--last", action="store_true", dest="print_only_last_block",
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	131 help="Print only the last block of every given input file")
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	132 ip.add_argument(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	133 "digest_files", nargs="+", metavar="TREESUM-DIGEST-FILE")
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	134
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	135 parser = argparse.ArgumentParser(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	136 description="Generate and verify checksums for directory trees.",
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	137 fromfile_prefix_chars='@',
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	138 add_help=False)
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	139
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	140 #
153 3505406ef9f3 Comment Franz Glasner <fzglas.hg@dom66.de> parents: 152 diff changeset	141 # Global options for all sub-commands.
3505406ef9f3 Comment Franz Glasner <fzglas.hg@dom66.de> parents: 152 diff changeset	142 # In a group because this allows a customized title.
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	143 #
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	144 gparser = parser.add_argument_group(title="Global Options")
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	145 gparser.add_argument(
191 1b8bc876146a Make "--debug" a global argument in treeview Franz Glasner <fzglas.hg@dom66.de> parents: 190 diff changeset	146 "--debug", action="store_true",
1b8bc876146a Make "--debug" a global argument in treeview Franz Glasner <fzglas.hg@dom66.de> parents: 190 diff changeset	147 help="Activate debug logging to stderr")
1b8bc876146a Make "--debug" a global argument in treeview Franz Glasner <fzglas.hg@dom66.de> parents: 190 diff changeset	148 gparser.add_argument(
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	149 "-v", "--version", action="version",
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	150 version="%s (rv:%s)" % (__version__, __revision__),
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	151 help="Show program's version number and exit")
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	152 gparser.add_argument(
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	153 "-h", "--help", action="help",
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	154 help="Show this help message and exit")
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	155
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	156 #
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	157 # Subcommands
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	158 #
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	159 subparsers = parser.add_subparsers(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	160 dest="subcommand",
145 073e0faea599 Optimize help output for subcommands Franz Glasner <fzglas.hg@dom66.de> parents: 144 diff changeset	161 title="Commands",
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	162 description="This tool uses subcommands. "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	163 "To see detailed help for a specific subcommand use "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	164 "the -h/--help option after the subcommand name. "
148 17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	165 "A list of valid commands and their short descriptions "
17d6e760143f Optimize help / usage output for the global options. Franz Glasner <fzglas.hg@dom66.de> parents: 147 diff changeset	166 "is listed below:",
145 073e0faea599 Optimize help output for subcommands Franz Glasner <fzglas.hg@dom66.de> parents: 144 diff changeset	167 metavar="COMMAND")
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	168
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	169 genparser = subparsers.add_parser(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	170 "generate",
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	171 help="Generate checksums for directory trees",
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	172 description="Generate checksums for directory trees.")
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	173 _populate_generate_arguments(genparser)
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	174 # And an alias for "generate"
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	175 genparser2 = subparsers.add_parser(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	176 "gen",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	177 help="Alias for \"generate\"",
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	178 description="Generate checksums for directory trees. "
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	179 "This is an alias to \"generate\".")
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	180 _populate_generate_arguments(genparser2)
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	181
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	182 infoparser = subparsers.add_parser(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	183 "info",
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	184 help="Print some information from given treesum digest file",
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	185 description="""Print some informations from given treesum digest files
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	186 to stdout."""
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	187 )
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	188 _populate_info_arguments(infoparser)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	189
147 ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	190 hparser = subparsers.add_parser(
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	191 "help",
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	192 help="Show this help message or a subcommand's help and exit",
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	193 description="Show this help message or a subcommand's help and exit.")
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	194 hparser.add_argument("help_command", nargs='?', metavar="COMMAND")
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	195
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	196 vparser = subparsers.add_parser(
144 b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	197 "version",
b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	198 help="Show the program's version number and exit",
b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	199 description="Show the program's version number and exit.")
b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	200
147 ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	201 # Parse leniently to just check for "version" and/or help
146 7d8df8311e3b Optimize argument parsing for the "version" command Franz Glasner <fzglas.hg@dom66.de> parents: 145 diff changeset	202 opts, _dummy = parser.parse_known_args(args=argv)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	203
144 b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	204 if opts.subcommand == "version":
b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	205 print("%s (rv:%s)" % (__version__, __revision__),
b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	206 file=sys.stdout)
174 fc1055878775 Use "return 0" instead of "sys.exit(0)" when printing version and help# Franz Glasner <fzglas.hg@dom66.de> parents: 173 diff changeset	207 return 0
147 ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	208 if opts.subcommand == "help":
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	209 if not opts.help_command:
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	210 parser.print_help()
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	211 else:
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	212 if opts.help_command == "generate":
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	213 genparser.print_help()
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	214 elif opts.help_command == "gen":
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	215 genparser2.print_help()
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	216 elif opts.help_command == "info":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	217 infoparser.print_help()
147 ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	218 elif opts.help_command == "version":
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	219 vparser.print_help()
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	220 elif opts.help_command == "help":
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	221 hparser.print_help()
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	222 else:
ed35f3c9e2b5 Add also a "help" subcommand to "treesum". Franz Glasner <fzglas.hg@dom66.de> parents: 146 diff changeset	223 parser.print_help()
174 fc1055878775 Use "return 0" instead of "sys.exit(0)" when printing version and help# Franz Glasner <fzglas.hg@dom66.de> parents: 173 diff changeset	224 return 0
144 b39f8082ced1 Make a "version" subcommand to also print the program's version number for "treesum" Franz Glasner <fzglas.hg@dom66.de> parents: 143 diff changeset	225
146 7d8df8311e3b Optimize argument parsing for the "version" command Franz Glasner <fzglas.hg@dom66.de> parents: 145 diff changeset	226 # Reparse strictly
7d8df8311e3b Optimize argument parsing for the "version" command Franz Glasner <fzglas.hg@dom66.de> parents: 145 diff changeset	227 opts = parser.parse_args(args=argv)
7d8df8311e3b Optimize argument parsing for the "version" command Franz Glasner <fzglas.hg@dom66.de> parents: 145 diff changeset	228
176 7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	229 # Minimal logging -- just for debugging - not for more "normal" use
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	230 logging.basicConfig(
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	231 level=logging.DEBUG if opts.debug else logging.WARNING,
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	232 stream=sys.stderr,
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	233 format="[%(asctime)s][%(levelname)s][%(process)d:%(name)s] %(message)s"
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	234 )
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	235 logging.captureWarnings(True)
7f5d05a625fd Implement preconditions for some debug logging Franz Glasner <fzglas.hg@dom66.de> parents: 174 diff changeset	236
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	237 return treesum(opts)
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	238
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	239
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	240 def gen_generate_opts(directories=[],
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	241 algorithm=util.default_algotag(),
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	242 append_output=False,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	243 base64=False,
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	244 comment=[],
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	245 follow_directory_symlinks=False,
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	246 full_mode=False,
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	247 logical=None,
150 f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	248 minimal=None,
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	249 mode=False,
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	250 mmap=None,
151 b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	251 mtime=False,
168 bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	252 output=None,
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	253 print_size=False,
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	254 size_only=False):
131 3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	255 opts = argparse.Namespace(
3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	256 directories=directories,
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	257 algorithm=util.argv2algo(algorithm),
131 3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	258 append_output=append_output,
3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	259 base64=base64,
135 dbf27681a1f6 Allow to put comments into the output with "--comment" Franz Glasner <fzglas.hg@dom66.de> parents: 134 diff changeset	260 comment=comment,
131 3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	261 follow_directory_symlinks=follow_directory_symlinks,
3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	262 logical=logical,
150 f84cf853da22 Implement "--minimal [TAG]" for treesum. Franz Glasner <fzglas.hg@dom66.de> parents: 149 diff changeset	263 minimal=minimal,
131 3a18d71d7c50 Implement --follow-directory-symlinks when walking a directory tree Franz Glasner <fzglas.hg@dom66.de> parents: 130 diff changeset	264 mmap=mmap,
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	265 metadata_full_mode=full_mode,
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	266 metadata_mode=mode,
151 b26c4290e928 Implement "--mtime" for treesum to include a file's mtime in a directory digest. Franz Glasner <fzglas.hg@dom66.de> parents: 150 diff changeset	267 metadata_mtime=mtime,
168 bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	268 output=output,
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	269 print_size=print_size,
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	270 size_only=size_only)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	271 return opts
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	272
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	273
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	274 def gen_info_opts(digest_files=[], last=False):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	275 opts = argparse.Namespace(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	276 digest_files=digest_files,
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	277 print_only_last_block=last)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	278 return opts
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	279
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	280
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	281 def treesum(opts):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	282 # XXX TBD: opts.check and opts.checklist (as in shasum.py)
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	283 if opts.subcommand in ("generate", "gen"):
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	284 return generate_treesum(opts)
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	285 elif opts.subcommand == "info":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	286 return print_treesum_digestfile_infos(opts)
142 71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	287 else:
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	288 raise RuntimeError(
71747e45b52c Prepare for using subcommands in "treesum.py". Franz Glasner <fzglas.hg@dom66.de> parents: 137 diff changeset	289 "command `{}' not yet handled".format(opts.subcommand))
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	290
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	291
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	292 def generate_treesum(opts):
149 f717854be1de Put the defaults generation when generating directory digests into "generate_treesum()" instead of "main()" Franz Glasner <fzglas.hg@dom66.de> parents: 148 diff changeset	293 # Provide defaults
f717854be1de Put the defaults generation when generating directory digests into "generate_treesum()" instead of "main()" Franz Glasner <fzglas.hg@dom66.de> parents: 148 diff changeset	294 if not opts.algorithm:
172 804a823c63f5 Now the selection of the default algorithm depends on availiability in hashlib Franz Glasner <fzglas.hg@dom66.de> parents: 170 diff changeset	295 opts.algorithm = util.argv2algo(util.default_algotag())
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	296 if not opts.directories:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	297 opts.directories.append(".")
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	298
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	299 if opts.output is None or opts.output == "-":
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	300 if hasattr(sys.stdout, "buffer"):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	301 out_cm = cm.nullcontext(sys.stdout.buffer)
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	302 else:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	303 out_cm = cm.nullcontext(sys.stdout)
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	304 else:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	305 if opts.append_output:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	306 out_cm = open(opts.output, "ab")
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	307 else:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	308 out_cm = open(opts.output, "wb")
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	309 out_cm = CRC32Output(out_cm)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	310
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	311 with out_cm as outfp:
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	312 for d in opts.directories:
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	313
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	314 V1DirectoryTreesumGenerator(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	315 opts.algorithm, opts.mmap, opts.base64, opts.logical,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	316 opts.follow_directory_symlinks,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	317 opts.metadata_mode,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	318 opts.metadata_full_mode,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	319 opts.metadata_mtime,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	320 opts.size_only,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	321 opts.print_size,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	322 minimal=opts.minimal).generate(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	323 outfp, d, comment=opts.comment)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	324
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	325
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	326 class V1DirectoryTreesumGenerator(object):
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	327
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	328 def __init__(self, algorithm, use_mmap, use_base64,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	329 handle_root_logical, follow_directory_symlinks,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	330 with_metadata_mode, with_metadata_full_mode,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	331 with_metadata_mtime, size_only, print_size,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	332 minimal=None,):
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	333 super(V1DirectoryTreesumGenerator, self).__init__()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	334 self._algorithm = algorithm
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	335 self._use_mmap = use_mmap
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	336 self._use_base64 = use_base64
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	337 self._handle_root_logical = handle_root_logical
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	338 self._follow_directory_symlinks = follow_directory_symlinks
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	339 self._with_metadata_mode = with_metadata_mode
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	340 self._with_metadata_full_mode = with_metadata_full_mode
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	341 self._with_metadata_mtime = with_metadata_mtime
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	342 self._size_only = size_only
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	343 self._print_size = print_size
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	344 self._minimal = minimal
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	345
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	346 def generate(self, outfp, root, comment=None):
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	347 """
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	348
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	349 :param outfp: a binary file with a "write()" and a "flush()" method
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	350
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	351 """
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	352 self._outfp = outfp
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	353 self._outfp.resetdigest()
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	354 self._outfp.write(format_bsd_line("VERSION", "1", None, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	355 self._outfp.flush()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	356
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	357 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	358 # Note: Given non-default flags that are relevant for
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	359 # directory traversal.
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	360 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	361 flags = []
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	362 if self._with_metadata_full_mode:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	363 flags.append("with-metadata-fullmode")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	364 elif self._with_metadata_mode:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	365 flags.append("with-metadata-mode")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	366 if self._with_metadata_mtime:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	367 flags.append("with-metadata-mtime")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	368 if self._handle_root_logical:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	369 flags.append("logical")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	370 if self._follow_directory_symlinks:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	371 flags.append("follow-directory-symlinks")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	372 if self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	373 flags.append("size-only")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	374 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	375 if self._print_size:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	376 flags.append("print-size")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	377 if flags:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	378 flags.sort()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	379 self._outfp.write(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	380 format_bsd_line("FLAGS", ",".join(flags), None, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	381
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	382 if self._minimal is None:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	383 # Write execution timestamps in POSIX epoch and ISO format
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	384 ts = int(time.time())
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	385 self._outfp.write(format_bsd_line("TIMESTAMP", ts, None, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	386 ts = (datetime.datetime.utcfromtimestamp(ts)).isoformat("T")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	387 self._outfp.write(format_bsd_line("ISOTIMESTAMP", ts, None, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	388
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	389 if comment:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	390 for line in comment:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	391 self._outfp.write(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	392 format_bsd_line("COMMENT", None, line, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	393
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	394 if self._minimal is not None:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	395 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	396 "ROOT", None, self._minimal if self._minimal else "", False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	397 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	398 self._outfp.write(format_bsd_line("ROOT", None, root, False))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	399 self._outfp.flush()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	400
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	401 if not self._handle_root_logical and os.path.islink(root):
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	402 linktgt = util.fsencode(os.readlink(root))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	403 linkdgst = self._algorithm[0]()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	404 linkdgst.update(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	405 util.interpolate_bytes(b"%d:%s,", len(linktgt), linktgt))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	406 dir_dgst = self._algorithm[0]()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	407 dir_dgst.update(b"1:L,")
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	408 dir_dgst.update(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	409 util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	410 b"%d:%s,", len(linkdgst.digest()), linkdgst.digest()))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	411 if self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	412 self._outfp.write(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	413 format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	414 "SIZE",
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	415 None,
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	416 "./@/",
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	417 False,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	418 0))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	419 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	420 self._outfp.write(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	421 format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	422 self._algorithm[1],
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	423 dir_dgst.digest(),
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	424 "./@/",
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	425 self._use_base64))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	426 self._outfp.flush()
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	427 self._outfp.write(format_bsd_line(
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	428 "CRC32", self._outfp.hexcrcdigest(), None, False))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	429 return
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	430
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	431 self._generate(os.path.normpath(root), tuple())
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	432 self._outfp.write(format_bsd_line(
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	433 "CRC32", self._outfp.hexcrcdigest(), None, False))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	434
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	435 def _generate(self, root, top):
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	436 logging.debug("Handling %s/%r", root, top)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	437 path = os.path.join(root, *top) if top else root
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	438 with walk.ScanDir(path) as dirscan:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	439 fsobjects = list(dirscan)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	440 fsobjects.sort(key=walk.WalkDirEntry.sort_key)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	441 dir_dgst = self._algorithm[0]()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	442 dir_size = 0
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	443 for fso in fsobjects:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	444 if fso.is_dir:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	445 if fso.is_symlink and not self._follow_directory_symlinks:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	446 linktgt = util.fsencode(os.readlink(fso.path))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	447 linkdgst = self._algorithm[0]()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	448 linkdgst.update(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	449 util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	450 b"%d:%s,", len(linktgt), linktgt))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	451 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	452 b"1:S,%d:%s,", len(fso.fsname), fso.fsname))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	453 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	454 # - no mtime and no mode for symlinks
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	455 # - also does not count for dir_size
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	456 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	457 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	458 b"%d:%s,",
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	459 len(linkdgst.digest()), linkdgst.digest()))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	460 opath = "/".join(top) + "/" + fso.name if top else fso.name
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	461 if self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	462 self._outfp.write(format_bsd_line(
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	463 "SIZE", None, "%s/./@/" % (opath,), False, 0))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	464 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	465 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	466 self._algorithm[1],
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	467 linkdgst.digest(),
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	468 "%s/./@/" % (opath,),
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	469 self._use_base64))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	470 self._outfp.flush()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	471 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	472 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	473 # Follow the symlink to dir or handle a "real" directory
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	474 #
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	475
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	476 # Get subdir data from recursing into it
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	477 sub_dir_dgst, sub_dir_size = self._generate(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	478 root, top + (fso.name, ))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	479
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	480 dir_size += sub_dir_size
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	481 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	482 b"1:d,%d:%s,", len(fso.fsname), fso.fsname))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	483 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	484 b"%d:%s,", len(sub_dir_dgst), sub_dir_dgst))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	485 if self._with_metadata_full_mode:
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	486 modestr = util.b(normalized_mode_str(fso.stat.st_mode))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	487 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	488 b"8:fullmode,%d:%s,", len(modestr), modestr))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	489 elif self._with_metadata_mode:
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	490 modestr = util.b(normalized_compatible_mode_str(
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	491 fso.stat.st_mode))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	492 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	493 b"4:mode,%d:%s,", len(modestr), modestr))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	494 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	495 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	496 b"1:f,%d:%s,", len(fso.fsname), fso.fsname))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	497 dir_size += fso.stat.st_size
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	498 if self._with_metadata_mtime:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	499 mtime = datetime.datetime.utcfromtimestamp(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	500 int(fso.stat.st_mtime))
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	501 mtime = util.b(mtime.isoformat("T") + "Z")
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	502 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	503 b"5:mtime,%d:%s,", len(mtime), mtime))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	504 if self._with_metadata_full_mode:
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	505 modestr = util.b(normalized_mode_str(fso.stat.st_mode))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	506 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	507 b"8:fullmode,%d:%s,", len(modestr), modestr))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	508 elif self._with_metadata_mode:
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	509 modestr = util.b(normalized_compatible_mode_str(
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	510 fso.stat.st_mode))
177 089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	511 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	512 b"4:mode,%d:%s,", len(modestr), modestr))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	513 if not self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	514 dgst = digest.compute_digest_file(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	515 self._algorithm[0], fso.path, use_mmap=self._use_mmap)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	516 dir_dgst.update(util.interpolate_bytes(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	517 b"%d:%s,", len(dgst), dgst))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	518 opath = "/".join(top) + "/" + fso.name if top else fso.name
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	519 if self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	520 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	521 "SIZE", None, opath, False, fso.stat.st_size))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	522 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	523 if self._print_size:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	524 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	525 self._algorithm[1], dgst, opath, self._use_base64,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	526 fso.stat.st_size))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	527 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	528 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	529 self._algorithm[1], dgst, opath,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	530 self._use_base64))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	531 self._outfp.flush()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	532
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	533 opath = "/".join(top) + "/" if top else ""
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	534 if self._size_only:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	535 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	536 "SIZE", None, opath, False, dir_size))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	537 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	538 if self._print_size:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	539 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	540 self._algorithm[1], dir_dgst.digest(), opath,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	541 self._use_base64, dir_size))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	542 else:
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	543 self._outfp.write(format_bsd_line(
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	544 self._algorithm[1], dir_dgst.digest(), opath,
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	545 self._use_base64))
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	546 self._outfp.flush()
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	547 return (dir_dgst.digest(), dir_size)
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	548
089c40240061 Add an alternate implementation for generating directory tree digests: Franz Glasner <fzglas.hg@dom66.de> parents: 176 diff changeset	549
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	550 class CRC32Output(object):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	551
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	552 """Wrapper for a minimal binary file contextmanager that calculates
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	553 the CRC32 of the written bytes on the fly.
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	554
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	555 Also acts as context manager proxy for the given context manager.
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	556
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	557 """
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	558
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	559 __slots__ = ("_fp_cm", "_fp", "_crc32")
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	560
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	561 def __init__(self, fp_cm):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	562 self._fp_cm = fp_cm
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	563 self._fp = None
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	564 self.resetdigest()
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	565
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	566 def __enter__(self):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	567 assert self._fp is None
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	568 self._fp = self._fp_cm.__enter__()
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	569 return self
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	570
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	571 def __exit__(self, *args):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	572 rv = self._fp_cm.__exit__(*args)
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	573 self._fp = None
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	574 return rv
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	575
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	576 def write(self, what):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	577 self._fp.write(what)
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	578 self._crc32 = zlib.crc32(what, self._crc32)
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	579
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	580 def flush(self):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	581 self._fp.flush()
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	582
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	583 def resetdigest(self):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	584 """Reset the current CRC digest"""
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	585 self._crc32 = zlib.crc32(b"")
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	586
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	587 def hexcrcdigest(self):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	588 """
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	589
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	590 :rtype: str
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	591
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	592 """
197 48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	593 return (hex(self.crcdigest())[2:]).upper()
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	594
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	595 def crcdigest(self):
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	596 """
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	597
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	598 :rtype: int
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	599
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	600 """
197 48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	601 if util.PY2:
48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	602 # Return the bitpattern as unsigned 32-bit number
48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	603 return (~self._crc32 ^ 0xFFFFFFFF)
48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	604 else:
48e2610978e5 Return the CRC-32 digest on Python2 as unsigned int instead of a signed one. Franz Glasner <fzglas.hg@dom66.de> parents: 193 diff changeset	605 return self._crc32
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	606
53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	607
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	608 def normalized_compatible_mode_str(mode):
157 27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	609 # XXX FIXME: Windows and "executable"
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	610 modebits = stat.S_IMODE(mode)
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	611 modestr = "%o" % (modebits,)
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	612 if not modestr.startswith("0"):
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	613 modestr = "0" + modestr
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	614 return modestr
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	615
27d1aaf5fe39 Implement "--mode" flag for "treesum.py" to consider file portable mode bits Franz Glasner <fzglas.hg@dom66.de> parents: 156 diff changeset	616
158 d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	617 def normalized_mode_str(mode):
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	618 modestr = "%o" % (mode,)
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	619 if not modestr.startswith("0"):
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	620 modestr = "0" + modestr
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	621 return modestr
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	622
d8cdd1985d43 Implement "--full-mode" for "treesum.py" Franz Glasner <fzglas.hg@dom66.de> parents: 157 diff changeset	623
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	624 def format_bsd_line(what, value, filename, use_base64, size=None):
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	625 ls = util.b(os.linesep)
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	626 if not isinstance(what, bytes):
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	627 what = what.encode("ascii")
8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	628 if what == b"TIMESTAMP":
129 bdd8ea43074b Output FLAGS as line "FLAGS = ..." instead of "FLAGS (...)" Franz Glasner <fzglas.hg@dom66.de> parents: 128 diff changeset	629 assert filename is None
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	630 return util.interpolate_bytes(b"TIMESTAMP = %d%s", value, ls)
179 53614a724bf0 Also write a (standard) CRC-32 checksum for each block of output Franz Glasner <fzglas.hg@dom66.de> parents: 178 diff changeset	631 if what in (b"ISOTIMESTAMP", b"FLAGS", b"VERSION", b"CRC32"):
129 bdd8ea43074b Output FLAGS as line "FLAGS = ..." instead of "FLAGS (...)" Franz Glasner <fzglas.hg@dom66.de> parents: 128 diff changeset	632 assert filename is None
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	633 return util.interpolate_bytes(b"%s = %s%s", what, util.b(value), ls)
129 bdd8ea43074b Output FLAGS as line "FLAGS = ..." instead of "FLAGS (...)" Franz Glasner <fzglas.hg@dom66.de> parents: 128 diff changeset	634 assert filename is not None
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	635 if what == b"COMMENT":
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	636 return util.interpolate_bytes(
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	637 b"COMMENT (%s)%s", util.b(filename, "utf-8"), ls)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	638 if not isinstance(filename, bytes):
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	639 filename = util.fsencode(filename)
170 8945be6b404e Mode for treesum.py to print only the size of files and the accumulated size of a directory: --size-only. Franz Glasner <fzglas.hg@dom66.de> parents: 169 diff changeset	640 if what == b"SIZE":
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	641 return util.interpolate_bytes(b"SIZE (%s) = %d%s", filename, size, ls)
128 7c646921a479 Add TIMESTAMP and ISOTIMESTAMP to the output Franz Glasner <fzglas.hg@dom66.de> parents: 127 diff changeset	642 if value is None:
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	643 return util.interpolate_bytes(b"%s (%s)%s", what, filename, ls)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	644 if use_base64:
128 7c646921a479 Add TIMESTAMP and ISOTIMESTAMP to the output Franz Glasner <fzglas.hg@dom66.de> parents: 127 diff changeset	645 value = base64.b64encode(value)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	646 else:
128 7c646921a479 Add TIMESTAMP and ISOTIMESTAMP to the output Franz Glasner <fzglas.hg@dom66.de> parents: 127 diff changeset	647 value = binascii.hexlify(value)
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	648 if filename != b"./@/":
127 6a50d02fe0ca Change the filename output: make it more consistent Franz Glasner <fzglas.hg@dom66.de> parents: 125 diff changeset	649 filename = util.normalize_filename(filename, True)
168 bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	650 if size is None:
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	651 return util.interpolate_bytes(
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	652 b"%s (%s) = %s%s", what, filename, value, ls)
168 bcc4441cf216 Implement "--print-size" to print file and accumulated directory sizes also. Franz Glasner <fzglas.hg@dom66.de> parents: 161 diff changeset	653 else:
173 e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	654 return util.interpolate_bytes(
e081b6ee5570 treesum.py now runs on Python3.4 also: use a workaround for its missing byte % formatting. Franz Glasner <fzglas.hg@dom66.de> parents: 172 diff changeset	655 b"%s (%s) = %s,%d%s", what, filename, value, size, ls)
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	656
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	657
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	658 class TreesumReader(object):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	659
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	660 """Reader to read and/or verify treesum digest files.
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	661
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	662 Supports the iterator and context manager protocol.
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	663
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	664 """
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	665
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	666 PATTERN0 = re.compile(br"\A[ \t]*\r?\n\Z") # empty lines
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	667 PATTERN1 = re.compile(br"\A(VERSION\|FLAGS\|TIMESTAMP\|ISOTIMESTAMP\|CRC32)[ \t]=[ \t]([^ \t]+)[ \t]*\r?\n\Z") # noqa: E501 line too long
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	668 PATTERN2 = re.compile(br"\A(ROOT\|COMMENT)[ \t]\((.)\)[ \t]*\r?\n\Z")
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	669 PATTERN3 = re.compile(br"\ASIZE[ \t]\((.)\)[ \t]=[ \t](\d+)[ \t]*\r?\n\Z") # noqa: E501 line too long
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	670 PATTERN4 = re.compile(br"\A([A-Za-z0-9_-]+)[ \t]\((.)\)[ \t]=[ \t]([A-Za-z0-9=+/]+)(,(\d+))?[ \t]*\r?\n\Z") # noqa: E501 line too long
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	671
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	672 def __init__(self, _fp, _filename, _own_fp):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	673 self._fp = _fp
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	674 self._own_fp = _own_fp
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	675 self._filename = _filename
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	676 self._line_no = 0
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	677 self._reset_crc()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	678 self._expect_crc = None # NOTE: tristate: None is different from False
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	679 self._current_algo_name = self._current_algo_digest_size = None
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	680
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	681 @classmethod
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	682 def from_path(cls_, path):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	683 """Open file at `path` and return a reader that owns the file object"""
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	684 return cls_(open(path, "rb"), path, True)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	685
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	686 @classmethod
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	687 def from_binary_buffer(cls_, binary_fp, filename):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	688 return cls_(binary_fp, filename, False)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	689
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	690 def __enter__(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	691 return self
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	692
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	693 def __exit__(self, *args):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	694 self.close()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	695
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	696 def close(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	697 if self._fp is not None:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	698 try:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	699 if self._own_fp:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	700 self._fp.close()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	701 finally:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	702 self._fp = None
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	703
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	704 def __iter__(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	705 return self
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	706
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	707 def __next__(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	708 rec = self.read_record()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	709 if rec is None:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	710 raise StopIteration()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	711 return rec
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	712
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	713 if util.PY2:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	714 next = __next__
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	715
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	716 def all_records(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	717 """Iterator over all remaining records"""
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	718 while True:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	719 rec = self.read_record()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	720 if rec is None:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	721 return
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	722 yield rec
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	723
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	724 def read_record(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	725 """Read and parse the "next" line.
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	726
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	727 :returns: `None` at EOF or the parsed contents of the line
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	728 :rtype: tuple or None
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	729
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	730 """
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	731 # Loop to skip empty lines
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	732 while True:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	733 line = self._get_next_line()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	734 if not line:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	735 #
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	736 # Skip for empty files at the very beginning.
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	737 # Check only after the first VERSION line.
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	738 #
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	739 if self._expect_crc is not None:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	740 if self._expect_crc:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	741 logging.warning("CRC32 is missing at EOF")
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	742 return None
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	743 if not self.PATTERN0.search(line):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	744 break
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	745 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	746 #
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	747 # At the beginning transparently skip an eventually embedded signify
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	748 # signature
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	749 #
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	750 if self._line_no == 1:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	751 if line.startswith(b"untrusted comment: "):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	752 line = self._get_next_line()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	753 if not line.endswith(b"\n"):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	754 raise binascii.Error("No valid signify signature value")
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	755 # Try to decode for an early error check
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	756 base64.b64decode(line[:-1])
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	757 mo = self.PATTERN1.search(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	758 if mo:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	759 if mo.group(1) == b"VERSION":
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	760 if self._expect_crc:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	761 logging.warning("CRC32 missing before line %d",
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	762 self._line_no)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	763 self._reset_crc()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	764 self._expect_crc = True
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	765 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	766 return ("VERSION", util.n(mo.group(2)))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	767 if mo.group(1) == b"CRC32":
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	768 # TODO: check
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	769 if self._expect_crc is None:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	770 logging.warning("Lone CRC32 before VERSION in line %d",
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	771 self._line_no)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	772 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	773 if self._expect_crc:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	774 if (self._hex_crc()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	775 != mo.group(2).decode("latin1").upper()):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	776 logging.warning(
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	777 "CRC32 mismatch in line %d:"
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	778 " expected: %s, given: %s",
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	779 self._line_no,
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	780 self._hex_crc(),
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	781 mo.group(2).decode("latin1").upper())
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	782 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	783 logging.warning("CRC32 before VERSION in line %d",
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	784 self._line_no)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	785 # Do not update the CRC here but reset the state
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	786 self._expect_crc = False
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	787 return ("CRC32", util.n(mo.group(2)))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	788 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	789 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	790 return (util.n(mo.group(1)), util.n(mo.group(2)))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	791 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	792 mo = self.PATTERN2.search(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	793 if mo:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	794 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	795 if mo.group(1) == b"COMMENT":
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	796 return ("COMMENT", util.u(mo.group(2), "utf-8"))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	797 elif mo.group(1) == b"ROOT":
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	798 return ("ROOT", mo.group(2))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	799 assert False, line
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	800 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	801 mo = self.PATTERN3.search(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	802 if mo:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	803 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	804 return ("SIZE", mo.group(1), int(util.n(mo.group(2)), 10))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	805 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	806 mo = self.PATTERN4.search(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	807 if mo:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	808 self._update_crc(line)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	809 algo_name = util.n(mo.group(1))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	810 if (len(mo.group(3)) ==
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	811 2 * self._get_digest_size(algo_name)):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	812 # hex
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	813 digest = binascii.unhexlify(mo.group(3))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	814 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	815 # base64
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	816 digest = base64.b64decode(mo.group(3))
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	817 if mo.group(4):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	818 size = int(util.n(mo.group(5)), 10)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	819 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	820 size = None
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	821 return (algo_name, mo.group(2), digest, size)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	822 else:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	823 assert False, line
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	824 return line
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	825
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	826 def _get_next_line(self):
189 959c6d37b014 Extend the max line length to read to 4096, which is something along PATH_MAX onn Linux Franz Glasner <fzglas.hg@dom66.de> parents: 188 diff changeset	827 line = self._fp.readline(4096) # along PATH_MAX on Linux
188 2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	828 if line:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	829 self._line_no += 1
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	830 return line
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	831
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	832 def _reset_crc(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	833 self._crc32 = zlib.crc32(b"")
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	834
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	835 def _update_crc(self, data):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	836 self._crc32 = zlib.crc32(data, self._crc32)
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	837
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	838 def _hex_crc(self):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	839 return (hex(self._crc32)[2:]).upper()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	840
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	841 def _get_digest_size(self, algo_name):
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	842 if self._current_algo_name == algo_name:
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	843 return self._current_algo_digest_size
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	844 h = util.algotag2algotype(algo_name)()
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	845 self._current_algo_name = algo_name
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	846 self._current_algo_digest_size = h.digest_size
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	847 return self._current_algo_digest_size
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	848
2784fdcc99e5 Implement basic parsing of treesum output. Franz Glasner <fzglas.hg@dom66.de> parents: 179 diff changeset	849
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	850 def print_treesum_digestfile_infos(opts):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	851 print_infos_for_digestfile(opts.digest_files, opts.print_only_last_block)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	852
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	853
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	854 def print_infos_for_digestfile(digest_files, print_only_last_block=True):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	855 for fn in digest_files:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	856 if fn == "-":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	857 if util.PY2:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	858 reader = TreesumReader.from_binary_buffer(sys.stdin)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	859 else:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	860 reader = TreesumReader.from_binary_buffer(sys.stdin.buffer)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	861 else:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	862 reader = TreesumReader.from_path(fn)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	863
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	864 with reader:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	865 root = flags = algorithm = digest = size = None
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	866 comments = []
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	867 in_block = False
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	868 block_no = 0
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	869 for record in reader:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	870 if record[0] == "VERSION":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	871 assert record[1] == "1"
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	872 # start a new block
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	873 in_block = True
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	874 block_no += 1
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	875 root = flags = algorithm = digest = size = None
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	876 comments = []
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	877 elif record[0] == "FLAGS":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	878 flags = record[1]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	879 elif record[0] == "ROOT":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	880 root = record[1]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	881 elif record[0] == "COMMENT":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	882 comments.append(record[1])
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	883 elif record[0] in ("TIMESTAMP", "ISOTIMESTAMP"):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	884 pass
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	885 elif record[0] == "CRC32":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	886 pass
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	887 # in_block = False
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	888 else:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	889 if not in_block:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	890 continue
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	891 # digest line or size line
193 fb36e71f6ba8 Change: path indicators for symlinks: ./@ -> ./@/ and /./@ -> /./@/ Franz Glasner <fzglas.hg@dom66.de> parents: 191 diff changeset	892 if not record[1] or record[1] == b"./@/":
190 7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	893 if record[0] == "SIZE":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	894 algorithm = "SIZE"
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	895 digest = None
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	896 size = record[2]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	897 else:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	898 algorithm = record[0]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	899 digest = record[2]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	900 size = record[3]
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	901 if not print_only_last_block:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	902 print_block_data(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	903 block_no,
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	904 root, flags, comments, algorithm, digest, size)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	905 root = flags = algorithm = digest = size = None
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	906 in_block = False
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	907 if print_only_last_block:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	908 if not in_block:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	909 if digest is not None or size is not None:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	910 print_block_data(
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	911 block_no,
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	912 root, flags, comments, algorithm, digest, size)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	913 else:
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	914 logging.warning("missing block end")
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	915
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	916
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	917 def print_block_data(block_no, tag, flags, comments, algorithm, digest, size):
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	918 digeststr = util.n(binascii.hexlify(digest)) if digest else "<no digest>"
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	919 sizestr = str(size) if size is not None else "<no size>"
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	920 print("BLOCK No %d:" % (block_no,))
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	921 print(" Tag:", tag)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	922 print(" Flags:", flags if flags else "<none>")
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	923 print(" Comments:", comments if comments else "")
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	924 print(" Algorithm:", algorithm)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	925 if algorithm != "SIZE":
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	926 print(" Digest:", digeststr)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	927 print(" Size:", sizestr)
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	928
7e0c25a31757 First implementation of "treeview info" to print some information from the treeview digest files Franz Glasner <fzglas.hg@dom66.de> parents: 189 diff changeset	929
124 3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	930 if __name__ == "__main__":
3bd3f32b5e60 A first version of "treesum" is working Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	931 sys.exit(main())

Mercurial > hgrepos > Python > apps > py-cutils

annotate cutils/treesum.py @ 197:48e2610978e5