Mercurial > hgrepos > Python2 > PyMuPDF

comparison mupdf-source/thirdparty/libjpeg/structure.txt @ 2:b50eed0cc0ef upstream

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
ADD: MuPDF v1.26.7: the MuPDF source as downloaded by a default build of PyMuPDF 1.26.4. The directory name has changed: no version number in the expanded directory now.
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 15 Sep 2025 11:43:07 +0200
parents
children
comparison
equal deleted inserted replaced
1:1d09e1dec1d9 2:b50eed0cc0ef
1 IJG JPEG LIBRARY: SYSTEM ARCHITECTURE
2
3 Copyright (C) 1991-2013, Thomas G. Lane, Guido Vollbeding.
4 This file is part of the Independent JPEG Group's software.
5 For conditions of distribution and use, see the accompanying README file.
6
7
8 This file provides an overview of the architecture of the IJG JPEG software;
9 that is, the functions of the various modules in the system and the interfaces
10 between modules. For more precise details about any data structure or calling
11 convention, see the include files and comments in the source code.
12
13 We assume that the reader is already somewhat familiar with the JPEG standard.
14 The README file includes references for learning about JPEG. The file
15 libjpeg.txt describes the library from the viewpoint of an application
16 programmer using the library; it's best to read that file before this one.
17 Also, the file coderules.txt describes the coding style conventions we use.
18
19 In this document, JPEG-specific terminology follows the JPEG standard:
20 A "component" means a color channel, e.g., Red or Luminance.
21 A "sample" is a single component value (i.e., one number in the image data).
22 A "coefficient" is a frequency coefficient (a DCT transform output number).
23 A "block" is an array of samples or coefficients.
24 An "MCU" (minimum coded unit) is an interleaved set of blocks of size
25 determined by the sampling factors, or a single block in a
26 noninterleaved scan.
27 We do not use the terms "pixel" and "sample" interchangeably. When we say
28 pixel, we mean an element of the full-size image, while a sample is an element
29 of the downsampled image. Thus the number of samples may vary across
30 components while the number of pixels does not. (This terminology is not used
31 rigorously throughout the code, but it is used in places where confusion would
32 otherwise result.)
33
34
35 *** System features ***
36
37 The IJG distribution contains two parts:
38 * A subroutine library for JPEG compression and decompression.
39 * cjpeg/djpeg, two sample applications that use the library to transform
40 JFIF JPEG files to and from several other image formats.
41 cjpeg/djpeg are of no great intellectual complexity: they merely add a simple
42 command-line user interface and I/O routines for several uncompressed image
43 formats. This document concentrates on the library itself.
44
45 We desire the library to be capable of supporting all JPEG baseline, extended
46 sequential, and progressive DCT processes. The library does not support the
47 hierarchical or lossless processes defined in the standard.
48
49 Within these limits, any set of compression parameters allowed by the JPEG
50 spec should be readable for decompression. (We can be more restrictive about
51 what formats we can generate.) Although the system design allows for all
52 parameter values, some uncommon settings are not yet implemented and may
53 never be; nonintegral sampling ratios are the prime example. Furthermore,
54 we treat 8-bit vs. 12-bit data precision as a compile-time switch, not a
55 run-time option, because most machines can store 8-bit pixels much more
56 compactly than 12-bit.
57
58 By itself, the library handles only interchange JPEG datastreams --- in
59 particular the widely used JFIF file format. The library can be used by
60 surrounding code to process interchange or abbreviated JPEG datastreams that
61 are embedded in more complex file formats. (For example, libtiff uses this
62 library to implement JPEG compression within the TIFF file format.)
63
64 The library includes a substantial amount of code that is not covered by the
65 JPEG standard but is necessary for typical applications of JPEG. These
66 functions preprocess the image before JPEG compression or postprocess it after
67 decompression. They include colorspace conversion, downsampling/upsampling,
68 and color quantization. This code can be omitted if not needed.
69
70 A wide range of quality vs. speed tradeoffs are possible in JPEG processing,
71 and even more so in decompression postprocessing. The decompression library
72 provides multiple implementations that cover most of the useful tradeoffs,
73 ranging from very-high-quality down to fast-preview operation. On the
74 compression side we have generally not provided low-quality choices, since
75 compression is normally less time-critical. It should be understood that the
76 low-quality modes may not meet the JPEG standard's accuracy requirements;
77 nonetheless, they are useful for viewers.
78
79
80 *** Portability issues ***
81
82 Portability is an essential requirement for the library. The key portability
83 issues that show up at the level of system architecture are:
84
85 1. Memory usage. We want the code to be able to run on PC-class machines
86 with limited memory. Images should therefore be processed sequentially (in
87 strips), to avoid holding the whole image in memory at once. Where a
88 full-image buffer is necessary, we should be able to use either virtual memory
89 or temporary files.
90
91 2. Near/far pointer distinction. To run efficiently on 80x86 machines, the
92 code should distinguish "small" objects (kept in near data space) from
93 "large" ones (kept in far data space). This is an annoying restriction, but
94 fortunately it does not impact code quality for less brain-damaged machines,
95 and the source code clutter turns out to be minimal with sufficient use of
96 pointer typedefs.
97
98 3. Data precision. We assume that "char" is at least 8 bits, "short" and
99 "int" at least 16, "long" at least 32. The code will work fine with larger
100 data sizes, although memory may be used inefficiently in some cases. However,
101 the JPEG compressed datastream must ultimately appear on external storage as a
102 sequence of 8-bit bytes if it is to conform to the standard. This may pose a
103 problem on machines where char is wider than 8 bits. The library represents
104 compressed data as an array of values of typedef JOCTET. If no data type
105 exactly 8 bits wide is available, custom data source and data destination
106 modules must be written to unpack and pack the chosen JOCTET datatype into
107 8-bit external representation.
108
109
110 *** System overview ***
111
112 The compressor and decompressor are each divided into two main sections:
113 the JPEG compressor or decompressor proper, and the preprocessing or
114 postprocessing functions. The interface between these two sections is the
115 image data that the official JPEG spec regards as its input or output: this
116 data is in the colorspace to be used for compression, and it is downsampled
117 to the sampling factors to be used. The preprocessing and postprocessing
118 steps are responsible for converting a normal image representation to or from
119 this form. (Those few applications that want to deal with YCbCr downsampled
120 data can skip the preprocessing or postprocessing step.)
121
122 Looking more closely, the compressor library contains the following main
123 elements:
124
125 Preprocessing:
126 * Color space conversion (e.g., RGB to YCbCr).
127 * Edge expansion and downsampling. Optionally, this step can do simple
128 smoothing --- this is often helpful for low-quality source data.
129 JPEG proper:
130 * MCU assembly, DCT, quantization.
131 * Entropy coding (sequential or progressive, Huffman or arithmetic).
132
133 In addition to these modules we need overall control, marker generation,
134 and support code (memory management & error handling). There is also a
135 module responsible for physically writing the output data --- typically
136 this is just an interface to fwrite(), but some applications may need to
137 do something else with the data.
138
139 The decompressor library contains the following main elements:
140
141 JPEG proper:
142 * Entropy decoding (sequential or progressive, Huffman or arithmetic).
143 * Dequantization, inverse DCT, MCU disassembly.
144 Postprocessing:
145 * Upsampling. Optionally, this step may be able to do more general
146 rescaling of the image.
147 * Color space conversion (e.g., YCbCr to RGB). This step may also
148 provide gamma adjustment [ currently it does not ].
149 * Optional color quantization (e.g., reduction to 256 colors).
150 * Optional color precision reduction (e.g., 24-bit to 15-bit color).
151 [This feature is not currently implemented.]
152
153 We also need overall control, marker parsing, and a data source module.
154 The support code (memory management & error handling) can be shared with
155 the compression half of the library.
156
157 There may be several implementations of each of these elements, particularly
158 in the decompressor, where a wide range of speed/quality tradeoffs is very
159 useful. It must be understood that some of the best speedups involve
160 merging adjacent steps in the pipeline. For example, upsampling, color space
161 conversion, and color quantization might all be done at once when using a
162 low-quality ordered-dither technique. The system architecture is designed to
163 allow such merging where appropriate.
164
165
166 Note: it is convenient to regard edge expansion (padding to block boundaries)
167 as a preprocessing/postprocessing function, even though the JPEG spec includes
168 it in compression/decompression. We do this because downsampling/upsampling
169 can be simplified a little if they work on padded data: it's not necessary to
170 have special cases at the right and bottom edges. Therefore the interface
171 buffer is always an integral number of blocks wide and high, and we expect
172 compression preprocessing to pad the source data properly. Padding will occur
173 only to the next block (block_size-sample) boundary. In an interleaved-scan
174 situation, additional dummy blocks may be used to fill out MCUs, but the MCU
175 assembly and disassembly logic will create or discard these blocks internally.
176 (This is advantageous for speed reasons, since we avoid DCTing the dummy
177 blocks. It also permits a small reduction in file size, because the
178 compressor can choose dummy block contents so as to minimize their size
179 in compressed form. Finally, it makes the interface buffer specification
180 independent of whether the file is actually interleaved or not.)
181 Applications that wish to deal directly with the downsampled data must
182 provide similar buffering and padding for odd-sized images.
183
184
185 *** Poor man's object-oriented programming ***
186
187 It should be clear by now that we have a lot of quasi-independent processing
188 steps, many of which have several possible behaviors. To avoid cluttering the
189 code with lots of switch statements, we use a simple form of object-style
190 programming to separate out the different possibilities.
191
192 For example, two different color quantization algorithms could be implemented
193 as two separate modules that present the same external interface; at runtime,
194 the calling code will access the proper module indirectly through an "object".
195
196 We can get the limited features we need while staying within portable C.
197 The basic tool is a function pointer. An "object" is just a struct
198 containing one or more function pointer fields, each of which corresponds to
199 a method name in real object-oriented languages. During initialization we
200 fill in the function pointers with references to whichever module we have
201 determined we need to use in this run. Then invocation of the module is done
202 by indirecting through a function pointer; on most machines this is no more
203 expensive than a switch statement, which would be the only other way of
204 making the required run-time choice. The really significant benefit, of
205 course, is keeping the source code clean and well structured.
206
207 We can also arrange to have private storage that varies between different
208 implementations of the same kind of object. We do this by making all the
209 module-specific object structs be separately allocated entities, which will
210 be accessed via pointers in the master compression or decompression struct.
211 The "public" fields or methods for a given kind of object are specified by
212 a commonly known struct. But a module's initialization code can allocate
213 a larger struct that contains the common struct as its first member, plus
214 additional private fields. With appropriate pointer casting, the module's
215 internal functions can access these private fields. (For a simple example,
216 see jdatadst.c, which implements the external interface specified by struct
217 jpeg_destination_mgr, but adds extra fields.)
218
219 (Of course this would all be a lot easier if we were using C++, but we are
220 not yet prepared to assume that everyone has a C++ compiler.)
221
222 An important benefit of this scheme is that it is easy to provide multiple
223 versions of any method, each tuned to a particular case. While a lot of
224 precalculation might be done to select an optimal implementation of a method,
225 the cost per invocation is constant. For example, the upsampling step might
226 have a "generic" method, plus one or more "hardwired" methods for the most
227 popular sampling factors; the hardwired methods would be faster because they'd
228 use straight-line code instead of for-loops. The cost to determine which
229 method to use is paid only once, at startup, and the selection criteria are
230 hidden from the callers of the method.
231
232 This plan differs a little bit from usual object-oriented structures, in that
233 only one instance of each object class will exist during execution. The
234 reason for having the class structure is that on different runs we may create
235 different instances (choose to execute different modules). You can think of
236 the term "method" as denoting the common interface presented by a particular
237 set of interchangeable functions, and "object" as denoting a group of related
238 methods, or the total shared interface behavior of a group of modules.
239
240
241 *** Overall control structure ***
242
243 We previously mentioned the need for overall control logic in the compression
244 and decompression libraries. In IJG implementations prior to v5, overall
245 control was mostly provided by "pipeline control" modules, which proved to be
246 large, unwieldy, and hard to understand. To improve the situation, the
247 control logic has been subdivided into multiple modules. The control modules
248 consist of:
249
250 1. Master control for module selection and initialization. This has two
251 responsibilities:
252
253 1A. Startup initialization at the beginning of image processing.
254 The individual processing modules to be used in this run are selected
255 and given initialization calls.
256
257 1B. Per-pass control. This determines how many passes will be performed
258 and calls each active processing module to configure itself
259 appropriately at the beginning of each pass. End-of-pass processing,
260 where necessary, is also invoked from the master control module.
261
262 Method selection is partially distributed, in that a particular processing
263 module may contain several possible implementations of a particular method,
264 which it will select among when given its initialization call. The master
265 control code need only be concerned with decisions that affect more than
266 one module.
267
268 2. Data buffering control. A separate control module exists for each
269 inter-processing-step data buffer. This module is responsible for
270 invoking the processing steps that write or read that data buffer.
271
272 Each buffer controller sees the world as follows:
273
274 input data => processing step A => buffer => processing step B => output data
275 | | |
276 ------------------ controller ------------------
277
278 The controller knows the dataflow requirements of steps A and B: how much data
279 they want to accept in one chunk and how much they output in one chunk. Its
280 function is to manage its buffer and call A and B at the proper times.
281
282 A data buffer control module may itself be viewed as a processing step by a
283 higher-level control module; thus the control modules form a binary tree with
284 elementary processing steps at the leaves of the tree.
285
286 The control modules are objects. A considerable amount of flexibility can
287 be had by replacing implementations of a control module. For example:
288 * Merging of adjacent steps in the pipeline is done by replacing a control
289 module and its pair of processing-step modules with a single processing-
290 step module. (Hence the possible merges are determined by the tree of
291 control modules.)
292 * In some processing modes, a given interstep buffer need only be a "strip"
293 buffer large enough to accommodate the desired data chunk sizes. In other
294 modes, a full-image buffer is needed and several passes are required.
295 The control module determines which kind of buffer is used and manipulates
296 virtual array buffers as needed. One or both processing steps may be
297 unaware of the multi-pass behavior.
298
299 In theory, we might be able to make all of the data buffer controllers
300 interchangeable and provide just one set of implementations for all. In
301 practice, each one contains considerable special-case processing for its
302 particular job. The buffer controller concept should be regarded as an
303 overall system structuring principle, not as a complete description of the
304 task performed by any one controller.
305
306
307 *** Compression object structure ***
308
309 Here is a sketch of the logical structure of the JPEG compression library:
310
311 |-- Colorspace conversion
312 |-- Preprocessing controller --|
313 | |-- Downsampling
314 Main controller --|
315 | |-- Forward DCT, quantize
316 |-- Coefficient controller --|
317 |-- Entropy encoding
318
319 This sketch also describes the flow of control (subroutine calls) during
320 typical image data processing. Each of the components shown in the diagram is
321 an "object" which may have several different implementations available. One
322 or more source code files contain the actual implementation(s) of each object.
323
324 The objects shown above are:
325
326 * Main controller: buffer controller for the subsampled-data buffer, which
327 holds the preprocessed input data. This controller invokes preprocessing to
328 fill the subsampled-data buffer, and JPEG compression to empty it. There is
329 usually no need for a full-image buffer here; a strip buffer is adequate.
330
331 * Preprocessing controller: buffer controller for the downsampling input data
332 buffer, which lies between colorspace conversion and downsampling. Note
333 that a unified conversion/downsampling module would probably replace this
334 controller entirely.
335
336 * Colorspace conversion: converts application image data into the desired
337 JPEG color space; also changes the data from pixel-interleaved layout to
338 separate component planes. Processes one pixel row at a time.
339
340 * Downsampling: performs reduction of chroma components as required.
341 Optionally may perform pixel-level smoothing as well. Processes a "row
342 group" at a time, where a row group is defined as Vmax pixel rows of each
343 component before downsampling, and Vk sample rows afterwards (remember Vk
344 differs across components). Some downsampling or smoothing algorithms may
345 require context rows above and below the current row group; the
346 preprocessing controller is responsible for supplying these rows via proper
347 buffering. The downsampler is responsible for edge expansion at the right
348 edge (i.e., extending each sample row to a multiple of block_size samples);
349 but the preprocessing controller is responsible for vertical edge expansion
350 (i.e., duplicating the bottom sample row as needed to make a multiple of
351 block_size rows).
352
353 * Coefficient controller: buffer controller for the DCT-coefficient data.
354 This controller handles MCU assembly, including insertion of dummy DCT
355 blocks when needed at the right or bottom edge. When performing
356 Huffman-code optimization or emitting a multiscan JPEG file, this
357 controller is responsible for buffering the full image. The equivalent of
358 one fully interleaved MCU row of subsampled data is processed per call,
359 even when the JPEG file is noninterleaved.
360
361 * Forward DCT and quantization: Perform DCT, quantize, and emit coefficients.
362 Works on one or more DCT blocks at a time. (Note: the coefficients are now
363 emitted in normal array order, which the entropy encoder is expected to
364 convert to zigzag order as necessary. Prior versions of the IJG code did
365 the conversion to zigzag order within the quantization step.)
366
367 * Entropy encoding: Perform Huffman or arithmetic entropy coding and emit the
368 coded data to the data destination module. Works on one MCU per call.
369 For progressive JPEG, the same DCT blocks are fed to the entropy coder
370 during each pass, and the coder must emit the appropriate subset of
371 coefficients.
372
373 In addition to the above objects, the compression library includes these
374 objects:
375
376 * Master control: determines the number of passes required, controls overall
377 and per-pass initialization of the other modules.
378
379 * Marker writing: generates JPEG markers (except for RSTn, which is emitted
380 by the entropy encoder when needed).
381
382 * Data destination manager: writes the output JPEG datastream to its final
383 destination (e.g., a file). The destination manager supplied with the
384 library knows how to write to a stdio stream or to a memory buffer;
385 for other behaviors, the surrounding application may provide its own
386 destination manager.
387
388 * Memory manager: allocates and releases memory, controls virtual arrays
389 (with backing store management, where required).
390
391 * Error handler: performs formatting and output of error and trace messages;
392 determines handling of nonfatal errors. The surrounding application may
393 override some or all of this object's methods to change error handling.
394
395 * Progress monitor: supports output of "percent-done" progress reports.
396 This object represents an optional callback to the surrounding application:
397 if wanted, it must be supplied by the application.
398
399 The error handler, destination manager, and progress monitor objects are
400 defined as separate objects in order to simplify application-specific
401 customization of the JPEG library. A surrounding application may override
402 individual methods or supply its own all-new implementation of one of these
403 objects. The object interfaces for these objects are therefore treated as
404 part of the application interface of the library, whereas the other objects
405 are internal to the library.
406
407 The error handler and memory manager are shared by JPEG compression and
408 decompression; the progress monitor, if used, may be shared as well.
409
410
411 *** Decompression object structure ***
412
413 Here is a sketch of the logical structure of the JPEG decompression library:
414
415 |-- Entropy decoding
416 |-- Coefficient controller --|
417 | |-- Dequantize, Inverse DCT
418 Main controller --|
419 | |-- Upsampling
420 |-- Postprocessing controller --| |-- Colorspace conversion
421 |-- Color quantization
422 |-- Color precision reduction
423
424 As before, this diagram also represents typical control flow. The objects
425 shown are:
426
427 * Main controller: buffer controller for the subsampled-data buffer, which
428 holds the output of JPEG decompression proper. This controller's primary
429 task is to feed the postprocessing procedure. Some upsampling algorithms
430 may require context rows above and below the current row group; when this
431 is true, the main controller is responsible for managing its buffer so as
432 to make context rows available. In the current design, the main buffer is
433 always a strip buffer; a full-image buffer is never required.
434
435 * Coefficient controller: buffer controller for the DCT-coefficient data.
436 This controller handles MCU disassembly, including deletion of any dummy
437 DCT blocks at the right or bottom edge. When reading a multiscan JPEG
438 file, this controller is responsible for buffering the full image.
439 (Buffering DCT coefficients, rather than samples, is necessary to support
440 progressive JPEG.) The equivalent of one fully interleaved MCU row of
441 subsampled data is processed per call, even when the source JPEG file is
442 noninterleaved.
443
444 * Entropy decoding: Read coded data from the data source module and perform
445 Huffman or arithmetic entropy decoding. Works on one MCU per call.
446 For progressive JPEG decoding, the coefficient controller supplies the prior
447 coefficients of each MCU (initially all zeroes), which the entropy decoder
448 modifies in each scan.
449
450 * Dequantization and inverse DCT: like it says. Note that the coefficients
451 buffered by the coefficient controller have NOT been dequantized; we
452 merge dequantization and inverse DCT into a single step for speed reasons.
453 When scaled-down output is asked for, simplified DCT algorithms may be used
454 that need fewer coefficients and emit fewer samples per DCT block, not the
455 full 8x8. Works on one DCT block at a time.
456
457 * Postprocessing controller: buffer controller for the color quantization
458 input buffer, when quantization is in use. (Without quantization, this
459 controller just calls the upsampler.) For two-pass quantization, this
460 controller is responsible for buffering the full-image data.
461
462 * Upsampling: restores chroma components to full size. (May support more
463 general output rescaling, too. Note that if undersized DCT outputs have
464 been emitted by the DCT module, this module must adjust so that properly
465 sized outputs are created.) Works on one row group at a time. This module
466 also calls the color conversion module, so its top level is effectively a
467 buffer controller for the upsampling->color conversion buffer. However, in
468 all but the highest-quality operating modes, upsampling and color
469 conversion are likely to be merged into a single step.
470
471 * Colorspace conversion: convert from JPEG color space to output color space,
472 and change data layout from separate component planes to pixel-interleaved.
473 Works on one pixel row at a time.
474
475 * Color quantization: reduce the data to colormapped form, using either an
476 externally specified colormap or an internally generated one. This module
477 is not used for full-color output. Works on one pixel row at a time; may
478 require two passes to generate a color map. Note that the output will
479 always be a single component representing colormap indexes. In the current
480 design, the output values are JSAMPLEs, so an 8-bit compilation cannot
481 quantize to more than 256 colors. This is unlikely to be a problem in
482 practice.
483
484 * Color reduction: this module handles color precision reduction, e.g.,
485 generating 15-bit color (5 bits/primary) from JPEG's 24-bit output.
486 Not quite clear yet how this should be handled... should we merge it with
487 colorspace conversion???
488
489 Note that some high-speed operating modes might condense the entire
490 postprocessing sequence to a single module (upsample, color convert, and
491 quantize in one step).
492
493 In addition to the above objects, the decompression library includes these
494 objects:
495
496 * Master control: determines the number of passes required, controls overall
497 and per-pass initialization of the other modules. This is subdivided into
498 input and output control: jdinput.c controls only input-side processing,
499 while jdmaster.c handles overall initialization and output-side control.
500
501 * Marker reading: decodes JPEG markers (except for RSTn).
502
503 * Data source manager: supplies the input JPEG datastream. The source
504 manager supplied with the library knows how to read from a stdio stream
505 or from a memory buffer; for other behaviors, the surrounding application
506 may provide its own source manager.
507
508 * Memory manager: same as for compression library.
509
510 * Error handler: same as for compression library.
511
512 * Progress monitor: same as for compression library.
513
514 As with compression, the data source manager, error handler, and progress
515 monitor are candidates for replacement by a surrounding application.
516
517
518 *** Decompression input and output separation ***
519
520 To support efficient incremental display of progressive JPEG files, the
521 decompressor is divided into two sections that can run independently:
522
523 1. Data input includes marker parsing, entropy decoding, and input into the
524 coefficient controller's DCT coefficient buffer. Note that this
525 processing is relatively cheap and fast.
526
527 2. Data output reads from the DCT coefficient buffer and performs the IDCT
528 and all postprocessing steps.
529
530 For a progressive JPEG file, the data input processing is allowed to get
531 arbitrarily far ahead of the data output processing. (This occurs only
532 if the application calls jpeg_consume_input(); otherwise input and output
533 run in lockstep, since the input section is called only when the output
534 section needs more data.) In this way the application can avoid making
535 extra display passes when data is arriving faster than the display pass
536 can run. Furthermore, it is possible to abort an output pass without
537 losing anything, since the coefficient buffer is read-only as far as the
538 output section is concerned. See libjpeg.txt for more detail.
539
540 A full-image coefficient array is only created if the JPEG file has multiple
541 scans (or if the application specifies buffered-image mode anyway). When
542 reading a single-scan file, the coefficient controller normally creates only
543 a one-MCU buffer, so input and output processing must run in lockstep in this
544 case. jpeg_consume_input() is effectively a no-op in this situation.
545
546 The main impact of dividing the decompressor in this fashion is that we must
547 be very careful with shared variables in the cinfo data structure. Each
548 variable that can change during the course of decompression must be
549 classified as belonging to data input or data output, and each section must
550 look only at its own variables. For example, the data output section may not
551 depend on any of the variables that describe the current scan in the JPEG
552 file, because these may change as the data input section advances into a new
553 scan.
554
555 The progress monitor is (somewhat arbitrarily) defined to treat input of the
556 file as one pass when buffered-image mode is not used, and to ignore data
557 input work completely when buffered-image mode is used. Note that the
558 library has no reliable way to predict the number of passes when dealing
559 with a progressive JPEG file, nor can it predict the number of output passes
560 in buffered-image mode. So the work estimate is inherently bogus anyway.
561
562 No comparable division is currently made in the compression library, because
563 there isn't any real need for it.
564
565
566 *** Data formats ***
567
568 Arrays of pixel sample values use the following data structure:
569
570 typedef something JSAMPLE; a pixel component value, 0..MAXJSAMPLE
571 typedef JSAMPLE *JSAMPROW; ptr to a row of samples
572 typedef JSAMPROW *JSAMPARRAY; ptr to a list of rows
573 typedef JSAMPARRAY *JSAMPIMAGE; ptr to a list of color-component arrays
574
575 The basic element type JSAMPLE will typically be one of unsigned char,
576 (signed) char, or short. Short will be used if samples wider than 8 bits are
577 to be supported (this is a compile-time option). Otherwise, unsigned char is
578 used if possible. If the compiler only supports signed chars, then it is
579 necessary to mask off the value when reading. Thus, all reads of JSAMPLE
580 values must be coded as "GETJSAMPLE(value)", where the macro will be defined
581 as "((value) & 0xFF)" on signed-char machines and "((int) (value))" elsewhere.
582
583 With these conventions, JSAMPLE values can be assumed to be >= 0. This helps
584 simplify correct rounding during downsampling, etc. The JPEG standard's
585 specification that sample values run from -128..127 is accommodated by
586 subtracting 128 from the sample value in the DCT step. Similarly, during
587 decompression the output of the IDCT step will be immediately shifted back to
588 0..255. (NB: different values are required when 12-bit samples are in use.
589 The code is written in terms of MAXJSAMPLE and CENTERJSAMPLE, which will be
590 defined as 255 and 128 respectively in an 8-bit implementation, and as 4095
591 and 2048 in a 12-bit implementation.)
592
593 We use a pointer per row, rather than a two-dimensional JSAMPLE array. This
594 choice costs only a small amount of memory and has several benefits:
595 * Code using the data structure doesn't need to know the allocated width of
596 the rows. This simplifies edge expansion/compression, since we can work
597 in an array that's wider than the logical picture width.
598 * Indexing doesn't require multiplication; this is a performance win on many
599 machines.
600 * Arrays with more than 64K total elements can be supported even on machines
601 where malloc() cannot allocate chunks larger than 64K.
602 * The rows forming a component array may be allocated at different times
603 without extra copying. This trick allows some speedups in smoothing steps
604 that need access to the previous and next rows.
605
606 Note that each color component is stored in a separate array; we don't use the
607 traditional layout in which the components of a pixel are stored together.
608 This simplifies coding of modules that work on each component independently,
609 because they don't need to know how many components there are. Furthermore,
610 we can read or write each component to a temporary file independently, which
611 is helpful when dealing with noninterleaved JPEG files.
612
613 In general, a specific sample value is accessed by code such as
614 GETJSAMPLE(image[colorcomponent][row][col])
615 where col is measured from the image left edge, but row is measured from the
616 first sample row currently in memory. Either of the first two indexings can
617 be precomputed by copying the relevant pointer.
618
619
620 Since most image-processing applications prefer to work on images in which
621 the components of a pixel are stored together, the data passed to or from the
622 surrounding application uses the traditional convention: a single pixel is
623 represented by N consecutive JSAMPLE values, and an image row is an array of
624 (# of color components)*(image width) JSAMPLEs. One or more rows of data can
625 be represented by a pointer of type JSAMPARRAY in this scheme. This scheme is
626 converted to component-wise storage inside the JPEG library. (Applications
627 that want to skip JPEG preprocessing or postprocessing will have to contend
628 with component-wise storage.)
629
630
631 Arrays of DCT-coefficient values use the following data structure:
632
633 typedef short JCOEF; a 16-bit signed integer
634 typedef JCOEF JBLOCK[DCTSIZE2]; an 8x8 block of coefficients
635 typedef JBLOCK *JBLOCKROW; ptr to one horizontal row of 8x8 blocks
636 typedef JBLOCKROW *JBLOCKARRAY; ptr to a list of such rows
637 typedef JBLOCKARRAY *JBLOCKIMAGE; ptr to a list of color component arrays
638
639 The underlying type is at least a 16-bit signed integer; while "short" is big
640 enough on all machines of interest, on some machines it is preferable to use
641 "int" for speed reasons, despite the storage cost. Coefficients are grouped
642 into 8x8 blocks (but we always use #defines DCTSIZE and DCTSIZE2 rather than
643 "8" and "64").
644
645 The contents of a coefficient block may be in either "natural" or zigzagged
646 order, and may be true values or divided by the quantization coefficients,
647 depending on where the block is in the processing pipeline. In the current
648 library, coefficient blocks are kept in natural order everywhere; the entropy
649 codecs zigzag or dezigzag the data as it is written or read. The blocks
650 contain quantized coefficients everywhere outside the DCT/IDCT subsystems.
651 (This latter decision may need to be revisited to support variable
652 quantization a la JPEG Part 3.)
653
654 Notice that the allocation unit is now a row of 8x8 coefficient blocks,
655 corresponding to block_size rows of samples. Otherwise the structure
656 is much the same as for samples, and for the same reasons.
657
658 On machines where malloc() can't handle a request bigger than 64Kb, this data
659 structure limits us to rows of less than 512 JBLOCKs, or a picture width of
660 4000+ pixels. This seems an acceptable restriction.
661
662
663 On 80x86 machines, the bottom-level pointer types (JSAMPROW and JBLOCKROW)
664 must be declared as "far" pointers, but the upper levels can be "near"
665 (implying that the pointer lists are allocated in the DS segment).
666 We use a #define symbol FAR, which expands to the "far" keyword when
667 compiling on 80x86 machines and to nothing elsewhere.
668
669
670 *** Suspendable processing ***
671
672 In some applications it is desirable to use the JPEG library as an
673 incremental, memory-to-memory filter. In this situation the data source or
674 destination may be a limited-size buffer, and we can't rely on being able to
675 empty or refill the buffer at arbitrary times. Instead the application would
676 like to have control return from the library at buffer overflow/underrun, and
677 then resume compression or decompression at a later time.
678
679 This scenario is supported for simple cases. (For anything more complex, we
680 recommend that the application "bite the bullet" and develop real multitasking
681 capability.) The libjpeg.txt file goes into more detail about the usage and
682 limitations of this capability; here we address the implications for library
683 structure.
684
685 The essence of the problem is that the entropy codec (coder or decoder) must
686 be prepared to stop at arbitrary times. In turn, the controllers that call
687 the entropy codec must be able to stop before having produced or consumed all
688 the data that they normally would handle in one call. That part is reasonably
689 straightforward: we make the controller call interfaces include "progress
690 counters" which indicate the number of data chunks successfully processed, and
691 we require callers to test the counter rather than just assume all of the data
692 was processed.
693
694 Rather than trying to restart at an arbitrary point, the current Huffman
695 codecs are designed to restart at the beginning of the current MCU after a
696 suspension due to buffer overflow/underrun. At the start of each call, the
697 codec's internal state is loaded from permanent storage (in the JPEG object
698 structures) into local variables. On successful completion of the MCU, the
699 permanent state is updated. (This copying is not very expensive, and may even
700 lead to *improved* performance if the local variables can be registerized.)
701 If a suspension occurs, the codec simply returns without updating the state,
702 thus effectively reverting to the start of the MCU. Note that this implies
703 leaving some data unprocessed in the source/destination buffer (ie, the
704 compressed partial MCU). The data source/destination module interfaces are
705 specified so as to make this possible. This also implies that the data buffer
706 must be large enough to hold a worst-case compressed MCU; a couple thousand
707 bytes should be enough.
708
709 In a successive-approximation AC refinement scan, the progressive Huffman
710 decoder has to be able to undo assignments of newly nonzero coefficients if it
711 suspends before the MCU is complete, since decoding requires distinguishing
712 previously-zero and previously-nonzero coefficients. This is a bit tedious
713 but probably won't have much effect on performance. Other variants of Huffman
714 decoding need not worry about this, since they will just store the same values
715 again if forced to repeat the MCU.
716
717 This approach would probably not work for an arithmetic codec, since its
718 modifiable state is quite large and couldn't be copied cheaply. Instead it
719 would have to suspend and resume exactly at the point of the buffer end.
720
721 The JPEG marker reader is designed to cope with suspension at an arbitrary
722 point. It does so by backing up to the start of the marker parameter segment,
723 so the data buffer must be big enough to hold the largest marker of interest.
724 Again, a couple KB should be adequate. (A special "skip" convention is used
725 to bypass COM and APPn markers, so these can be larger than the buffer size
726 without causing problems; otherwise a 64K buffer would be needed in the worst
727 case.)
728
729 The JPEG marker writer currently does *not* cope with suspension.
730 We feel that this is not necessary; it is much easier simply to require
731 the application to ensure there is enough buffer space before starting. (An
732 empty 2K buffer is more than sufficient for the header markers; and ensuring
733 there are a dozen or two bytes available before calling jpeg_finish_compress()
734 will suffice for the trailer.) This would not work for writing multi-scan
735 JPEG files, but we simply do not intend to support that capability with
736 suspension.
737
738
739 *** Memory manager services ***
740
741 The JPEG library's memory manager controls allocation and deallocation of
742 memory, and it manages large "virtual" data arrays on machines where the
743 operating system does not provide virtual memory. Note that the same
744 memory manager serves both compression and decompression operations.
745
746 In all cases, allocated objects are tied to a particular compression or
747 decompression master record, and they will be released when that master
748 record is destroyed.
749
750 The memory manager does not provide explicit deallocation of objects.
751 Instead, objects are created in "pools" of free storage, and a whole pool
752 can be freed at once. This approach helps prevent storage-leak bugs, and
753 it speeds up operations whenever malloc/free are slow (as they often are).
754 The pools can be regarded as lifetime identifiers for objects. Two
755 pools/lifetimes are defined:
756 * JPOOL_PERMANENT lasts until master record is destroyed
757 * JPOOL_IMAGE lasts until done with image (JPEG datastream)
758 Permanent lifetime is used for parameters and tables that should be carried
759 across from one datastream to another; this includes all application-visible
760 parameters. Image lifetime is used for everything else. (A third lifetime,
761 JPOOL_PASS = one processing pass, was originally planned. However it was
762 dropped as not being worthwhile. The actual usage patterns are such that the
763 peak memory usage would be about the same anyway; and having per-pass storage
764 substantially complicates the virtual memory allocation rules --- see below.)
765
766 The memory manager deals with three kinds of object:
767 1. "Small" objects. Typically these require no more than 10K-20K total.
768 2. "Large" objects. These may require tens to hundreds of K depending on
769 image size. Semantically they behave the same as small objects, but we
770 distinguish them for two reasons:
771 * On MS-DOS machines, large objects are referenced by FAR pointers,
772 small objects by NEAR pointers.
773 * Pool allocation heuristics may differ for large and small objects.
774 Note that individual "large" objects cannot exceed the size allowed by
775 type size_t, which may be 64K or less on some machines.
776 3. "Virtual" objects. These are large 2-D arrays of JSAMPLEs or JBLOCKs
777 (typically large enough for the entire image being processed). The
778 memory manager provides stripwise access to these arrays. On machines
779 without virtual memory, the rest of the array may be swapped out to a
780 temporary file.
781
782 (Note: JSAMPARRAY and JBLOCKARRAY data structures are a combination of large
783 objects for the data proper and small objects for the row pointers. For
784 convenience and speed, the memory manager provides single routines to create
785 these structures. Similarly, virtual arrays include a small control block
786 and a JSAMPARRAY or JBLOCKARRAY working buffer, all created with one call.)
787
788 In the present implementation, virtual arrays are only permitted to have image
789 lifespan. (Permanent lifespan would not be reasonable, and pass lifespan is
790 not very useful since a virtual array's raison d'etre is to store data for
791 multiple passes through the image.) We also expect that only "small" objects
792 will be given permanent lifespan, though this restriction is not required by
793 the memory manager.
794
795 In a non-virtual-memory machine, some performance benefit can be gained by
796 making the in-memory buffers for virtual arrays be as large as possible.
797 (For small images, the buffers might fit entirely in memory, so blind
798 swapping would be very wasteful.) The memory manager will adjust the height
799 of the buffers to fit within a prespecified maximum memory usage. In order
800 to do this in a reasonably optimal fashion, the manager needs to allocate all
801 of the virtual arrays at once. Therefore, there isn't a one-step allocation
802 routine for virtual arrays; instead, there is a "request" routine that simply
803 allocates the control block, and a "realize" routine (called just once) that
804 determines space allocation and creates all of the actual buffers. The
805 realize routine must allow for space occupied by non-virtual large objects.
806 (We don't bother to factor in the space needed for small objects, on the
807 grounds that it isn't worth the trouble.)
808
809 To support all this, we establish the following protocol for doing business
810 with the memory manager:
811 1. Modules must request virtual arrays (which may have only image lifespan)
812 during the initial setup phase, i.e., in their jinit_xxx routines.
813 2. All "large" objects (including JSAMPARRAYs and JBLOCKARRAYs) must also be
814 allocated during initial setup.
815 3. realize_virt_arrays will be called at the completion of initial setup.
816 The above conventions ensure that sufficient information is available
817 for it to choose a good size for virtual array buffers.
818 Small objects of any lifespan may be allocated at any time. We expect that
819 the total space used for small objects will be small enough to be negligible
820 in the realize_virt_arrays computation.
821
822 In a virtual-memory machine, we simply pretend that the available space is
823 infinite, thus causing realize_virt_arrays to decide that it can allocate all
824 the virtual arrays as full-size in-memory buffers. The overhead of the
825 virtual-array access protocol is very small when no swapping occurs.
826
827 A virtual array can be specified to be "pre-zeroed"; when this flag is set,
828 never-yet-written sections of the array are set to zero before being made
829 available to the caller. If this flag is not set, never-written sections
830 of the array contain garbage. (This feature exists primarily because the
831 equivalent logic would otherwise be needed in jdcoefct.c for progressive
832 JPEG mode; we may as well make it available for possible other uses.)
833
834 The first write pass on a virtual array is required to occur in top-to-bottom
835 order; read passes, as well as any write passes after the first one, may
836 access the array in any order. This restriction exists partly to simplify
837 the virtual array control logic, and partly because some file systems may not
838 support seeking beyond the current end-of-file in a temporary file. The main
839 implication of this restriction is that rearrangement of rows (such as
840 converting top-to-bottom data order to bottom-to-top) must be handled while
841 reading data out of the virtual array, not while putting it in.
842
843
844 *** Memory manager internal structure ***
845
846 To isolate system dependencies as much as possible, we have broken the
847 memory manager into two parts. There is a reasonably system-independent
848 "front end" (jmemmgr.c) and a "back end" that contains only the code
849 likely to change across systems. All of the memory management methods
850 outlined above are implemented by the front end. The back end provides
851 the following routines for use by the front end (none of these routines
852 are known to the rest of the JPEG code):
853
854 jpeg_mem_init, jpeg_mem_term system-dependent initialization/shutdown
855
856 jpeg_get_small, jpeg_free_small interface to malloc and free library routines
857 (or their equivalents)
858
859 jpeg_get_large, jpeg_free_large interface to FAR malloc/free in MSDOS machines;
860 else usually the same as
861 jpeg_get_small/jpeg_free_small
862
863 jpeg_mem_available estimate available memory
864
865 jpeg_open_backing_store create a backing-store object
866
867 read_backing_store, manipulate a backing-store object
868 write_backing_store,
869 close_backing_store
870
871 On some systems there will be more than one type of backing-store object
872 (specifically, in MS-DOS a backing store file might be an area of extended
873 memory as well as a disk file). jpeg_open_backing_store is responsible for
874 choosing how to implement a given object. The read/write/close routines
875 are method pointers in the structure that describes a given object; this
876 lets them be different for different object types.
877
878 It may be necessary to ensure that backing store objects are explicitly
879 released upon abnormal program termination. For example, MS-DOS won't free
880 extended memory by itself. To support this, we will expect the main program
881 or surrounding application to arrange to call self_destruct (typically via
882 jpeg_destroy) upon abnormal termination. This may require a SIGINT signal
883 handler or equivalent. We don't want to have the back end module install its
884 own signal handler, because that would pre-empt the surrounding application's
885 ability to control signal handling.
886
887 The IJG distribution includes several memory manager back end implementations.
888 Usually the same back end should be suitable for all applications on a given
889 system, but it is possible for an application to supply its own back end at
890 need.
891
892
893 *** Implications of DNL marker ***
894
895 Some JPEG files may use a DNL marker to postpone definition of the image
896 height (this would be useful for a fax-like scanner's output, for instance).
897 In these files the SOF marker claims the image height is 0, and you only
898 find out the true image height at the end of the first scan.
899
900 We could read these files as follows:
901 1. Upon seeing zero image height, replace it by 65535 (the maximum allowed).
902 2. When the DNL is found, update the image height in the global image
903 descriptor.
904 This implies that control modules must avoid making copies of the image
905 height, and must re-test for termination after each MCU row. This would
906 be easy enough to do.
907
908 In cases where image-size data structures are allocated, this approach will
909 result in very inefficient use of virtual memory or much-larger-than-necessary
910 temporary files. This seems acceptable for something that probably won't be a
911 mainstream usage. People might have to forgo use of memory-hogging options
912 (such as two-pass color quantization or noninterleaved JPEG files) if they
913 want efficient conversion of such files. (One could improve efficiency by
914 demanding a user-supplied upper bound for the height, less than 65536; in most
915 cases it could be much less.)
916
917 The standard also permits the SOF marker to overestimate the image height,
918 with a DNL to give the true, smaller height at the end of the first scan.
919 This would solve the space problems if the overestimate wasn't too great.
920 However, it implies that you don't even know whether DNL will be used.
921
922 This leads to a couple of very serious objections:
923 1. Testing for a DNL marker must occur in the inner loop of the decompressor's
924 Huffman decoder; this implies a speed penalty whether the feature is used
925 or not.
926 2. There is no way to hide the last-minute change in image height from an
927 application using the decoder. Thus *every* application using the IJG
928 library would suffer a complexity penalty whether it cared about DNL or
929 not.
930 We currently do not support DNL because of these problems.
931
932 A different approach is to insist that DNL-using files be preprocessed by a
933 separate program that reads ahead to the DNL, then goes back and fixes the SOF
934 marker. This is a much simpler solution and is probably far more efficient.
935 Even if one wants piped input, buffering the first scan of the JPEG file needs
936 a lot smaller temp file than is implied by the maximum-height method. For
937 this approach we'd simply treat DNL as a no-op in the decompressor (at most,
938 check that it matches the SOF image height).
939
940 We will not worry about making the compressor capable of outputting DNL.
941 Something similar to the first scheme above could be applied if anyone ever
942 wants to make that work.