container.h File Reference

File formats. More...

Go to the source code of this file.

Classes
struct	lzma_mt
	Multithreading options. More...

Macros
#define	LZMA_PRESET_DEFAULT   UINT32_C(6)
	Default compression preset. More...
#define	LZMA_PRESET_LEVEL_MASK   UINT32_C(0x1F)
	Mask for preset level. More...
#define	LZMA_PRESET_EXTREME   (UINT32_C(1) << 31)
	Extreme compression preset. More...
#define	LZMA_TELL_NO_CHECK   UINT32_C(0x01)
#define	LZMA_TELL_UNSUPPORTED_CHECK   UINT32_C(0x02)
#define	LZMA_TELL_ANY_CHECK   UINT32_C(0x04)
#define	LZMA_IGNORE_CHECK   UINT32_C(0x10)
#define	LZMA_CONCATENATED   UINT32_C(0x08)

Functions
	LZMA_API (uint64_t) lzma_easy_encoder_memusage(uint32_t preset) lzma_nothrow lzma_attr_pure
	Calculate approximate memory usage of easy encoder. More...
	LZMA_API (lzma_ret) lzma_easy_encoder(lzma_stream *strm
	Initialize .xz Stream encoder using a preset number. More...
	LZMA_API (size_t) lzma_stream_buffer_bound(size_t uncompressed_size) lzma_nothrow
	Calculate output buffer size for single-call Stream encoder. More...

Variables
uint32_t	preset
uint32_t lzma_check check lzma_nothrow	lzma_attr_warn_unused_result
lzma_check	check
lzma_check const lzma_allocator *	allocator
lzma_check const lzma_allocator const uint8_t *	in
lzma_check const lzma_allocator const uint8_t size_t	in_size
lzma_check const lzma_allocator const uint8_t size_t uint8_t *	out
lzma_check const lzma_allocator const uint8_t size_t uint8_t size_t *	out_pos
lzma_check const lzma_allocator const uint8_t size_t uint8_t size_t size_t out_size	lzma_nothrow
const lzma_filter *	filters
uint64_t	memlimit
uint32_t	flags
uint32_t const lzma_allocator const uint8_t size_t *	in_pos

Detailed Description

File formats.

Definition in file container.h.

Macro Definition Documentation

LZMA_CONCATENATED

#define LZMA_CONCATENATED   UINT32_C(0x08)

This flag enables decoding of concatenated files with file formats that allow concatenating compressed files as is. From the formats currently supported by liblzma, only the .xz format allows concatenated files. Concatenated files are not allowed with the legacy .lzma format.

This flag also affects the usage of the ‘action’ argument for lzma_code(). When LZMA_CONCATENATED is used, lzma_code() won't return LZMA_STREAM_END unless LZMA_FINISH is used as ‘action’. Thus, the application has to set LZMA_FINISH in the same way as it does when encoding.

If LZMA_CONCATENATED is not used, the decoders still accept LZMA_FINISH as ‘action’ for lzma_code(), but the usage of LZMA_FINISH isn't required.

Definition at line 515 of file container.h.

LZMA_IGNORE_CHECK

#define LZMA_IGNORE_CHECK   UINT32_C(0x10)

This flag makes lzma_code() not calculate and verify the integrity check of the compressed data in .xz files. This means that invalid integrity check values won't be detected and LZMA_DATA_ERROR won't be returned in such cases.

This flag only affects the checks of the compressed data itself; the CRC32 values in the .xz headers will still be verified normally.

Don't use this flag unless you know what you are doing. Possible reasons to use this flag:

• Trying to recover data from a corrupt .xz file.
• Speeding up decompression, which matters mostly with SHA-256 or with files that have compressed extremely well. It's recommended to not use this flag for this purpose unless the file integrity is verified externally in some other way.

Support for this flag was added in liblzma 5.1.4beta.

Definition at line 498 of file container.h.

LZMA_PRESET_DEFAULT

#define LZMA_PRESET_DEFAULT   UINT32_C(6)

Default compression preset.

It's not straightforward to recommend a default preset, because in some cases keeping the resource usage relatively low is more important that getting the maximum compression ratio.

Definition at line 31 of file container.h.

LZMA_PRESET_EXTREME

#define LZMA_PRESET_EXTREME   (UINT32_C(1) << 31)

Extreme compression preset.

This flag modifies the preset to make the encoding significantly slower while improving the compression ratio only marginally. This is useful when you don't mind wasting time to get as small result as possible.

This flag doesn't affect the memory usage requirements of the decoder (at least not significantly). The memory usage of the encoder may be increased a little but only at the lowest preset levels (0-3).

Definition at line 60 of file container.h.

LZMA_PRESET_LEVEL_MASK

#define LZMA_PRESET_LEVEL_MASK   UINT32_C(0x1F)

Mask for preset level.

This is useful only if you need to extract the level from the preset variable. That should be rare.

Definition at line 40 of file container.h.

LZMA_TELL_ANY_CHECK

#define LZMA_TELL_ANY_CHECK   UINT32_C(0x04)

This flag makes lzma_code() return LZMA_GET_CHECK as soon as the type of the integrity check is known. The type can then be got with lzma_get_check().

Definition at line 474 of file container.h.

LZMA_TELL_NO_CHECK

#define LZMA_TELL_NO_CHECK   UINT32_C(0x01)

This flag makes lzma_code() return LZMA_NO_CHECK if the input stream being decoded has no integrity check. Note that when used with lzma_auto_decoder(), all .lzma files will trigger LZMA_NO_CHECK if LZMA_TELL_NO_CHECK is used.

Definition at line 457 of file container.h.

LZMA_TELL_UNSUPPORTED_CHECK

#define LZMA_TELL_UNSUPPORTED_CHECK   UINT32_C(0x02)

This flag makes lzma_code() return LZMA_UNSUPPORTED_CHECK if the input stream has an integrity check, but the type of the integrity check is not supported by this liblzma version or build. Such files can still be decoded, but the integrity check cannot be verified.

Definition at line 466 of file container.h.

Function Documentation

LZMA_API() [1/3]

LZMA_API

(

lzma_ret

)

Initialize .xz Stream encoder using a preset number.

Single-call .xz Stream decoder.

Initialize .lzma decoder (legacy file format)

Decode .xz Streams and .lzma files with autodetection.

Initialize .xz Stream decoder.

Single-call .xz Stream encoder.

Initialize .lzma encoder (legacy file format)

Initialize multithreaded .xz Stream encoder.

Initialize .xz Stream encoder using a custom filter chain.

Single-call .xz Stream encoding using a preset number.

This function is intended for those who just want to use the basic features if liblzma (that is, most developers out there).

Parameters

strm	Pointer to lzma_stream that is at least initialized with LZMA_STREAM_INIT.
preset	Compression preset to use. A preset consist of level number and zero or more flags. Usually flags aren't used, so preset is simply a number [0, 9] which match the options -0 ... -9 of the xz command line tool. Additional flags can be be set using bitwise-or with the preset level number, e.g. 6 | LZMA_PRESET_EXTREME.
check	Integrity check type to use. See check.h for available checks. The xz command line tool defaults to LZMA_CHECK_CRC64, which is a good choice if you are unsure. LZMA_CHECK_CRC32 is good too as long as the uncompressed file is not many gigabytes.

Returns

- LZMA_OK: Initialization succeeded. Use lzma_code() to encode your data.

• LZMA_MEM_ERROR: Memory allocation failed.
• LZMA_OPTIONS_ERROR: The given compression preset is not supported by this build of liblzma.
• LZMA_UNSUPPORTED_CHECK: The given check type is not supported by this liblzma build.
• LZMA_PROG_ERROR: One or more of the parameters have values that will never be valid. For example, strm == NULL.

If initialization fails (return value is not LZMA_OK), all the memory allocated for *strm by liblzma is always freed. Thus, there is no need to call lzma_end() after failed initialization.

If initialization succeeds, use lzma_code() to do the actual encoding. Valid values for ‘action’ (the second argument of lzma_code()) are LZMA_RUN, LZMA_SYNC_FLUSH, LZMA_FULL_FLUSH, and LZMA_FINISH. In future, there may be compression levels or flags that don't support LZMA_SYNC_FLUSH.

The maximum required output buffer size can be calculated with lzma_stream_buffer_bound().

Parameters

preset	Compression preset to use. See the description in lzma_easy_encoder().
check	Type of the integrity check to calculate from uncompressed data.
allocator	lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free().
in	Beginning of the input buffer
in_size	Size of the input buffer
out	Beginning of the output buffer
out_pos	The next byte will be written to out[*out_pos]. *out_pos is updated only if encoding succeeds.
out_size	Size of the out buffer; the first byte into which no data is written to is out[out_size].

Returns

- LZMA_OK: Encoding was successful.

• LZMA_BUF_ERROR: Not enough output buffer space.
• LZMA_UNSUPPORTED_CHECK
• LZMA_OPTIONS_ERROR
• LZMA_MEM_ERROR
• LZMA_DATA_ERROR
• LZMA_PROG_ERROR

Parameters

strm	Pointer to properly prepared lzma_stream
filters	Array of filters. This must be terminated with filters[n].id = LZMA_VLI_UNKNOWN. See filter.h for more information.
check	Type of the integrity check to calculate from uncompressed data.

Returns

- LZMA_OK: Initialization was successful.

• LZMA_MEM_ERROR
• LZMA_UNSUPPORTED_CHECK
• LZMA_OPTIONS_ERROR
• LZMA_PROG_ERROR

This provides the functionality of lzma_easy_encoder() and lzma_stream_encoder() as a single function for multithreaded use.

The supported actions for lzma_code() are LZMA_RUN, LZMA_FULL_FLUSH, LZMA_FULL_BARRIER, and LZMA_FINISH. Support for LZMA_SYNC_FLUSH might be added in the future.

Parameters

strm	Pointer to properly prepared lzma_stream
options	Pointer to multithreaded compression options

Returns

- LZMA_OK

• LZMA_MEM_ERROR
• LZMA_UNSUPPORTED_CHECK
• LZMA_OPTIONS_ERROR
• LZMA_PROG_ERROR

The .lzma format is sometimes called the LZMA_Alone format, which is the reason for the name of this function. The .lzma format supports only the LZMA1 filter. There is no support for integrity checks like CRC32.

Use this function if and only if you need to create files readable by legacy LZMA tools such as LZMA Utils 4.32.x. Moving to the .xz format is strongly recommended.

The valid action values for lzma_code() are LZMA_RUN and LZMA_FINISH. No kind of flushing is supported, because the file format doesn't make it possible.

Returns

- LZMA_OK

• LZMA_MEM_ERROR
• LZMA_OPTIONS_ERROR
• LZMA_PROG_ERROR

Parameters

filters	Array of filters. This must be terminated with filters[n].id = LZMA_VLI_UNKNOWN. See filter.h for more information.
check	Type of the integrity check to calculate from uncompressed data.
allocator	lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free().
in	Beginning of the input buffer
in_size	Size of the input buffer
out	Beginning of the output buffer
out_pos	The next byte will be written to out[*out_pos]. *out_pos is updated only if encoding succeeds.
out_size	Size of the out buffer; the first byte into which no data is written to is out[out_size].

Returns

- LZMA_OK: Encoding was successful.

• LZMA_BUF_ERROR: Not enough output buffer space.
• LZMA_UNSUPPORTED_CHECK
• LZMA_OPTIONS_ERROR
• LZMA_MEM_ERROR
• LZMA_DATA_ERROR
• LZMA_PROG_ERROR

Parameters

strm	Pointer to properly prepared lzma_stream
memlimit	Memory usage limit as bytes. Use UINT64_MAX to effectively disable the limiter. liblzma 5.2.3 and earlier don't allow 0 here and return LZMA_PROG_ERROR; later versions treat 0 as if 1 had been specified.
flags	Bitwise-or of zero or more of the decoder flags: LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, LZMA_TELL_ANY_CHECK, LZMA_CONCATENATED

Returns

- LZMA_OK: Initialization was successful.

• LZMA_MEM_ERROR: Cannot allocate memory.
• LZMA_OPTIONS_ERROR: Unsupported flags
• LZMA_PROG_ERROR

This decoder autodetects between the .xz and .lzma file formats, and calls lzma_stream_decoder() or lzma_alone_decoder() once the type of the input file has been detected.

Parameters

strm	Pointer to properly prepared lzma_stream
memlimit	Memory usage limit as bytes. Use UINT64_MAX to effectively disable the limiter. liblzma 5.2.3 and earlier don't allow 0 here and return LZMA_PROG_ERROR; later versions treat 0 as if 1 had been specified.
flags	Bitwise-or of flags, or zero for no flags.

Returns

- LZMA_OK: Initialization was successful.

• LZMA_MEM_ERROR: Cannot allocate memory.
• LZMA_OPTIONS_ERROR: Unsupported flags
• LZMA_PROG_ERROR

Parameters

strm	Pointer to properly prepared lzma_stream
memlimit	Memory usage limit as bytes. Use UINT64_MAX to effectively disable the limiter. liblzma 5.2.3 and earlier don't allow 0 here and return LZMA_PROG_ERROR; later versions treat 0 as if 1 had been specified.

Valid ‘action’ arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. There is no need to use LZMA_FINISH, but it's allowed because it may simplify certain types of applications.

Returns

- LZMA_OK

• LZMA_MEM_ERROR
• LZMA_PROG_ERROR

Parameters

memlimit	Pointer to how much memory the decoder is allowed to allocate. The value pointed by this pointer is modified if and only if LZMA_MEMLIMIT_ERROR is returned.
flags	Bitwise-or of zero or more of the decoder flags: LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, LZMA_CONCATENATED. Note that LZMA_TELL_ANY_CHECK is not allowed and will return LZMA_PROG_ERROR.
allocator	lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free().
in	Beginning of the input buffer
in_pos	The next byte will be read from in[*in_pos]. *in_pos is updated only if decoding succeeds.
in_size	Size of the input buffer; the first byte that won't be read is in[in_size].
out	Beginning of the output buffer
out_pos	The next byte will be written to out[*out_pos]. *out_pos is updated only if decoding succeeds.
out_size	Size of the out buffer; the first byte into which no data is written to is out[out_size].

Returns

- LZMA_OK: Decoding was successful.

• LZMA_FORMAT_ERROR
• LZMA_OPTIONS_ERROR
• LZMA_DATA_ERROR
• LZMA_NO_CHECK: This can be returned only if using the LZMA_TELL_NO_CHECK flag.
• LZMA_UNSUPPORTED_CHECK: This can be returned only if using the LZMA_TELL_UNSUPPORTED_CHECK flag.
• LZMA_MEM_ERROR
• LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. The minimum required memlimit value was stored to *memlimit.
• LZMA_BUF_ERROR: Output buffer was too small.
• LZMA_PROG_ERROR

LZMA_API() [2/3]

LZMA_API

(

size_t

)

Calculate output buffer size for single-call Stream encoder.

When trying to compress uncompressible data, the encoded size will be slightly bigger than the input data. This function calculates how much output buffer space is required to be sure that lzma_stream_buffer_encode() doesn't return LZMA_BUF_ERROR.

The calculated value is not exact, but it is guaranteed to be big enough. The actual maximum output space required may be slightly smaller (up to about 100 bytes). This should not be a problem in practice.

If the calculated maximum size doesn't fit into size_t or would make the Stream grow past LZMA_VLI_MAX (which should never happen in practice), zero is returned to indicate the error.

Note

The limit calculated by this function applies only to single-call encoding. Multi-call encoding may (and probably will) have larger maximum expansion when encoding uncompressible data. Currently there is no function to calculate the maximum expansion of multi-call encoding.

Calculate output buffer size for single-call Stream encoder.

This is equivalent to lzma_stream_buffer_bound() but for .xz Blocks. See the documentation of lzma_stream_buffer_bound().

Definition at line 73 of file block_buffer_encoder.c.

{
    uint64_t ret = lzma_block_buffer_bound64(uncompressed_size);
 
#if SIZE_MAX < UINT64_MAX
    // Catch the possible integer overflow on 32-bit systems.
    if (ret > SIZE_MAX)
        return 0;
#endif
 
    return ret;
}

References HEADERS_BOUND, LZMA_VLI_MAX, my_min, SIZE_MAX, and uncompressed_size.

LZMA_API() [3/3]

LZMA_API

(

uint64_t

)

Calculate approximate memory usage of easy encoder.

Calculate approximate memory usage of multithreaded .xz encoder.

Calculate approximate decoder memory usage of a preset.

This function is a wrapper for lzma_raw_encoder_memusage().

Parameters

preset

Compression preset (level and possible flags)

Returns

Number of bytes of memory required for the given preset when encoding. If an error occurs, for example due to unsupported preset, UINT64_MAX is returned.

This function is a wrapper for lzma_raw_decoder_memusage().

Parameters

preset

Compression preset (level and possible flags)

Returns

Number of bytes of memory required to decompress a file that was compressed using the given preset. If an error occurs, for example due to unsupported preset, UINT64_MAX is returned.

Since doing the encoding in threaded mode doesn't affect the memory requirements of single-threaded decompressor, you can use lzma_easy_decoder_memusage(options->preset) or lzma_raw_decoder_memusage(options->filters) to calculate the decompressor memory requirements.

Parameters

options

Compression options

Returns

Number of bytes of memory required for encoding with the given options. If an error occurs, for example due to unsupported preset or filter chain, UINT64_MAX is returned.

Calculate approximate memory usage of easy encoder.

Get the uncompressed size of the file.

Get the total size of the file.

Get the total size of the Blocks.

Get the total size of the Stream.

Get the size of the Index field as bytes.

Get the number of Blocks.

Get the number of Streams.

Calculate the memory usage of an existing lzma_index.

On disk, the size of the Index field depends on both the number of Records stored and how big values the Records store (due to variable-length integer encoding). When the Index is kept in lzma_index structure, the memory usage depends only on the number of Records/Blocks stored in the Index(es), and in case of concatenated lzma_indexes, the number of Streams. The size in RAM is almost always significantly bigger than in the encoded form on disk.

This function calculates an approximate amount of memory needed hold the given number of Streams and Blocks in lzma_index structure. This value may vary between CPU architectures and also between liblzma versions if the internal implementation is modified.

This is a shorthand for lzma_index_memusage(lzma_index_stream_count(i), lzma_index_block_count(i)).

This returns the total number of Blocks in lzma_index. To get number of Blocks in individual Streams, use lzma_index_iter.

This is needed to verify the Backward Size field in the Stream Footer.

If multiple lzma_indexes have been combined, this works as if the Blocks were in a single Stream. This is useful if you are going to combine Blocks from multiple Streams into a single new Stream.

This doesn't include the Stream Header, Stream Footer, Stream Padding, or Index fields.

When no lzma_indexes have been combined with lzma_index_cat() and there is no Stream Padding, this function is identical to lzma_index_stream_size(). If multiple lzma_indexes have been combined, this includes also the headers of each separate Stream and the possible Stream Padding fields.

Definition at line 29 of file crc64_fast.c.

{
    crc = ~crc;
 
#ifdef WORDS_BIGENDIAN
    crc = bswap64(crc);
#endif
 
    if (size > 4) {
        while ((uintptr_t)(buf) & 3) {
            crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc);
            --size;
        }
 
        const uint8_t *const limit = buf + (size & ~(size_t)(3));
        size &= (size_t)(3);
 
        while (buf < limit) {
#ifdef WORDS_BIGENDIAN
            const uint32_t tmp = (crc >> 32)
                    ^ aligned_read32ne(buf);
#else
            const uint32_t tmp = crc ^ aligned_read32ne(buf);
#endif
            buf += 4;
 
            crc = lzma_crc64_table[3][A(tmp)]
                ^ lzma_crc64_table[2][B(tmp)]
                ^ S32(crc)
                ^ lzma_crc64_table[1][C(tmp)]
                ^ lzma_crc64_table[0][D(tmp)];
        }
    }
 
    while (size-- != 0)
        crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc);
 
#ifdef WORDS_BIGENDIAN
    crc = bswap64(crc);
#endif
 
    return ~crc;
}

References A, A1, aligned_read32ne(), B, bswap64, C, D, limit, lzma_crc64_table, S32, S8, and autogen_x86imm::tmp.

Variable Documentation

allocator

uint32_t const lzma_allocator* allocator

Definition at line 293 of file container.h.

check

lzma_check check

Definition at line 292 of file container.h.

Referenced by block_buffer_encode(), compare_list(), do_rd_expr(), indx(), inflateValidate(), LZMA_API(), lzma_check_finish(), lzma_check_init(), lzma_check_update(), lzma_sha256_finish(), lzma_sha256_init(), lzma_sha256_update(), main(), parse_real(), process(), rd_expr(), rd_expr_and(), rd_expr_equal(), rd_expr_or(), rd_expr_shift(), rd_expr_unequal(), rd_expr_xor(), rd_factor(), rd_term(), rd_value(), rz_dyldcache_get_objc_opt_info(), stream_encoder_init(), and xz_compress().

filters

const lzma_filter* filters

Definition at line 315 of file container.h.

Referenced by alone_decode(), alone_encoder_init(), arm_coder_init(), armthumb_coder_init(), block_encode_uncompressed(), code(), compress(), get_options(), ia64_coder_init(), init_encoder(), LZMA_API(), lzma_delta_coder_init(), lzma_delta_decoder_init(), lzma_delta_encoder_init(), lzma_lz_decoder_init(), lzma_lz_encoder_init(), lzma_lzma2_decoder_init(), lzma_lzma2_encoder_init(), lzma_lzma_decoder_init(), lzma_lzma_encoder_init(), lzma_mt_block_size(), lzma_next_filter_init(), lzma_raw_coder_init(), lzma_raw_coder_memusage(), lzma_simple_arm_decoder_init(), lzma_simple_arm_encoder_init(), lzma_simple_armthumb_decoder_init(), lzma_simple_armthumb_encoder_init(), lzma_simple_coder_init(), lzma_simple_ia64_decoder_init(), lzma_simple_ia64_encoder_init(), lzma_simple_powerpc_decoder_init(), lzma_simple_powerpc_encoder_init(), lzma_simple_sparc_decoder_init(), lzma_simple_sparc_encoder_init(), lzma_simple_x86_decoder_init(), lzma_simple_x86_encoder_init(), main(), message_filters_show(), message_filters_to_str(), parse_block_header(), powerpc_coder_init(), rz_main_rz_asm(), sparc_coder_init(), start(), stream_decode(), stream_encoder_init(), stream_encoder_mt_init(), stream_encoder_update(), test3(), validate_chain(), and x86_coder_init().

flags

uint32_t flags

Definition at line 628 of file container.h.

in

uint32_t const lzma_allocator const uint8_t* in

Definition at line 294 of file container.h.

in_pos

uint32_t const lzma_allocator const uint8_t size_t* in_pos

Definition at line 630 of file container.h.

in_size

uint32_t const lzma_allocator const uint8_t size_t size_t in_size

Definition at line 294 of file container.h.

lzma_attr_warn_unused_result

uint32_t const lzma_allocator const uint8_t size_t size_t uint8_t size_t size_t out_size lzma_nothrow lzma_attr_warn_unused_result

Definition at line 260 of file container.h.

lzma_nothrow

lzma_check const lzma_allocator const uint8_t size_t uint8_t size_t size_t out_size lzma_nothrow

Definition at line 295 of file container.h.

memlimit

uint64_t memlimit

Definition at line 537 of file container.h.

Referenced by auto_decoder_init(), decompress_empty(), hardware_memlimit_get(), index_decoder_init(), index_decoder_reset(), lzma_alone_decoder_init(), LZMA_API(), lzma_stream_decoder_init(), message_mem_needed(), parse_indexes(), and test_code().

out

uint32_t const lzma_allocator const uint8_t size_t size_t uint8_t* out

Definition at line 295 of file container.h.

out_pos

uint32_t const lzma_allocator const uint8_t size_t size_t uint8_t size_t* out_pos

Definition at line 295 of file container.h.

preset

uint32_t preset

Definition at line 259 of file container.h.

Referenced by crc_init_preset(), get_preset(), init_encoder(), LZMA_API(), lzma_easy_preset(), main(), rz_hash_crc_preset(), set_lzma(), and xz_compress().