hardware.h File Reference

Hardware information. More...

Go to the source code of this file.

Functions
	LZMA_API (uint64_t) lzma_physmem(void) lzma_nothrow
	Get the total amount of physical memory (RAM) in bytes. More...
	LZMA_API (uint32_t) lzma_cputhreads(void) lzma_nothrow
	Get the number of processor cores or threads. More...

Detailed Description

Hardware information.

Since liblzma can consume a lot of system resources, it also provides ways to limit the resource usage. Applications linking against liblzma need to do the actual decisions how much resources to let liblzma to use. To ease making these decisions, liblzma provides functions to find out the relevant capabilities of the underlying hardware. Currently there is only a function to find out the amount of RAM, but in the future there will be also a function to detect how many concurrent threads the system can run.

Note

On some operating systems, these function may temporarily load a shared library or open file descriptor(s) to find out the requested hardware information. Unless the application assumes that specific file descriptors are not touched by other threads, this should have no effect on thread safety. Possible operations involving file descriptors will restart the syscalls if they return EINTR.

Definition in file hardware.h.

Function Documentation

LZMA_API() [1/2]

LZMA_API

(

uint32_t

)

const

Get the number of processor cores or threads.

This function may be useful when determining how many threads to use. If the hardware supports more than one thread per CPU core, the number of hardware threads is returned if that information is available.

On success, the number of available CPU threads or cores is returned. If this information isn't available or an error occurs, zero is returned.

Get the number of processor cores or threads.

Calculate CRC32.

Although not all Check IDs have a check algorithm associated, the size of every Check is already frozen. This function returns the size (in bytes) of the Check field with the specified Check ID. The values are: { 0, 4, 4, 4, 8, 8, 8, 16, 16, 16, 32, 32, 32, 64, 64, 64 }

If the argument is not in the range [0, 15], UINT32_MAX is returned.

Calculate CRC32 using the polynomial from the IEEE 802.3 standard.

Parameters

buf	Pointer to the input buffer
size	Size of the input buffer
crc	Previously returned CRC value. This is used to calculate the CRC of a big buffer in smaller chunks. Set to zero when starting a new calculation.

Returns

Updated CRC value, which can be passed to this function again to continue CRC calculation.

LZMA_API() [2/2]

LZMA_API

(

uint64_t

)

Get the total amount of physical memory (RAM) in bytes.

This function may be useful when determining a reasonable memory usage limit for decompressing or how much memory it is OK to use for compressing.

Returns

On success, the total amount of physical memory in bytes is returned. If the amount of RAM cannot be determined, zero is returned. This can happen if an error occurs or if there is no code in liblzma to detect the amount of RAM on the specific operating system.

Get the total amount of physical memory (RAM) in bytes.

Calculate approximate memory usage of easy encoder.

Get the uncompressed size of the file.

Get the total size of the file.

Get the total size of the Blocks.

Get the total size of the Stream.

Get the size of the Index field as bytes.

Get the number of Blocks.

Get the number of Streams.

Calculate the memory usage of an existing lzma_index.

On disk, the size of the Index field depends on both the number of Records stored and how big values the Records store (due to variable-length integer encoding). When the Index is kept in lzma_index structure, the memory usage depends only on the number of Records/Blocks stored in the Index(es), and in case of concatenated lzma_indexes, the number of Streams. The size in RAM is almost always significantly bigger than in the encoded form on disk.

This function calculates an approximate amount of memory needed hold the given number of Streams and Blocks in lzma_index structure. This value may vary between CPU architectures and also between liblzma versions if the internal implementation is modified.

This is a shorthand for lzma_index_memusage(lzma_index_stream_count(i), lzma_index_block_count(i)).

This returns the total number of Blocks in lzma_index. To get number of Blocks in individual Streams, use lzma_index_iter.

This is needed to verify the Backward Size field in the Stream Footer.

If multiple lzma_indexes have been combined, this works as if the Blocks were in a single Stream. This is useful if you are going to combine Blocks from multiple Streams into a single new Stream.

This doesn't include the Stream Header, Stream Footer, Stream Padding, or Index fields.

When no lzma_indexes have been combined with lzma_index_cat() and there is no Stream Padding, this function is identical to lzma_index_stream_size(). If multiple lzma_indexes have been combined, this includes also the headers of each separate Stream and the possible Stream Padding fields.

Get the total amount of physical memory (RAM) in bytes.

Calculate approximate memory usage of multithreaded .xz encoder.

Calculate approximate decoder memory usage of a preset.

This function is a wrapper for lzma_raw_encoder_memusage().

Parameters

preset

Compression preset (level and possible flags)

Returns

Number of bytes of memory required for the given preset when encoding. If an error occurs, for example due to unsupported preset, UINT64_MAX is returned.

This function is a wrapper for lzma_raw_decoder_memusage().

Parameters

preset

Compression preset (level and possible flags)

Returns

Number of bytes of memory required to decompress a file that was compressed using the given preset. If an error occurs, for example due to unsupported preset, UINT64_MAX is returned.

Since doing the encoding in threaded mode doesn't affect the memory requirements of single-threaded decompressor, you can use lzma_easy_decoder_memusage(options->preset) or lzma_raw_decoder_memusage(options->filters) to calculate the decompressor memory requirements.

Parameters

options

Compression options

Returns

Number of bytes of memory required for encoding with the given options. If an error occurs, for example due to unsupported preset or filter chain, UINT64_MAX is returned.

Calculate approximate memory usage of easy encoder.

Get the uncompressed size of the file.

Get the total size of the file.

Get the total size of the Blocks.

Get the total size of the Stream.

Get the size of the Index field as bytes.

Get the number of Blocks.

Get the number of Streams.

Calculate the memory usage of an existing lzma_index.

On disk, the size of the Index field depends on both the number of Records stored and how big values the Records store (due to variable-length integer encoding). When the Index is kept in lzma_index structure, the memory usage depends only on the number of Records/Blocks stored in the Index(es), and in case of concatenated lzma_indexes, the number of Streams. The size in RAM is almost always significantly bigger than in the encoded form on disk.

This function calculates an approximate amount of memory needed hold the given number of Streams and Blocks in lzma_index structure. This value may vary between CPU architectures and also between liblzma versions if the internal implementation is modified.

This is a shorthand for lzma_index_memusage(lzma_index_stream_count(i), lzma_index_block_count(i)).

This returns the total number of Blocks in lzma_index. To get number of Blocks in individual Streams, use lzma_index_iter.

This is needed to verify the Backward Size field in the Stream Footer.

If multiple lzma_indexes have been combined, this works as if the Blocks were in a single Stream. This is useful if you are going to combine Blocks from multiple Streams into a single new Stream.

This doesn't include the Stream Header, Stream Footer, Stream Padding, or Index fields.

When no lzma_indexes have been combined with lzma_index_cat() and there is no Stream Padding, this function is identical to lzma_index_stream_size(). If multiple lzma_indexes have been combined, this includes also the headers of each separate Stream and the possible Stream Padding fields.

Definition at line 29 of file crc64_fast.c.

{
    crc = ~crc;
 
#ifdef WORDS_BIGENDIAN
    crc = bswap64(crc);
#endif
 
    if (size > 4) {
        while ((uintptr_t)(buf) & 3) {
            crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc);
            --size;
        }
 
        const uint8_t *const limit = buf + (size & ~(size_t)(3));
        size &= (size_t)(3);
 
        while (buf < limit) {
#ifdef WORDS_BIGENDIAN
            const uint32_t tmp = (crc >> 32)
                    ^ aligned_read32ne(buf);
#else
            const uint32_t tmp = crc ^ aligned_read32ne(buf);
#endif
            buf += 4;
 
            crc = lzma_crc64_table[3][A(tmp)]
                ^ lzma_crc64_table[2][B(tmp)]
                ^ S32(crc)
                ^ lzma_crc64_table[1][C(tmp)]
                ^ lzma_crc64_table[0][D(tmp)];
        }
    }
 
    while (size-- != 0)
        crc = lzma_crc64_table[0][*buf++ ^ A1(crc)] ^ S8(crc);
 
#ifdef WORDS_BIGENDIAN
    crc = bswap64(crc);
#endif
 
    return ~crc;
}

References assert(), lzma_index_hash_s::blocks, blocks, lzma_block::check, lzma_next_coder_s::coder, lzma_block::compressed_size, lzma_index_hash_info::count, crc64_init(), crc64_table, decoder_find(), encoder_find(), filters, lzma_options_easy::filters, get_options(), make_dist_html::groups, lzma_block::header_size, INDEX_GROUP_SIZE, lzma_index_hash_info::index_list_size, index_size(), lzma_stream::internal, limit, LZMA_BLOCK_HEADER_SIZE_MAX, LZMA_BLOCK_HEADER_SIZE_MIN, LZMA_CHECK_ID_MAX, lzma_easy_preset(), LZMA_MEMUSAGE_BASE, LZMA_OK, lzma_outq_memusage(), lzma_raw_coder_memusage(), lzma_vli_is_valid, LZMA_VLI_MAX, LZMA_VLI_UNKNOWN, lzma_next_coder_s::memconfig, mythread_once, lzma_internal_s::next, NULL, options, preset, streams, strm, tuklib_physmem(), UINT32_MAX, UINT64_MAX, unpadded_size, UNPADDED_SIZE_MAX, UNPADDED_SIZE_MIN, and lzma_block::version.