LZ4 Streaming API Example : Line by Line Text Compression

by Takayuki Matsuoka

blockStreaming_lineByLine.c is LZ4 Straming API example which implements line by line incremental (de)compression.

Please note the following restrictions :

• Firstly, read "LZ4 Streaming API Basics".
• This is relatively advanced application example.
• Output file is not compatible with lz4frame and platform dependent.

What's the point of this example ?

• Line by line incremental (de)compression.
• Handle huge file in small amount of memory
• Generally better compression ratio than Block API
• Non-uniform block size

How the compression works

First of all, allocate "Ring Buffer" for input and LZ4 compressed data buffer for output.

(1)
    Ring Buffer
 
    +--------+
    | Line#1 |
    +---+----+
        |
        v
     {Out#1}
 
 
(2)
    Prefix Mode Dependency
          +----+
          |    |
          v    |
    +--------+-+------+
    | Line#1 | Line#2 |
    +--------+---+----+
                 |
                 v
              {Out#2}
 
 
(3)
          Prefix   Prefix
          +----+   +----+
          |    |   |    |
          v    |   v    |
    +--------+-+------+-+------+
    | Line#1 | Line#2 | Line#3 |
    +--------+--------+---+----+
                          |
                          v
                       {Out#3}
 
 
(4)
                        External Dictionary Mode
                +----+   +----+
                |    |   |    |
                v    |   v    |
    ------+--------+-+------+-+--------+
          |  ....  | Line#X | Line#X+1 |
    ------+--------+--------+-----+----+
                            ^     |
                            |     v
                            |  {Out#X+1}
                            |
                          Reset
 
 
(5)
                                    Prefix
                                    +-----+
                                    |     |
                                    v     |
    ------+--------+--------+----------+--+-------+
          |  ....  | Line#X | Line#X+1 | Line#X+2 |
    ------+--------+--------+----------+-----+----+
                            ^                |
                            |                v
                            |            {Out#X+2}
                            |
                          Reset

Next (see (1)), read first line to ringbuffer and compress it by LZ4_compress_continue(). For the first time, LZ4 doesn't know any previous dependencies, so it just compress the line without dependencies and generates compressed line {Out#1} to LZ4 compressed data buffer. After that, write {Out#1} to the file and forward ringbuffer offset.

Do the same things to second line (see (2)). But in this time, LZ4 can use dependency to Line#1 to improve compression ratio. This dependency is called "Prefix mode".

Eventually, we'll reach end of ringbuffer at Line::X (see (4)). This time, we should reset ringbuffer offset. After resetting, at Line::X+1 pointer is not adjacent, but LZ4 still maintain its memory. This is called "External Dictionary Mode".

In Line::X+2 (see (5)), finally LZ4 forget almost all memories but still remains Line::X+1. This is the same situation as Line#2.

Continue these procedure to the end of text file.

How the decompression works

Decompression will do reverse order.

• Read compressed line from the file to buffer.
• Decompress it to the ringbuffer.
• Output decompressed plain text line to the file.
• Forward ringbuffer offset. If offset exceedes end of the ringbuffer, reset it.

Continue these procedure to the end of the compressed file.