ZMat - Portable Data Compression for MATLAB/Octave

ZMat

Portable data compression library and toolbox for MATLAB/Octave

Version 0.9.9 C Library MATLAB/Octave GPL v3 NIH Funded

🗜️ What is ZMat?

ZMat provides both an easy-to-use C-based data compression library (libzmat) and a portable MEX function enabling multiple compression algorithms in MATLAB and GNU Octave.

It is fast and compact, processing large arrays within fractions of a second, making it ideal for scientific computing and data storage applications.

🏆 NeuroJSON Project

ZMat is part of the NeuroJSON project (neurojson.org), funded by US NIH grant U24-NS124027. It provides essential compression infrastructure for JSONLab, JNIfTI, and other NeuroJSON tools.

6+
Years Active
7
Compression Methods
5
Blosc2 Codecs

⚙️ Supported Compression Methods

ZMat supports 7 major compression algorithms plus 5 blosc2 meta-compressor variants:

Method Speed Ratio Best For
lz4 ⚡⚡⚡⚡⚡ ⭐⭐ Real-time processing
zlib/gzip ⚡⚡⚡⚡ ⭐⭐⭐ Balanced performance
zstd ⚡⚡⚡ ⭐⭐⭐⭐ Modern applications
lz4hc ⚡⚡⚡ ⭐⭐⭐ High compression LZ4
lzma/lzip ⭐⭐⭐⭐⭐ Maximum compression
blosc2* ⚡⚡⚡⚡ ⭐⭐⭐⭐ N-D array data
base64 ⚡⚡⚡⚡⚡ - Encoding only

⚡ Performance Characteristics

🚀 Ultra Fast - LZ4

  • Fastest compression/decompression
  • Ideal for real-time applications
  • Lower compression ratio
  • Default level: 1 (1-9)

⚖️ Balanced - Zlib/Gzip/Zstd

  • Excellent speed/ratio balance
  • Industry standard algorithms
  • Wide compatibility
  • Default level: 6 (1-9)

🎯 Maximum Compression - LZMA

  • Highest compression ratio
  • Slower compression speed
  • Best for archival storage
  • Default level: 5 (1-9)

🔬 Scientific Data - Blosc2

  • Optimized for N-D arrays
  • Byte-shuffling support
  • Multi-threaded compression
  • Multiple codec options

Key Features

Comprehensive compression toolkit

🔧

C Library + MEX

Both libzmat C library and MATLAB/Octave MEX bindings included

🌍

Cross-Platform

Windows, Linux, macOS support with pre-compiled binaries

High Performance

Process large arrays in fractions of a second

📦

Multiple Codecs

7 compression methods + 5 blosc2 variants + base64

🔀

Type Preservation

Automatically restore original data type and dimensions

🎚️

Configurable Levels

Adjustable compression levels for all algorithms

Code Examples

Quick start with ZMat

% Compress data with default zlib method
[compressed, info] = zmat(eye(10));

% Decompress using the info structure
original = zmat(compressed, info);

% Or decompress explicitly
original = zmat(compressed, 0);

% Verify data integrity
isequal(original, eye(10))  % returns true

% Set compression level (-1 to -9)
[compressed, info] = zmat(data, -9, 'zlib');  % max compression
% Try different compression methods
data = rand(100, 100);

% Zlib (default, balanced)
[zlib_out, info] = zmat(data, 1, 'zlib');

% Gzip (similar to zlib)
[gzip_out, info] = zmat(data, 1, 'gzip');

% LZ4 (fastest)
[lz4_out, info] = zmat(data, 1, 'lz4');

% LZMA (highest compression)
[lzma_out, info] = zmat(data, 1, 'lzma');

% Zstd (modern, flexible)
[zstd_out, info] = zmat(data, 1, 'zstd');
% Blosc2 meta-compressor optimized for N-D arrays
data = uint8(rand(100, 100, 50) * 255);

% Blosc2 with different internal codecs
[b_lz4, info] = zmat(data, 1, 'blosc2lz4');
[b_zlib, info] = zmat(data, 1, 'blosc2zlib');
[b_zstd, info] = zmat(data, 1, 'blosc2zstd');

% With multi-threading and byte-shuffling
[compressed, info] = zmat(data, -6, 'blosc2lz4', ...
    'nthread', 4, 'shuffle', 1, 'typesize', 1);

% Decompress
original = zmat(compressed, info);
/* test_zmat.c - Compress data using libzmat */
#include "zmatlib.h"
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main() {
    unsigned char input[] = "ZMat compression test data";
    size_t inputsize = strlen((char*)input);
    unsigned char *output = NULL;
    size_t outputsize = 0;
    int status = 0;
    
    /* Compress: zipid=0 (zlib), clevel=-6 */
    zmat_run(inputsize, input, &outputsize, &output, 
             0, &status, -6);
    
    printf("Compressed: %zu -> %zu bytes\n", inputsize, outputsize);
    
    /* Decompress: clevel=0 for decompression */
    unsigned char *decoded = NULL;
    size_t decodedsize = 0;
    zmat_run(outputsize, output, &decodedsize, &decoded,
             0, &status, 0);
    
    printf("Decompressed: %s\n", decoded);
    
    free(output);
    free(decoded);
    return 0;
}

📝 Compile Command:

gcc -o test_zmat test_zmat.c -I../include -L../lib -lzmat -lz -llzma -llz4

📋 Makefile:

CC = gcc
CFLAGS = -I../include -O3 -Wall
LDFLAGS = -L../lib -lzmat -lz -llzma -llz4 -lzstd

test_zmat: test_zmat.c
	$(CC) $(CFLAGS) -o $@ $< $(LDFLAGS)

clean:
	rm -f test_zmat

🏗️ CMakeLists.txt:

cmake_minimum_required(VERSION 3.3)
project(zmat_example C)

find_library(ZMAT_LIB zmat PATHS ../lib)
find_library(Z_LIB z)
find_library(LZMA_LIB lzma)
find_library(LZ4_LIB lz4)

add_executable(test_zmat test_zmat.c)
target_include_directories(test_zmat PRIVATE ../include)
target_link_libraries(test_zmat ${ZMAT_LIB} ${Z_LIB} ${LZMA_LIB} ${LZ4_LIB})
! test_zmat.f90 - Use ZMat in Fortran90
program zmat_demo
    use iso_c_binding
    use zmat_module
    implicit none
    
    character(len=100) :: input_str = "ZMat Fortran compression test"
    integer(c_size_t) :: input_len, output_len, decoded_len
    integer(c_int) :: zipid, status, clevel
    type(c_ptr) :: output_ptr, decoded_ptr
    character(kind=c_char), pointer :: output(:), decoded(:)
    integer :: i
    
    ! Set compression parameters
    input_len = len_trim(input_str)
    zipid = 0          ! 0=zlib, 1=gzip, 3=lzma, 5=lz4, 7=zstd
    clevel = -6        ! compression level (-1 to -9)
    
    ! Compress data
    call zmat_run(input_len, input_str, output_len, &
                  output_ptr, zipid, status, clevel)
    
    call c_f_pointer(output_ptr, output, [output_len])
    print *, "Compressed:", input_len, "->", output_len, "bytes"
    
    ! Decompress (clevel=0 for decompression)
    call zmat_run(output_len, output, decoded_len, &
                  decoded_ptr, zipid, status, 0)
    
    call c_f_pointer(decoded_ptr, decoded, [decoded_len])
    print *, "Decompressed:", (decoded(i), i=1,decoded_len)
    
end program zmat_demo

📝 Compile Command:

gfortran -o test_zmat test_zmat.f90 zmat_fortran.f90 \
         -I../include -L../lib -lzmat -lz -llzma -llz4

📋 Makefile:

FC = gfortran
FFLAGS = -I../include -O3 -Wall
LDFLAGS = -L../lib -lzmat -lz -llzma -llz4 -lzstd

SOURCES = test_zmat.f90 zmat_fortran.f90

test_zmat: $(SOURCES)
	$(FC) $(FFLAGS) -o $@ $^ $(LDFLAGS)

clean:
	rm -f test_zmat *.mod *.o

🏗️ CMakeLists.txt:

cmake_minimum_required(VERSION 3.3)
project(zmat_fortran Fortran)

enable_language(Fortran)

find_library(ZMAT_LIB zmat PATHS ../lib)
find_library(Z_LIB z)
find_library(LZMA_LIB lzma)

add_executable(test_zmat test_zmat.f90 zmat_fortran.f90)
target_include_directories(test_zmat PRIVATE ../include)
target_link_libraries(test_zmat ${ZMAT_LIB} ${Z_LIB} ${LZMA_LIB})
% Base64 encoding and decoding
str = 'ZMat compression test';

% Encode to base64 (iscompress=2 removes trailing newline)
encoded = char(zmat(str, 2, 'base64'));

% Decode from base64
decoded = char(zmat(encoded, 0, 'base64'));

% Keep all newlines with iscompress=3
encoded_full = char(zmat(str, 3, 'base64'));

% Combine compression + base64 encoding
data = rand(50, 50);
[compressed, info] = zmat(data, 1, 'zlib');
base64_compressed = char(zmat(compressed, 2, 'base64'));

API Reference

Simple and powerful interface

Main Function

  • zmat - Compress/decompress data

Single unified function for all compression operations

Compression Methods

  • 'zlib' - Zlib/zip compression
  • 'gzip' - Gzip compression
  • 'lzma' - LZMA compression
  • 'lzip' - Lzip compression
  • 'lz4' - LZ4 fast compression
  • 'lz4hc' - LZ4 high compression
  • 'zstd' - Zstandard compression

Blosc2 Meta-Compressor

  • 'blosc2blosclz' - Blosc LZ codec
  • 'blosc2lz4' - Blosc + LZ4
  • 'blosc2lz4hc' - Blosc + LZ4HC
  • 'blosc2zlib' - Blosc + Zlib
  • 'blosc2zstd' - Blosc + Zstd

Encoding

  • 'base64' - Base64 encode/decode

C Library API

  • zmat_run - Main compression function
  • zmat_encode - Encoding wrapper
  • zmat_decode - Decoding wrapper

Available in libzmat.a / libzmat.so

Options

  • 'nthread' - Thread count (blosc2)
  • 'typesize' - Bytes per element
  • 'shuffle' - Byte shuffling (blosc2)

Installation

Multiple installation options

📦 Manual Installation

Download and add to MATLAB/Octave path:

addpath('/path/to/zmat');
rehash;

Pre-compiled MEX files included

🐧 Debian/Ubuntu

Install via package manager:

# For Octave/MATLAB toolbox
sudo apt-get install octave-zmat matlab-zmat

# For C library development
sudo apt-get install libzmat1 libzmat1-dev

🎩 Fedora

Install official packages:

# For Octave toolbox
sudo dnf install octave-zmat

# For C library
sudo dnf install zmat zmat-devel zmat-static

🔧 From Source

Clone and compile:

git clone https://github.com/NeuroJSON/zmat.git
cd zmat/src
make mex  # for MATLAB
make oct  # for Octave

🏗️ CMake Build

Advanced compilation:

cd zmat/src
mkdir build && cd build
cmake ../
make

🪟 Windows MinGW

In MATLAB, run:

cd zmat/src
compilezmat

Requires MinGW-w64 compiler

Ready to Compress Your Data?

Fast, portable, and easy to use

6+
Years Development
0.9.9
Current Version
12
Total Codecs

Supported by NIH Grant U24-NS124027

GPL v3 License

Powered by Habitat