Showing revision 2.0
ZMat - Portable Data Compression for MATLAB, Python & C

ZMat

Portable data compression library for MATLAB/Octave, Python and C/Fortran

v1.0.0 MATLAB / Octave Python C Library Fortran90 GPL v3 NIH Funded

🗜️ What is ZMat?

ZMat provides a fast, portable C compression library (libzmat) with bindings for MATLAB/Octave (MEX) and Python (C extension). A single unified API gives you access to 12 codecs with zero external dependencies.

🏆 NeuroJSON Project

Part of the NeuroJSON project (neurojson.org), funded by NIH grant U24-NS124027. Powers compression in JSONLab, JNIfTI and other NeuroJSON tools.

12
Total Codecs
3
Language Bindings
6+
Years Active

⚙️ Supported Codecs

MethodSpeedRatioBest For
lz4⚡⚡⚡⚡⚡⭐⭐Real-time / streaming
zlib / gzip⚡⚡⚡⚡⭐⭐⭐Balanced; wide compatibility
zstd⚡⚡⚡⭐⭐⭐⭐Modern high-throughput
lz4hc⚡⚡⚡⭐⭐⭐LZ4 with better ratio
lzma / lzip⭐⭐⭐⭐⭐Maximum compression
blosc2*⚡⚡⚡⚡⭐⭐⭐⭐N-D numeric arrays
base64⚡⚡⚡⚡⚡Text-safe encoding

* blosc2 variants: blosclz, lz4, lz4hc, zlib, zstd

⚡ Performance Guide

🚀 Ultra-Fast — LZ4

  • Fastest compress & decompress
  • Ideal for real-time pipelines
  • Level 1–9

⚖️ Balanced — zlib / zstd

  • Industry-standard algorithms
  • Python stdlib interoperable
  • Level 1–9 (zstd up to 22)

🎯 Smallest — LZMA

  • Highest compression ratio
  • Best for archival/cold storage
  • Level 1–9

🔬 Scientific — Blosc2

  • Optimized for N-D arrays
  • Byte-shuffle pre-filter
  • Multi-threaded (nthread)

Key Features

One library, every language you need

🔧

libzmat C Library

Static & dynamic library. Embed directly in any C/C++/Fortran project.

📐

MATLAB / Octave MEX

Single zmat() call for compress, decompress, encode and decode.

🐍

Python Extension

pip install zmat — pure C extension, zero runtime dependencies.

🌍

Cross-Platform

Windows, Linux, macOS. Pre-compiled MEX & Python wheels available.

📦

12 Codecs Built-in

zlib, gzip, lzma, lzip, lz4, lz4hc, zstd, 5× blosc2, base64.

🔀

Type Preservation

MATLAB info struct restores original array type and dimensions.

Code Examples

Choose your language

% ── Compress with default zlib ──────────────────────────────────
[compressed, info] = zmat(eye(10));

% Decompress using the info struct (restores type + dimensions)
original = zmat(compressed, info);

% Or decompress manually (iscompress = 0)
original = zmat(compressed, 0);
isequal(original, eye(10))   % → 1

% ── Compression level ───────────────────────────────────────────
% iscompress = -N sets compression level N (1–9)
data = rand(500, 500);
[c1, i1] = zmat(data, -1, 'zlib');   % fast, larger
[c9, i9] = zmat(data, -9, 'zlib');   % slow, smaller
fprintf('Level 1: %d B  Level 9: %d B\n', numel(c1), numel(c9));

% ── Round-trip any array type ───────────────────────────────────
img = uint8(rand(256,256,3) * 255);
[cimg, iimg] = zmat(img, 1, 'lz4');
restored = zmat(cimg, iimg);    % uint8 256×256×3 restored
% ── Benchmark all codecs on the same data ───────────────────────
data   = rand(500, 500);       % 2 MB of doubles
codecs = {'zlib','gzip','lzma','lzip','lz4','lz4hc','zstd'};

for k = 1:numel(codecs)
    t = tic;
    [cdata, info] = zmat(data, 1, codecs{k});
    tc = toc(t);
    t = tic;
    orig = zmat(cdata, info);
    td = toc(t);
    fprintf('%-8s  %6.1f KB  ratio=%.2f  c=%.3fs  d=%.3fs\n', ...
        codecs{k}, numel(cdata)/1024, numel(cdata)/numel(data)/8, tc, td);
end

% ── Decompression with explicit method ──────────────────────────
[czstd, ~] = zmat(data, 1, 'zstd');
orig = zmat(czstd, 0, 'zstd');     % pass method explicitly
assert(isequal(orig, data));
% ── Blosc2 meta-compressor (optimized for N-D numeric arrays) ───
vol = uint16(rand(128,128,64) * 65535);   % 3-D volume, uint16

% Basic blosc2 variants
[b_lz4,  i1] = zmat(vol, 1, 'blosc2lz4');
[b_zstd, i2] = zmat(vol, 1, 'blosc2zstd');
[b_zlib, i3] = zmat(vol, 1, 'blosc2zlib');

fprintf('lz4: %d  zstd: %d  zlib: %d bytes\n', ...
    numel(b_lz4), numel(b_zstd), numel(b_zlib));

% Advanced: 4 threads, byte-shuffle, typesize=2 (uint16)
[best, info] = zmat(vol, -6, 'blosc2zstd', ...
    'nthread', 4, 'shuffle', 1, 'typesize', 2);

% Decompress — info carries nthread / shuffle / typesize
restored = zmat(best, info);
assert(isequal(restored, vol));
% ── Base64 encode / decode ──────────────────────────────────────
str = 'ZMat compression test';

% Encode  (iscompress=2 strips trailing newline)
encoded = char(zmat(str, 2, 'base64'));
disp(encoded)   % → 'Wk1hdCBjb21wcmVzc2lvbiB0ZXN0'

% Decode
decoded = char(zmat(encoded, 0, 'base64'));
assert(strcmp(decoded, str));

% Keep newlines every 76 chars (iscompress=3 = RFC 2045 style)
encoded_full = char(zmat(str, 3, 'base64'));

% Chain: compress then base64-encode for JSON embedding
data  = rand(50, 50);
[cmp, info] = zmat(data, 1, 'zlib');
b64  = char(zmat(cmp, 2, 'base64'));   % safe for JSON strings

% Reverse: base64-decode then decompress
raw  = zmat(uint8(b64), 0, 'base64');
orig = zmat(raw, info);
assert(isequal(orig, data));
import zmat

# ── Compress / decompress (default: zlib) ───────────────────────
data       = b"ZMat compression test data" * 1000
compressed = zmat.compress(data)
restored   = zmat.decompress(compressed)
assert restored == data

print(ff"Original: {len(data):,} B → Compressed: {len(compressed):,} B  "
      f"(ratio {len(compressed)/len(data):.3f})")

# ── Compression level ───────────────────────────────────────────
# low-level zmat() call: iscompress=1 default, -N sets level
c_fast = zmat.zmat(data, iscompress=1,  method='zlib')   # default
c_max  = zmat.zmat(data, iscompress=-9, method='zlib')   # max
print(f"Level 1: {len(c_fast)} B  Level 9: {len(c_max)} B")

# Both round-trip correctly
assert zmat.decompress(c_fast) == data
assert zmat.decompress(c_max)  == data

# ── bytearray input also works ──────────────────────────────────
buf = bytearray(b"buffered data" * 500)
assert zmat.decompress(zmat.compress(buf)) == bytes(buf)
import zmat, time

data = bytes(range(256)) * 4000   # ~1 MB, moderately compressible

methods = ['zlib', 'gzip', 'lzma', 'lzip', 'lz4', 'lz4hc', 'zstd']

print(f"{'Method':<10} {'Size':>8}  {'Ratio':>6}  {'Comp':>8}  {'Decomp':>8}")
print("-" * 50)

for m in methods:
    t0 = time.perf_counter()
    c  = zmat.compress(data, method=m)
    tc = time.perf_counter() - t0

    t0 = time.perf_counter()
    d  = zmat.decompress(c, method=m)
    td = time.perf_counter() - t0

    assert d == data, ff"{m} round-trip failed!"
    print(ff"{m:<10} {len(c):>8}  {len(c)/len(data):>6.3f}  {tc:>7.4f}s  {td:>7.4f}s")

# ── Method names are case-insensitive ───────────────────────────
c1 = zmat.compress(data, method='ZLIB')
c2 = zmat.compress(data, method='zlib')
assert c1 == c2
import zmat

# ── Blosc2 meta-compressor (ideal for numeric arrays) ───────────
import struct
# Simulate a 3-D uint16 volume (128×128×64)
vol = bytes(range(256)) * (128*128*64 * 2 // 256)

# All five blosc2 variants
for m in ['blosc2blosclz','blosc2lz4','blosc2lz4hc','blosc2zlib','blosc2zstd']:
    c = zmat.compress(vol, method=m)
    d = zmat.decompress(c, method=m)
    assert d == vol
    print(ff"{m:<18}  {len(c):>8} B  ratio={len(c)/len(vol):.3f}")

# ── Advanced: multi-threading + byte-shuffle ────────────────────
# typesize=2 → uint16 elements; shuffle=1 → byte-shuffle filter
c_adv = zmat.zmat(vol, iscompress=-6, method='blosc2zstd',
                  nthread=4, shuffle=1, typesize=2)
d_adv = zmat.zmat(c_adv, iscompress=0, method='blosc2zstd')
assert d_adv == vol
print(ff"Advanced blosc2zstd: {len(c_adv)} B")
import numpy as np
import zmat

# ── Compress / restore a NumPy array ────────────────────────────
arr = np.random.rand(1000, 1000)           # 8 MB float64
raw = arr.tobytes()                          # zmat works on bytes

c = zmat.compress(raw, method='lz4')
print(f"lz4: {len(raw):,} → {len(c):,} B")

restored = np.frombuffer(
    zmat.decompress(c, method='lz4'),
    dtype=arr.dtype
).reshape(arr.shape)
assert np.array_equal(arr, restored)

# ── Blosc2 + shuffle for integer arrays ─────────────────────────
vol   = np.arange(128*128*64, dtype=np.uint16).reshape(128,128,64)
cbest = zmat.zmat(vol.tobytes(), iscompress=1, method='blosc2zstd',
                  typesize=2, shuffle=1)    # typesize = itemsize
dbest = zmat.zmat(cbest, iscompress=0, method='blosc2zstd')
vrest = np.frombuffer(dbest, dtype=np.uint16).reshape(vol.shape)
assert np.array_equal(vol, vrest)

# ── Structured dtype round-trip ─────────────────────────────────
dt  = np.dtype([('x', np.float32), ('y', np.float32), ('z', np.float32)])
pts = np.zeros(100000, dtype=dt)
c2  = zmat.compress(pts.tobytes(), method='zstd')
print(f"Struct array: {pts.nbytes:,} → {len(c2):,} B")
import zmat, zlib, gzip, base64, io

# ── Base64 encode / decode ──────────────────────────────────────
data    = b"ZMat Python base64 test"
encoded = zmat.encode(data, method='base64')
decoded = zmat.decode(encoded, method='base64')
assert decoded == data

# zmat base64 matches Python stdlib (strip newlines for comparison)
assert encoded.replace(b'\n', b'') == base64.b64encode(data)

# Chain: compress → base64  (useful for JSON / XML embedding)
payload = b"scientific data" * 5000
c   = zmat.compress(payload, method='zlib')
b64 = zmat.encode(c, method='base64')
print(f"base64 string: {len(b64)} chars")

# Reverse
c2  = zmat.decode(b64, method='base64')
out = zmat.decompress(c2, method='zlib')
assert out == payload

# ── stdlib interoperability ─────────────────────────────────────
text = b"interoperability test" * 500

# zmat zlib ↔ Python zlib
assert zlib.decompress(zmat.compress(text, method='zlib')) == text
assert zmat.decompress(zlib.compress(text), method='zlib')  == text

# zmat gzip ↔ Python gzip
cg = zmat.compress(text, method='gzip')
with gzip.open(io.BytesIO(cg), 'rb') as f:
    assert f.read() == text
/* test_zmat.c – compress and decompress with libzmat */
#include "zmatlib.h"
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main() {
    unsigned char input[] = "ZMat C compression example";
    size_t inputsize = strlen((char*)input);
    unsigned char *output  = NULL;
    unsigned char *decoded = NULL;
    size_t outputsize = 0, decodedsize = 0;
    int    status    = 0;

    /* zipid: 0=zlib 1=gzip 3=lzma 5=lz4 7=zstd 8-12=blosc2* */
    /* clevel: -1 to -9 to set level;  0 = decompress          */
    zmat_run(inputsize, input, &outputsize, &output, 0, &status, -6);
    printf("Compressed  : %zu → %zu bytes\n", inputsize, outputsize);

    zmat_run(outputsize, output, &decodedsize, &decoded, 0, &status, 0);
    printf("Decompressed: %s\n", decoded);

    free(output);
    free(decoded);
    return 0;
}
! test_zmat.f90 – Fortran90 binding via zmat_module
program zmat_demo
    use iso_c_binding
    use zmat_module
    implicit none

    character(len=40)         :: input_str = "ZMat Fortran demo"
    integer(c_size_t)          :: input_len, output_len, decoded_len
    integer(c_int)             :: zipid, status, clevel
    type(c_ptr)                :: out_ptr, dec_ptr
    character(kind=c_char), pointer :: out_arr(:), dec_arr(:)
    integer :: i

    input_len = len_trim(input_str)
    zipid  = 0    ! 0=zlib, 5=lz4, 7=zstd …
    clevel = -6   ! compression level

    ! Compress
    call zmat_run(input_len, input_str, output_len, out_ptr, zipid, status, clevel)
    call c_f_pointer(out_ptr, out_arr, [output_len])
    print *, "Compressed:", input_len, "→", output_len, "bytes"

    ! Decompress (clevel=0)
    call zmat_run(output_len, out_arr, decoded_len, dec_ptr, zipid, status, 0)
    call c_f_pointer(dec_ptr, dec_arr, [decoded_len])
    print *, "Decompressed:", (dec_arr(i), i=1,decoded_len)
end program zmat_demo

GCC / Clang

gcc -o test_zmat test_zmat.c -I../include -L../lib -lzmat -lpthread -lm

CMake

cmake_minimum_required(VERSION 3.3)
project(zmat_example C)

find_library(ZMAT_LIB zmat PATHS ../lib)
add_executable(test_zmat test_zmat.c)
target_include_directories(test_zmat PRIVATE ../include)
target_link_libraries(test_zmat ${ZMAT_LIB} pthread m)

Fortran

gfortran -o test_zmat test_zmat.f90 zmat_fortran.f90 \
    -I../include -L../lib -lzmat -lpthread -lm

API Reference

Consistent interface across all languages

MATLAB / Octave

zmat.m / zipmat.mex
  • zmat(data)compress with default zlib
  • zmat(data, 1, method)compress with named codec
  • zmat(data, -N, method)compress at level N (1–9)
  • zmat(data, 0, method)decompress
  • zmat(data, info)decompress & restore type/size
  • [out, info] = zmat(…)info carries type, size, method
  • zmat(…, 'nthread', N)blosc2 thread count
  • zmat(…, 'typesize', B)bytes per element (shuffle)

Python

import zmat
  • zmat.compress(data, method, level)compress bytes / bytearray
  • zmat.decompress(data, method)decompress bytes
  • zmat.encode(data, method)encode (e.g. base64)
  • zmat.decode(data, method)decode
  • zmat.zmat(data, iscompress, method,
      nthread, shuffle, typesize) low-level full-control interface

C Library

zmatlib.h
  • zmat_run(insize, in, &outsize, &out,
      zipid, &status, clevel) main compress / decompress call
  • zmat_encode(…)compress wrapper
  • zmat_decode(…)decompress wrapper

Codec Names

zipid (C) / method (MATLAB & Python)
  • 0 / 'zlib'zlib/zip (default)
  • 1 / 'gzip'gzip format
  • 2 / 'base64'base64 encoding
  • 3 / 'lzma'LZMA
  • 4 / 'lzip'lzip format
  • 5 / 'lz4'LZ4 fast
  • 6 / 'lz4hc'LZ4 high compression
  • 7 / 'zstd'Zstandard
  • 8–12 / 'blosc2*'blosclz, lz4, lz4hc, zlib, zstd

Installation

Pick your platform and language

🐍 Python — PyPI

Zero dependencies. Wheels for Linux, macOS, Windows.

pip install zmat

Build from source:

git clone https://github.com/NeuroJSON/zmat.git cd zmat/python && pip install .

📐 MATLAB / Octave — Manual

Download ZIP, then add to path:

addpath('/path/to/zmat'); rehash;

Pre-compiled MEX files included for Win/Linux/macOS

🐧 Debian / Ubuntu

Install from package manager:

sudo apt-get install octave-zmat matlab-zmat # C library + headers sudo apt-get install libzmat1 libzmat1-dev

🎩 Fedora

sudo dnf install octave-zmat # C library sudo dnf install zmat zmat-devel zmat-static

🔧 From Source (MEX)

git clone https://github.com/NeuroJSON/zmat.git cd zmat/src make mex # MATLAB make oct # Octave

🏗️ CMake

cd zmat/src mkdir build && cd build cmake ../ make

Builds libzmat.a + zipmat.mex automatically

Ready to Compress?

Fast · Portable · Zero dependencies

1.0.0
Current Version
12
Codecs Built-in
3
Language Bindings

Supported by NIH Grant U24-NS124027  ·  GPL v3 License

Copyright © 2019–2026 Qianqian Fang <q.fang@neu.edu>

Powered by Habitat