Binary XML encode/decode

About

inspired by tsoding

this project tries to give an answer for embedding binary data in XML. it uses self-closing XML tags to represent the binary data. Hours were spent on this project to make sure that it as highly optimized as possible, and for that reason there are some limitations.

What I've learned

• existence of zmm registers, as well as k mask registers (on cpus supporting avx512)
• ability to mmap redirected input/output, using /proc/self/fd/{0,1}
• how to use the perf program
• my own form of error handling in asm
• read/write with a big enough buffer are usually faster than other fancy syscalls
• syscalls
  • fork
  • lseek
  • madvise
  • mmap
  • munmap
  • pwrite64
  • pwritev
  • writev
• instructions
  • cmov
  • movzx
  • popcnt
  • pusha (that it can't be used on 64 bit)
  • vmovdqa64
  • vmovdqu64
  • vpbroadcast
  • vperm
  • vpextr
  • vpgather
  • vpmovzx
  • vpopcnt
  • vpscatter
  • vpshuf
  • vzeroall

Efficiency

the efficiency of the encoding method can be calculated using the following equation:

1 - (encoded_size / original_size)

and when multiplied by 100, you get the percentage.

best case: all ones

1 - 48 = -4700%

worst case: all zeros

1 - 56 = -5500%

average case: equal number of zeros and ones

1 - 52 = -5100%

Performance

My machine contains the following:

• Void Linux x86-64 (kernel version 6.12.41_1)
• AMD Ryzen AI 9 365 @ 5.04 GHz (10 cores, 20 threads, L1: 800 KiB, L2: 10 MiB, L3: 24 MiB)
• 24 GiB DDR4 RAM @ 3205 MHz
• 1TB (953 GiB) NVME M.2 PCIe (Read 4500 MB/s, Write 3600 MB/s, Read IOPS 600K, Write IOPS 650K)

encoding

139.4ms for reading 14.39 MB and writing 736.88 MB.

• read: 103.228 MB/s
• write: 5286.083 MB/s

decoding

103.3ms for reading 736.88 MB and writing 14.39 MB

• read: 7133.398 MB/s
• write: 139.303 MB/s

Getting Started

Dependencies

• x86-64 architecture
• linux kernel
• fasm
• make
• cpu with avx512

no need for libc :)

Example

after building the programs, we can run the following commands to demonstrate how this format works:

echo -n "Hello World!" > hello.txt
build/encode < hello.txt > hello.xml

firstly, we create a hello.txt file with the content Hello World!, and by using echo's -n flag we ensure that it won't insert a \n at the end of the file. then, we encode the content. we read from hello.txt and write to hello.xml the encoded XML.

looking into hello.xml, we should see the following content:

<zero/><zero/><zero/><one/><zero/><zero/><one/><zero/><one/><zero/><one/><zero/><zero/><one/><one/><zero/><zero/><zero/><one/><one/><zero/><one/><one/><zero/><zero/><zero/><one/><one/><zero/><one/><one/><zero/><one/><one/><one/><one/><zero/><one/><one/><zero/><zero/><zero/><zero/><zero/><zero/><one/><zero/><zero/><one/><one/><one/><zero/><one/><zero/><one/><zero/><one/><one/><one/><one/><zero/><one/><one/><zero/><zero/><one/><zero/><zero/><one/><one/><one/><zero/><zero/><zero/><one/><one/><zero/><one/><one/><zero/><zero/><zero/><one/><zero/><zero/><one/><one/><zero/><one/><zero/><zero/><zero/><zero/><one/><zero/><zero/>

very fast and very efficient encoding.

Build

make

Clean

make clean

Usage

Encoding

build/encode < [file to encode] > [encoded file]

Decoding

build/decode < [file to decode] > [decoded file]