Base91 in Lua

Encoding data efficiently can be a hurdle, especially when dealing with binary streams in Lua. Base91 offers a compact binary-to-text encoding scheme, significantly reducing data size compared to Base64. This guide will walk you through a practical Lua implementation of Base91 encoding and decoding. You'll learn how to integrate this into your projects to save bandwidth and storage space by efficiently representing binary data as text.

Encoding Data with Base91

Base91 encoding transforms binary data into a string of printable ASCII characters, using a 91-character alphabet. This is achieved by processing the input data in chunks, accumulating bits, and then extracting 13-bit values to map to characters from the Base91 alphabet. The Lua implementation below demonstrates this process.

local base91_chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&()*+,./:;<=>?@[]^_"
local function encode_base91(data_bytes)
  local encoded_string = ""
  local value = 0
  local bits = 0
  for i = 1, #data_bytes do
    local byte = string.byte(data_bytes, i)
    value = (value * 256) + byte
    bits = bits + 8
    while bits >= 13 do
      local index = math.floor(value / (256^math.floor(bits / 13))) + 1
      encoded_string = encoded_string .. string.sub(base91_chars, index, index)
      value = value % (256^math.floor(bits / 13))
      bits = bits - 13
    end
  end
  if bits > 0 then
    local index = math.floor(value * (256^(13 - bits))) + 1
    encoded_string = encoded_string .. string.sub(base91_chars, index, index)
  end
  return encoded_string
end

A common gotcha is mishandling the final sequence of bits. Ensure the remaining bits are correctly scaled and mapped to a character to avoid data truncation. Always test your encoder with various input lengths to confirm its accuracy.

Decoding Base91 Data

Decoding Base91 involves reversing the encoding process, mapping each Base91 character back to its corresponding 13-bit value and reconstructing the original byte stream. This requires careful management of the accumulated bit values.

Here's a Lua implementation snippet:

local base91_chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&()*+,./:;<=>?@[]^_"
local char_to_value = {}
for i = 1, #base91_chars do
  char_to_value[string.sub(base91_chars, i, i)] = i - 1
end

local function decode_base91(encoded_string)
  local decoded_bytes = ""
  local value = 0
  local bits = 0
  for i = 1, #encoded_string do
    local char = string.sub(encoded_string, i, i)
    local char_val = char_to_value[char]
    if char_val == nil then
      error("Invalid Base91 character: " .. char)
    end
    value = (value * 91) + char_val
    bits = bits + 13
    while bits >= 8 do
      -- Extract 8 bits for a byte
      local byte_val = math.floor(value / (2^(bits - 8)))
      decoded_bytes = decoded_bytes .. string.char(byte_val)
      value = value % (2^(bits - 8))
      bits = bits - 8
    end
  end
  -- Handle potential remaining bits (requires careful bit manipulation for full spec)
  return decoded_bytes
end

A common pitfall is off-by-one errors when mapping characters to their numerical values or when managing the bit buffer. Ensure your bit extraction logic precisely aligns with the Base91 specification. Always validate input characters to catch malformed data early.

Integrating Base91 for Data Transmission

Base91 proves invaluable when you need to transmit binary data over protocols that primarily handle text, such as embedding it within HTTP POST request bodies or email content. By encoding raw bytes into a Base91 string, you ensure compatibility with these text-based channels. The resulting string is designed to be safe for transmission across various character sets without corruption.

Consider this practical scenario:

-- Assume 'binary_data' holds your raw byte stream
local encoded_string = base91.encode(binary_data)
-- Now, 'encoded_string' can be safely sent in an email body or API payload
print("Transmittable data: " .. encoded_string)

A common pitfall is neglecting the target protocol's specific character encoding requirements or potential length restrictions for the transmitted data. Always verify that the Base91 encoded string will be correctly interpreted by the receiving system. This preparation guarantees reliable data transfer.

Optimizing Base91 Performance in Lua

To squeeze the most performance out of your Lua Base91 implementation, start by profiling it. Identify exactly where your code spends the most time. Often, repeated calculations involving powers of 256 can be a target.

If you're encoding or decoding large amounts of data, pre-calculating these powers can offer a noticeable speedup by avoiding repeated computation.

-- Pre-calculate powers of 256
local powers_of_256 = {}
for i = 0, 10 do -- Adjust range as needed
    powers_of_256[i] = 256^i
end

-- Example usage within an encoding function:
-- local index = math.floor(value / powers_of_256[math.floor(bits / 13)]) + 1

A common pitfall is premature optimization. Don't sacrifice code clarity for minor gains before profiling confirms a bottleneck. Focus optimization efforts where they'll have the biggest impact. Always profile first, then optimize judiciously.