Anemll Anemll

Live decode TPS and TTFT for Pi

This guide adds a model-neutral throughput indicator to Pi. It was validated against @earendil-works/pi-coding-agent 0.85.1.

The extension works with local and hosted models because it observes Pi's standard assistant-stream events. It does not read vLLM logs or metrics and does not depend on a particular model name.

M5 W8A8 FFN expansion: splits and prefill vectors

The complete 2048 -> 8192 projection becomes about 2.1–2.5x faster when split into 2/4/8 output-channel branches on this M5. TP2 is a strong default for this tested shape; more splits do not consistently improve it. These are sequential branches in one ANE graph, not distributed tensor parallelism.

All measurements below include the full output: one native evaluation of all branches and concatenation. Compile/load time and external application overhead are excluded. This is an expansion projection, not a full gated FFN (no gate, activation, or down projection).

Shared weight scale

Vectors	TP1 µs	TP2 µs	TP4 µs	TP8 µs	TP2 speedup	TP2 µs/vector

KernelDMA 1 MiB prefetch notch — portable profile

Post: Getting 50 GB/s Back Out of the ANE (Eileen Yoon, 2026-08-10). This repo’s Bonsai 27B export is not on the notch. Use this note plus the profiler to check other chips (M1 Max, M3 Ultra, M4, M5, …) and to confirm ANEC has no 1 MiB KernelDMA-split flag.

Gist (same scripts): https://gist.github.com/Anemll/39f657dc48b402747bdd96458edd415f

Kimi Delta Attention (KDA): Algorithm and vLLM PR #27654

This note explains the Kimi Delta Attention algorithm and maps it to the merged implementation in vllm-project/vllm#27654. The analysis is based on the PR's final head commit f48a1cd, the Kimi Linear paper, and the reference Flash Linear Attention implementation.

1. Executive summary

Pi Coding Agent + Apple Foundation Models

This gist shows a working local Pi provider setup for Apple's fm serve Chat Completions endpoint.

It supports both Apple Foundation Models exposed by the fm CLI:

fm/system: on-device Apple Foundation Model, configured as 4K context
fm/pcc: Private Cloud Compute model, configured as 32K context

Pi Coding Agent + Apple Foundation Models

This gist shows a working local Pi provider setup for Apple's fm serve Chat Completions endpoint.

It supports both Apple Foundation Models exposed by the fm CLI:

fm/system: on-device Apple Foundation Model, configured as 4K context
fm/pcc: Private Cloud Compute model, configured as 32K context

ANE INT8 W8A8 Benchmark: ~1.7-1.9x FP16 Throughput on Apple Silicon

Demonstrates that Apple Neural Engine (ANE) achieves significantly higher throughput with INT8 W8A8 quantization vs FP16, consistent with native INT8 datapath support.

Results (M5, h17g, single ANE cluster)

Summary

Method	FP16	INT8 W8A8	Ratio

	import FoundationModels
	import Playgrounds
	import Foundation

	let session = LanguageModelSession()
	let start = Date()
	let response = try await session.respond(to: "What is Apple Neural Engine and how to use it?")
	let responseText = response.content // Replace 'value' with the actual property name from LanguageModelSession.Response<String> that holds the string payload.
	print(responseText)
	let end = Date()