Thanks! 
It was a fun challenge getting an AI agent to pair with me while adhering to strict c-struct padding and memory alignment.Let me know if anything sticks out you&#x27;d like to discuss deeper!

I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function &amp; Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn&#x27;t happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs.I have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw pointers pass straight through down to the low C level. This avoids primitive array cloning and heap churn.I mapped out the native structures from llama.cpp and whisper.cpp while matching the compiler&#x27;s padding to maintain safe memory access.I bundle pre-compiled native binaries in the jar for easy deployment.This execution engine provides the foundation I need for work I&#x27;m doing on a spatio-temporal memory layer (L-TABB) to replace RAGs. I&#x27;d love to get technical feedback to polish any issues while I continue working on the next layer. 
Deep-dives from anyone hacking on Project Panama or low-latency systems in modern JDK would be very appreciated!I&#x27;m much better with code than prose, so I&#x27;ll let the code do most of the talking.Happy Hacking!
&#x2F;DavidCode: <a href="https:&#x2F;&#x2F;libargus.cc" rel="nofollow">https:&#x2F;&#x2F;libargus.cc</a>
Project Landing Page: <a href="https:&#x2F;&#x2F;projectargus.cc" rel="nofollow">https:&#x2F;&#x2F;projectargus.cc</a>

Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)

Comments

exabrial

KingJoker