- Published on
Show HN: Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/Nvidia
- Authors
- Name
- DeepHorus
Janus, a new open-source tool showcased on Hacker News's Show HN forum, offers developers a single Go binary that runs GGUF large language models locally while exposing an OpenAI-compatible API. The project eliminates the need for Python, Docker, or Ollama, delivering local inference through llama.cpp backed by Vulkan — supporting AMD, Intel, and NVIDIA GPUs — with a CPU fallback for systems without compatible graphics hardware. Users simply place a .gguf file in a models folder and launch the executable, which serves requests at 127.0.0.1:8990.
The tool arrives amid growing demand for privacy-preserving, offline AI workflows, where developers want to keep model inference on their own machines rather than relying on cloud providers. Janus targets this audience by packaging everything into one executable: on Windows, users run a build.ps1 script that downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe, while Linux users build with Go and require libllama.so on LD_LIBRARY_PATH. A bundled modelget.exe helper downloads GGUF files directly from repositories such as meta-llama/Llama-3.2-3B-Instruct on Hugging Face.
Janus exposes standard OpenAI endpoints — /v1/chat/completions and /v1/models — at base URL http://127.0.0.1:8990/v1 with no API key required, and supports both standard and streaming responses via curl, PowerShell, or any script. It can be wired into coding assistants like Cursor and Cline as a local backend. Additional features include hot-swapping models without restarting, automatic chat template detection from GGUF metadata, and reasoning support that splits thinking tokens into a separate reasoning_content field.
Storage requirements scale with model size: users should plan for roughly 2–8 GB per model, plus about 50 MB for Janus and its llama.dll. Configuration is handled through a simple .env file where INFERENCE_BACKEND selects vulkan or cpu and JANUS_MODEL_PATH points to the chosen model, alongside options like JANUS_MAX_TOKENS set to 4096. The codebase is organized into cmd/janus for the server, cmd/modelget for downloads, and internal packages for the engine, DLL bridge, and single-instance guard.
By removing dependency-heavy setup, Janus lowers the barrier for developers who previously needed containerized stacks to run local models. The project is MIT-licensed and hosted on GitHub at Vibra-Ingenn/Janus, with contributions invited through a CONTRIBUTING.md file. Streaming support and compatibility with existing OpenAI clients mean teams can swap a base URL and keep their current tooling.
Future development will likely focus on broadening hardware support, refining model management, and expanding community contributions. As local inference gains traction for cost and privacy reasons, lightweight single-binary tools like Janus could shape how developers integrate open-weight models into everyday workflows.
