run local llm Reviews | Reddit Scout

Jan.ai is a local LLM provider that offers Vulkan acceleration on AMD GPUs.
It is not officially supported but users have reported success using the LocalAI option with Jan.ai.

Anythingllm: Users appreciate the stability and simplicity of Anythingllm, with some finding it better than h2ogpt. Some users wish for more features related to data analysis and connecting to databases.
- Pros: Stable, simple, self-hosted, "bring your own everything" approach.
- Cons: Issues with UI/UX, difficult setup with Docker, lack of clear documentation.
PrivateGPT: Users had positive experiences with PrivateGPT, finding it effortless and easy to use compared to manual setups.
Text-Generation-WebUI: Praised for its ease of use and speed, especially for GPU-only inference on NVIDIA.
KoboldCPP: Known for its compatibility with llama.cpp and ease of use for running inference with updated models.
Exllama2+ExUI: Reported as the best for GPU-only inference on NVIDIA cards but can be complex to set up.
Mistral-Nemo and Gemini Pro: Mentioned as LLM providers but not considered game-changers by users.
LM Playground: An app that can run smaller model LLMs effectively, with a voice interface.

Users encountered various challenges during installation on Windows, such as issues with building wheels for chroma-hnwslib and compatibility with Python 3.11.
Workarounds included installing a C++ compiler, using Python 3.10 for compatibility, and creating a .bat file to ensure the correct Python version.
Llama.cpp: Some users faced errors related to llama.cpp not being detected, which was resolved by rebuilding the application after deleting the build folder.

Users expressed interest in features like structured data extraction, source code analysis, support for multiple document collections, SQL integration, and graph-based technologies like GraphRAG.
Some users suggested exploring paid options instead of self-hosting due to hardware constraints.

Products Mentioned