Setting Up Local Language Models for Your App
Your app relies on two essential models: Embeddings and Text Generation. While OpenAI's default models work seamlessly, you have the flexibility to switch providers or even run the models locally.
Step 1: Configure Environment Variables
Navigate to the .env file or set the following environment variables:
LLM_NAME=<your Text Generation model>
API_KEY=<API key for Text Generation>
EMBEDDINGS_NAME=<LLM for Embeddings>
EMBEDDINGS_KEY=<API key for Embeddings>
VITE_API_STREAMING=<true or false>You can omit the keys if users provide their own. Ensure you set LLM_NAME and EMBEDDINGS_NAME.
Step 2: Choose Your Models
Options for LLM_NAME:
- openai (More details (opens in a new tab))
- anthropic (More details (opens in a new tab))
- manifest (More details (opens in a new tab))
- cohere (More details (opens in a new tab))
- llama.cpp (More details (opens in a new tab))
- huggingface (Arc53/DocsGPT-7B by default)
- sagemaker (Mode details (opens in a new tab))
Note: for huggingface you can choose any model inside application/llm/huggingface.py or pass llm_name on init, loads
Options for EMBEDDINGS_NAME:
- openai_text-embedding-ada-002
- huggingface_sentence-transformers/all-mpnet-base-v2
- huggingface_hkunlp/instructor-large
- cohere_medium
If you want to be completely local, set EMBEDDINGS_NAME to huggingface_sentence-transformers/all-mpnet-base-v2.
For llama.cpp Download the required model and place it in the models/ folder.
Alternatively, for local Llama setup, run setup.sh and choose option 1. The script handles the DocsGPT model addition.
Step 3: Local Hosting for Privacy
If working with sensitive data, host everything locally by setting LLM_NAME, llama.cpp or huggingface, use any model available on Hugging Face, for llama.cpp you need to convert it into gguf format.
That's it! Your app is now configured for local and private hosting, ensuring optimal security for critical data.
