A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA.

Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.

• Dropdown menu for switching between models
• Notebook mode that resembles OpenAI's playground
• Chat mode for conversation and role-playing
• Instruct mode compatible with various formats, including Alpaca, Vicuna, Open Assistant, Dolly, Koala, ChatGLM, MOSS, RWKV-Raven, Galactica, StableLM, WizardLM, Baize, MPT, and INCITE
• Multimodal pipelines, including LLaVA and MiniGPT-4
• Markdown output for GALACTICA, including LaTeX rendering
• Nice HTML output for GPT-4chan
• Custom chat characters
• Advanced chat features (send images, get audio responses with TTS)
• Very efficient text streaming
• Parameter presets
• LLaMA model
• 4-bit GPTQ mode
• LoRA (loading and training)
• llama.cpp
• RWKV model
• 8-bit mode
• Layers splitting across GPU(s), CPU, and disk
• CPU mode
• FlexGen
• DeepSpeed ZeRO-3
• API with streaming and without streaming
• Extensions - see the user extensions list

Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder.

• The source codes are here: https://github.com/oobabooga/one-click-installers
• There is no need to run the installers as admin.
• AMD doesn't work on Windows.
• Huge thanks to @jllllll, @ClayShoaf, and @xNul for their contributions to these installers.

Recommended if you have some experience with the command line.

On Windows, I additionally recommend carrying out the installation on WSL instead of the base system: WSL installation guide.

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands:

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

Source: https://educe-ubc.github.io/conda.html

conda create -n textgen python=3.10.9
conda activate textgen

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

• MacOS users: oobabooga#393
• AMD users: https://rentry.org/eq3hg

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt

The base installation covers transformers models (AutoModelForCausalLM and AutoModelForSeq2SeqLM specifically) and llama.cpp (GGML) models.

To use 4-bit GPU models, the additional installation steps below are necessary:

GPTQ models (4 bit mode)

As an alternative to the recommended WSL method, you can install the web UI natively on Windows using this guide. It will be a lot harder and the performance may be slower: Windows installation guide.

ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
docker compose up --build

You need to have docker compose v2.17 or higher installed in your system. To see how to install docker compose itself, see the guide in here.

Contributed by @loeken in #633

From time to time, the requirements.txt changes. To update, use this command:

conda activate textgen
cd text-generation-webui
pip install -r requirements.txt --upgrade

Models should be placed inside the models/ folder.

Hugging Face is the main place to download models. These are some examples:

• Pythia
• OPT
• GALACTICA
• GPT-J 6B

You can automatically download a model from HF using the script download-model.py:

python download-model.py organization/model

For example:

python download-model.py facebook/opt-1.3b

If you want to download a model manually, note that all you need are the json, txt, and pytorch*.bin (or model*.safetensors) files. The remaining files are not necessary.

You can drop these directly into the models/ folder, making sure that the file name contains ggml somewhere and ends in .bin.

GPT-4chan has been shut down from Hugging Face, so you need to download it elsewhere. You have two options:

• Torrent: 16-bit / 32-bit
• Direct download: 16-bit / 32-bit

The 32-bit version is only relevant if you intend to run the model in CPU mode. Otherwise, you should use the 16-bit version.

After downloading the model, follow these steps:

1. Place the files under models/gpt4chan_model_float16 or models/gpt4chan_model.
2. Place GPT-J 6B's config.json file in that same folder: config.json.
3. Download GPT-J 6B's tokenizer files (they will be automatically detected when you attempt to load GPT-4chan):

python download-model.py EleutherAI/gpt-j-6B --text-only

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

http://localhost:7860/?__theme=dark

Optionally, you can use the following command-line flags:

Out of memory errors? Check the low VRAM guide.

Inference settings presets can be created under presets/ as text files. These files are detected automatically at startup.

By default, 10 presets by NovelAI and KoboldAI are included. These were selected out of a sample of 43 presets after applying a K-Means clustering algorithm and selecting the elements closest to the average of each cluster.

Visualization

Make sure to check out the documentation for an in-depth guide on how to use the web UI.

https://github.com/oobabooga/text-generation-webui/tree/main/docs

Pull requests, suggestions, and issue reports are welcome.

You are also welcome to review open pull requests.

Before reporting a bug, make sure that you have:

1. Created a conda environment and installed the dependencies exactly as in the Installation section above.
2. Searched to see if an issue already exists for the issue you encountered.

• Gradio dropdown menu refresh button, code for reloading the interface: https://github.com/AUTOMATIC1111/stable-diffusion-webui
• Verbose preset: Anonymous 4chan user.
• NovelAI and KoboldAI presets: https://github.com/KoboldAI/KoboldAI-Client/wiki/Settings-Presets
• Code for early stopping in chat mode, code for some of the sliders: https://github.com/PygmalionAI/gradio-ui/