Topic: AI models (LLMs) on your desktop computer

This is the third time we talked about an AI-topic. If you want to read the notes of a previous HackyHour about AI, look here and here.

Speaker: Nils Seipel
Notes: Christian Krippes

Nils provided his slides. Make sure to look into it, because not all graphics and code examples are included in the notes. Slides are in German.

What are LLMs - Large Language Models

LLMs are a kind of GPT GPT=Generative Pretrained Transformer
creating text that (often) looks and feels very naturally
reproducing learned propabilities for the next word in a text.
no knowledge inside
biased in its answers

Hardware requirements for ChatGPT

Reportedly 800 GB of VRAM (RAM of your graphic card).
Many high-end graphic cards, with a price tag of 15k€+ each.
Problem: not feasible for hobbists, scientists or, home-users.

Development

Game-changer: LLaMA (Large Language Model Meta AI)

“Foundational model” available with different amounts of parameters 7B,13B,33B,65B (B=Billion)
Developed by Meta and released as open source (non-commercial). Meta held back the model weights, which could be called the “knowledge” of the model. These weights were then leaked.
Input can be given via prompts.
Further improvements came through llama.cpp.
LLaMA.cpp works on a modern laptop and even on smartphones and the Raspberry Pi but very slowly.
Thousands of models build up on LLaMA; there is literally a new model every day.

This Arstechnica article describes the developement and background story in more detail.

Quantisation of a model seems to be key to decreasing hardware requirements.
With 8-bit quantisation, LLaMA with 7B parameters required 10GB VRAM and with 65B, it required 80GB.
4-bit quantisation cuts this nearly in half, so you get 7B model->6GB VRAM and 65B model -> 40GB VRAM.
For comparison, a reasonable consumer graphics card for around ~300-400 € has up to 12GB VRAM. For around 1700 you get 24GB.
Those 4-bit quantised models are available as a single file, ready for download.

Diversity explosion

Successor to LLaMA is Alpaca followed by Vicuna and severel others
There is also WizardLM, OpenAssistant, and many more.
Different model architectures, e.g StableLM, RedPajama
Visual models: LLaVA, LaVIN, MiniGPT-4

Use cases

Avoid API costs.
Lots of AI configuration aspects are in your hands.
Build your own small ChatAI.
Research its behaviour.
Experiment with your own data that you don’t want to share with a cloud provider
“Retrain” it with your domain-specific data.
Use it for edgy things (not recommended).

How to use it in practice

For code example, please take a look at the slides in the PDF document. Using a LLM locally is as easy:

pip install llama-cpp-python
Download a model from https://huggingface.co/TheBloke.
Run it in Python (see the slides for the code)

It is important to look up what “talking” behaviour the model was trained on. The model expects input in the same way; otherwise, you’ll likely get weird results.

E.g a model was trained with the following input setup.

PROMPT: Blablabla
ASSISTANT:

Security, Info, and Tools

Security:

Prompt injection is a serious issue, so be careful if you use AI in a real world scenario. See https://simonwillison.net/2023/May/2/prompt-injection-explaine

Info:

Subreddit with guides and help: https://www.reddit.com/r/LocalLLaMA/
One-click-installer: https://github.com/oobabooga/text-generation-webui
Oobabooga supports different models and is an easy-to-use web-frontend.

Tools:

4-bit models for graphic cards -> GPTQ-for-LLaMA:

https://github.com/qwopqwop200/GPTQ-for-LLaMa

GGML-Models on CPU or with parts in GPU -> llama.cpp:

https://github.com/ggerganov/llama.cpp

easier way: python-package https://github.com/abetlen/llama-cpp-python
OpenAI-like API
Microsoft Guidance
LangChain
Own data sources can be questionable via ChatBot. This is possible through so called “embeddings”. See https://github.com/Appointat/Chat-with-Document-s-using-ChatGPT-API-and-Text-Embedding
https://open-assistant.io
https://chat.lmsys.org/ -> ChatBot test page for different models

Other:

Follow up on a discussion we had. Tranlator AI “invents” universal language.

https://www.wired.co.uk/article/google-ai-language-create