Davide Quack
488 lines
16 KiB
Markdown
488 lines
16 KiB
Markdown
_Leggi il leggimi in [Italiano](README.IT.md)._
|
||
|
||
|
||
# Colibrì Guide - Local Inference Engine
|
||
|
||
A simple guide to running **Colibrì**, a local inference engine based on GLM 5.2, without needing programming knowledge. If you already have Docker installed, you are well on your way.
|
||
|
||
---
|
||
|
||
## 📋 Table of Contents
|
||
|
||
* [What is Colibrì?](https://www.google.com/search?q=#what-is-colibr%C3%AC)
|
||
* [Requirements](https://www.google.com/search?q=#requirements)
|
||
* [Hardware](https://www.google.com/search?q=#hardware)
|
||
* [Software](https://www.google.com/search?q=#software)
|
||
* [How to get started](https://www.google.com/search?q=#how-to-get-started)
|
||
* [Step 1: Download the model](https://www.google.com/search?q=#step-1-download-the-model)
|
||
* [Step 2: Download the Colibrì Dockerfile](https://www.google.com/search?q=#step-2-download-the-colibr%C3%AC-dockerfile)
|
||
* [Step 3: Build the Docker image](https://www.google.com/search?q=#step-3-build-the-docker-image)
|
||
* [Step 4: Start Colibrì](https://www.google.com/search?q=#step-4-start-colibr%C3%AC)
|
||
* [What does that command mean?](https://www.google.com/search?q=#what-does-that-command-mean)
|
||
* [Using Colibrì](https://www.google.com/search?q=#using-colibr%C3%AC)
|
||
* [Entering a Linux console inside the container](https://www.google.com/search?q=#entering-a-linux-console-inside-the-container)
|
||
* [Troubleshooting](https://www.google.com/search?q=#troubleshooting)
|
||
* [Technical notes](https://www.google.com/search?q=#technical-notes)
|
||
* [Frequently Asked Questions](https://www.google.com/search?q=#frequently-asked-questions)
|
||
* [Support and contributions](https://www.google.com/search?q=#support-and-contributions)
|
||
* [Tests on my PC](https://www.google.com/search?q=#tests-on-my-pc)
|
||
|
||
---
|
||
|
||
## What is Colibrì?
|
||
|
||
Colibrì is an application that allows you to run an artificial intelligence model (GLM 5.2) directly on your computer, without connecting to external servers. It is also possible to run it in Docker, which isolates the application from the rest of the system.
|
||
|
||
> **Important note**: The model is very large. Expect to wait several minutes for an answer to a simple question, especially with low RAM. At the end of this readme, you will see the result on my PC (without a discrete graphics card), and I reach 0.01 tokens per second.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
### Hardware
|
||
|
||
| RAM Memory | Works? | Notes |
|
||
| --- | --- | --- |
|
||
| < 16 GB | ❌ No | Insufficient memory |
|
||
| 24 GB | ⚠️ Maybe | Possible, needs testing |
|
||
| 32 GB | ✅ Yes | The minimum (but see memory section for Windows) |
|
||
| 48+ GB | ✅ Yes | Better |
|
||
|
||
Additionally: a **fast SSD** is essential. Colibrì uses the disk as additional memory. Having an NVidia graphics card is even better.
|
||
|
||
### Software
|
||
|
||
* **Docker Desktop** (Windows, Mac, Linux) — [download here](https://www.google.com/search?q=https://www.docker.com/products/docker-desktop/)
|
||
* **Python** (only if you want to download the model yourself)
|
||
* Windows: [python.org](https://www.google.com/search?q=https://www.python.org) or Microsoft Store
|
||
* Linux: `apt-get install python3 python3-pip`
|
||
* Mac: [python.org](https://www.google.com/search?q=https://www.python.org) or Homebrew
|
||
|
||
No build environment is needed. Everything happens inside the Docker container.
|
||
|
||
---
|
||
|
||
## How to get started
|
||
|
||
### Step 1: Download the model
|
||
|
||
The GLM 5.2 model is approximately **360 GB**. Choose one of these methods:
|
||
|
||
#### Method A: Using Python (recommended)
|
||
|
||
1. **Install the library for Hugging Face:**
|
||
```bash
|
||
python -m pip install -U huggingface_hub[cli]
|
||
```
|
||
|
||
On Linux, use `python3` instead of `python`.
|
||
2. **Download the model** (open the terminal in the folder where you want to save it):
|
||
```bash
|
||
hf_download mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp --local-dir .
|
||
```
|
||
|
||
**Example**: if you want to save it in `C:\LLM\models\glm-5.2` (Windows):
|
||
* Open PowerShell in that folder
|
||
* Copy and paste the command above
|
||
* Wait (a long time)
|
||
|
||
#### Method B: Without Python (only if necessary)
|
||
|
||
If you are on Windows and cannot get it to work with Python:
|
||
|
||
* Download manually from [Hugging Face](https://www.google.com/search?q=https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp)
|
||
* Unzip into a folder (e.g., `C:\LLM\models\glm-5.2`)
|
||
|
||
---
|
||
|
||
### Step 2: Download the Colibrì Dockerfile
|
||
|
||
1. Go to: [https://github.com/JustVugg/colibri/blob/main/docker/Dockerfile](https://www.google.com/search?q=https://github.com/JustVugg/colibri/blob/main/docker/Dockerfile)
|
||
2. Click the **Download** button (⬇️ icon) in the top right
|
||
3. Save the file in a folder (e.g., `C:\LLM\Colibrì`)
|
||
|
||
---
|
||
|
||
### Step 3: Build the Docker image
|
||
|
||
Open the terminal (PowerShell on Windows, Terminal on Mac/Linux) **in the folder where you saved the Dockerfile** and type:
|
||
|
||
**Windows:**
|
||
|
||
```bash
|
||
docker build -t colibri-i .
|
||
```
|
||
|
||
**Linux/Mac:**
|
||
|
||
```bash
|
||
sudo docker build -t colibri-i .
|
||
```
|
||
|
||
Wait for it to finish (a few minutes). If everything goes well, you will see: `Successfully tagged colibri-i:latest`
|
||
|
||
> **If you want to receive repository updates**: First delete the old image with `docker rmi colibri-i` and rebuild.
|
||
|
||
---
|
||
|
||
### Step 4: Start Colibrì
|
||
|
||
Open the terminal and type the command below (replace `C:\LLM\models\glm-5.2` with the actual path on your PC):
|
||
|
||
**Windows** (PowerShell):
|
||
|
||
```bash
|
||
$MODEL_PATH="C:\LLM\models\glm-5.2"
|
||
docker run --rm -it --name colibri-c `
|
||
-v "$MODEL_PATH`:/app/glm-5.2" `
|
||
-e COLI_MODEL=/app/glm-5.2 `
|
||
colibri-i ./coli chat
|
||
```
|
||
|
||
**Mac/Linux** (Terminal/Bash):
|
||
|
||
```bash
|
||
MODEL_PATH="/path/to/glm-5.2"
|
||
docker run --rm -it --name colibri-c \
|
||
-v "$MODEL_PATH:/app/glm-5.2" \
|
||
-e COLI_MODEL=/app/glm-5.2 \
|
||
colibri-i ./coli chat
|
||
```
|
||
|
||
**Example for Linux:**
|
||
|
||
```bash
|
||
MODEL_PATH="/home/user/LLM/glm-5.2"
|
||
docker run --rm -it --name colibri-c \
|
||
-v "$MODEL_PATH:/app/glm-5.2" \
|
||
-e COLI_MODEL=/app/glm-5.2 \
|
||
colibri-i ./coli chat
|
||
```
|
||
|
||
---
|
||
|
||
### What does that command mean?
|
||
|
||
| Part | Explanation |
|
||
| --- | --- |
|
||
| `docker run` | Starts a container |
|
||
| `--rm` | Deletes the container when you close it |
|
||
| `-it` | Interactive mode (you can write and read) |
|
||
| `-v "PATH:/app/glm-5.2"` | Mounts your model inside the container |
|
||
| `-e COLI_MODEL=/app/glm-5.2` | Tells Colibrì where to find the model |
|
||
| `colibri-i` | Name of the Docker image |
|
||
| `./coli chat` | Starts Colibrì in chat mode |
|
||
|
||
---
|
||
|
||
### Using Colibrì
|
||
|
||
Once started, you will see a prompt like this:
|
||
|
||
```
|
||
──────────────────────────────────────────────────────────
|
||
type and press Enter · Ctrl-C stops the answer · :more continues · :reset clears memory · :q exits
|
||
|
||
```
|
||
|
||
**Useful commands:**
|
||
|
||
* `Write a question + Enter` → Receive the answer
|
||
* `Ctrl + C` → Stop the answer
|
||
* `:reset` → Clear conversation memory
|
||
* `:q` → Exit
|
||
|
||
**Usage example:**
|
||
|
||
```
|
||
› How many inhabitants does China have?
|
||
|
||
China is currently the most populous country in the world.
|
||
The population is approximately 1.41 billion people.
|
||
|
||
```
|
||
|
||
The model understands **Italian, English, Chinese, and other languages**, although it is optimized for English and Chinese.
|
||
|
||
---
|
||
|
||
## Entering a Linux console inside the container
|
||
|
||
If you want to explore the container as if it were a normal Linux machine:
|
||
|
||
```bash
|
||
docker run --rm -it --name colibri-c \
|
||
-v "MODEL_PATH:/app/glm-5.2" \
|
||
-e COLI_MODEL=/app/glm-5.2 \
|
||
colibri-i /bin/bash
|
||
```
|
||
|
||
Now you are inside Linux. Type `exit` to leave.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### ❌ "Docker not found"
|
||
|
||
**Cause**: Docker is not installed or the terminal does not recognize it.
|
||
|
||
**Solution**:
|
||
|
||
1. Reinstall [Docker Desktop](https://www.google.com/search?q=https://www.docker.com/products/docker-desktop/)
|
||
2. Restart your computer
|
||
3. Open a new terminal and try again
|
||
|
||
---
|
||
|
||
### ❌ "Out of memory" or container closes immediately
|
||
|
||
**Cause**: Your computer does not have enough RAM, or on Windows, WSL is using less memory than available.
|
||
|
||
**Solution for Windows (WSL):**
|
||
|
||
1. Open PowerShell and check the memory available to WSL:
|
||
```bash
|
||
wsl
|
||
cat /proc/meminfo | grep MemTotal
|
||
exit
|
||
```
|
||
|
||
|
||
Divide the number by 1,073,741,824 (which is 1024³) to get it in GB.
|
||
2. If WSL uses less than what you have, create a configuration file:
|
||
* Open a text editor (Notepad is fine)
|
||
* Copy this:
|
||
```ini
|
||
[wsl2]
|
||
memory=24GB
|
||
processors=12
|
||
swap=16GB
|
||
|
||
```
|
||
|
||
* Save the file with the name: `.wslconfig` (with the dot)
|
||
* Place it in: `C:\Users\YourUsername\`
|
||
|
||
3. Restart WSL from PowerShell:
|
||
```bash
|
||
wsl --shutdown
|
||
wsl
|
||
```
|
||
|
||
4. Check again:
|
||
```bash
|
||
# cat /proc/meminfo | grep MemTotal
|
||
# exit
|
||
```
|
||
|
||
**Solution for Mac/Linux**: Increase the RAM available to Docker from the Docker Desktop settings, or add more RAM to the computer.
|
||
|
||
---
|
||
|
||
### ❌ The answer is very slow
|
||
|
||
**Possible causes**:
|
||
|
||
1. The disk is slow
|
||
2. You have low RAM
|
||
3. Colibrì is using the disk as additional memory (normal)
|
||
|
||
**How to check disk speed:**
|
||
|
||
**Windows** (PowerShell as administrator):
|
||
|
||
```bash
|
||
winsat disk -drive C
|
||
```
|
||
|
||
Change `C` with your disk letter.
|
||
|
||
**Linux/Mac** (Terminal):
|
||
|
||
```bash
|
||
sudo hdparm -Tt /dev/sda
|
||
```
|
||
|
||
Change `/dev/sda` with your disk (check with `lsblk` on Linux).
|
||
|
||
A **modern NVMe SSD** reaches 15 GB/sec. If yours is under 2-3 GB/sec, it is slow.
|
||
|
||
---
|
||
|
||
### ❌ "Permission denied" on Linux
|
||
|
||
**Cause**: Docker requires administrator permissions.
|
||
|
||
**Solution - Option 1** (quick):
|
||
|
||
```bash
|
||
sudo docker build -t colibri-i .
|
||
sudo docker run ... (as above, with sudo in front)
|
||
```
|
||
|
||
**Solution - Option 2** (permanent):
|
||
|
||
```bash
|
||
sudo usermod -aG docker $USER
|
||
# Restart the computer
|
||
docker run ... (without sudo)
|
||
```
|
||
---
|
||
|
||
### ❌ "Image not found" or error during build
|
||
|
||
**Cause**: The Dockerfile is corrupted or not in the right folder.
|
||
|
||
**Solution**:
|
||
|
||
1. Verify that the Dockerfile is in the folder where you open the terminal:
|
||
```bash
|
||
ls Dockerfile # Mac/Linux
|
||
dir Dockerfile # Windows
|
||
```
|
||
|
||
2. Redownload the Dockerfile from the GitHub repository
|
||
3. Delete the old image: `docker rmi colibri-i`
|
||
4. Retry the build
|
||
|
||
---
|
||
|
||
### ❌ "hf_download: command not found"
|
||
|
||
**Cause**: The Hugging Face library is not installed correctly.
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
pip install -U huggingface_hub[cli]
|
||
# or on Linux/Mac:
|
||
pip3 install -U huggingface_hub[cli]
|
||
```
|
||
|
||
Then retry the `hf_download` command.
|
||
|
||
---
|
||
|
||
### ❌ The model does not download (timeout or network errors)
|
||
|
||
**Causes**: Slow or unstable connection, Hugging Face temporarily unavailable.
|
||
|
||
**Solution**:
|
||
|
||
1. Wait and retry the `hf_download` command
|
||
2. If it continues, download manually from [here](https://www.google.com/search?q=https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp)
|
||
3. Unzip the ZIP file into the desired folder
|
||
|
||
---
|
||
|
||
## Technical notes
|
||
|
||
### Why is the disk important?
|
||
|
||
Colibrì uses the disk as "additional virtual RAM" (paging). A **fast** disk is crucial for decent performance.
|
||
|
||
* **NVMe SSD** (recommended): 1-15 GB/sec
|
||
* **SATA SSD**: 0.5-1 GB/sec
|
||
* **Rotational hard disk**: 0.05-0.1 GB/sec ❌ (too slow)
|
||
|
||
If your disk is slow, the answers will be very slow even with a lot of RAM.
|
||
|
||
---
|
||
|
||
### Recommended default configuration for WLS on Windows
|
||
|
||
If you have **exactly 32 GB of RAM** and are using Windows, it is very likely that WLS by default is set to consume no more than 16 GB of RAM. We need to increase this limit [Troubleshooting](https://www.google.com/search?q=#troubleshooting) . In my case I adopted this configuration:
|
||
|
||
```ini
|
||
[wsl2]
|
||
memory=24GB
|
||
processors=12
|
||
swap=16GB
|
||
|
||
```
|
||
|
||
That is, in my case, I left 8 GB of RAM and 4 CPUs to Windows and gave 24 GB and 12 processors to WSL + Linux.
|
||
|
||
---
|
||
|
||
## Frequently Asked Questions
|
||
|
||
**Q: What if I have less than 32 GB of RAM?**
|
||
|
||
A: It is likely that it will not work well. You can try if you have 24 GB, but it is not guaranteed.
|
||
|
||
**Q: Can I increase the response speed?**
|
||
|
||
A: Yes, partly:
|
||
|
||
* Use a fast NVMe SSD
|
||
* Increase RAM
|
||
* Reduce the complexity of questions
|
||
* Use `:reset` to clear memory and lighten the load
|
||
|
||
**Q: Can I use Colibrì without Docker?**
|
||
|
||
A: Colibrì was born that way, but this guide assumes Docker. To build from source, see the GitHub repository.
|
||
|
||
**Q: How much internet connection do I need after downloading the model?**
|
||
|
||
A: Zero. Colibrì works completely offline.
|
||
|
||
---
|
||
|
||
## Support and contributions
|
||
|
||
If you find errors or have suggestions for improving this guide, open an issue or a pull request on the Colibrì GitHub repository.
|
||
|
||
Have fun! 🐦
|
||
|
||
---
|
||
|
||
## Tests on my PC
|
||
|
||
In the first case, I asked a question in Italian; in the second, in Japanese; and in the third, I repeated the question in Japanese but requested an answer in Italian.
|
||
|
||
```
|
||
PS C:\quack\llm\colibri\docker> docker run --rm -it --name colibri-c -v "C:\quack\llm\models\glm-5.2:/app/glm-5.2" -e COLI_MODEL=/app/glm-5.2 colibri-i ./coli chat
|
||
|
||
▄▀▀▀▄ ▄ colibrì v1.0
|
||
▄▄▄▄▀▀▀▀▄▀▀ tiny engine, immense model
|
||
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
|
||
▀▀▀▀ chat · glm-5.2 · ram -GB · topp off
|
||
▀
|
||
──────────────────────────────────────────────────────────
|
||
type and press Enter · Ctrl-C stops the answer · :more continues · :reset clears memory · :q exits
|
||
|
||
╭────────────────────────────────────────────────────────────────────────────────────────────────╮
|
||
│ › Quanti abitanti ha la Cina? │
|
||
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
|
||
|
||
◆ colibrì
|
||
La Cina è attualmente il paese più popoloso al mondo (sebbene, secondo alcune stime recenti, sia stata ormai superata dall'India).
|
||
|
||
La popolazione totale della Repubblica Popolare Cinese è di circa 1,41 miliardi di abitanti (dati del 2020-2022 circa).
|
||
└─ 76 tok · 0.04 tok/s · hit 3% · RSS 15.9 GB · 2012s
|
||
|
||
╭────────────────────────────────────────────────────────────────────────────────────────────────╮
|
||
│ › 漫画「ワンピース」の主人公の名前を教えてください。名前だけで、それ以上のコメントはありません。 │
|
||
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
|
||
|
||
◆ colibrì
|
||
ルフィ
|
||
└─ 2 tok · 0.01 tok/s · hit 1% · RSS 16.7 GB · 260s
|
||
|
||
╭────────────────────────────────────────────────────────────────────────────────────────────────╮
|
||
│ › 漫画「ワンピース」の主人公の名前を教えてください。名前だけで、それ以上のコメントはありません。イタリア語で返信 │
|
||
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
|
||
|
||
◆ colibrì
|
||
Il nome del protagonista di One Piece è Monkey D. Luffy.
|
||
└─ 14 tok · 0.02 tok/s · hit 2% · RSS 17.3 GB · 593s
|
||
|
||
╭────────────────────────────────────────────────────────────────────────────────────────────────╮
|
||
│ ›
|
||
|
||
```
|
||
|
||
[source: 1] |