#320 - How to install Docker in Ubuntu and create a RAG AI model

- Start a fresh Ubuntu VM with a good amount of RAM and CPU.

- In the settings, remove Power Saving in "Power" and remove all the screen locks in "Privacy and Security". 

- Launch these terminal commands:

sudo apt update
sudo apt upgrade
sudo apt install open-vm-tools open-vm-tools-desktop

- Build essential network tools:

sudo apt install -y curl git build-essential net-tools

- Install Docker:

sudo apt install -y docker.io docker-compose-v2
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
sudo reboot

docker run hello-world

- Install Ollama a lightweight, open-source platform used to download, run, and manage large language models locally:

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

- Get the LLM models from Ollama. 

ollama pull qwen2.5:14b
ollama pull llama3.1:8b
ollama pull nomic-embed-text
ollama list

Note: Pulling qwen (~9GB) through WiFi was failing, therefore a wired Ethernet connection was used instead. 

- Configure the AnythingLLM container:

export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
  --cap-add SYS_ADMIN \
  --name anythingllm \
  --add-host=host.docker.internal:host-gateway \
  -v ${STORAGE_LOCATION}:/app/server/storage \
  -v ${STORAGE_LOCATION}/.env:/app/server/.env \
  -e STORAGE_DIR="/app/server/storage" \
  mintplexlabs/anythingllm

- Launch AnythingLLM on the web browser (http://localhost:3001) and select Ollama LLM provider to run LLMs locally.

- Change the Ollama Base URL from localhost to http://host.docker.internal:11434 and select the desired model from the dropdown list. Note that it may be required to restart the browser for the dropdown to update.

- Performance issues when running models entirely on CPU. At these rates 0.5-2tok/s it is too slow to work with.

ollama ps
NAME           ID              SIZE     PROCESSOR    CONTEXT    UNTIL              
qwen2.5:14b    7cdf5a0187d5    12 GB    100% CPU     16384      4 minutes from now