Ollama Installation Ubuntu – Run LLMs Locally

Learn how to install Ollama on Ubuntu Linux. This guide provides a beginner-friendly, step-by-step tutorial for running large language models locally with AMD and ARM64 support.

Large Language Models (LLMs) like GPT-3.5 or Llama 2 7B are transforming the way we interact with technology – from generating creative content to automating complex tasks. However, accessing these powerful models traditionally required significant computational resources and cloud service subscriptions. Ollama changes that. It's an open-source tool that allows you to easily download and run LLMs directly on your Linux computer, dramatically reducing costs and improving privacy.

This guide is for anyone who wants to experiment with LLMs without the complexity of setting up a dedicated server. Whether you’re a beginner just starting with AI or an experienced developer looking for a local development environment, this comprehensive tutorial will walk you through the entire installation process on Ubuntu Linux. 

We'll cover AMD GPU support and ARM64 options, ensuring you can run Ollama effectively regardless of your hardware configuration. This is about more than just installing software; it’s about unlocking powerful AI capabilities locally.

Prerequisites

Before diving in, let's ensure you have everything needed to successfully install Ollama:

Hardware:

• A Linux system (Ubuntu 20.04 or later is recommended).
• This guide assumes a minimum of 8GB RAM, although running larger models will require significantly more.
• An AMD GPU *strongly* recommended for optimal performance (especially with larger models). While Ollama can run on CPU alone, inference speeds will be considerably slower.

Software:

• Ubuntu 20.04 or later installed and configured.
• `curl`: A command-line tool for transferring data with URLs. Usually pre-installed. 
• `git`: For downloading the Ollama repository. Install using: `sudo apt update && sudo apt install git` 
• Accounts Needed:** None – this is an open-source project.

Knowledge Requirements:

• Basic familiarity with the command line (terminal) and navigating Linux directories. Comfort with installing software from source is helpful, but not strictly required.

Estimated Setup Time:

30 - 60 minutes (depending on internet speed and GPU availability).

Other Requirements:

• Operating System

      - Ubuntu 20.04+ (Preferred for current support & tooling)

• Network Connection

      - Stable Internet (Required for downloading model files) 

• Command Line Access

      - Terminal (Necessary for executing commands)

Step-by-Step Tutorial:

Let’s get started! Follow these steps to install Ollama on your Ubuntu system.

Step 1: Add the Ollama Repository

Ollama doesn't have a standard package available in Ubuntu’s repositories, so we need to add its official repository. This ensures you always get the latest version.

On the command line run the following command:

"curl -fsSL https://ollama.com/install.sh | sh"

This command downloads the installation script from Ollama's website and executes it directly in your terminal. 

The script automatically configures your system to use the official Ollama repository, allowing you to install Ollama using `apt`.

Make sure you have a stable internet connection during this step. If the download fails, double-check the URL. 

Step 2: Install Ollama

Now that the repository is added, you can use apt to install Ollama.

run the following commands:

"sudo apt update && sudo apt install ollama"

`apt update` updates your package lists, and `apt install ollama` downloads and installs the Ollama software from the newly configured repository.

This provides a clean and reliable way to install Ollama on your system.

You'll be prompted for your sudo password. Ensure you enter the correct password. If you encounter errors during installation, check the terminal output for specific error messages. 

Step 3: Start the Ollama Server (Optional - for local experimentation)

The `ollama serve` command starts a local server that allows you to interact with LLMs through your terminal. This is useful for initial testing and development.

run the following commands:

"ollama serve"

This command initiates the Ollama server, which will download model files (if they aren’t already present) and start a local inference engine.  Allows you to interact with a running LLM directly in your terminal.

You may see warnings about not having GPU acceleration enabled – this is normal if you don't have an AMD GPU configured (more on that below).

Step 4: Download and Run a Model (Example - Llama 2 7B)

Let’s download and run the popular Llama 2 model.

run the following commands:

"ollama pull llama2"

This command downloads the `llama2` model from Ollama's servers and places it in your local storage. The first time you run this, it will take a significant amount of time as it downloads the model files. Subsequent runs will be much faster if the model is already cached locally. This allows you to actually use the LLM that was just installed.

The initial download can take a long time (potentially several hours) depending on your internet connection speed. To interact with the Llama 2 model, simply type: "bash ollama run llama2"

You’ll then be presented with a prompt where you can start typing and receive responses from the Llama 2 model.

Technical Details

How it Works (Behind the Scenes):

Ollama leverages Docker containers to isolate the LLM environment, ensuring consistency across different systems. It efficiently manages model downloads, caching, and inference execution. The `ollama serve` command runs a systemd service that automatically restarts the server if it crashes.

Key Concepts

Inference Engine: The core component responsible for running the LLM and generating responses.

Model Weights: The data representing the trained knowledge of the LLM.

Containerization (Docker): Ensures consistent execution environments, regardless of the host system.

Performance Considerations

GPU acceleration is crucial for performance with larger models. Utilize AMD GPUs for significantly faster inference speeds. CPU-based inference will be much slower. RAM requirements increase with model size; 32GB+ recommended for larger models like Llama 2.

Security Considerations

Running LLMs locally reduces the risk of exposing sensitive data to external servers. However, you are still responsible for securing your own system and ensuring that the downloaded models haven't been tampered with.

Troubleshooting Section 

Problem #1:

• Error Message: curl: (73) Failed to extract URL.
• Possible Cause(s): Network Issues, corrupted download.
• Solution(s): Check your internet connection; retry the download command.

Problem #2:

• Error Message: "Error: Could not load model"
• Possible Cause(s): Model file missing or corrupted.
• Solution(s): Try downloading the model again (`ollama pull `). Verify download integrity.

Problem #3:

• Error Message: None, just Slow Inference Speed.
• Possible Cause(s): CPU-based inference, insufficient RAM.
• Solution(s): Install an AMD GPU; ensure sufficient RAM for the model size; download a much smaller size model version.

Problem #4:

• Error Message:  "Ollama Server Error"
• Possible Cause(s): Systemd issues, resource constraints
• Solution(s): Restart the systemd service: `sudo systemctl restart ollama`

 

Best Practices

Recommended Settings:

1. Use an AMD GPU for optimal performance.
2. Allocate sufficient RAM (32GB+ recommended).
3. Regularly update Ollama to benefit from bug fixes and new features.

Optimization Tips:

1. Cache downloaded models locally to avoid repeated downloads.
2. Experiment with different model sizes based on your hardware capabilities.

Security Recommendations:

1. Keep your system updated with the latest security patches.
2. Monitor the logs for any suspicious activity.
3. Consider restricting access to the Ollama server if you are running it in a public environment.

Here are some practical applications you can build using Ollama on your local Ubuntu system.

1. Creative Writing Assistant: Use Llama 2 or another model to generate stories, poems, or scripts based on your prompts. 
2. Code Generation Tool: Utilize an LLM for generating code snippets or entire programs based on natural language descriptions. 
3. Personal Chatbot: Create a local chatbot powered by Ollama that you can use for casual conversation and information retrieval.

FAQs (Frequently Asked Questions)

1. What hardware do I need to run Ollama? – You’ll need a Linux system, preferably with an AMD GPU. Minimum 8GB RAM is recommended, but larger models require more. 
2. Is Ollama free to use? – Yes, Ollama is open-source and completely free to use! 
3. How do I update Ollama? - Use the same commands as installing it: `curl -fsSL https://ollama.com/install.sh | sh` followed by `sudo apt update && sudo apt install ollama`. 
4. Can I run Ollama on Windows? – Yes, please refer here on how to install Ollama in Windows. 
5. What’s the difference between `ollama serve` and running a model directly with `ollama run` ? - `ollama serve` starts a dedicated server process for your LLM, making it accessible from other applications. `ollama run ` runs the model directly in your terminal session and is ideal for quick experimentation.