1. Harbor User Guide

This is a high-level guide of the key functionality for getting started with Harbor and managing the main services.

You'll find many more detailed examples in the Services and Harbor CLI Reference.

Getting Started

Start Harbor with default services:

harbor up

This will start Ollama and Open WebUI by default. These two are default services

Tip

You can configure default services using harbor defaults

Access the Open WebUI interface:

harbor open

harbor open uses harbor url under the hood. You can specify a service name to open a specific service URL:

# See the URL
harbor url <service handle>
harbor open <service handle>

To add additional services, use:

harbor up <service handle>

Tip

Every service in Harbor will have its own dedicated "handle" that can be used with most compatible commands. You'll find all the handles in the Services section. For example, Open WebUI has the handle webui.

For example:

harbor up searxng

Default Services

Harbor allows you to specify the set of services that will be started by default. You can manage the default services using the harbor defaults command. It includes frontends, backends and satellite services that may be included in your Harbor setup.

# Show the list of default services
harbor defaults

# Add a new default service
harbor defaults add searxng

# Swap a default backend
harbor defaults rm ollama
harbor defaults add vllm

Running LLMs

Ollama

Ollama is the default LLM backend due to its convenience for the end-user, it's main benefit is that it determines system resources and features very well automatically.

List available models:

harbor ollama list

Pull a new model:

harbor ollama pull <model_name>

Tip

You can run any ollama commands via harbor ollama ...

llama.cpp

Llama.cpp own cache:

harbor llamacpp model <full hugging face gguf URL>

# Example
harbor llamacpp model https://huggingface.co/cognitivecomputations/dolphin-2.9.4-llama3.1-8b-gguf/blob/main/dolphin-2.9.4-llama3.1-8b-Q4_K_S.gguf

The model will be downloaded and cached on the next harbor up with llamacpp service.

Tip

You can quickly jump to HuggingFace model lookup with harbor hf find command:

harbor hf find dolphin gguf

Use shared HuggingFace hub cache:

# Download
harbor hf download <user/repo> <file.gguf>

# Locate the file
harbor find file.gguf

# Set the path to the model
harbor config set llamacpp.model.specifier -m /app/models/<path to file.gguf>

vLLM

Set model:

harbor vllm model <user/repo>

# Example
harbor vllm model google/gemma-2-2b-it

Tip

You can set HuggingFace Hub token to access gated/private models:

harbor hf token <your HF token>

This token will be pre-configured for services that might need it for such purpose.

Other backends

Please refer to the Backends section in the Services documentation for even more backends and their configuration.

Configuring Services

There are three layers of configuration in Harbor:

Service CLIs

For the most common/frequent configs, Harbor provides CLI aliases, one such example is configuration of service models:

# Setting service model via CLI alias
harbor llamacpp model https://huggingface.co/lm-kit/gemma-2-2b-gguf/blob/main/gemma-2-2B-Q8_0.gguf
harbor tgi model google/gemma-2-2b-it
harbor vllm model google/gemma-2-2b-it
harbor aphrodite model google/gemma-2-2b-it
harbor tabbyapi model google/gemma-2-2b-it-exl2
harbor mistralrs model google/gemma-2-2b-it
harbor opint model google/gemma-2-2b-it
harbor cmdh model google/gemma-2-2b-it
harbor fabric model google/gemma-2-2b-it
harbor parler model parler-tts/parler-tts-large-v1
harbor airllm model meta-llama/Meta-Llama-3.1-8B-Instruct
harbor txtai rag model llama3.1:8b-instruct-q6_K
harbor aider model llama3.1:8b-instruct-q6_K
harbor chatui model llama3.1:8b-instruct-q6_K
harbor aichat model llama3.1:8b-instruct-q6_K

Note

As you can see above, services accept different values for the model specifier, depending on their internal configuration. Refer to Services docs for detailed instructions on how to configure each service.

Versions:

# Setting service version via CLI alias
harbor vllm version 0.5.3
harbor webui version main
harbor mistralrs version 0.3
harbor chatui version latest
harbor comfyui version latest-cuda

Tip

Most Service CLIs will also have dedicated help entries showing all available options:

harbor <service> --help

Harbor Config

Service CLIs are aliases for the underlying Harbor Config. It can be accessed directly via harbor config command:

# Usage help
harbor config --help

harbor config works with the .env file as a key/value store, allowing you to set and get configuration values for services.

# See all configuration options
harbor config ls
# WEBUI_NAME    Harbor
# VLLM_VERSION  v0.5.3
# ... Many more options

# Get a specific value, accepts various aliases
harbor config get VLLM_VERSION
harbor config get vllm_version
harbor config get vllm.version

# Set a value, accepts same aliases as above
harbor config set webui.name v0.5.3

See a more detailed overview in the harbor config CLI reference

Using configuration files

Most of the services can be configured via either .env or a specific yaml, json, toml configuration (refer to service docs for specifics).

Harbor will either track where such files are stored on the host or will contain them in its workspace.

# Show the path to the Harbor workspace
harbor home

# Open workspace in the file manager
open $(harbor home)

# Shortcut for VS Code users
harbor vscode

You can find all the external tracked configuration folders by listing related configurations:

harbor config ls | grep CONFIG_PATH

Certain services will also have a dedicated cache folder, which can be found in the same way:

harbor config ls | grep CACHE

When you have a specific config key, you can open it in the file manager using:

open $(eval echo "$(harbor config get hf.cache)")

Harbor Profiles

When you have multiple configurations for different use-cases, you can save them as profiles for easy switching. Profiles include everything that can be configured via harbor config (most of the settings configured via CLI) and are stored in the Harbor workspace. They are just a way to swap between .env files using command line.

# Use a profile
$ ▼ h profile use default
21:49:20 [INFO] Profile 'default' loaded.

# Check settings specific to the vLLM service
$ ▼ h config ls | grep VLLM
VLLM_CACHE                     ~/.cache/vllm
VLLM_HOST_PORT                 33911
VLLM_VERSION                   v0.6.0
VLLM_MODEL                     microsoft/Phi-3.5-mini-instruct
VLLM_EXTRA_ARGS
VLLM_ATTENTION_BACKEND         FLASH_ATTN
VLLM_MODEL_SPECIFIER           --model microsoft/Phi-3.5-mini-instruct

# Switch to another profile
$ ▼ h profile use phimoe
21:49:42 [INFO] Profile 'phimoe' loaded.

# vLLM settings are different now
$ ▼ h config ls | grep VLLM
VLLM_CACHE                     ~/.cache/vllm
VLLM_HOST_PORT                 33911
VLLM_VERSION                   v0.6.1.post2
VLLM_MODEL                     microsoft/Phi-3.5-MoE-instruct
VLLM_EXTRA_ARGS                --max-model-len 1024 --trust-remote-code --cpu-offload-gb 56 --enforce-eager --gpu-memory-utilization 0 --device cpu
VLLM_ATTENTION_BACKEND         FLASH_ATTN
VLLM_MODEL_SPECIFIER           --model microsoft/Phi-3.5-MoE-instruct

There are a few considerations when using profiles:

• When the profile is loaded, modifications are not saved by default and will be lost when switching to another profile (or reloading the current one). Use harbor profile save <name> to persist the changes after making them
• Profiles are stored in the Harbor workspace and can be shared between different Harbor instances
• Profiles are not versioned and are not guaranteed to work between different Harbor versions
• You can also edit profiles as .env files in the workspace, it's not necessary to use the CLI

Using Satellite Services

SearXNG (Web Search)

Start SearXNG

harbor up searxng

Open WebUI will automatically use it for Web RAG functionality

Text-to-Speech

Start TTS service:

harbor up tts

Configure voices in tts/config/voice_to_speaker.yaml

Other Satellites

Please refer to the Satellites section in the Services documentation for even more additional services and their configuration in Harbor.

Accessing Service URLs

Get URL for any service:

harbor url <service handle>

Viewing Logs

harbor logs <service handle>

Stopping Services

Stop all services:

harbor down