Building AI literacy starts with hands-on experimentation. Free APIs let you test real workflows before you spend.
Quick answer
If you just want the easiest path:
- Ollama is the best default starting point for most people.
- LM Studio is the best GUI-first option.
- MLX / mlx-lm is the most Apple-native route on Apple Silicon Macs.
- llama.cpp is the best tool to learn if you want more control.
- AirLLM is more advanced and is mainly interesting when you want to push oversized models onto limited hardware.
Why local models matter
Local models matter because they teach you more than prompt engineering alone. Once you run a model on your own machine, you start to understand the real trade-offs that shape AI products: model size, quantization, context length, startup time, token speed, offline use, privacy, and hardware limits.
They also open up use cases that cloud APIs do not always handle well. You may want an offline assistant, local document analysis, private note search, an always-on coding helper, or a zero-API-cost demo app. Local models are not the right answer for everything, but they are an important part of AI literacy.
Google’s Gemma documentation is a good example of how mainstream this has become: Google explicitly documents Gemma running with Ollama and notes that frameworks like llama.cpp and Ollama can run quantized Gemma models on a laptop or other small device, even without a GPU.
Where to download local models
The main place to get local models is Hugging Face. The Hugging Face Hub describes itself as a platform with over 2 million models, along with datasets and demos. In practice, it is the default place to browse model cards, licenses, formats, and community conversions.
You can also download model files directly from the command line with the Hugging Face client:
hf download HuggingFaceH4/zephyr-7b-betaFor Gemma, Google’s official docs point users to both Kaggle and Hugging Face. For Qwen, the official Hugging Face organization states that Qwen is the large language model family built by Alibaba Cloud.
Tool-specific catalogs also exist. LM Studio has a built-in downloader for supported models from Hugging Face, and Ollama has its own model workflow through the CLI and model library.
Best ways to run local models
1. Ollama
Best for: beginners, local APIs, fast setup, and simple workflows.
Ollama is the easiest place to start for most people. Its official docs say it runs on Windows, macOS, and Linux, and on Windows it runs as a native application with NVIDIA and AMD Radeon GPU support. It also serves a local API on http://localhost:11434.
The basic CLI is as simple as:
ollama run gemma32. LM Studio
Best for: people who want a GUI, easier model browsing, and a local OpenAI-style endpoint.
LM Studio is the most beginner-friendly graphical route. Its docs say it supports Mac, Windows, and Linux, lets you search and download models from Hugging Face, can run llama.cpp (GGUF) models and, on Apple Silicon Macs, also supports MLX. It can also serve local models through OpenAI-compatible endpoints.
If you want local AI without living in the terminal, LM Studio is often the easiest route.
3. llama.cpp
Best for: advanced users, API serving, GGUF files, and deeper local-AI understanding.
If you want to learn the local-model ecosystem more seriously, learn llama.cpp. Its README documents a lightweight, OpenAI-compatible HTTP server and also supports downloading models directly from Hugging Face.
Typical examples look like:
llama-server -m model.gguf --port 8080
llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080That makes llama.cpp ideal when you want direct control over model files, GGUF quantizations, and local API serving.
4. MLX / mlx-lm
Best for: Apple Silicon Macs.
Apple’s MLX is built for Apple Silicon, and the mlx-lm project describes itself as a package for generating text and fine-tuning large language models on Apple Silicon with MLX. The README also highlights Hugging Face Hub integration and quantization support.
The mlx-lm README documents commands like:
mlx_lm.chat
mlx_lm.generate --prompt "How tall is Mt Everest?"5. AirLLM
Best for: advanced experiments with oversized models on limited hardware.
AirLLM is more specialized. Its README says the project optimizes inference memory usage and advertises running 70B models on a single 4GB GPU, which makes it interesting if your main problem is fitting very large models onto hardware that would normally be too small.
The official quickstart begins with:
pip install airllmWindows, macOS, and Linux recommendations
Windows
On Windows, the easiest route is usually Ollama or LM Studio. Ollama’s Windows docs say it runs as a native Windows application and supports both NVIDIA and AMD Radeon GPUs. LM Studio supports both x64 and ARM Windows systems and recommends at least 16GB RAM and 4GB dedicated VRAM.
macOS
On macOS, the best choice depends on whether you have Apple Silicon. Ollama supports Apple M-series Macs, while LM Studio supports Apple Silicon and MLX, and Apple’s MLX / mlx-lm stack is specifically designed for Apple Silicon hardware.
Linux
On Linux, Ollama and llama.cpp are the most practical starting points. Ollama’s official Linux install script is: curl -fsSL https://ollama.com/install.sh | sh.
How to choose a model for your RAM and GPU
A practical rule of thumb is: weight memory ≈ parameter count × bits per weight / 8
| Your hardware | Practical starting point |
|---|---|
| 8 GB RAM / unified memory | 1B–3B models, ideally 4-bit |
| 16 GB RAM / unified memory | 3B–8B models are usually the sweet spot |
| 24 GB RAM / unified memory | 7B–14B becomes much more comfortable |
| 8 GB VRAM GPU | Start around 7B–8B at 4-bit |
| 12 GB VRAM GPU | 8B–14B at 4-bit is usually realistic |
| 24 GB VRAM GPU | 14B–32B at 4-bit becomes realistic |
| 32 GB+ VRAM or heavy offloading | 32B–70B experimentation becomes possible |
Common mistakes beginners make
1. Starting with the biggest model instead of the right model
The biggest model your machine can barely load is rarely the best starting point. Smaller models that fit well often feel faster, more stable, and more useful.
2. Ignoring context size
Context length is not free. Long context consumes memory, which is why VRAM and context are linked in tools like Ollama.
3. Confusing “downloaded” with “compatible”
Just because you downloaded a model does not mean it fits your runtime. Check whether you need GGUF, MLX-compatible weights, or a Transformers checkpoint.
Frequently asked questions
What is a local AI model?
A local AI model is a model whose weights are downloaded to your own computer and run on your own CPU, GPU, or unified memory instead of being called through a hosted API.
Which tool is easiest for beginners?
For most beginners, Ollama is the easiest place to start. If you want a graphical interface, LM Studio is often the easiest GUI-first option.
Final takeaway
The best local-AI setup is not the most complex one. It is the one that matches your machine and your goal. If you are learning, start simple. Use Ollama or LM Studio. Move to MLX if you are on Apple Silicon and want a more native route. Learn llama.cpp if you want deeper control.
Explore the latest AI jobs on AI Career, or continue learning in our Learn AI section.
Frequently Asked Questions
No. Once you have downloaded the model weights and the necessary software (like Ollama or LM Studio), you can run the AI entirely offline. This is one of the primary benefits of local AI for privacy and accessibility.
Yes. Many tools like Ollama and llama.cpp are optimized to run on CPUs using system RAM. While it may be slower than a dedicated GPU, smaller models (1B-8B parameters) are very usable on modern laptop CPUs, especially on Apple Silicon Macs.
It depends on the model size. While massive models like GPT-4 or Claude 3.5 Sonnet are still superior in general reasoning, large local models (70B+) are highly competitive. Smaller local models (7B-14B) are often excellent for specific tasks like coding, summarization, or private document analysis.