Local LLMs suck. That’s a big statement, and it’s a bold claim, but it’s kind of true.

My Claude subscription just ran out. For an entire week, I managed to burn through my usage in just two days, and now I’m stuck coding manually because I’m not about to spend another $200 on a new account. I can wait a week and do a lot of the work myself.

While doing that, I had an interesting idea. Why not try a local LLM?

They seem promising. I’ve seen a lot of videos where people say that local LLMs are the future. There are models like Gemma 4 from Google, Qwen 3.5, Gemma 2.6 and 2.7, and GLM models that are widely available and can supposedly run on your own machine.

But the reality is that most developers have fairly ordinary computers: 16–32 GB of RAM, 1 TB of storage, a mid-range CPU, and maybe a dedicated GPU. Some have an Intel Core i7 from the 12th or 13th generation, while others use a base MacBook or another lower-end machine. Not everyone can afford a $2,000–$5,000 AI workstation.

So the question is: what can we actually run on these ordinary machines? Can we realistically use a local LLM?

Well, the answer is a little more complicated—and a little more interesting.

Before we begin with anything else, I want to be 100% clear that I love open source, and I think these open-weight models are incredibly important for competition and innovation. They push AI development forward much faster, and there’s no doubt that, without them, the industry wouldn’t be as far ahead as it is today.

With that said, let’s dive in.

TL;DR

Local LLMs have improved dramatically, and yes, you can run them on normal consumer hardware. A laptop like my base M4 MacBook Pro with 16 GB of unified memory can run smaller open-weight models such as Qwen 3 8B and similar models without much trouble.

However, running a local LLM and having a useful AI coding assistant are two completely different things. These smaller models are excellent for lightweight tasks like fixing grammar, generating Git commit messages, summarizing notes, organizing knowledge bases, and handling simple coding assistance.

The limitations become obvious when you move into serious software engineering workflows. Large codebases, multi-file debugging, complex refactoring, image understanding, and advanced tool usage require larger models with bigger context windows and significantly more hardware.

Local LLMs are not truly free. While the models themselves may be open and downloadable, the hardware, electricity, maintenance, and setup time all come with a cost.

For developers who enjoy experimenting and building their own AI workflows, local LLMs are exciting and valuable. But for most developers looking for a replacement for tools like Claude, GPT, or other frontier AI assistants, consumer hardware still has a long way to go.

Can You Actually Run a Local LLM?

The short answer is yes. You can absolutely run different LLMs locally on your machine.

For example, if you have an NVIDIA GPU with around 4–8 GB of VRAM, you can comfortably run models like Phi-4 Mini, Qwen 3 8B (Q4), and other lightweight coding models. These are perfectly suitable for simple developer workflows such as single-line code autocompletion, basic code refactoring, Git commit message generation, and other lightweight programming tasks. Their biggest limitation isn’t necessarily speed but their relatively small context window and limited reasoning ability, which makes them less effective once your project starts becoming more complex.

As your hardware becomes more capable, the size of the models you can run also increases. Machines with 12–16 GB of VRAM, such as an RTX 4060, RTX 5070, or even an M5 Pro MacBook Pro with around 18–24 GB of unified memory, can comfortably handle coding models in the 14B to 24B parameter range. Models like DeepSeek Coder V2 Lite and Qwen 2.5 Coder become practical choices here. At this level, local models become genuinely useful for everyday software development. They are capable of reviewing multiple files, generating unit tests, completing larger functions, refactoring existing code, writing documentation, and assisting with medium-sized software projects without constantly running into context limitations.

If you move into the 24 GB VRAM range, such as an RTX 5090, or use a machine like an M5 Max with 64 GB of unified memory, you can start running much larger reasoning models such as Qwen 3 32B or DeepSeek R1 Distill 32B. These models are capable of handling significantly more complex workflows, including real-world debugging sessions, tool calling, code analysis, type-safe API generation, software architecture discussions, design planning, and much longer reasoning chains. At this point, local models begin to feel much closer to what cloud-hosted AI assistants can provide.

Finally, if you own extremely powerful hardware, such as dual RTX 5090s, multiple enterprise GPUs, or cloud instances equipped with high-end accelerators, you can realistically run models in the 70B+ parameter category. Models like Llama 3 70B and larger variants of DeepSeek R1 become practical. These models are designed for large-scale architecture planning, complex system design, enterprise-level refactoring, long-running reasoning tasks, and even multi-agent software engineering workflows.

Hardware Overview

The Reality of Open Models

That is the biggest advantage of owning more powerful hardware—you gain access to significantly larger models. However, despite how impressive these open-source models have become, they still don’t come close to GPT-5.5, Codex, or Claude Opus when it comes to overall reasoning ability, coding performance, and consistency.

The gap is certainly getting smaller. Every few months, models like Gemma, Qwen, DeepSeek, and GLM continue to improve. With each release, they catch up a little more to today’s frontier models. However, by the time they approach the capabilities of the previous generation, companies like OpenAI and Anthropic have already released another generation of even more capable models. As a result, the gap, although shrinking, still exists.

Different Models, Different Strengths

Every one of these models comes with its own strengths and weaknesses. They differ in architecture, training methodology, tokenizer design, context length, quantization support, and hardware optimization. Some are better at coding, others excel at reasoning, while some are designed specifically for multilingual tasks or long-context workloads.

Many of the newer Chinese open-source models are also trained using hardware that differs significantly from the NVIDIA-centric infrastructure most developers are familiar with. They often rely on domestically developed AI accelerators and entirely different training stacks. That is a fascinating topic on its own, but it’s also a discussion that’s beyond the scope of this article.

The Question That Actually Matters

So the real question isn’t whether local LLMs can run on consumer hardware—they clearly can.

The real question is this: if I’m using a base M4 MacBook Pro with 16 GB of unified memory, can I realistically run a local model that helps me with everyday junior developer tasks? Can it write Git commit messages, organize my Obsidian vault, manage my notes, build a personal wiki, summarize documentation, and generally help me organize my thoughts? These aren’t particularly difficult tasks, but they’re important enough that I don’t necessarily want to spend Claude Opus or GPT-5.5 money on them every single day.

And if the answer is yes for me, then can my teammates with similar laptops use the exact same setup as well?

Setting Up a Local LLM on a Mac

With all that said, let’s actually put it to the test.

As I mentioned earlier, I’m using a base M4 MacBook Pro with 16 GB of unified memory. The obvious question is: can I run a local LLM on this machine? And if I can, how exactly do I do it?

With that question in mind, let’s start from the basics.

To run any local LLM on your machine, you need three things. First, you need the model itself. Second, you need a server that can load and serve the model. Third, you need a client or interface that communicates with the server, sends requests to the model, and integrates it with your coding environment or other applications.

There are countless options available for each of these components. The internet is full of different model repositories, inference servers, and desktop applications, each with its own strengths and weaknesses.

However, if you’re using a Mac, there is one technology that stands out: MLX.

MLX is Apple’s machine learning framework designed specifically for Apple Silicon. Many open-source models are converted into the MLX format, allowing them to take full advantage of Apple’s unified memory architecture. Instead of relying solely on GPU memory, MLX enables the CPU and GPU to efficiently share the same pool of memory, making it possible to run surprisingly capable models even on base MacBooks.

If you’re on Windows, however, the story is completely different.

Installing LM Studio and Setting Up My First Local LLM

With the theory out of the way, it was finally time to see whether this would actually work on my machine.

I started by installing LM Studio, and the installation process was surprisingly smooth. After downloading the application, it walked me through a short onboarding process. One thing I really liked was that LM Studio automatically detected my system specifications, including the amount of unified memory and the overall hardware available. Based on those resources, it recommended a model that it believed would run well on my machine. In my case, it suggested a Gemma 3 quantized model.

However, before downloading anything, I asked ChatGPT which local model would be the best choice for my hardware. My primary goal wasn’t to replace Claude or GPT-5.5—I simply wanted something capable of handling lightweight development tasks such as writing Git commit messages, reviewing pull requests, performing simple code reviews, and helping with other junior-level development work.

The recommendation was Qwen 3 8B, quantized to Q4, which is generally considered one of the best performance-to-memory trade-offs for consumer hardware. That sounded reasonable, so I downloaded the model.

Once the download was complete, I increased the model’s context window to the maximum value that LM Studio allowed. After that, I installed OpenCode, which would act as the client—or, more accurately, the harness—for interacting with the model.

As I mentioned earlier, there are three pieces involved in running a local LLM. First, there’s the model itself. Second, there’s the inference server that loads and serves the model. Finally, there’s the client application that communicates with that server and provides a usable interface inside your coding workflow. In my setup, LM Studio handled the inference server, while OpenCode became the interface that streamed responses and integrated with my development environment.

Once everything was installed, the remaining setup was straightforward. Inside LM Studio, I opened the Developer tab and enabled the local server. By default, it started an OpenAI-compatible API server on localhost:1234, which meant any compatible client could immediately connect to it.

The model booted without any issues. Before using it, I enabled a few performance-related options, including KV cache and parallelism, to squeeze a little more performance out of the hardware. After that, I started chatting with the model directly inside LM Studio just to verify that everything was working correctly.

One thing became immediately obvious, though. Unlike cloud services, this isn’t a serverless environment where resources are allocated on demand. The entire model is loaded directly into your machine’s memory. On my 16 GB M4 MacBook Pro, my memory usage jumped from around 5 GB with just my browser open to roughly 11–12 GB as soon as the model finished loading. That’s a significant increase, but it was still manageable and left enough memory for the rest of my development workflow.

With the server running, I switched over to OpenCode, entered the local API URL, configured the API key, selected the model, and finished the remaining setup. Within minutes, everything was connected.

The first few interactions were surprisingly smooth. I could send prompts, receive streaming responses, and use the local model almost exactly like I would use a cloud-hosted API. At that point, I finally had a fully working local coding assistant running entirely on my laptop.

The Downsides

And there you have it—the model was finally running on my local machine. I could chat with it, ask questions, and use it just like any other AI assistant.

Honestly, I could have ended the blog post right there.

But there are downsides.

You probably already know the biggest one: your hardware.

Even on my base machine, yes, I can run these models. But here’s the thing—they’re simply not good at complex tasks.

By “complex,” I mean giving the model an entire TypeScript codebase and expecting it to find bugs, explain the issue, and fix it. I actually tried this with Wryte.xyz, the platform I built for writing and scheduling content.

I gave it a fairly simple prompt, almost identical to something I’d normally send to Claude:

In this project, when I am writing content and hit the Publish button, if the text length is too long, the content spills out of the dialog box, causing poor UI and UX. Can you resolve this UI/UX bug for me?

And before anyone comments, yes—I know I could fix a bug like this myself. I was bored, I was lazy, and that’s exactly why I wanted to see how capable these local LLMs really were.

The result was… disappointing.

Yeah… it didn’t work.

Part of that was my fault. I hadn’t configured the context window properly, and the model I chose didn’t support images. Even if it had, the context window was simply too small to understand the entire project or make reliable tool calls. That’s when I realized what these smaller models are actually built for.

They’re great as chat models. They’re good at fixing grammar, rewriting text, writing Git commit messages, summarizing notes, or handling the small jobs that don’t justify spending Claude or GPT money. But asking them to reason through an entire production codebase is expecting far too much—at least on consumer hardware.

Sure, with significantly more RAM and a much larger GPU, the story changes. But I’m trying to stay realistic. Most developers—probably 80% of us—aren’t building dedicated AI workstations. We just want tools that work on the laptop we already own.

There’s another thing nobody really talks about either: concurrency.

During my testing, everything was basically one request at a time. I couldn’t comfortably run multiple sessions, and that became frustrating very quickly.

And please, for the love of God, stop saying local LLMs are “free.” The model might be free to download, but the hardware isn’t. Neither is the electricity. If you’re tying up an expensive machine all day running inference, you’re paying for it somehow.

So, are local LLMs worth it?

Absolutely—if you enjoy tinkering. They’re fantastic for learning, experimenting, and handling lightweight everyday tasks.

But if your goal is serious end-to-end software engineering, large-scale coding, image generation, or video generation, then on ordinary consumer hardware, they’re simply not there yet.

There are plenty of YouTube channels that go much deeper into local LLMs and hardware optimization, so if this interests you, they’re well worth watching.

Anyway… this turned into a much bigger rant than I expected.

Until next time,

Goodbye, nerds.

Frequently Asked Questions (FAQ)

Can I run a local LLM on a normal laptop?

Yes, you can run local LLMs on normal laptops. Machines with 16 GB to 32 GB of RAM can run smaller open-weight models, especially quantized versions. However, the size and capability of the model depend heavily on your available memory and hardware.

What is the best local LLM for a 16 GB MacBook Pro?

For a 16 GB MacBook Pro, smaller quantized models like Qwen 3 8B, Gemma, Phi, and similar models are good options. They are suitable for lightweight coding tasks, writing assistance, and personal productivity workflows.

Can a local LLM replace Claude or GPT for coding?

Not on most consumer hardware. Local LLMs can handle simple coding tasks, but they struggle with large codebases, complex debugging, long context understanding, and advanced software engineering workflows compared to frontier cloud models.

Are local LLMs actually free?

No. While many open-weight models are free to download, running them requires hardware, electricity, storage, and maintenance. The cost is transferred from cloud subscriptions to your own infrastructure.

What hardware do I need to run a good local coding LLM?

It depends on the model size. Smaller models can run on 8–16 GB of memory, while larger coding and reasoning models often require 24 GB, 48 GB, or more of GPU memory.

Are local LLMs good for software developers?

Yes, but mainly for specific tasks. They are useful for generating commit messages, explaining code, writing documentation, summarizing information, and assisting with small coding tasks. They are currently not a complete replacement for advanced AI coding assistants.

Why are local LLMs slower than cloud AI models?

Cloud AI providers run models on large clusters of specialized hardware. A local machine usually has much less compute power, meaning smaller models, slower inference, and fewer simultaneous requests.

Should developers use local LLMs?

Developers should use local LLMs if they want privacy, experimentation, customization, or lightweight AI assistance. For complex development workflows, cloud-based AI models are still generally more capable.