Typically when most people think of AI, they think of tools like ChatGPT, Claude, or Gemini. Some web or desktop UI with a chat interface. If you’re on the coding side of things, you may be working more with a TUI (text user interface) in the command line. You’ll hear terms like models, agents, or harnesses get tossed around. This list of vocab can go on and on; reinforcement learning, fine-tuning, prompt engineering, context windows, and much much more.
“Yesterday”, it was all about prompts and prompt engineering. Today, it’s about context, the context windows, memory, and tokens. Tomorrow, it’ll likely be all about the harness. Who knows what it will be in 9 months, 18 months, or 3 years from now. But, that future is a bit where my ADHD likes to live. What could we care about in the near future? What will be the next stage of AI and agents? What would we need in order to have that “iPhone moment” for AI?
NOTE: This is not at all me “reading the tea leaves” or taking a guess at the future. This is purely day dreaming out loud and (maybe) hoping for something interesting to happen.
Primer
Before we get into the fun fantasy of “what could be”, let’s create a bit of clarity around some of the current patterns in use within the industry today. I think this is important as it helps pull together the moving parts.
First point of clarity… I’ll focus on LLMs and the agentic frameworks around them. This is fairly portable to small local models, but most of what we’ll be going through will apply to both.
We also have an agentic stack. In this stack, we likely know that the model is the LLM. What we may not be familiar with is the harness, from a terminology standpoint. But you actually use it every time you engage with an LLM. The harness is the “engine” that sits between where you type in your prompt plus see the outputs and what the LLM “sees”. This engine is simply a program that orchestrates between you, the LLM, and any tooling that may be appropriate to accomplish the task. That one line makes it sound like not that big of a deal, while the reality is far more complicated than that. Remember that even though we talk about agentic memory, LLMs do not actually have memory. Each time you interact with one, the harness has to pull together the entire context. This is all of the inputs and outputs from the conversation, the tooling used, and any relevant data that was part of the session. It’s essentially replaying the entire conversation up to the current point plus any new information, data, or context that has been introduced since the last interaction. Now, I am oversimplifying things a bit, but it’s fairly safe to think of it this way.
We could easily walk down the tokenomics rabbit hole, how caching works, AI credits vs raw tokens, etc. This is an entire topic all on its own. But for now, we’ll keep it focused on the harness itself.
Now that we have this picture of the engine that sits between us and the models, there’s one benefit worth calling out. Providers can train their models with some “awareness” of the harness they’ll run in. We don’t have full access to their processes, so we can’t know exactly how it all works under the hood, but a model trained to use its harness effectively can better understand the context it’s operating in. Strapping a different harness on top (e.g. Hermes, GitHub Copilot) won’t necessarily make things better or worse. The point is simply that aligning the model and the harness can unlock optimizations.
NOTE: I tend to use GitHub Copilot as my primary harness (for the moment), so my perspective is somewhat influenced by its capabilities and limitations.
One last thing before we dive into that future state is around the term distillation. We’re not referencing the typical process to make distilled water. This is in reference to the industry practice of a technique where the intent is to transfer the learnings of a larger pre-trained model, known as the teacher model, to a target student model. This allows the student model to benefit from the knowledge and capabilities of the teacher model, often resulting in a smaller, more efficient model that retains much of the performance of the larger model. Now, I say “often resulting in a smaller, more efficient model” as that leans into the intent and industry standard definition, but it is a tool employed within the industry to produce other competing LLMs. It is also worth noting that distillation can be applied not just to the models themselves but also to the interaction patterns and optimizations learned within the harness, potentially creating a more portable and efficient agentic package.
Portable Agentic Package
Now for the future…
I just mentioned the model training on the harness aspect of things as well as the potential for distillation. It’s these 2 factors that lead into what I’d like to see happen. The vision is to produce smaller, more distributable units. A purpose-built package would include both the model and the harness, created for distribution and ease of use.
You want a personal assistant that helps manage your emails and schedule? How about a coding assistant for the language and frameworks you’re using in your current project? What about a specialist in analysis of a specific area of scientific research? Or perhaps an opponent in a strategic game? Instead of loading up a generalist model and strapping some generic harness on top of it, you could simply download a single package that is just enough for your needs. Less generalization means you have less wasted capacity and more efficiency towards the activity in scope.
What if you were a developer working on 2 different projects in parallel, while also needing a personal assistant to manage your schedule and email backlog? How about a specialized orchestrator, being a specialized harness + model package as well, which would be running in the background. It understands the state and context in the activity you’re working on and coordinates the loading and unloading of the other specialized agents.
Similar to the concept of downloading an app from an app store or from a software repository, a person could pull down a radio-dj.papi (Portable Agentic Package Instance) package onto their device. Click on it, kicks off a harness that includes the interface for interacting with the “agent”, loads the model into the memory, and makes it ready for immediate use. Instead of relying on Spotify and their algorithm, you talk to your own personalized DJ. Let it know where your collection of music is stored and what mood you’re in. Instant vibes.
Working on the next cyberpunk novel? How about pulling down a cyberpunk-editor.papi and cyberpunk-critic.papi? The editor jumps in to help you with shaping and refining of your manuscript, while the critic provides feedback, suggestions, and points out potential plot holes or inconsistencies. We already crowdsource feedback from editors, readers, and critics. Now you’ve got instant access earlier in the process.
The potential here is for a more modular and personalized AI ecosystem. Instead of being locked into a single, monolithic model that tries to do everything, folks can mix and match specialized lightweight agents that best suit their needs in the moment. This could lead to a more vibrant and diverse landscape, where innovation is driven by the community and individuals rather than a few centralized providers. It also opens up opportunities for niche markets and specialized applications that may not be commercially viable for large, generalist models to support. Ideally, it also reduces the hardware burden. Instead of needing hundreds of gigabytes in memory and storage for a single multipurpose model, you have access to on-demand targeted capabilities as needed.
Until then
We have some patterns today with skills, MCP, and agent plugins, but these are a bit of a band-aid solution on top of things. It still relies on a centralized model provider, either self-hosted or through a vendor. Do I think the centralized, general model (approach) will ever go away? No. I think we still need that for quite some time, especially when building out the specialist agents ecosystem. You need the raw horsepower of the large, generalist models to support the smaller, specialized agents.
Hell, maybe we’ll even revisit physical media for this. Just imagine swapping out an agent on a portable device like you’re swapping out a cassette on a walkman.