How to turn tribal knowledge into an AI agent
How I turned organizational and tacit knowledge into a local AI assistant that helps Saleor prepare approach documents faster.
This article was originally published on LinkedIn.
As a Solution Architect at Saleor Commerce, a big part of my job is turning a discovery call into a deal and a successful go-live. The artifact that bridges those two things is an approach document: a presentation walking through the proposed implementation architecture, tailored to the client’s specific pain points, integrations, and constraints.
We put our heart into these. But they require a lot of work, so no wonder we tried to augment it with AI. The results were, unfortunately, poor. We’d ask Claude or ChatGPT for help with structure, wording, solution brainstorming. The chatbot did the best it could given its understanding of the problem, which was poor.
The results were suboptimal because we tried to use a general agent for a task that required a lot of domain knowledge and understanding of the process. It required a knowledge agent instead - one that knows your process, not just your prompt.
So I asked myself: if I had to onboard a highly intelligent new hire to do this job, what would I give them? Where would I point them to read?
In this exploration, I not only created a pipeline for extracting encyclopedic knowledge (like how to do something in Saleor) but also what I thought was tacit knowledge (unwritten, unspoken, tribal) but turned out to be knowledge nobody had written down yet.
I combined it with some patterns from agentic engineering and - in result - created an assistant that speeds up our team’s development of an approach document by at least 50%.
Let me walk you through some generic patterns I discovered you can apply to building a knowledge agent in your line of work:
Selecting the right tool
We already established that a generic chatbot is not the right tool for the job. If so, then what is?
Let’s inspect our requirements. We need the ability to:
- Store some domain/organizational knowledge.
- Establish recurring tasks/processes in an easy-to-run way.
- Use tools, like web search or executing a script.
- Learn and improve.
Since I am a software developer, I reached for Claude Code.
When I started building the Solution Agent, it was a bit unusual to use CC for something other than coding. Now, it’s recognized in a broader market, which resulted in Anthropic publishing Claude Cowork - essentially an interface to Claude Code for non-technical knowledge workers. It can write or read files in your filesystem, set up recurring (via schedules) or structured tasks (via custom commands), or use skills.
This process can be fully local - no deployment, no access management, no integration with existing tools. It’s definitely not as satisfying as pinging your custom agent on Slack via @SolutionAgentBot but it gets the job done.
Extracting domain knowledge
Given how digital some of our communication already is, chances are your chatbot’s missing context is up for grabs. Most of our discussions already happen online, tasks are written in Linear, customer calls are processed in Attio. All these produce artifacts you - or your agent - has access to.
Don’t bother with full automation at an early stage. In the beginning, you can do the extraction manually, which is what I do with the Solution Agent.
I go to Attio to download the meeting transcript and I export a discovery board from Excalidraw. I run a sanitization script to remove any potential PII but I still read through everything myself to be sure. An agent doesn’t have to be fully autonomous to be productive.
That’s the dynamic input to the process: what’s different between each run. What about what remains the same?
In the case of an approach document, that’s:
- Knowledge about Saleor (e.g., what is our API surface, what webhook events do we support)
- Saleor architectural patterns (e.g., how to solve a specific pricing problem)
- Structure of the presentation and other artifacts (e.g., what slides do we usually have, what goes first and second)
The first one is the simplest. Given how capable frontier models are at running tools, like web search, I don’t even bother with hooking a documentation MCP like Context7. I just point to Saleor documentation or its llms.txt and the agent is effective at surfacing the relevant docs.
The next two are more interesting because these are pieces of organizational, contextual knowledge. Here is how I solved it:
- I provided a broad context in the prompt. I described the process roughly, its milestones, how to validate it etc. If this sounds like a lot of writing, try dictating (I use Wispr Flow).
- I asked the model to interview me, something along the lines of: “Don’t assume you know anything about the process, ask me all the tiny details, propose possible answers in A, B, C or custom text input. Write me back with your understanding of the process”.
- I showed/embedded the examples of the work. In my case, I anonymized a bunch of approach documents we produced in the past and I pasted it into the chat with Claude Code. It used the image analysis tool to OCR each image, out of 30-something, and inferred the process. I exported our presentation template in .png and dropped it into the repo as well, so it can peak at how each slide look when in doubt.
Breaking down the process
Software engineering was the original inspiration for the tool selection, so why not for the entire process, too?
So far, writing code is our best referential point on how agents can perform long-running, complex tasks, while assisted/steered by humans. This practice has lately been named agentic engineering. Many words were written on its best practices. What’s universally agreed is that Markdown (text) is a format both humans and AI agents are comfortable using.
This results in a popular workflow where, when faced with a complicated task, the AI coding agent will first present a detailed plan.md - a document the human user has to review and eventually accept. Only then does the actual implementation start.
This pattern has been expanded to more complex workflows, where each step produces its own kind of Markdown document. In the context of software engineering, I wrote about it in my article “Staying out of the LLM dumb zone”.
For our knowledge agent, we can apply a similar pattern. My approach document process is sequential, meaning it always goes in the same order of steps:
- Research (what company is this)
- Analyze the meeting transcript
- Propose a solution (apply Saleor & architecture knowledge)
- Prepare slides content
I wanted my agentic process to mimic this structure because it’s easier to review and provide feedback to the output of each step individually, rather than the sum of it.
That’s why I reached for Claude Code custom commands/skills. Each step is its own command that produces a specific document I have to review and accept. Here is the example of my .claude/commands/1_company_research.md step:
---
allowed-tools: WebSearch, WebFetch, Read, Write, Glob, Grep, AskUserQuestion
---
# Company Research
Research the prospect company before processing discovery materials.
## Execution Instructions
When this command is invoked, execute the following workflow:
### 1. Get Company Information
Use AskUserQuestion to prompt for the store/company information:
- Ask for either a URL of their store/website OR the company name
- This is a required first step
### 2. Research the Company
...
**Gather the following context:**
- What does the company do? (products/services)
- What industry are they in?
- What is their business model? (B2B, B2C, D2C, marketplace, etc.)
...
**Source Attribution (CRITICAL):**
- For EVERY fact you include in the output, tag it with the source it came from
- Use inline source tags: `[Source: {URL}]` for website-sourced facts, `[Source: web search]` for search results
- If a fact is inferred or interpreted (not directly stated), mark it as `[Interpretation]`
- When multiple sources confirm the same fact, cite the most authoritative one (company website > search results)
### 3. Write Output
Write the research to `output/1_company-research.md` in this format:
````markdown
# Company Research: {Company Name}
## Overview
{2-3 sentence summary of the company}
...
````
Claude Code created this command for me, based on our conversation. I only adjusted it where it missed the mark.
Compounding effect
If you got this far, you might be thinking: woah, that’s a lot of work.
That’s true. I would only put in the effort to automate it when it’s a high-stakes, high-leverage game. The comforting part is that all you need to set up a basic process is a Claude window. It can do all the setup for you and you can continue to improve it (as I did).
But what’s important is: it really pays off to make this investment because what starts as a simple 1:1 agentic process recreation doesn’t have to end there.
Since Claude Opus now had the full context of the process, I could ask it to provide feedback on what we can improve, and we already made some significant improvements thanks to that. But, more importantly, that “agentization” allowed us to expand to new steps that wouldn’t be possible prior.
Recently, I updated the Solution Agent with a new command: implementation blueprint. I thought: we are gathering all the knowledge about the client, everything is in text already, all the implementation details, all the edge cases, and then we condense it into a presentation that has to be simple because it’s not an implementation manual, it’s an overview. There is a lot of valuable information going to waste there.
So now, along with the presentation, we attach the implementation blueprint: the relevant links to docs, the webhook events, GraphQL queries and mutations, sketches of an integration. Everything is aligned with the milestones we walk our clients through anyway: building the catalog, managing the fulfillment, creating a checkout flow. It’s not a replacement for a solution architecture or customer support, it’s an augmentation.
This document is a rather dry list of references, code examples, diagrams. Nothing sophisticated that requires a human soul poured into it. We simply wouldn’t write it ourselves because it would be too much work. But because of all the effort we’ve put into creating the agent and extracting the knowledge, we can now reap the benefits.
Human in the loop
Let’s finish on a humanistic note. This entire process is built from the human knowledge we’ve gathered over the years doing solution architecture at Saleor. It’s highly conversational - the AI isn’t “one-shotting” anything. It can only work with a human in the loop - the agent itself isn’t sending an email with the actual presentation with no governance.
But even with those restrictions and challenges, it’s a great help. It writes the first draft of the research, I catch what it gets wrong. It proposes the solution architecture, I push back where it misreads the client. The relationship, the judgment, the hard conversation - that stays human. On purpose.
Let me know what type of knowledge agents you’ve deployed at work or in your personal life. If you are still a bit lost with the subject, DM me and I’ll happily help.