Elvin Garcia · ORGANISMIC
You asked it something that mattered.
Not a recipe, not a rewrite of an email. A real question — about a decision you had been carrying, or a diagnosis, or a contract, or a person you love.
The answer came back quickly, and it sounded measured, specific, and useful. It named considerations you had not thought of. It was appropriately careful in the places where care was warranted. It was, as far as you could tell, good.
And then there was a small moment that most people do not examine.
You had no way to know where any of it came from.
Not in the sense of wondering whether it was wrong. Not because you had reason to think it was wrong. That is not the feeling. The feeling is stranger and harder to name: you had just been handed something to act on, and there was nothing attached to it — no author, no date, no basis, no way to look behind the answer and see what it stood on.
Most people notice that for about a second and then get on with their day, because there is nothing obvious to do with it and the answer was useful anyway.
I want to spend a few thousand words on that second.
What is actually behind the answer
It is worth being accurate about the mechanism, because the popular accounts are wrong in a way that matters.
Unless a system tells you it is searching the web or drawing on documents you have given it, an ordinary answer is not a lookup. Nothing goes and consults a library, retrieves a document, and reports back. That is a natural assumption and it is not what happens.
What happens is closer to this. The system was shaped, over an enormous quantity of text, into something that can produce fluent and often accurate continuations of language. The shaping worked. It is why the answers are as good as they are, and they are frequently very good.
But the material that did the shaping is not carried along with the answer. It was not a source that got consulted; it was an influence that got absorbed, distributed across the whole system in a form that no longer resembles the documents it came from. The answer does not arrive with a source-level account of which material shaped it — and the people building these systems are still working to understand, in a reliable and complete way, how particular internal processes lead to particular answers.[1]
That is not a scandal and I am not going to treat it as one. It is how the thing works. That opacity is not a cover-up. It is a consequence of the way these systems are built, and reducing it remains a serious technical problem rather than a solved feature.
I am also not going to tell you that the answers are untrustworthy. This essay is not about whether a particular answer is accurate. It is about what accompanies an answer when you need to decide how much weight to give it.
It is about something that used to come attached and no longer does.
What quietly stopped coming with the answer
Many of the knowledge objects we grew up with made more of their basis visible.
A textbook had an author, a date, an edition, and a bibliography. An encyclopaedia listed contributors. A manual said who wrote it and when it was last revised. A newspaper article named its reporter and, usually, where the information came from. A doctor could tell you what they were looking at and what would change their mind.
None of that guaranteed correctness. Plenty of published knowledge has been confidently wrong, and a bibliography has never made a bad argument good. That is not the claim.
The claim is that you could locate a piece of knowledge in the world. You could see how old it was, who stood behind it, what it rested on, and — crucially — you could form some judgment about how much weight to put on it. A 1987 edition told you something. An author with a stake in the outcome told you something. A sentence with three citations and a sentence with none told you something.
We rarely treated any of this as a feature. It was simply what many knowledge objects looked like, and we rarely expected knowledge to arrive without it.
And we did not decide to give it up. Nobody made that trade deliberately. We were handed something so much more capable, so much faster, and so much more available than what we had, that the thing which quietly stopped coming with it was not missed for some time.
I am not arguing that we should go back. The exchange was, on most questions, a good one.
I am arguing that it was an exchange, and that most people have never been told what was on the other side of it.
The same thing, made visible
Here is what the other condition looks like, because a description is not enough. You have to see one.
Below is a small object — not a product, not a system, and not something I am selling. It is one way a piece of knowledge can arrive with its basis attached. Read it and then I will say what it is doing.
It concerns a question that a great many families face, and it concerns no family in particular.
The question
Is it time for our father to move into assisted living?
What the current record supports
There have been two falls in eleven months, both at home, neither requiring hospitalisation. Medication is being taken but the pattern has become irregular since the pillbox was reorganised in March. He is eating, though shopping has been intermittent since he stopped driving in January. A primary-care assessment in June recorded no cognitive concern; a nurse’s note in the care record in August mentions “some confusion about appointment times.” He has said twice that he wants to stay in the house. He has said once, after the second fall, that he doesn’t want to be a burden.
Why it supports it
The falls and the medication pattern are directly observed and recorded by two people independently. The eating and shopping are reported by one family member and not corroborated. The stated wishes are first-hand and recent. The nurse’s note is second-hand as recorded here and has not been read in full. The primary-care assessment is two months old and predates the second fall.
What could change the judgment
A formal needs assessment, which has not been requested. A medication review, which has been discussed and not booked. Whether the confusion about appointment times is isolated or a pattern — nobody has been tracking it. What in-home support is actually available locally and at what cost, which nobody has looked into. Whether the second fall had a cause that is treatable. And whether “I don’t want to be a burden” was a statement of preference or a response to distress in the room.
What this does not decide for you
Whether the risk is acceptable. That is a judgment about a person’s life, and the people who know him have to make it.
Whether his stated wish to remain at home is his settled preference or a fear of the alternative — and whether the difference should change the answer.
What the family can actually sustain, which no assessment measures and only they can say.
What he would want if he could see the whole picture, which is the question underneath all of the others and cannot be resolved by any of the above.
Notice what that object does not do.
It does not answer the question. It does not lean toward the move and it does not lean toward waiting. If you read it looking for a recommendation you will not find one, and that is not evasion — it is the honest position, because nothing in that record decides it.
Notice also what it makes possible that a fluent answer does not.
You can see what is carrying weight and what isn’t. Two falls observed by two people is a different kind of fact from one relative’s impression about shopping. Both might be true. They are not the same evidence and the card does not pretend they are.
You can see the age of things. The primary-care assessment predates the second fall. That matters, and it is the sort of thing a smooth answer absorbs without comment.
You can see what has not been done. No needs assessment. No medication review. Nobody tracking the confusion. Several of the things that could most change the judgment are simply unmeasured, and the card says so instead of reasoning past them.
And you can see where the decision actually lives. Not in the record. In a family, weighing a person’s dignity against a risk, with information that will never be complete.
A confident answer to that question — and you can get one in four seconds — would say yes or not yet, would sound compassionate, and would be indistinguishable in tone from one produced with all of the above in view. That is the whole problem in one sentence.
Why this matters more the closer it gets
For most questions, none of this is worth thinking about. If you want a marinade, a summary of a long document, or a first draft of an awkward email, the basis is irrelevant and asking about it would be tiresome. Use the thing. It is very good.
The property becomes important somewhere along a gradient, and the gradient runs roughly from using an answer to relying on one.
Using an answer means you will look at it, apply your own judgment, and notice if it is wrong. Relying on one means you will act, and the action will be difficult to reverse, and you will not have an independent way to check first.
And here the pattern inverts in a way worth noticing.
When a question carries real consequence, the appetite for a fast and confident answer tends to rise at exactly the moment the patience to examine it falls. That is not a failure of character and I am not exempt from it. It is ordinary. You are worried, you want to know what to do, and something has just told you clearly and kindly what to do.
The questions where the basis matters most are the questions where you are least disposed to ask about it.
What you can actually do
Three things, and none of them requires you to change how you work or to buy anything.
Notice which kind of question you are asking. That discrimination is most of the value here, and almost nobody makes it consciously. Am I using this answer, or am I about to rely on it? You will find the honest answer is usually obvious, and that you have been treating both kinds the same way.
For the ones you will rely on, do not stop at the answer. Ask yourself what you would need to check before acting. If sources are offered, open them — and read enough to see whether they actually say what the answer says.
I want to be careful here, because there is a version of this advice that is worse than none. If you ask one of these systems to explain its reasoning, the account you receive is not a reliable audit trail of how the original answer was produced. It may be useful. It may even be a good explanation. But models can produce plausible accounts that do not faithfully describe the internal processes that led to an answer,[2] and treating one as a look behind the curtain would be a more confident kind of mistake than the one you started with.
The point of asking is not to open the box. It is to reduce how much you are resting on it.
And for one thing that matters to you, make a note that keeps its basis attached.
Not a system. Not a framework. A page. What you currently believe, where each part of it came from, how strong that is, what you don’t know, what has gone wrong before, and what still has to be decided by a person rather than looked up.
It need not take long, and it does something odd the first time you try it. You may find how much of what you were confident about was assembled from things you cannot now locate. Which is not a comfortable discovery, and is considerably better made at a kitchen table than in the middle of a decision.
What you are left with
You will ask a machine something tomorrow, and so will I, and it may be useful.
Nothing here is an argument against that. These systems are genuinely useful and I use them for a great deal, including work that matters to me. The unease at the top of this essay is not a warning sign to be heeded by using them less.
It is a signal that two things you have been treating as one are actually two.
An answer can be useful without being examinable. Often, for ordinary tasks, that is a perfectly good arrangement. But usefulness and inspectability are different properties, and you have been receiving one of them almost exclusively for the last few years — from something so capable that the absence of the other has been easy not to notice.
You can trust what you cannot inspect. People do it constantly and reasonably. Pilots, surgeons, and the person who serviced your brakes are all trusted rather than inspected, and the world would stop if we insisted otherwise.
But you should know which one you have.
And on the questions that will not come again — the ones about a parent, a child, a business you built, a thing you cannot take back — it is worth asking for something more than fluency.
Elvin Garcia is the founder of ORGANISMIC, a publishing house for capability you own rather than rent. organismic.org
Notes
The description of how these systems produce answers is deliberately non-technical. Foundation models learn statistical patterns from training material and generate responses using learned parameters, rather than consulting a source library at the moment of ordinary generation.[1] The caution that an explanation supplied after an answer is not necessarily a faithful account of the model’s internal process is supported by current interpretability research.[2]
The claim card describes no actual family. Its details are constructed to be ordinary, and any resemblance to a particular situation is a consequence of how common the situation is.
[1] OpenAI, How ChatGPT and Our Foundation Models Are Developed.
[2] Anthropic, Tracing the Thoughts of a Large Language Model, March 2025.