The questions to ask an agentic AI provider are about control: where it runs, whose keys it holds, what waits for a yes, what is logged, which models work, how you pause it and how you leave. IBM's 2026 research found 92% of organizations with an AI-related breach lacked proper AI access controls.
The questions to ask an agentic AI provider are about control before features: where it runs, whose keys it holds, what waits for a person's yes, what gets logged, which models do the work, how you pause it, how drift is caught and how you leave. IBM's 2026 breach research found 92% of organizations with an AI-related breach lacked proper AI access controls.
An agentic AI provider sells software that does more than answer: it takes a job, works through steps, uses your tools and acts inside limits someone set. That is why the buying questions differ from the ones for a chat tool or a SaaS app. The software will act in your name, so the questions are about who sets its limits and what record it leaves.
We have run LTFI, our agentic business operating system, in live production at Kief Studio for a year and a half, and Brian walked through it on our October 23 webinar. Since then, the lookalikes have started arriving, and agent washing covers how to spot them. This post is the next step: the questions to ask an agentic AI provider, written as the checklist I would bring to the call with any provider, including us. It sits on top of the general vendor work in how to actually evaluate a technology vendor and the security questions in what to ask your vendors about security, so it covers only what changes when the software can act.
There are eight, and each one has an answer the provider can show on screen. Use the table as the agenda for the first call, then work through the sections below for what a good answer sounds like.
| # | Question | A good answer | A warning sign |
|---|---|---|---|
| 1 | Where does it run, and who else runs there? | A named server and boundary, and who operates it | "The cloud" |
| 2 | Whose keys and whose identity does each agent use? | Its own identity, keys in a vault, scoped permissions | Agents sign in as you |
| 3 | What goes ahead, what waits for a yes, and who decides? | A written approval line only a person can change | "You can review anything" |
| 4 | What is logged, and can a person read it? | Who asked, which agent, which tools, who approved | A chat transcript |
| 5 | Which models do which jobs, and how is drift caught? | Choice per job, written rules and a second check | One model, no monitoring |
| 6 | Can you pause every agent, and cap spending? | One control, shown live; caps per person | A support ticket |
| 7 | How do you leave, and what do you keep? | A full export on a stated timeline | "We have never had anyone leave" |
| 8 | What does your contract say about safeguards? | Security measures written into the agreement | Terms only on a web page |
The order matters. The first three decide what the agents can reach and do, and a provider that cannot answer them clearly is not ready for the rest of the conversation.
Ask for a plain answer: your own server, a shared cloud tenant, or the provider's general platform alongside every other customer, and what the agents can reach from there. On May 1, 2026, CISA and its Five Eyes partners told organizations to "Avoid granting broad or unrestricted access, especially to sensitive data or critical systems" and to "Begin with agentic AI use cases that are low-risk and non-sensitive."
A good answer names the machine, the boundary around it and who operates it. A weak answer says "the cloud" and moves on. If the provider cannot draw the boundary, you cannot tell your insurer, your auditor or your clients where their records sit.
This one shaped how we built LTFI. Each company gets a dedicated server that Kief Studio runs, and no customer shares hosting with another. When Brian showed the support portal on the webinar, he explained why it lives inside each company's own system: support requests carry sensitive data, so "it sits on your box, not ours." Five Eyes agentic AI guidance for a small business turns the CISA paper into first steps.
Each agent should act under its own identity, with only the permissions its job needs, and the keys it uses should sit somewhere no agent or ordinary user can read them. An agent that signs in with your account is indistinguishable from you in every log, the problem described in what a non-human identity for an AI agent is.
This is where most companies are exposed. IBM's 2026 research, published July 29, 2026, found that only 40% of organizations reported using access controls on AI models and data, and fewer than half are actively securing non-human identities.
Ask the provider to show permissions per person, not only per agent. Brian's example on the webinar was a contractor who needs the shared files but has no reason to touch the AI system, and the settings screen let him grant one without the other. His other example belongs in every sales call: "if somebody is doing social media posting, do they really need access to financial data? Probably not."
The provider should hand you a written approval line: which actions run on their own, which stop for a person, which only a person can ever do, and who is allowed to change that line. If an agent or a regular user can loosen its own limits, the line is decoration.
The useful answers are specific. Sending email waits. Publishing waits. Moving money waits. Changing permissions is person-only. Approving every tiny step is its own failure, because people stop reading what they approve, the pattern covered in AI agent governance that checks the sequence, not each click.
In our own use, the line that earns its keep is the one on anything that leaves the company. On the webinar, Brian asked the chief of staff to email the leads that had no next step. The outreach teammate drafted the messages, and the system stopped with "Emails will wait for your approval before they go out." He read them, approved them, and they went. He also said plainly that a company can switch that gate off. We keep it on; his example of what skipping it costs was a double-booked schedule.
The log should say who asked, which agent did the work, which tools it called, which model ran, what it cost and who approved the result, in a form a person can read without an engineer. Among the agentic risks CISA's May 2026 release names are privilege creep and obscure event records, so ask for a sample log from a real job before you sign.
Then ask what the log is for. A log that only proves work happened gets read after an incident. A log that shows where a draft went wrong gets used every week. Brian described ours as being "for more than just proof of work or proof of effort": when an output misses, the record shows which step to tune.
I lean on that log more than I expected to. When Brian had shipped a run of changes and I was working on something else, I asked the system what had happened on that project and got a summary drawn from the work itself. A provider whose log cannot answer that kind of question will not answer an auditor's either.
You should be able to choose the model for each kind of job, bring your own provider or a local model, and see how the provider notices when a model's behavior changes. A provider locked to one model is asking you to accept its pricing and its quality on its schedule. Own versus rent AI explains why the workflow should stay yours while the model stays swappable.
Drift is the reason to ask the second half. Researchers at Stanford and Berkeley found GPT-4 answered the same prime-number questions with 84% accuracy in March 2023 and 51% in June 2023 (Chen, Zaharia and Zou, revised October 2023). The same product name can behave differently from one season to the next.
A good answer involves written company rules, a second model checking work against those rules before it counts as done, and logs a person reviews. On the webinar, Brian showed jobs assigned to different models by task, one for planning, one for writing, one for research, one for checks and grades. How to catch AI agent drift goes deeper on the monitoring side.
Yes should be the answer to both, shown live. One control should stop every agent while you work out what went wrong, and spending should be cappable per person, so one enthusiastic user cannot run up a month of model costs in a weekend.
Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027 (reported by MarTech, April 29, 2026). That coverage points to organizations lacking "the governance to manage what happens when something goes wrong." Caps and a pause are part of that governance.
The pause matters for a reason that has nothing to do with emergencies. If an agent starts a task with bad context, Brian's advice on the webinar was to pause them all while you fine-tune, then turn them back on. "You can pause everybody" is the line I would want any provider to be able to say and show.
The provider should describe an exit before you sign: which records come back, in what format, on what timeline, and at what cost. Contacts, files, content, rules and logs should all be yours. Vendor contract exit terms that decide what leaving costs sets out the clauses to ask for.
Ask about independence while you are there. Brian's view on the webinar was that a company using the system "should be independent," and that he wants as little access as possible to a customer's own accounts. A provider that needs to stay inside your accounts to keep things running has made leaving harder by design.
If the agents will touch personal information about Massachusetts residents, the state already requires vendor diligence and a contract. 201 CMR 17.03 calls for "Taking reasonable steps to select and retain third-party service providers that are capable of maintaining appropriate security measures" and "Requiring such third-party service providers by contract to implement and maintain such appropriate security measures."
An agentic AI provider that reads your CRM is one of those service providers. For a business in Shrewsbury, Worcester or anywhere in Central Massachusetts, the written answers to the questions to ask an agentic AI provider belong in your file, next to the plan described in Massachusetts WISP requirements under 201 CMR 17.00. The wider state picture is in Massachusetts AI regulations for business. This is general information, not legal advice.
Every question in the table has an answer we can show on screen, because LTFI was meticulously built from scratch by Kief Studio as one whole system: the CRM, projects, content, sites, security and AI staff together, run by the team that built it.
People talk to one chief of staff, which hands work to the AI teammate who owns it, and company rules are written once for every teammate to follow. Bring this list to any provider call, ours included. To see the answers in a working system, the October 23 webinar replay walks through it, and ltfi.ai has the walkthrough. Wherever your business is, we can set LTFI up remotely, as we have for clients around the world for more than 13 years; in Worcester, Shrewsbury and the nearby towns, we can also do it in person.
Start with where it runs, whose keys and identity each agent uses, and which actions wait for a person's approval. Those three decide what the agents can reach and do; logging, models, the pause and the exit come next.
For business records, a dedicated server gives a clear boundary you can describe to clients, insurers and auditors. Whatever the setup, the provider should be able to say exactly where your data sits and what the agents can reach.
A named person, on a line the company sets. Drafts and research can usually go ahead, while anything that leaves the company, spends money or changes permissions should wait for a person's yes.
If the provider handles personal information about Massachusetts residents, 201 CMR 17.00 expects you to choose a provider that can protect it and to require those safeguards by contract. An attorney can tell you what your contract needs.
The person who owns the work the agents will do, and whoever keeps your security and records. The first can judge the approval line; the second can judge the identity, logging and exit answers.
Ask it to show identity, approvals, logs, a pause and a data export working on a real job. A thin wrapper over a chat window or someone else's agent usually cannot show all five.
Agent washing is selling a chatbot or script as an AI agent. Gartner estimates only about 130 of thousands of agentic AI vendors offer real agentic features. Seven on-screen tests, the FTC and SEC cases so far, Massachusetts law, and what a year and a half of running agents taught us.
Only 13% of organizations think they have the right AI agent governance in place, Gartner reported in April 2026. AI governance for agentic AI is six controls built into where agents run: identity, permissions, approvals, audit, data ownership and model choice.
AI agent drift is the slow change in what an agent does after launch. Researchers found one model's prime number accuracy fell from 84% in March 2023 to 51% in June 2023 on the same questions. The four causes, five checks that catch it, and how we handle it in LTFI.
Work With Us
Kief Studio builds, protects, automates, and supports full-stack systems for businesses up to $50M ARR.
Newsletter
Strategy, psychology, AI adoption, and the patterns that actually compound. No spam, easy to leave.
Subscribe
