How to Audit What AI Is Actually Doing in Your Business
51% of small business owners are AI explorers, testing tools without fully committing, and 34% of them see no clear return yet. Here's how to audit what AI is actually doing for you, and what's just noise.
How to Audit What AI Is Actually Doing in Your Business
Most businesses didn't adopt AI strategically. They adopted it the way they adopt everything: one person tried a tool, told a colleague, and now there are six subscriptions nobody's tracking, three overlapping use cases, and no measurement of whether any of it is working.
According to a May 2025 Reimagine Main Street survey of 947 small businesses, run with 14 partner organizations including the National Small Business Association and released in June 2025, 51% of small business owners are "AI explorers": testing or researching AI tools without having fully committed. A separate Salesforce survey found that 91% of SMBs using AI report a revenue boost, but when you dig into the methodology, "revenue boost" is self-reported and rarely tied to specific tools or workflows. The gap between "we're using AI" and "we know what AI is doing for us" is where most businesses live right now.
That gap is expensive. The subscriptions themselves are usually cheap (most individual subscriptions are under $50/month); the cost comes from the hidden costs unaudited tools create: redundant subscriptions, inconsistent outputs, security exposure from data pasted into unknown platforms, and the opportunity cost of using mediocre tools when better options exist.
The same Reimagine Main Street survey found that 34% of explorers don't yet see a clear use case or return on investment, and 74% would adopt with clearer evidence of that return. An audit is how you produce that evidence for your own business, from your own tools and your own numbers.
The fix is straightforward. Run an audit. Here's how.
The short version: to audit what AI is actually doing, you list every AI tool your people use, score each one on time saved, quality, cost and risk, and decide whether to double down, investigate or cut. The first audit takes an afternoon. After that, it's about an hour a quarter.
List every AI tool, including free ones and the AI features inside software you already pay for.
Record who uses it, for what task, what it costs and what it replaced.
Score each tool from 1 to 5 on time saved, quality, cost and risk.
Check what each tool can read, where its data goes and who approved it.
Sort the tools into double down, investigate or cut, then put the next audit on the calendar.
Step 1: Build the Inventory
You can't evaluate what you haven't listed. The first step is identifying every AI tool in use across your organization, and this is harder than it sounds, because people adopt AI tools individually, often without telling anyone. A Census Bureau study of November 2025 to January 2026 survey data found that in 29% of US firms with any AI use, the use came only from workers' tasks, with no formal adoption reported by the firm.
In nearly three in ten US firms using AI, the only use the Census Bureau found was by workers, which is the use an inventory has to ask about.
How AI shows up in US firms that use it (U.S. Census Bureau, CES Working Paper 26-25R, The Microstructure of AI Diffusion (Bonney, Breaux, Dinlersoz et al.), October 2026 revision of the April 2026 paper; BTOS AI supplement, reference period Nov 2025 to Jan 2026)
Where AI use was reported
Firms with any AI use (%)
Firm and workers
61%
Workers only
29%
Firm only
10%
Send a simple question to everyone in the company: "What AI tools do you use for work? Include anything you've tried in the last 90 days, even if you stopped using it." You'll get a mix of ChatGPT, Claude, Gemini, Copilot, Jasper, Grammarly, Otter, Fireflies, Midjourney, and tools you've never heard of. That's the point.
For each tool, capture five things:
Tool name and which plan (free, pro, team, enterprise)
Who uses it: by name, not department
What they use it for: specific tasks, not categories
What it costs: monthly, per seat, or usage-based
What it replaced: the previous method for that task (manual, different tool, outsourced, not done at all)
That last column is the one people skip, and it's the most important. You can't measure ROI without a baseline. If the AI writing assistant replaced an hour of drafting per day, that's measurable. If it replaced "nothing, we just didn't do this before," that's a different kind of value and needs to be evaluated differently.
The inventory is the foundation when you audit what AI is actually doing. Most businesses discover they have more AI tools in use than anyone realized, and more overlap than anyone intended.
Step 2: Score Each Tool on Four Dimensions
Once you have the inventory, score each tool on a simple 1–5 scale across four dimensions. Don't overcomplicate this. Gut-informed scoring from the person who uses the tool daily is more accurate than a committee-designed rubric nobody fills out honestly.
Time saved
Does this tool measurably reduce time on the task it's used for? A 5 means "cuts the task by 75% or more." A 1 means "takes about the same amount of time, just distributed differently." Be honest. Some AI tools create a feeling of productivity without actually reducing total time, especially tools that generate output you then have to heavily edit. In Gallup's August 2026 survey, 52% of frequent AI users said they spend less time gathering information since using AI, while 21% spend more time reviewing and correcting work.
Reviewing and correcting work is the task where time saved and time added nearly cancel out: 23% of frequent users spend less time on it and 21% spend more.
Where frequent AI users say they spend less time (Gallup, "How AI Is Changing How Employees Spend Time at Work", Nov 16, 2026; web survey of 21,648 employed US adults on the Gallup Panel, Aug 5 to 19, 2026)
Task
Frequent AI users spending less time (%)
Searching for or gathering information
52%
Writing or drafting
47%
Analyzing data
38%
Manual or administrative tasks
29%
Reviewing and correcting work
23%
Learning new skills and tools
18%
Quality impact
Is the output better, worse, or the same as the previous method? Quality is subjective, so define it for each task. For email drafts, quality might mean "tone-appropriate and factually accurate." For data analysis, it might mean "catches patterns a human would miss." For content generation, it might mean "requires less than 20% editing." A tool that saves time but degrades quality isn't a net win.
Cost
What does this tool cost relative to the value it produces? Include the subscription cost, but also the time cost of managing it, the learning curve for new users, and any integration work. A $20/month tool that saves 10 hours is excellent. A $200/month tool that saves 10 hours might still be worth it, but the math is tighter. A $20/month tool that saves 30 minutes and nobody uses consistently is just a recurring charge.
Risk
What's the exposure from using this tool? Risk covers data handling (where does the input data go?), accuracy (what happens when the output is wrong?), and dependency (what breaks if this tool disappears?). A meeting transcription tool that stores recordings on its servers is higher risk than a local grammar checker. A coding assistant that generates plausible-looking but subtly broken code is higher risk than one that generates spreadsheet formulas you can verify. Score inversely: a 5 means low risk, a 1 means significant unmitigated exposure.
The U.S. Chamber of Commerce Foundation's Main Street AI Monitor, an Ipsos survey of 1,070 small business workers in May 2026, found that 41% point to a skills gap as a barrier to using AI, second only to privacy and security concerns at 47%. The same share, 41%, said it's unclear how AI applies to their specific business. Scoring tools this way surfaces a specific version of that gap: your team might have the skills to use a tool but not the skills to evaluate whether it's working. The audit closes that gap by making evaluation explicit.
The two barriers after privacy, an unclear use and a skills gap, are the ones a tool-by-tool score answers directly.
What small business workers say holds back AI use (U.S. Chamber of Commerce Foundation, Main Street AI Monitor (with Ipsos), June 17, 2026; survey of 1,070 small business workers, May 2026)
Barrier
Workers citing the barrier (%)
Privacy or security concerns
47%
Unclear how AI applies
41%
Skills gap
41%
Step 3: Sort Into Three Buckets
With scores in hand, every tool falls into one of three categories. The scoring makes this obvious: you don't need a framework, you need honesty.
Double down
Tools scoring 4–5 across most dimensions. These are doing real work, saving real time, and the risk is managed. The action here is to invest more: train more people, integrate them deeper, and explore premium tiers if the free version is the bottleneck. If you've already figured out where AI fits in your business, these are the tools that proved it.
Investigate
Tools with mixed scores: strong on some dimensions, weak on others. A tool that saves significant time but scores a 2 on quality needs a different workflow, not removal. Maybe the person using it needs better prompting skills. Maybe it's being used for the wrong task. Maybe a competitor tool handles that specific use case better. These tools get a 30-day focused evaluation: define what "working" looks like, measure it, and decide.
Cut
Tools scoring 1–2 across multiple dimensions. Cancel the subscription, remove the integration, and move on. Sunk cost is real (people resist cutting tools they spent time learning), but a tool that doesn't save time, doesn't improve quality, and carries risk is overhead rather than an asset.
Identifying which tools to cut is the easier part; actually cutting them is harder. People develop attachment to tools they invested time in, even when the evidence says the tool isn't working. This is where the audit needs teeth: someone with authority has to enforce the results.
The three-bucket sort is where the audit becomes actionable. Most businesses find some of their AI tools belong in the "cut" category.
Step 4: Check for Gaps and Overlaps
The inventory almost always reveals two things: tasks where multiple people use different AI tools for the same purpose, and tasks where AI could help but nobody's tried it.
Overlaps are easy to fix. If three people use three different transcription tools, standardize on one. The consolidation saves money, simplifies IT management, and means the team can share tips and workflows instead of each operating in isolation. This is the same logic behind evaluating AI tools without getting sold: the best tool for your team isn't necessarily the best tool on the market. It's the one that fits your workflow, your budget, and your risk tolerance.
Gaps are more interesting. The audit process itself generates them. When one team member describes how AI handles their meeting notes, another realizes they've been doing that manually. When someone shows how they use AI for data cleanup, a colleague with a messier spreadsheet problem sees the application. Along with the evaluation, the audit works as internal knowledge transfer.
To turn those gaps into something you can act on, list the recurring tasks nobody has tried AI on yet and score them before anyone buys a new tool. Qurtoo's task inventory playbook scores each task on how often it happens, the risk of a mistake and how easily the output can be checked, then picks three low-risk pilots. That's the opportunity side of the audit, and it's usually where the next quarter's double-down list comes from.
Step 5: Schedule the Next One
An AI audit isn't a one-time event, because the tools, your team's skills and the use cases all keep changing. Run this quarterly: same spreadsheet, same four scoring dimensions, updated inventory. The cadence matters because AI tools improve rapidly. A tool that scored a 2 on quality six months ago might score a 4 today. A tool that was best-in-class in January might have three better competitors by July.
Quarterly also matches the pace of subscription renewals. Most AI tools bill monthly or annually. A quarterly audit means you catch underperforming tools before the annual renewal and have data to negotiate enterprise agreements for the tools that prove their value.
The businesses getting real value from AI are the ones who know exactly what each tool does, what it costs, and whether it's earning its place, however many tools they have. That knowledge comes from measuring, and vendor marketing can't supply it.
The whole strategy is to build the spreadsheet, score the tools, cut what doesn't work, double down on what does, and do it again in 90 days.
What the Audit Reveals About Your Organization
Beyond the tool-level decisions, the audit produces a higher-order insight: how your organization actually adopts technology. If the inventory shows one person using eight tools and everyone else using zero, that points to how the team works more than to the technology. If every tool in use is free-tier, that might mean the team is cost-conscious, or it might mean nobody has enough conviction in a tool to request budget for it.
These patterns inform more than AI strategy. They tell you how your team evaluates new tools, how information spreads internally, and where decision-making authority sits in practice versus on paper. The same discipline that makes a good AI audit makes a good system-building practice: alongside managing tools, you're building an organizational habit of evaluating and adopting technology deliberately, and that habit is worth more than any individual tool.
Quarterly audits compound. Each cycle, the tool stack gets tighter, the team gets sharper, and the ROI gets easier to measure.
What is an AI audit, and which kind does a small business need?
An AI audit is a structured review of where AI is used in your business, what it costs, what it can reach and whether it produces the results people assume. The five steps above are the version most small businesses need first: a usage audit. The U.S. Chamber of Commerce's 2026 Empowering Small Business report says two in three small businesses (66%) now use AI, so for most owners the question has moved from whether to use it to what it's doing.
The phrase "AI audit" covers several different jobs, and it helps to know which one you mean before you pay anyone or block out a day:
Usage audit: which tools people use, for which tasks, and whether each one earns its place. This is the audit in the steps above.
Spend audit: what you pay for AI across every card, expense report and invoice, broken down by team.
Data and access audit: what each tool can read, where your data goes, and whether that meets your obligations under Massachusetts law. People sometimes call this an AI safety or security audit.
Output audit: whether the drafts, answers and decisions AI produces are accurate, checked by a person and logged.
Visibility audit: what ChatGPT, Gemini, Copilot and other assistants tell people about your company.
Opportunity audit: where in your workflows AI could save time that nobody has tried yet.
Most small businesses need the usage, spend and data audits every quarter, and an output audit for any AI that touches customers or money. A formal third-party audit against a published standard makes sense when a customer, an insurer or a regulator asks for one. My guide to an AI audit for business covers when that outside review is worth it.
How companies audit AI spending across teams
AI spending is hard to see because it rarely arrives as one line item. It shows up as $20 personal subscriptions on company cards, as seats added to a plan someone set up two years ago, as usage-based bills that rise with volume, and as AI add-ons folded into software you already pay for. To audit AI spending by team, pull from every place money leaves the business, then group it.
Export twelve months of charges from every company card, the bank account and your expense tool. Search the descriptions for AI vendors and for words like "AI", "GPT", "copilot" and "assistant".
Pull reimbursed expenses separately. Personal subscriptions that get expensed are where most unlisted AI tools turn up.
Open the billing page of each software suite you pay for (Google Workspace, Microsoft 365, your CRM, your help desk) and note any AI add-on, tier upgrade or credit pack.
For usage-based tools, export the last three months of usage so you can see the trend, as well as the latest bill.
Tag every line with the team and the person who approved it. A charge with no owner goes on the investigate list automatically.
The result is one sheet: tool, team, owner, monthly cost, billing type and renewal date. Put it next to the inventory from Step 1 and the gaps show quickly: tools that are billed but nobody listed, and tools people listed that nobody pays for (usually a free tier holding company data). Brian's guide to finding your software spend per employee in an afternoon walks through the same card-statement method for every subscription, and it pairs well with the AI spend audit.
Renewal dates matter more than the totals. A spend audit that lands a month before an annual renewal gives you time to cut seats, move to a cheaper tier or negotiate. One that lands the week after the charge means another year of paying before you can act on what you found.
How to audit what AI tools can see and where your data goes
Every tool in the inventory has an access question attached to it: what can it read, what does it keep, and who said yes. This is the part of the audit that protects customers, and it's the part most often skipped, because it means opening admin consoles instead of asking people.
Start with connected apps. Google Workspace and Microsoft 365 both let an administrator see which third-party apps people have granted access to their mail, calendar and files. Meeting note takers, writing assistants and AI agents often ask for broad permissions, such as reading all mail or all files, when someone connects them. List each grant, the account that approved it and what it can reach. A personal account holding a grant to company data is a finding on its own.
Then answer five questions for each tool, from the vendor's own documentation or from the vendor directly:
Is our input used to train the vendor's models, and is that setting on or off for our plan?
How long are prompts, files and recordings kept, and can we delete them?
Where is the data stored, and which subprocessors can see it?
Can we remove one person's access without breaking everyone else's work?
Is there a log of what the tool did, and can we export it?
People reach for unapproved tools when the approved path is slower. The LTFI guide to shadow AI in a small business explains how to find those tools by asking without blame, then give people an approved option that's easier than the workaround. That's what keeps the data and access audit honest from one quarter to the next, because people keep reporting what they use.
In Massachusetts, this part of the audit carries legal weight if any tool touches personal information about state residents. Under 201 CMR 17.00, section 17.03(2)(f) requires a business to oversee service providers by "taking reasonable steps to select and retain third-party service providers that are capable of maintaining appropriate security measures to protect such personal information" and by "requiring such third-party service providers by contract to implement and maintain such appropriate security measures for personal information." Section 17.03(2)(i) requires "reviewing the scope of the security measures at least annually or whenever there is a material change in business practices that may reasonably implicate the security or integrity of records containing personal information." Connecting an AI tool to customer records is the kind of change that clause describes. My guide to Massachusetts WISP requirements explains where AI tools fit in your written program.
How to audit AI output and AI-assisted decisions
Scoring a tool on quality in Step 2 tells you how it performs on average. Auditing the output tells you what it did on specific days, for specific customers. You need that for any AI that drafts customer replies, summarizes contracts, sorts applicants, approves refunds or touches money.
A workable output audit for a small team has four parts:
A log. Every AI-assisted action that leaves the building is recorded with the input, the output, the time and the person who approved it. If the tool can't produce that log, keep a simple one yourself for the tasks that matter most.
A sample. Each month, pull 20 items at random and have someone who knows the work check them against the source. That's small enough to get done and large enough to show a repeating problem.
An error list. Every mistake found goes on one list with what went wrong and what changed afterward, so the same error isn't found twice.
An edit rate. Track how often a person changes or rejects what the AI produced. A rising edit rate is often the first sign that a model, a prompt or the incoming data has changed.
The edit rate is where Gallup's finding shows up in practice: 21% of frequent AI users spend more time reviewing and correcting work. If your output audit shows people rewriting most of what a tool drafts, the time saved you scored in Step 2 is smaller than it looked. AI behavior can also shift over time without anyone touching your setup, which I cover in how to catch AI agent drift.
The same records make AI-enabled work auditable for someone else, whether that's an insurer, a customer's security questionnaire or your accountant: a log with a person's name on every approval, and a record of what you checked and when.
How to audit what AI assistants say about your business
A different kind of AI audit looks outward: what ChatGPT, Gemini, Copilot, Claude and Perplexity tell people who ask about your company, your services or your industry. Customers increasingly start there, and the answers can be out of date or simply wrong.
Write ten questions a real customer would ask: who offers your service in your town, what it usually costs, whether you're reputable, what you're known for.
Ask each assistant the same questions in a fresh session, signed out where you can, and save the full answer and any sources it cites.
Mark each answer as correct, incomplete or wrong, and note which pages (yours or someone else's) it relied on.
Fix the source rather than arguing with the answer. If an assistant repeats an old address, the old address is still published somewhere it trusts.
Repeat each quarter with the same questions, so you can see whether the answers move.
Teams that publish often may want to check monthly. For most small businesses, quarterly matches the usage audit and catches a problem before it spreads. How assistants choose what to cite is its own topic, covered in how AI assistants decide which businesses to recommend.
An AI audit checklist you can copy
Here's the whole audit on one page. Add it to the spreadsheet from Step 1 as a second tab and check it off each quarter.
Inventory question sent to everyone, including "anything you've tried in the last 90 days".
AI features inside existing software listed (email, documents, CRM, help desk, phones).
Each tool has an owner, a task, a cost, a plan level and a note on what it replaced.
Each tool scored 1 to 5 on time saved, quality, cost and risk by the person who uses it daily.
Card, bank and expense exports searched for AI charges, with every charge matched to a tool and a team.
Renewal dates recorded for every paid tool.
Connected-app grants reviewed in your email and file admin consoles.
Training, retention and storage settings confirmed for every tool that sees customer or employee data.
Output sample checked for any AI that writes to customers or touches money.
Assistant answers about the business checked and saved.
Double down, investigate and cut lists signed off by someone with the authority to cancel.
Next audit date on the calendar.
For a reference point on the governance side, the NIST AI Risk Management Framework organizes AI risk work into four functions (Govern, Map, Measure and Manage). Its version 1.0, published in January 2023, is still the current edition while NIST works on a revision. ISO/IEC 42001:2023 is the international standard for an AI management system, and it's what a formal certification audit measures against. Neither is required for a small business, but both rest on the same building blocks as this checklist: an inventory, named owners, measured results and a regular review.
You can't improve what you don't measure. Output logging on every AI-powered feature in production is the minimum: when something generates a bad result, you need to trace what happened and why.
The audit isn't a one-time event. AI systems drift as their inputs change. Quarterly reviews of output quality, accuracy rates, and edge case handling are the operational discipline that keeps AI trustworthy.
Frequently asked questions about how to audit what AI is actually doing
How long does a full AI audit take?
The first one takes 2–4 hours, depending on company size. Most of that time is collecting the inventory; once people report what they're using, the scoring and sorting move quickly. Subsequent quarterly audits take about an hour because the spreadsheet already exists and you're updating rather than building from scratch.
What if my team resists reporting which AI tools they use?
This usually means people are worried they'll be told to stop using a tool they find helpful, or they're concerned about being judged for using AI at all. Frame the audit as optimization, not enforcement. The goal is to find what's working and invest more in it, not to police individual tool choices. Anonymizing the initial survey can help, though you'll need names eventually to consolidate overlapping tools.
Should I audit free AI tools, or only paid ones?
Audit everything, including free tools. Free tools still carry costs: time spent using them, data exposure from what's pasted into them, and the opportunity cost of using a free tier when a paid tier would be significantly more effective. Free also doesn't mean permanent. If a tool your team depends on changes its free tier or shuts down, you need to know about that dependency before it becomes a disruption.
How do I handle AI tools embedded in software we already use (like AI features in Google Workspace or Microsoft 365)?
Include them in the audit, but score them differently. Embedded AI features don't carry a separate subscription cost, so the cost dimension is effectively zero. Focus the scoring on whether people actually use the feature, whether it improves the task, and whether the data handling meets your standards. Many embedded AI features are turned on by default and used by no one; that's low-risk, but it's still worth knowing. Others are used daily and quietly handling sensitive data in ways the value-versus-cost calculation should account for.
How much does an AI audit cost?
An internal AI audit like the one above costs staff time: about 2 to 4 hours for the first one and about an hour a quarter after that, plus the few minutes each person spends answering the inventory question. An outside AI audit is priced by scope. A usage and spend review for a small team is a short engagement. A formal audit against ISO/IEC 42001 or a customer's security requirements takes longer and costs more, because it needs evidence, interviews and a written report. Get the scope in writing before you agree to a price. I break down how consultants charge for this kind of work in what it costs to hire an AI consultant.
Can AI do an audit?
AI can do parts of an audit well, and a person still has to own the result. It handles the clerical work: searching a year of card statements for AI vendors, grouping charges by team, turning a pile of survey replies into one inventory table, and summarizing vendor privacy terms so you know which paragraph to read closely. It's weaker at judgment: deciding whether a tool's output is good enough, whether a data grant is acceptable, or which tool to cut when two people depend on it. Check every AI-made summary against its source before it goes into the audit.
How often should you audit AI answers for accuracy?
For AI that writes to customers or makes decisions about money, check a sample every month. For what public AI assistants say about your business, quarterly is enough for most small businesses, and monthly makes sense if you publish often or change your services. Whatever the cadence, use the same questions or the same sample size each time so one audit can be compared with the last.
How do I know if our AI spend is actually producing anything?
Compare each tool's cost with the baseline in the "what it replaced" column from Step 1. If a tool replaced an hour of drafting a day and the person using it confirms it still does, you can put a number on it. If nobody can say what the tool replaced or what changed after it arrived, the honest answer is that you don't know yet, and the tool goes in the investigate bucket for a 30-day measured trial.
Do AI monitoring tools actually show business impact?
Most monitoring and admin dashboards show activity: logins, prompts, seats in use, credits consumed. Activity tells you a tool is being used, which is useful for finding seats nobody touches. It doesn't tell you whether the work got faster or better. Business impact comes from comparing that activity with the baseline you recorded in the inventory and with the output sample, which is why the audit needs both the dashboard and the people who do the work.
Which AI audit is most suitable for a small business?
Start with the usage audit and the spend audit, because they're quick and often pay for themselves through cancelled subscriptions. Add the data and access audit if any tool sees customer, employee or financial records, which in Massachusetts brings 201 CMR 17.00 into play. Add an output audit for any AI that talks to customers. A formal standards-based audit can wait until a customer, an insurer or a regulator asks for one.
How do I audit what AI is actually doing across my whole organization?
Start with the people, then check the records, because neither view is complete on its own. Ask everyone the inventory question from Step 1, then compare the answers with three things you already have: card and expense exports showing AI charges, the connected-app grants in your email and file admin consoles, and the AI features switched on inside software you already pay for. Anything that shows up in the records but not in the answers is a conversation to have with that person, not a violation to punish. For AI that writes to customers or touches money, add the monthly output sample, so you see what it produced and not only that it ran.
How to audit AI spending without buying a tool first. Zylo's 2026 index found ChatGPT is now the most expensed app, so the audit starts in card statements and expense reports, then moves to API keys, owners and a monthly review.
AI pilots stall before production for boring reasons: no owner, no definition of done, no data path, no kill switch. CISA 1 May 2026: start low-risk. NIST AI RMF: Govern, Map, Measure, Manage.
AI agent governance fails when you approve each click and miss the path. Five in-policy refunds can still empty the drawer. Store intent, cap the run, gate pay/send/delete.