Because AI does not open your website the way you do. It reads the code, not the design, and most agency sites fail one of six checks before the machine ever gets to your content. Here they are in the order they happen, with what our audits found for each.
Last updated: September 23, 2026
Check one: can the crawler read anything at all?
This is the one that ends the conversation before it starts. A lot of modern sites, and most page builders, ship a page that is empty until the browser runs JavaScript to fill it in. You never notice, because your browser runs the JavaScript. Many AI crawlers do not, because rendering is expensive and they have a billion pages to get through. They fetch the page, see nothing, and move on.
Of the agency sites we have audited with the readability check, 13 percent came back empty. Every one of them was a custom or unidentifiable build; none of the named insurance-website vendors shipped one. The sites the crawler could not read scored a median of 31 on the audit. The readable ones scored 50. The gap is not a coincidence, it is the machine telling you it could not see you.
Your name, address, phone, and services have to live in basic HTML. If you do not know whether they do, the audit tells you in about three minutes, and it is the first thing on the report if the answer is no.
Check two: is the crawler even allowed in?
Every site has a small file called robots.txt that tells crawlers what they may read. Privacy and security plugins, especially on WordPress, add a blanket rule that blocks GPTBot, ClaudeBot and PerplexityBot by default. The agency never asked for it and nobody knows it happened.
A blocked crawler cannot cite you. Perplexity in particular is extremely sensitive to this; block its bot and you are gone from that engine entirely. The audit reads your robots.txt and names any AI crawler that is blocked. The fix is a two-line edit and it is the single highest-impact item on the whole list when it applies.
Check three: does your site say what you are, in code?
Schema markup is a block of code that tells a machine, in its own language, what a page is about. There is a specific type for insurance agencies, called InsuranceAgency. Only 35 percent of the sites we audit use it. Another third have some schema, usually whatever the builder added by default, and a third have none.
Without it, the machine has to guess whether Smith and Sons is an insurance agency, a law firm, or a family history project. It does not guess in your favor; it prefers a business it can identify with confidence. This is a developer task, an hour or two, and it moves the needle more than any redesign.
Check four: is there a licensed human on the page?
Insurance is what the engines call your-money-or-your-life content. They apply a stricter trust filter to it than to almost anything else, and the filter starts with a simple question: is there a real, licensed person standing behind this advice?
Sixty-two percent of the sites we audit name at least one agent. Only 6 percent show a license number anywhere. A contact form with no names, no faces, and no credentials scores near zero on this test, and the machine will happily recommend the agency next door that lists its producers with their NPNs.
Check five: is there a clean answer to pull?
AI works in chunks. A single page that says “we offer auto, home, life and commercial” gives it nothing to extract. Each line you write needs its own page, with its own local detail and its own questions answered in the first sentence, because the machine quotes the sentence that answers the question and skips the paragraph of history in front of it.
The captives cannot write your county’s hail season or the wind deductible on the coast from a national headquarters. That specificity is the one thing an independent has that they do not. Generic is invisible. Specific is winnable.
Check six: do other sources agree you exist?
The last check is off your site entirely. Does your Google Business Profile match your website? Is there a BBB profile? Does the state association directory have your working web address? AI trusts what other sites say about you more than what you say about yourself, and when your Google profile says one name and your site says another, the machine sees two businesses and trusts neither.
Is this worth the trouble yet? On the four agency sites we track, AI assistants sent 8 completed quote requests between September and mid-October, 7 of them from ChatGPT. Small, but real, and the full source breakdown is in where insurance agency website leads actually come from.
What do you do about it?
Run the checks in order, because the order is the order the machine uses. Readability first, permissions second, schema third, people fourth, content fifth, validation sixth. The free audit runs the first five automatically and reports the sixth honestly. The checklist has the fix for each, and most of them are free.
If check one fails, stop reading the rest of the list. A site the crawler cannot read is not fixable by adding schema to it; it needs to be rebuilt in plain HTML. That is the one case where we will tell you a new site is cheaper than the old one, and it is the case our sites were built for.