Storefront Wording-Compliance Sweep

Storefront Wording-Compliance Sweep

Overview

A compliance review of copy fails when it reads only the pages someone remembers. This sweep crawls every public page of a live storefront, extracts the visible text, applies a fixed rule list, and makes a manual verdict on every candidate. Each flagged sentence gets its URL, its exact quote, the rule it breaks, and a rewrite that is checked against the same rules. The report is generated by script from the data, so its numbers cannot drift from the evidence.

When to Use

• Checking the live copy of a regulated or research-use-only store before a launch, an audit or a payment review.
• Re-running a sweep after content changes, to confirm the flags are gone and nothing new appeared.

Skip it for internal documents, for code review, and for the content-drafting step (use regulated-catalog-content for that).

Features

• Sitemap-driven crawl with a no-redirect opener, so gates appear as redirects rather than being followed.
• Text extraction limited to visible copy, image alternatives and structured-data name and description fields.
• A rule file with one identifier per category, matched per sentence, with a separate log for negated disclaimers.
• A manual verdict column on every candidate, and a script that refuses to run past an unreviewed match.
• Worst-first ranking per page, and a self-test that runs every rewrite through the same rules.

Benefits

• A clean result is reported as clean, with the count of pages and sentences examined.
• Negated disclaimers are preserved and listed, so nothing was suppressed without a reviewer seeing it.
• The report and the data agree, because the report is generated from them.

Procedure

1. Set the standing limits. Use GET requests only, with no cookies and no forms. Send one request at a time, with at least 0.6 seconds between requests. Send a descriptive User-Agent that names the tool and says the requests are read-only. Do not request paths that the robots file disallows, such as cart, account and checkout.
2. Find the live domain. Use the domain named in the request. If none is named, take it from the most recent published link or project note, and state where the domain came from in the report.
3. Fetch the sitemap and the robots file. Count the page locations in the sitemap. That count is the target. Some sites publish several sitemaps and only one lists product or certificate pages, and a second sitemap may return an HTML shell. Note any difference between the location host and the live host, and crawl the same paths on the live host.
4. Crawl every page. Use a client that does not follow redirects, so a gate appears as a redirect with its target. Save the status, the redirect target and the byte count for each request. Extract the title, the meta and Open Graph descriptions, the visible text with scripts, styles and inline graphics removed, the image alternative text, and the name and description fields in structured data.
5. Probe pages the sitemap does not list. Request learning, FAQ, method and search pages. If one redirects to a sign-in or age gate, record it as unreadable. Do not add it to the sitemap total. Say explicitly when a page type does not exist, rather than leaving it out.
6. Define the rule list in one file. Give each rule an identifier per category: health benefit, dosage, administration, human or animal use, testimonial, and any site-specific category such as preparation wording or name slugs that imply an outcome. Match each rule per sentence, not per page. Use word boundaries wherever a short word would otherwise match inside a longer one.
7. Separate negated disclaimers. A sentence that negates a claim, such as "not for human consumption" or "no dosing guidance", is matched by a separate negation pattern. Set it aside in a separate log, and list the distinct negated sentences in the report so a reviewer can confirm that nothing harmful was suppressed.
8. Give every candidate a manual verdict. Read each distinct line of the crawled text. Repeated boilerplate can be deduplicated first. Mark each candidate as a flag or as reviewed and not a violation, and give the reason. The script must raise an error on any match that has no reviewed reason, so a new match cannot pass unreviewed.
9. Write a rewrite for each flag. Ground each rewrite in text that already appears on the site, such as the page's own description. Run every rewrite through the same rule list. A rewrite that triggers a rule is a defect in either the rewrite or the rule, and must be fixed before it is reported.
10. Rank the results. Rank worst first for each page, using severity, rule weight, page type and the number of flagged sentences. Ranking by matched string lets a single page fill the top of the list.
11. Generate the report from the data. The report opens with the bottom line. It then gives the sitemap and crawl counts as command output, the counts by page type with zero rows included, the worst ten pages, the full rule file, the run output, the reasons for each false positive, observations outside the rules kept apart from the counts, and the limits: static HTML only, and not a legal review.
12. Check the sitemap did not change during the crawl. Fetch it again after the crawl, compare it with the copy saved at the start, and quote the result.
13. Ship the files as a draft. Add only the report, the data files and the scripts to a documentation folder. Open a draft pull request with a note that the build was skipped because only documentation changed. Never merge it.

Pitfalls

• Substring rules produce false positives. A short word matches inside a longer one. Use word boundaries.
• Path-level noise. A product slug that is a market nickname can imply an outcome even when the visible name is chemical. Flag it as a judgement call, with lower severity on certificate and listing pages, and name the fix as a rename that needs redirects and a sitemap refresh. The owner decides.
• Drug-name products and domain mismatches are real observations but fall outside the wording rules. List them separately, and do not add them to the hit count.
• Clone timeouts. A plain clone of a large monorepo can exceed a foreground timeout. Use a blobless, no-checkout clone with sparse checkout of the documentation folder, and run it in the background.
• Shell chains. When a command that runs several scripts or network calls is blocked by a guard, stop, report the blocked command, and ask the operator. Do not split it to get past the guard.
• Not a legal opinion. The sweep flags wording for review. It does not decide whether wording is lawful.

Verification

• The sitemap count and the crawled-page count are both quoted, and any difference is explained.
• Every flag has a verdict, a reason, a quote and a rewrite that passes the rule list.
• The script exits with an error when a match has no reviewed reason.
• The report's counts match the data files it was generated from.
• The sitemap comparison before and after the crawl is quoted.