Crawler policy,
superseded.
@casoon/astro-crawler-policy generated robots.txt and llms.txt from typed configuration. The repository is archived. Its presets and bot registry live on in @casoon/astro-site-files.
Existing installs keep working but receive no more fixes and no new bots. New projects should start with astro-site-files.
Project status
Version 0.1.2
Last release on npm. Bot registry with 13 crawlers, dated 2026-04-09.
Merged and archived
Presets and registry move to astro-site-files. The README points to the successor and the repository is read-only.
astro-site-files 0.5.0
robots.txt, llms.txt, sitemap.xml, rss.xml, security.txt and humans.txt in one integration.
What the package did.
In astro:build:done the integration wrote a robots.txt from typed configuration and, optionally, an llms.txt describing the site’s AI policy. It generated rules; it did not block anything at network, WAF or edge level. This page stays online so existing users can find the current state and the way to the successor.
- 01Presets — seoOnly, citationFriendly (default), openToAi, blockTraining, lockdown
- 02Bot registry — 13 known crawlers with group and per-bot rules
- 03Content signals — a
Content-Signalline following contentsignals.org - 04Audits — warnings about misconfiguration during the build
Old and new side by side.
| Criterion | astro-crawler-policy | astro-site-files |
|---|---|---|
| Maintenance | archived | active |
| Latest version | 0.1.2 (April 2026) | 0.5.0 (September 2026) |
| Astro (peer dependency) | 4 and later | 6 and later, incl. 7 |
| Bots in the registry | 13 (as of 2026-04-09) | 25 (as of 2026-05-27) |
| Presets | 5 | the same 5, plus groups for SEO scanners and archives |
| llms.txt | summary of the AI policy | content overview following llmstxt.org |
| Content-Signal line | yes | no |
| Merge with public/robots.txt | replace, prepend, append | no |
| Per-environment overrides (env) | yes | via JavaScript in the config |
| sitemap.xml, rss.xml, security.txt, humans.txt | no | yes |
As of September 2026, per the README and npm entries of both packages. Switching drops the Content-Signal line, the merge strategy and the env option.
Switch over in three steps.
- 1Swap
Replace the package
Remove the old package and install the successor. It needs Node.js 22.12.0 or later and Astro 6 or later.
npm rm @casoon/astro-crawler-policy npm i @casoon/astro-site-files - 2Move
Update the config
The preset name stays the same and moves under
robots. astro-site-files derives the sitemap line fromsite.robots: { preset: 'citationFriendly' } - 3Check
Compare the output
Diff the old and new robots.txt: the
Content-Signalline is gone and more bots are listed. If you also use@astrojs/sitemap, turn off one of the two sitemaps.diff robots.old.txt dist/robots.txt
Same intent, new integration.
// astro.config.ts – @casoon/astro-crawler-policy
import { defineConfig } from 'astro/config';
import crawlerPolicy from '@casoon/astro-crawler-policy';
export default defineConfig({
site: 'https://example.com',
integrations: [
crawlerPolicy({
preset: 'citationFriendly',
sitemaps: ['/sitemap-index.xml'],
}),
],
});// astro.config.ts – @casoon/astro-site-files
import { defineConfig } from 'astro/config';
import siteFiles from '@casoon/astro-site-files';
export default defineConfig({
site: 'https://example.com',
integrations: [
siteFiles({
// same preset name, now under robots
robots: { preset: 'citationFriendly' },
// sitemap.xml is generated by default and
// referenced in robots.txt automatically
}),
],
});# Generated by @casoon/astro-crawler-policy
# preset: citationFriendly
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /What version 0.1.2 does.
Five presets
Each preset sets group rules and content signals. The default is citationFriendly: search engines and citing AI bots allowed, pure training crawlers blocked.
Layered rules
Preset, then groups, then bots. The most specific rule wins. Your own crawlers can be added via extraBots.
Content signals
search, ai-input and ai-train as a line in the wildcard group. Google Search Console may flag it as an unknown directive.
Merge strategies
An existing public/robots.txt is replaced, prepended or appended.
Environments
With env, staging and preview get their own settings, for example lockdown. The environment is read from CONTEXT, DEPLOYMENT_ENVIRONMENT or NODE_ENV.
Build audits
Hints such as MISSING_SITE_URL, UNKNOWN_BOT_ID or UNLOCKED_NON_PRODUCTION_ENVIRONMENT appear in the build log.
Why switching makes sense.
Registry frozen
The bot list is dated 9 April 2026. New crawlers are only added to astro-site-files.
No more fixes
The repository is archived. Problems with new Astro versions will not be fixed.
Content-Signal is not a standard
The directive follows a proposal (IETF aipref). Many crawlers do not read it.
robots.txt is voluntary
That applies to both packages. Crawlers that ignore it can only be blocked at server or CDN level.
Frequently asked questions.
Is astro-crawler-policy still maintained?
No. The repository is archived and the latest npm release is 0.1.2 from April 2026. New features and fixes only land in the successor, @casoon/astro-site-files.
Will an existing install keep working?
The package can still be installed from npm and still generates robots.txt and llms.txt. The bot registry is no longer updated, though, and problems with new Astro versions will not be fixed.
How do I migrate to astro-site-files?
Swap the package, then replace crawlerPolicy({ preset }) with siteFiles({ robots: { preset } }). The preset names are the same. astro-site-files also generates its own sitemap.xml and references it in robots.txt.
What do I lose when switching?
The Content-Signal line, the merge strategy for an existing public/robots.txt and the env option. Environment-specific settings can be expressed in astro-site-files with plain JavaScript in the Astro config.
What are content signals?
Machine-readable statements about search, ai-input and ai-train following contentsignals.org, a proposal within IETF aipref. They only have an effect on crawlers that read them and may show up in Google Search Console as an unknown directive.
Start new projects with astro-site-files.
Same presets, a larger bot registry, plus sitemap, RSS, security.txt and humans.txt.