Import a website as knowledge
Scan a website for free, choose how many pages to bring in, and review every generated document before it becomes agent knowledge.
Point KlicForge at a website and it reads the pages, writes a set of knowledge documents from what it found, and lets you review each one before anything becomes agent knowledge. It is one of three ways to create a knowledge source, alongside writing content directly and uploading files.
Scanning a website
From Knowledge Library, select Add Source → Import a website, enter the site's address, and select Scan website.
The scan finds the site's pages and reports how many there are, without importing anything or costing you anything — you always see the size and price of an import before you commit to it. By default it stays on the site's main domain and skips blog and news pages; both are toggles you can turn on before scanning if you want them included.
Choosing how many pages to import
After a scan, pick how many pages to bring in — KlicForge suggests round numbers (10, 25, 50, 100) based on the site's size and your remaining monthly allowance, or you can import every page found. Whatever you choose is the most you can be charged; pages the site fails to return are never charged.
Select Start import to begin. KlicForge reads the selected pages and organizes them into documents — this takes a few minutes for a large site.
A scan is only good for 24 hours. If you wait longer than that before starting the import, scan the site again.
What gets imported — and what doesn't
| Included by default | Excluded |
|---|---|
| The homepage, always | Cart, checkout, login, search and other transactional pages |
| Ordinary content pages on the site's main domain | Subdomains — turn on Allow subdomains to include them |
| Blog and news pages — turn on Include blog to include them | |
| Linked PDFs and other documents — use Upload Files for these |
Only one import can be running for a given website at a time — scanning a site that already has an import in progress is blocked until it finishes.
Reviewing before it becomes knowledge
Once KlicForge finishes organizing what it read, each generated document appears as a draft for you to review — nothing is searchable by an agent yet. For each one you can:
- Edit its title, content, category or tags before deciding.
- Add it, which turns it into a real knowledge source.
- Discard it, which throws it away.
You can also add or discard everything at once. Discarding is final — getting a document back means running the import again.
Documents created this way are treated as ordinary knowledge — visible to any agent you grant the source to, including public-facing ones. Review content before adding it if the site has pages you would not want an agent repeating.
If the documents don't look right — topics grouped oddly, or split in a way that doesn't match how you'd expect to search for them — edit the ones worth keeping, discard the rest, and import the site again if you still need what you threw away. A fresh import reads the pages again, so it is charged again; edit before you discard where you can.
Importing the same site again
Sites change, so you can re-import one you have already brought in. KlicForge compares what it just read against the documents your earlier import produced for that same site, and tells you what each one will do before you decide:
| Label | What it means |
|---|---|
| (no label) | Nothing like it exists yet — Add creates a new document |
| Updates “…” | It matches one you already have — Update refreshes that document in place |
| Unchanged since last import | The page has not changed — Dismiss it, there is nothing to write |
| You edited “…” | You changed that document yourself after importing it |
Updating refreshes the existing document rather than adding a second copy of it, so agents already granted that knowledge keep working and your library does not fill up with near-identical pages. Add All applies every recommendation in one go.
Where you have edited a document yourself, KlicForge never overwrites it silently — you choose Keep mine or Use new version, and Add All leaves those for you rather than deciding.
Documents from the earlier import that nothing matched this time are listed under No longer on the site. They are not removed automatically: a page missing from one crawl may simply not have been reached. Remove them yourself once you are sure.
A re-import reads and charges for the pages again — the comparison happens after the pages are read, so it saves you a cluttered knowledge library rather than interactions.
What you're billed for
Scanning a site, choosing a page count, and reviewing documents are all free. You are charged one interaction per page actually imported — pages that fail to load, get skipped, or turn out to be duplicates of a page you already have are never charged. See interaction counting for the full billing rules and worked examples.