Back to Blog
Data EngineeringData CollectionWeb CrawlingPage DiscoveryAI Data Pipeline

Designing an Intelligent Data Collection System (Part 2): Page Discovery and How to Continuously Find High-Value Pages

Long-term data collection should not traverse an entire site from a single entry point without limits. This article explains the role of the page discovery stage and how to identify, classify, and continuously maintain high-value pages, creating a stable input for crawling, snapshots, and structured extraction.

RaytonX
9 min read

In the previous article, we split long-term data collection into a complete data pipeline:

Source entry
→ Page discovery
→ Page crawling
→ Content snapshot
→ Structured extraction
→ Data merging
→ Human review
→ Production data

Page discovery is the first step where the system really starts to understand a target website.

The problem it solves is not how to download a page, but:

Which pages on a website are worth managing and collecting over the long term?

If this step is handled poorly, crawling, storage, and AI parsing downstream will all be dragged down by large amounts of low-value content.


Why Not Just Crawl the Entire Website

Recursively following every link from the home page is the most intuitive approach.

But real websites usually contain plenty of content unrelated to the target data, such as:

  • News and event archives
  • Tag pages and pagination
  • Login entries
  • Privacy policies
  • Social media links
  • Duplicate language pages
  • Images, scripts, and other static assets

If the system crawls the whole site by default, several problems soon appear:

  • The number of pages grows quickly
  • The same content appears under multiple URLs
  • Large numbers of low-value pages enter the parsing flow
  • Request pressure on the website increases
  • Storage and model costs get out of control

Long-term data collection is therefore better served by discovering and filtering pages first, then deciding which ones enter the rest of the flow.


What the Page Discovery Stage Does

Page discovery starts from a known website entry, extracts candidate links from the page, and identifies high-value pages among them.

It is usually responsible for:

  1. Visiting the website entry
  2. Extracting navigation, body, and resource links
  3. Filtering out clearly irrelevant URLs
  4. Normalizing candidate links
  5. Determining page types
  6. Writing valid pages to the page management table

It is not responsible for:

  • Deep parsing of business fields
  • Determining whether page content has changed
  • Calling an LLM
  • Merging data from multiple pages
  • Updating production data

These boundaries let page discovery rules evolve independently without affecting crawling and parsing logic.


What Counts as a High-Value Page

There is no universal definition of a high-value page. It depends on the specific data goal.

Structurally, however, these pages usually have the following characteristics:

  • They contain stable factual information
  • They are directly related to the target entity
  • Their updates have business meaning
  • They can support structured field extraction
  • They are worth rechecking periodically

For example, common high-value pages on a public organization's website might include:

  • Home page
  • About page
  • Product or service pages
  • Pricing page
  • Contact page
  • Application or process guides
  • PDF documents
  • Public downloads

By comparison, news, event, tag, and archive pages usually have lower structured value, unless the system exists to collect that content.

The goal of page discovery is not to find every page on a website, but to build a maintainable set of pages relevant to the business goal.


How to Identify High-Value Pages

Page type classification usually does not rely on a single signal. It combines several sources.

Common signals include:

  • URL path
  • Link text
  • Page title
  • File type
  • Page depth
  • Position in the navigation
  • Whether it belongs to the target domain

For example, all of these can help determine a page's purpose:

/pricing
/fees
/contact
/about
/products
/downloads
/document.pdf

Link text matters just as much.

Even when a URL has no obvious meaning, text like "Pricing", "Contact Us", or "Download Brochure" can provide a strong classification signal.

In real projects, a common approach is to do an initial classification with explainable rules, rather than relying on an LLM to judge every link from the start.

The reasons are simple:

  • Rules cost less
  • They run faster
  • Their results are easier to explain
  • Page discovery usually does not need deep semantic understanding

Pages that cannot be classified reliably can be marked as unknown instead of being discarded. They can be reassessed later based on crawl results or improved rules.


URL Normalization Is the Foundation of Page Discovery

The same page often appears in different forms:

https://example.com/contact
https://example.com/contact/
https://example.com/contact#form
https://example.com/contact?utm_source=menu

If URLs are saved exactly as found, the system will treat these as different pages.

Candidate links therefore need to be normalized before they are written.

Common steps include:

  • Converting relative paths to absolute URLs
  • Removing fragments
  • Removing known tracking parameters
  • Unifying the hostname format
  • Unifying trailing slash rules
  • Filtering out mailto: and tel:
  • Filtering out off-site advertising and social media links
  • Filtering out login pages and clearly irrelevant resources

The normalized URL is used for deduplication, but the original link is still worth keeping.

The original link text, discovery location, and source page can help explain:

  • Why the system discovered this page
  • Whether it came from the navigation, body, or footer
  • What the page type judgment was based on

The page management table can apply a unique constraint on "entity ID + normalized URL" to prevent duplicate pages from the same website being written repeatedly.


How the Home Page and Sitemap Work Together

Both the home page and the sitemap can serve as discovery entries, but they provide different information.

The home page and navigation are closer to what a website actively presents to its users, so they usually reflect how important a page is.

A sitemap is better for improving page recall, but you cannot assume that every URL in it is worth collecting. Some sitemaps include:

  • News archives
  • Category pages
  • Tag pages
  • Duplicate language pages
  • Dead URLs
  • Large amounts of low-value content

A safer approach is therefore to:

  • Use the home page and navigation as the main discovery sources
  • Use the sitemap to fill in missed pages
  • Run all candidate links through the same filtering and classification
  • Avoid adding every sitemap URL directly to the crawl queue

The home page helps determine priority and the sitemap improves coverage. They work better as complements than as replacements for each other.


Page Discovery Needs to Run Periodically

Page discovery is not a one-time task.

Websites change continuously, for example:

  • Navigation structures are adjusted
  • New pricing pages are added
  • New PDFs are published
  • Original pages are replaced
  • Content moves to new paths
  • Some links are removed

If page discovery runs only once during system initialization, newly added pages will never enter the collection flow.

The system should therefore rerun page discovery periodically.

During repeated discovery:

  • Existing pages update their last-seen time
  • New pages enter the page management table
  • Existing pages keep their crawl history
  • New pages can enter the crawl flow promptly
  • Pages missing for a long time can move to a pending confirmation state

The focus here is on continuously maintaining the page set, not on regenerating a page list on every run.


Page Types Affect Downstream Processing

Page types are not just for classification and display. They also shape what happens later.

For example, a page type can determine:

  • The crawl frequency
  • The parsing method
  • Which schema to use
  • Whether LLM calls are allowed
  • The processing priority
  • Whether the data is highly time-sensitive

Pricing, application, and announcement pages may need more frequent checks, while company history or basic introduction pages usually change less often.

PDF pages and ordinary HTML pages may also use different methods for generating body text.

So although page discovery does not handle specific fields, it provides important context for scheduling and structured extraction.


Do Not Treat Classification Failure as Discovery Failure

Page classification is not always accurate.

Some websites have URLs with no semantic meaning and vague link text, such as:

/page-12
/view?id=428
/content/details

Discarding these pages outright may lose useful information.

A better approach is to:

  • Save the page
  • Mark it as unknown
  • Give it a lower priority
  • Reassess it based on the title or body after crawling
  • Reclassify it through later rule improvements

The success criterion for page discovery should not be that every page is classified accurately, but rather:

Valuable pages can enter the system, and uncertain pages are not discarded too early.


Design Principles for Page Discovery

Discover Only, Do Not Parse

Page discovery only builds the page set. It does not extract specific business fields.

Prefer Explainable Rules

Prioritize path, link text, file type, and page position when making judgments.

Use Normalization for Deduplication

Normalized URLs identify identical pages, but should not overwrite the original discovery information.

Use Sitemaps as a Supplement

Sitemaps can improve coverage, but they cannot replace page filtering.

Allow Unclassified Pages to Stay

Pages that cannot be judged can go to unknown and wait for later processing.

Keep Maintaining the Page Set

Page discovery should run periodically to adapt to changes in website structure and content entry points.


Summary

The page discovery stage solves a scoping problem.

It is not about crawling as many pages as possible, but about continuously maintaining a set of pages relevant to the data goal while websites keep changing.

A simplified page discovery flow looks like this:

Website entry
→ Extract candidate links
→ Normalize URLs
→ Filter irrelevant links
→ Classify pages
→ Deduplicate and save pages

The output of this stage is not structured data, but a set of source pages that can be reliably scheduled and continuously tracked.

The next article moves on to the first versioning step after a page enters the system:

Snapshots: why content versions are at the core of the whole system.