When collecting public website data at scale, information is usually spread across web pages, PDFs, images, and other formats. Page structures differ from site to site, and there is little consistency in how data is formatted or presented.
That makes this work fundamentally different from traditional scraping based on fixed XPath or CSS selectors. Faced with highly heterogeneous sources, a crawler has to evolve from a one-off page parser into an intelligent processing pipeline that can continuously discover, collect, parse, clean, and update data.
This article introduces a staged architecture for intelligent data collection. It suits scenarios where public website data is unstructured, non-standardized, and needs long-term maintenance.
Why Traditional Crawlers Struggle with Long-Term Collection
A common crawler workflow looks like this:
Request page
→ Parse content
→ Extract fields
→ Write to database
There is nothing wrong with this flow, but it puts several responsibilities into a single execution.
The page is parsed as soon as it is downloaded, and the parsed result goes straight into production data. Usually only logs are kept in between, with little record of the full source content or the processing that took place.
At a small scale, this simplification lowers development costs. As the system runs for longer, however, some problems start to appear.
Pages and Data Change Together
Websites are not fixed data sources.
Page paths can change, navigation can gain new items, PDF files can be replaced, and some data may move from the home page to a dedicated page. Maintaining a fixed set of URLs and parsing rules makes it hard to keep up with these changes.
Even when a URL stays the same, only a small part of the page may have been updated. The system needs to distinguish between:
- A page being revisited
- The page code changing
- The useful information on the page changing
If every crawl reparses the content and overwrites the data, the system does a lot of repeated work and struggles to tell whether an update has any business meaning.
Extracted Results Lose Their Source
After structured data is written to the database, usually only the final value remains.
For example, an address, price, or contact detail may already be in a production table, but the system may be unable to quickly answer:
- Which page did this value come from?
- When was the page crawled?
- What did the original content say at the time?
- Which version of the parsing rules was used?
- Why did it overwrite the previous value?
Without this information, the only way to investigate a data problem is to revisit the current page. But that page may have changed, making it impossible to reconstruct what happened.
Parsing Is Coupled to Production Writes
When a parser updates production data directly, errors spread quickly.
Traditional rules may extract the wrong field after a page structure changes, and an LLM may return a well-formed result with no real supporting evidence. Without a clear boundary between automated extraction and production writes, it is hard to balance efficiency and trustworthiness.
A Completed Task Does Not Mean Correct Data
A successful HTTP request, a clean Worker exit, and a JSON payload that passes validation only show that a technical step finished.
The page may have returned a login screen, content extraction may have come back empty, the LLM may have misread the context, and multiple sources may conflict.
Long-term data collection therefore cannot measure quality by task success rate alone. It needs to save the inputs, outputs, and status of each stage so that problems can be traced back to a specific step.
From Crawler Scripts to a Data Pipeline
A cleaner design splits the collection process into several relatively independent stages:
Source entry
→ Page discovery
→ Page crawling
→ Content snapshot
→ Structured extraction
→ Data merging
→ Human review
→ Production data
The point is not to make the system more complex, but to stop different responsibilities from interfering with each other.
Each stage solves one main problem and saves its result as a stable input for the next stage.
The full data flow can be abstracted as:
Seed Source
↓
Source Page
↓
Content Snapshot
↓
Extraction Record
↓
Merge Candidate
↓
Review Result
↓
Production Data
This flow contains three different kinds of data: source content ≠ automated extraction results ≠ production facts
Source content describes what a website actually provided at a point in time. Automated extraction results describe what rules or models identified from that source. Production facts describe what the system finally adopted after validation and confirmation.
Keeping these three layers separate is the foundation for source tracking, incremental updates, reparsing, and human review.
Stage 1: Source Entries
A pipeline usually starts from a set of known entry points.
An entry can be:
- An organization's official website
- A product catalog home page
- A public data page
- A document download page
- A confirmed domain
A source entry only needs to hold basic information, such as a business entity identifier and a website address.
It should not also take on page lists, crawl status, parsing results, and production data. Otherwise, the entry table gradually turns into a catch-all task table full of runtime fields, which becomes hard to maintain.
The goal of this stage is simple:
Define where the system starts looking for data.
Stage 2: Page Discovery
A website usually contains many pages, but not all of them are worth collecting over the long term.
News archives, event pages, tag pagination, privacy policies, and login entries may have no direct value for the target data. Traversing an entire site without limits increases the number of requests, storage costs, and downstream parsing costs.
The page discovery stage therefore identifies the pages worth managing, such as:
- Basic introduction pages
- Product or service pages
- Pricing pages
- Contact pages
- Public documents
- PDF download pages
- Other pages containing target fields
Page discovery and page crawling need to be separate.
Page discovery answers:
Which pages deserve to enter the system?
Page crawling answers:
What is the current content of this page?
Once separated, link filtering rules can change independently, without touching crawling or parsing. When a site adds a new high-value page, the next discovery run can bring it into the system.
Specific discovery strategies, URL normalization, and page classification are covered in Part 2 of this series.
Stage 3: Page Crawling
The page crawling stage is responsible for obtaining source content.
It usually needs to handle:
- HTML pages
- PDF files
- Images
- Redirects
- Different response types
- Network failures
- Generating page body content
This stage only fetches and prepares content. It should not extract business fields directly.
There are two reasons for this.
First, crawling and parsing change at different rates. Website requests, redirects, and file downloads are collection concerns, while field recognition is a parsing concern. Combining them means that changes to parsing rules affect crawling logic.
Second, page content is the basis for everything that follows. Only after the source is stored reliably can the system reparse it, compare versions, and trace results.
A successful crawl does not necessarily produce new data either. A page may load normally while its useful content is identical to the last crawl. Whether it needs further processing is decided at the content versioning stage.
Stage 4: Content Snapshots
A content snapshot represents the version of a page's content at a specific point in time.
Pages and snapshots are two different concepts:
Source Page
├── Snapshot A
├── Snapshot B
└── Snapshot C
A page is the object that is managed and scheduled over the long term.
A snapshot is one specific content record.
A snapshot is usually associated with:
- The source page
- The crawl time
- The original URL and final URL
- The content type
- The original file
- The body text available for parsing
- A content version identifier
Once a snapshot is created, its content should remain as immutable as possible. Even if extraction rules, models, or schemas change later, existing records should not be overwritten directly.
This enables several capabilities:
- Preserving a page's version history
- Determining whether content has changed
- Reprocessing old content with new rules
- Comparing different parsing versions
- Tracing the source of production fields
- Reproducing a past processing run
Across the pipeline, snapshots connect external websites to internal structured data and are a key foundation for source tracking.
Snapshots do more than preserve page history. They also underpin incremental updates and reparsing, which is covered in Part 3 of this series.
Stage 5: Structured Extraction
Once source content enters the system, it needs to be converted into a unified data structure.
Traditional crawlers mainly rely on:
- CSS selectors
- XPath
- Regular expressions
- Fixed field mappings
- Page-specific parsers
These methods are still very useful for stable pages.
LLMs are better suited to:
- Content with widely varying page structures
- Natural language descriptions
- PDF documents
- Tables and semi-structured data
- Different expressions of the same field
An AI data pipeline therefore does not mean that everything must be processed by an LLM. More commonly, the parsing method is chosen according to content type and stability.
The input to structured extraction should be a saved snapshot, not a temporary variable from the crawl.
Its output is not final production data either, but a structured candidate record:
Content Snapshot
→ Structured Extraction
→ Candidate Record
A candidate record needs to stay linked to its source snapshot and record the parsing method or model version. That way, even if the model output is wrong, the original source remains intact.
This stage mainly answers:
What structured information does the current page explicitly provide?
It is not responsible for choosing the right answer across pages, and production data should not influence the model's results in reverse.
Model input preparation, prompts, schemas, and result validation are discussed in later articles.
Stage 6: Data Merging
Information about a single entity is usually spread across multiple pages.
For example:
Home page
├── Name
└── Introduction
Contact page
├── Address
├── Phone
└── Email
Pricing page
├── Amount
├── Period
└── Scope
These pages may complement each other, or they may conflict.
One page may still show an old phone number while another has been updated. Different language versions may format addresses differently. A PDF may belong to last year while the web page shows current data.
Extraction from a single page should therefore not generate a complete entity directly, and conflicts should not be resolved early inside the model.
A cleaner division of responsibilities is:
- The extraction stage keeps an independent result for each page
- The merging stage groups candidate values by field
- Consistent results can be aggregated automatically
- Inconsistent results keep their sources and go to review
This avoids hiding source differences inside a merged result that is hard to explain.
How to compare, normalize, and resolve conflicts between multiple sources is covered later in this series.
Stage 7: Human Review
For data that requires high trustworthiness, there usually needs to be a clear boundary between automated extraction results and production data.
Human review does not mean entering every field by hand.
The automated system can first handle:
- Page change detection
- Candidate field extraction
- Basic format normalization
- Merging consistent results
- Identifying conflicts
- Comparing with production data
Reviewers then only need to focus on:
- Fields where sources conflict
- Time-sensitive data
- Enumerations that cannot be determined automatically
- Production fields that need to be overwritten or deleted
- Results without sufficient source evidence
The goal is not to put a person in every step, but to focus human judgment on the parts automation cannot handle reliably.
Only confirmed results enter production, and automated extraction records remain as historical evidence.
The specifics of the review interaction, data comparison, and write boundaries are discussed in a separate article.
Stage 8: Production Data
Production data represents the facts the system currently accepts and uses externally.
Its role differs from the intermediate data in the pipeline.
The pipeline stores:
- Source pages
- Historical content
- Automated extraction results
- Data candidates
- Processing status
- Call audit records
The production database stores:
- Currently effective data
- Confirmed relationships
- Structures designed for business queries
- Fields needed for external display
With the two separated, collection and parsing logic can keep evolving without directly affecting live data.
Production data should ideally keep the necessary source references or change summaries, so that a field can be traced back along this path:
Production Data
→ Approved Result
→ Extraction Record
→ Content Snapshot
→ Source Page
The complete source chain does not have to live in a single table, but the system should keep enough relationships to prevent final values from being separated from their evidence.
Where Task Queues Fit in the Architecture
Page discovery, page crawling, and AI extraction are all generally well suited to asynchronous execution.
A task queue can handle:
- Queuing
- Concurrency control
- Failure retries
- Delayed execution
- Worker distribution
But a queue only solves how tasks run. It cannot decide on its own which tasks should run.
In this architecture, a clean division of responsibilities looks like this:
Database
↓
Dispatcher
↓
Task Queue
↓
Worker
↓
Database
The database stores business state, the Dispatcher finds tasks that meet the conditions, the queue distributes them, and Workers do the actual crawling or parsing.
This first article only establishes where these components sit. It does not go into scheduling state, retries, leases, or idempotency.
How the Dispatcher and Workers cooperate using BullMQ is covered in Part 5 of this series.
What This Architecture Solves
Splitting a single crawl into a data pipeline brings several capabilities.
1. Sources Can Be Traced
A structured result is no longer an isolated value. It can be linked to a specific page and content version.
2. Content Can Be Reprocessed
After parsing rules, prompts, or schemas change, existing snapshots can be reparsed without immediately revisiting external websites.
3. Page Changes Can Be Judged Independently
Checking a page and updating data are no longer the same thing. Only meaningful content changes need further processing.
4. Automated Results Do Not Pollute Production Data
Structured extraction first produces candidate records, which enter production only after merging and review.
5. Each Stage Can Evolve Independently
Page discovery, crawling, content processing, AI extraction, and review can each be improved without rewriting the whole pipeline.
6. Runs Are Easier to Recover
As long as stage status and intermediate results are saved, the system can continue from existing data after an interruption instead of crawling everything again.
How tasks recover from repeated execution, Worker interruptions, and inconsistent queue state is covered in the article on idempotent design.
Core Design Principles
This architecture can be summarized into a few relatively stable principles.
Keep Stage Responsibilities Single
Page discovery does not parse fields, page crawling does not call models directly, and model extraction does not write directly to production data.
Save Intermediate Results
What is worth keeping is not just the final data, but also source pages, content snapshots, and structured candidates.
Separate Source Content from Production Facts
Sources describe what a website provided. Production data describes what the system finally adopted. The two are not the same.
Treat Automated Results as Candidates by Default
Whether they come from fixed rules or an LLM, automated extraction results can be affected by page structure, input quality, and parsing version.
Queues Handle Execution, Databases Hold State
Task queues can be cleared or rebuilt. Whether business work is finished should be determined by the results and status in the database.
Every Result Should Be Explainable
The system needs to know not only whether a task succeeded, but also where the data came from, which rules generated it, and at which stage a problem occurred.
Summary
Traditional crawlers focus on a single request and field extraction, while long-term data collection needs to manage the full data lifecycle.
Overall, this AI data pipeline design does not try to make one component solve every problem. Instead, clear stage boundaries give sources, content versions, automated extraction results, and production facts their own separate responsibilities.
Later articles will discuss why each stage is needed, how to design it, and what problem it solves in the overall data flow.
The next article starts with the first active step in the pipeline:
Page discovery: how to continuously find high-value pages.