Blog

Technical insights, problem-solving notes, and best practices from real-world projects.

Data EngineeringData CollectionWeb Crawling

Designing an Intelligent Data Collection System (Part 2): Page Discovery and How to Continuously Find High-Value Pages

Long-term data collection should not traverse an entire site from a single entry point without limits. This article explains the role of the page discovery stage and how to identify, classify, and continuously maintain high-value pages, creating a stable input for crawling, snapshots, and structured extraction.

9 min read
Read more
Data EngineeringData CollectionAI Data Pipeline

Designing an Intelligent Data Collection System (Part 1): From One-Off Crawls to a Continuously Running Data Pipeline

Traditional crawlers work well for one-off page scraping and field extraction, but long-running data collection also has to handle page discovery, content versions, structured parsing, source tracking, and production data boundaries. This article takes an architectural view of how to extend a single crawl into a sustainable AI data pipeline.

15 min read
Read more
ShopifyWebhookNestJS

Shopify-to-Medusa | Phase One of a Progressive Migration

Rather than rebuilding an entire Shopify store from scratch, we adopted a progressive migration strategy. The first phase focuses on building a reliable data synchronization layer using webhooks, asynchronous workers, scheduled snapshots, retry mechanisms, dead letter queues, and message replay. This foundation keeps Shopify running while continuously syncing products, customers, and orders into an internal database, enabling future modules to be migrated safely and incrementally.

6 min read
Read more
KafkaClickHouse

Kafka + ClickHouse High-Performance Analytics in Practice — Performance Challenges and Solutions Amid Data Growth

As our business continued to grow, user scale and data volumes expanded exponentially. The original monolithic synchronous write approach began to expose significant performance bottlenecks. This post shares how we evolved from a single MongoDB architecture to a Kafka + ClickHouse analytics platform in production, and the lessons we learned.

3 min read
Read more