Case study
Turning thirty scrapers and messy product data into a clean, searchable deals site
Shoppers wanted every fashion sale in one place. The data came from more than thirty brand websites, each with its own layout and its own idea of what a category means. We built the scrapers, the pipeline and the AI that sorts it all every night.
Quick facts
- Industry
- Ecommerce
- Product type
- Consumer website with admin panel
- Platform
- Web, works on phones
- Timeline
- Live, refreshed every night
- Status
- Live
At a glance
What changed
37+ brands collected nightly
Scrapers for more than thirty-seven brand websites run every night without anyone starting them.
Consistent categories
A three-stage pipeline classifies every product the same way, whatever the brand called it.
Thousands of fixes
The pipeline corrected thousands of products that simple rules had put in the wrong category.
Hosting under control
Moving to a self-managed server brought hosting costs under control.
The client
The client and the situation
The client runs a deal aggregator for fashion shoppers in Pakistan. The idea is simple: instead of visiting dozens of brand websites during sale season, shoppers visit one site that shows discounted items from all of them, sorted into clear categories they can filter.
The value of the site depends entirely on its data. If products are missing, out of date or in the wrong category, shoppers stop trusting it. That made data quality the core of the product, not a back-office detail.
The challenge
The challenge
Every brand website is built differently. Some list products in plain HTML, some load them with scripts, and layouts change without notice. Each brand also names things its own way. One brand's "kurta" is another's "shirt", and a product title might mention three garment types at once.
Collecting the data was hard. Classifying it was harder. Simple keyword rules put thousands of products in the wrong category. A shopper filtering for one garment type would see others mixed in, and good deals were hidden in categories where nobody looked.
Before
How the work was done before
Category assignment started with keyword rules. When a keyword rule misfired, products landed in the wrong category, and each mistake had to be found and fixed by hand. With thousands of products and new ones every night, manual fixing could not keep up.
Hosting was a second concern. The site ran on a managed platform, and for a product with nightly data refreshes and seasonal peaks, the client wanted costs that were predictable and under its own control.
Objectives
What the project had to achieve
-
Collect sale items from every supported brand automatically, every night.
-
Store products in a clean, consistent structure, whatever the source.
-
Classify every product into the right category reliably, at scale.
-
Give the team an admin panel to fix edge cases and tune the rules.
-
Bring hosting costs under control without hurting speed.
The solution
What we built
We built scrapers for more than thirty-seven brand websites and scheduled them to run every night on GitHub Actions at 3 AM UTC. Products are stored in PostgreSQL through Drizzle ORM. A trigger in the database enriches each product as it arrives, filling derived fields so the site can filter and sort quickly.
Classification runs in three stages. First, fast rules: category keyword maps and brand overrides that handle the clear cases. Second, a zero-shot classifier model for products the rules cannot place with confidence. Third, a large language model as a fallback for the hardest cases. Each stage only handles what the stage before could not, which keeps cost low and quality high. An admin panel lets the team review results, adjust keyword maps and set brand overrides.
-
Brand scrapers
Scrapers for 37+ brand websites, each written for that site's structure.
-
Nightly schedule
All scrapers run every night on GitHub Actions at 3 AM UTC.
-
Clean product store
PostgreSQL with Drizzle ORM and a consistent product structure.
-
Enrichment trigger
A database trigger fills derived fields as each product arrives.
-
Rules first
Category keyword maps and brand overrides handle the clear cases.
-
Zero-shot classifier
A classifier model places products the rules cannot.
-
Language model fallback
A large language model handles the hardest cases last.
-
Admin panel
Tools to review classifications, edit keyword maps and set brand overrides.
-
Deals grid and filters
Shoppers browse and filter sale items across all brands.
UX and UI
The UX and UI approach
Shoppers come for deals, so the home page is a grid of discounted items with the discount clear on each card. Filters by category, brand and price are designed to work well on a phone. Shoppers buy on the brand's own website.
The admin panel is built for the team that keeps the data clean. It shows products by category and classification stage, so the team can spot where rules or models need tuning, and fix a whole group of products at once instead of one by one.
Architecture
How the pieces fit together
Every night, scheduled jobs on GitHub Actions run the scrapers against each brand website. Products are written to PostgreSQL through Drizzle ORM, where a trigger enriches them. The classification pipeline then processes new and changed products: rules first, the zero-shot classifier next, the language model last. The website and the admin panel both read from the same database.
The site first ran on Vercel. We moved it to a self-managed server running Coolify, which gave the client predictable hosting costs and full control over the environment.
AI-accelerated
Where AI sped up the build
Honest about what AI did, and what people did.
-
AI in the product: the zero-shot classifier and the language model fallback are part of the running system, classifying products every night.
-
Fixing classification at scale: the classifier and language model stages corrected thousands of products that keyword rules had misplaced.
-
Cost control by design: cheap rules run first and the language model runs last, so AI cost stays proportional to the hard cases.
-
People stay in charge: the team reviews results and tunes keyword maps and brand overrides in the admin panel.
Integrations
Integrations
-
Brand websites
Product data collected from 37+ brand websites.
-
GitHub Actions
Scheduled nightly runs for all scrapers.
-
AI models
A zero-shot classifier and a large language model in the classification pipeline.
-
Coolify
Self-managed hosting after the move from Vercel.
Screens
Concept screens
Simple mockups of the described interface. Real screenshots will replace them.
Concept screen
Results
The results
Sale items from more than thirty brands are now collected every night without anyone starting a job. Products are classified consistently, whatever each brand called them, and the three-stage pipeline fixed thousands of products that keyword rules alone had placed in the wrong category. Shoppers filtering for one garment type now see that garment type.
Moving from Vercel to a self-managed Coolify server brought hosting costs under control and made them predictable. We do not publish exact figures here. If the client shares them, they will appear on this page.
What the team can do now
-
Refresh every brand's sale items automatically each night.
-
Classify new products consistently, with AI handling the hard cases.
-
Fix groups of products and tune rules from the admin panel.
-
Add new brands by writing one scraper, not rebuilding the pipeline.
-
Run the site on hosting with predictable costs.
Before and after
Before and after
Many sources and keyword guesses
-
Sale items are spread across dozens of brand websites.
-
Each brand names and groups products its own way.
-
Keyword rules put thousands of products in the wrong category.
-
Hosting costs are set by a managed platform.
A nightly pipeline with AI sorting
-
Scrapers for 37+ brands run every night on a schedule.
-
Products land in one clean database and are enriched on arrival.
-
Rules, a classifier and a language model sort every product.
-
Self-managed hosting keeps costs under control.
Technology
Technology stack
- PostgreSQL
- Drizzle ORM
- Web scrapers
- GitHub Actions
- Zero-shot classifier
- LLM fallback
- Coolify
What comes next
What we would do next
We would next add scraper health checks that alert the team when a brand's product count drops sharply, which usually means the website changed. We would also add price history per product, so shoppers can see whether a sale price is really low. That builds on our price monitoring work.
On the AI side, we would measure the classifier and the language model on a labeled sample every month, and move any category the rules handle badly into the model stages. That is the kind of regular AI model evaluation that keeps quality from drifting as brands change their websites.
Solutions in this project
Problems this project solved
Price monitoring
Collect competitor prices automatically, match them to your products and alert you when something changes.
AI automation and integrations
Connect the tools you already use so data moves once, correctly, and AI reads the parts that arrive as text or documents.
AI-assisted content workflows
Briefs, first drafts, edits and repurposing in a workflow where people set the angle and approve every word.
Keep exploring
Related case studies, services and industries
Related services
View all related services- Web applications Custom browser-based software for the work your team does every day, from booking tools to operations systems.
- Admin panels The back-office screens your team uses to manage users, content, orders and settings without a developer.
- LLM integration Add a large language model to software you already have, with the guardrails, costs and logging handled.
- Database design Data models that stay fast and correct as your business grows, with backups and access rules in place.
- AI Product Development AI agents, knowledge assistants, copilots and document automation built into the way your team already works.
- Automation and Integrations CRM, payment and API integrations, MCP connections and workflows that stop people copying data by hand.
Solutions
View all solutions- Price monitoring Collect competitor prices automatically, match them to your products and alert you when something changes.
- AI automation and integrations Connect the tools you already use so data moves once, correctly, and AI reads the parts that arrive as text or documents.
- AI-assisted content workflows Briefs, first drafts, edits and repurposing in a workflow where people set the angle and approve every word.
- Forecasting dashboards Forecast demand from your sales history and seasonality, and flag items to reorder before they run out.
- Feedback analysis Read every review, survey and ticket, group them by theme and sentiment, and show what customers keep asking for.
- AI translation workflows Translate websites, products and support content quickly with AI, a shared glossary and native-speaker review where it matters.
Industries
View all industries- Retail Stock forecasting, sales dashboards, feedback analysis and store tools for retailers with shops and online sales.
- Ecommerce Online stores, product data cleanup, support assistants, price monitoring and order automation for online retailers.
- Education and elearning Course platforms, student portals, content workflows and AI tutoring assistants for schools and training companies.
- SaaS and startups MVPs, subscription billing, AI features, multi-tenant platforms and scaling support for founders and product teams.
- Vacation rentals Booking websites, owner and guest portals, turnover scheduling and guest messaging for rental operators.
- Property management Tenant portals, maintenance tracking, owner statements and inbox automation for property managers.
Case studies
View all case studies- Beach rental booking system A long-running booking system kept earning while it was extended, tested and given a safe, gradual path to a modern stack.
- Goal and effort tracking app A product design built around handwritten daily proof, locked plans and honest self-scoring. The build is in progress.
- Facility management platform Every request, inspection and cost for every building now lives in one place, with a clear owner and status.
- Vacation rental operations system One calendar per property, cleaning tasks created from bookings, and owner statements built from the same records.
- Payroll system for an IT services company Payroll is calculated from recorded inputs, reviewed once and sent as payslips in a single batch, with every change logged.
- Tax filing service platform Clients upload documents and see where their case stands, and staff work from one queue with deadlines and reminders.
Guides and articles
View all guides and articles- How to choose an AI model for your business A step-by-step way to pick an AI model by testing candidates on your own tasks, data rules and budget.
- How to keep customer data safe when using AI Practical steps to protect customer data when you use AI services, from data terms to access and logging.
- Prompt writing basics for business teams How to write prompts that get consistent, useful results from AI tools, with templates your team can reuse.
AI models
View all ai models- GPT-6 Luna OpenAI's most efficient model for focused, high-volume tasks, with vision, tool calling and structured outputs at the lowest price level in the family.
- GPT-4.1 An older OpenAI model described as its smartest non-reasoning model, strong at following instructions and calling tools, with a very long context window.
- DeepSeek V4.1 Flash DeepSeek's default, best-value model: open weights under MIT, image input, tool calling, JSON output, optional thinking and a 1 million token window.
- Qwen3.8-Max Alibaba's current Qwen flagship for long autonomous coding and professional work, with image and video input, tool calling, JSON Schema output and a 1 million token window.
- General purpose All-round language models for writing, summarizing, support, extraction and most everyday business tasks.
- Image and video Models that generate or edit images and video from text and reference images.
Glossary terms
View all glossary terms- Zero-shot classification Zero-shot classification is sorting text or images into categories an AI model was not specifically trained on, using only the category names or descriptions.
- Large language model A large language model, or LLM, is an AI model trained on vast amounts of text that can understand and generate language, and often images and code.
- Natural language processing Natural language processing is the field of AI that deals with understanding and generating human language, in text or speech.
- Prompt A prompt is the instruction and context you give an AI model to get the output you want.
- AI copilot An AI copilot is an assistant built into a tool people already use, which suggests, drafts and explains while the person stays in control.
- Artificial intelligence Artificial intelligence is the broad field of building software that performs tasks that normally need human judgment, such as understanding language or images.