MarketLens
An AI-powered product research agent that gathers product data from Daraz, OLX, and the wider web, analyzes prices, compares new versus used markets, and produces a final market intelligence report.
The Problem Context
Conducting competitive product research across regional e-commerce platforms and classifieds (like Daraz and OLX) requires tedious manual search queries, copying price outliers, distinguishing brand-new retail items from secondhand listings, and synthesizing price elasticity by hand.
- ›E-commerce sites have drastically different HTML markup, search algorithms, and pagination mechanics.
- ›Classified listings frequently lack structured attributes, mixing accessories, parts, and complete items under confusing headlines.
- ›Search results from open web queries contain noisy duplicates and irrelevant sponsored links.
Engineering Architecture
Developed MarketLens: an automated research agent that coordinates concurrent scrapers across Daraz, OLX, and open-web search (DDGS), normalizes inconsistent product specs and currencies, classifies listings into new vs. used tiers, and feeds clean data into an LLM synthesis engine for executive market intelligence reports.
Execution Sequence:
Interactive Architecture & Data Flow
Parallel multi-source data ingestion pipeline with automated condition classification and LLM synthesis.
Target product search terms and boundary filters submitted.
Parallel crawlers query Daraz, OLX, and DDGS Search.
Cleans irregular text, strips currencies, standardizes attributes.
Categorizes retail inventory vs private classifieds.
Computes quartile ranges, outliers, and average discounts.
Synthesizes structured findings into an actionable brief.
Key Technical Challenges
Multi-Source Data Inconsistency
Problem:OLX presents unstructured classified text while Daraz presents structured catalog cards.
Accurate New vs. Used Differentiation
Problem:Sellers on classified sites often label used products as '10/10 condition' or 'box packed'.
Filtering Outlier & Fake Listings
Problem:Zero-price listings, bait ads, or accessory parts skew average market valuations.
Verified Technology Stack
Design Decisions & Rationales
Decoupled Fetchers with Resilient Async HTTP
If one marketplace experiences rate-limiting or downtime, the agent continues aggregating data from remaining sources gracefully.
Structured Intermediate JSON before LLM Synthesis
Feeding the model pre-aggregated statistics rather than 100 raw webpage dumps significantly reduces token costs while preventing calculation errors.
Technical Results
- ✓Built multi-source product scraping across Daraz, OLX, and search engines.
- ✓Automated new vs. used price segmentation and outlier detection.
- ✓Generated structured market analysis reports synthesizing retail spreads and secondary market dynamics.
Lessons Learned
- •LLMs excel at high-level market commentary and synthesis, but statistical metrics (medians, spreads) should always be calculated deterministically in code.
Future Improvements
- •Implement scheduled price tracking cron jobs to monitor price drops over weeks.
- •Add automated currency conversion for international marketplaces.