Crawling and scraping: definitions
Crawling means automatically browsing websites to discover and read pages; scraping, or extraction, means turning the content of those pages into structured data — a price, a specification or a document. Both underpin product data: this is how manufacturer records, catalogues and competitor pages become usable.
Crawling is not extraction
The two terms are often confused. Crawling answers “where is the information?”: finding the product page, PDF technical data sheet or catalogue. Scraping answers “what does it say?”: extracting structured values in the right format. A serious system does both — and keeps the link to the source page for every value. Without that traceability, extracted data cannot be verified.
What AI agents change
Historically, extracting a website required a dedicated development — one crawler per site, liable to break after every redesign. Agents change the economics of the problem: an agent navigates, understands the page as a person would, finds the data even when the structure changes, and can discover where a product is sold by itself. Extraction moves from a site-by-site coding problem to a page-by-page judgement problem — see product matching and product data enrichment.
The legal framework in France
Collecting publicly accessible data is an established practice — it underpins price monitoring and product data — but it takes place within a legal framework. Known points of attention include database rights (extracting a substantial part of another party’s database is protected by the French Intellectual Property Code), website terms of use, the GDPR whenever personal data is involved — which is not the case for a product specification — and fair collection practices, including reasonable frequencies and avoiding disruption to the visited website. Serious collection is designed around these constraints from the outset, not afterwards. When a specific use is uncertain, the answer should come from legal counsel, not a technical provider.