How to Scrape Websites Without Using AI Tokens: A Deterministic Approach

There is a growing habit in research and data teams: paste a web page into ChatGPT, ask it to extract the data you need, and copy the output into a spreadsheet.

It feels convenient. And for one or two pages, it is fine. But at any real scale - dozens of profiles, hundreds of company pages, thousands of directory listings - this approach turns into an expensive, inconsistent mess.

You do not need AI to scrape websites. You need a scraper. AI is what you use after you have the data.

This guide is about doing the extraction step with no model in the loop at all.

What this guide covers

How deterministic extraction actually works: where the data sits in a page, which retrieval method suits which case, how to handle repeated templates and pagination, how to validate output, and - importantly - when this approach is the wrong choice.

Written for people using a point-and-click extractor rather than writing scrapers, so the patterns below matter more than the syntax. You do not need to write code to use any of this, but knowing what the tool is doing underneath is what lets you fix it when a page fights back.

What it does not cover: pipelines that keep a model. If yours does and you want it cheaper rather than gone, optimising an LLM extraction pipeline is the right guide. If you have not yet decided which approach fits, start with choosing an extraction architecture.

Where the data actually is

Before choosing a method, find out where the values live. Most people go straight to clicking on visible text, which is often the hardest route.

Embedded JSON. Many sites ship their data as structured JSON inside the page - a <script type="application/ld+json"> block, or a state blob the page uses to render itself. This is the best case by a wide margin: already structured, already typed, and far more stable than visual layout because it is not tied to how anything looks. Check for it first, always.

Metadata tags. <meta> tags and Open Graph properties carry titles, descriptions, canonical URLs, and often publication dates. Stable and trivially extractable.

Tables. <table> markup maps directly to rows and columns with no interpretation needed. Reliable when present.

Rendered text. The visible content, reached by selector. The most common route and the most fragile - it is tied to layout, so it breaks when the design changes.

The general rule: prefer the most structured source available, because structure equals stability. A site can redesign completely and leave its embedded JSON untouched.

Targeting elements

When you do need rendered text, how you target it determines how long it keeps working.

Selectors anchored to meaning survive redesigns. Selectors anchored to appearance do not.

/* Fragile - styling classes change with any redesign */
.css-1x7ky2 > div:nth-child(3) > span.text-sm

/* Sturdier - anchored to structure and semantics */
[data-testid='company-name']
article h1

Label anchoring - “the value next to the word Founded” - is the sturdiest pattern of the three, but CSS has no text-matching selector, so it needs either XPath or a short DOM traversal:

// XPath: the <dd> following the <dt> whose text is "Founded"
document.evaluate(
	'//dt[normalize-space()="Founded"]/following-sibling::dd[1]',
	document,
	null,
	XPathResult.FIRST_ORDERED_NODE_TYPE,
	null
).singleNodeValue;

// Equivalent DOM traversal
[...document.querySelectorAll('dt')].find((dt) => dt.textContent.trim() === 'Founded')
	?.nextElementSibling;

Some scraping tools accept dt:contains("Founded") + dd for this. That is jQuery/Sizzle syntax, not standard CSS - it will not work in querySelectorAll or anywhere else relying on the browser’s native selector engine, so only use it where the tool documents support for it.

Three habits that decide whether a template lasts a month or a year:

Prefer data attributes. data-testid, data-field, and similar hooks exist for automated testing, so teams avoid changing them casually.

Anchor to labels, not positions. The label-anchored lookup above survives a reordered layout. “The third span” does not.

Avoid deep chains. Every level in a selector path is something that can change. Short and specific beats long and precise.

Point-and-click tools generate selectors automatically, and they often generate the fragile kind - the styling-class chain in the first example. If a template keeps breaking, this is usually why, and most tools let you edit the selector by hand.

Repeated templates and pagination

Deterministic extraction pays off through repetition: define once, run many times. Two structures come up constantly.

List pages. Identify the repeating container - the element that appears once per result - then define fields relative to it rather than to the page. That way one definition handles ten results or fifty, and a page returning a different number of rows does not break it.

Pagination. Prefer URL patterns where they exist (?page=2), since they are predictable and resumable after a failure. Where content loads on scroll or via a “load more” button, the extractor has to trigger loading before reading, and you need a stop condition - a page count, a target record count, or the absence of new results. Without one, an infinite-scroll page will happily run forever.

Always record the source URL alongside every extracted row. It costs one column and it is the only way to check a value later or resume a partial run.

Validating output

Deterministic extraction fails loudly, which is its main advantage over a model - but only if you look.

A selector matching nothing returns empty, not wrong. So validate on completeness rather than correctness: what proportion of rows have a value for each field? Record that proportion per field on a run you trust, then compare every later run against it. For example, suppose a field is populated on nearly every row for months and then, after a redesign, on almost none - that collapse is the layout change announcing itself. The absolute numbers do not matter; the size of the drop does.

Then apply format checks - emails contain @, dates parse, URLs resolve - and a plausibility pass on a sample. Fields that quietly capture the wrong element often look structurally valid, so ten rows read by eye is worth more than it sounds.

When deterministic extraction is the wrong tool

It is worth being direct about the limits, because forcing this approach where it does not fit wastes more time than the tokens save.

Pages with no shared template. If every page differs, you are writing a template per page. Rules need repetition to pay for themselves.

Data that is not on the page. If the answer requires combining several fields, reading between the lines, or applying outside knowledge, no selector reaches it. Extract the inputs deterministically, then let a model do that step.

Interpretation tasks. Classifying a company’s industry from a description, judging seniority from an unusual job title, or summarising - these are model tasks by nature.

Aggressively dynamic layouts. Some sites vary structure per visit through A/B tests or personalisation. Selectors that work in your browser may not work in someone else’s.

One-off jobs. Setting up a template for twenty pages you will never revisit is rarely worth it. Rules are an investment that repetition amortises.

The honest summary: deterministic extraction is excellent at retrieval from predictable structure and incapable of interpretation. Most real workflows want both, which is why the extraction step and the reasoning step are worth keeping separate.

How browser-based scraping works differently

Browser-based scraping reads data directly from the page’s DOM - the underlying structure of HTML elements that make up what you see on screen.

Instead of sending the page to a model and asking it to figure out what is relevant, a DOM-based scraper uses CSS selectors to target specific elements: the h2 tag that contains a company name, the span that holds a phone number, the a link that contains a profile URL.

The result is fast, consistent, and token-free. The scraper does not need to interpret anything. It reads the element, extracts the value, and moves on. Run it across 200 pages and you get 200 rows of identically structured data.

The tradeoff is setup time. Traditional web scrapers require writing code - Python with requests and BeautifulSoup, or Playwright for JavaScript-heavy sites. You have to inspect the HTML, write selectors, handle edge cases, and maintain the script when the site layout changes.

Modern browser extensions solve this by making the setup visual. Instead of writing selectors, you click on the elements you want to extract. The tool generates the selectors for you, runs them against the page, and produces the structured output.

How Fetchr’s custom scraper works

Fetchr is a Chrome extension that includes both an automatic LinkedIn extractor and a custom scraper for most websites and pages you can access in your browser.

The custom scraper works through a visual, three-step process:

Step 1 - Pick the repeating row

Most data collection involves a repeating pattern: a list of profiles, a directory of companies, a search results page full of records. Fetchr’s picker lets you hover over one of these repeating items on the page. As you move your cursor, Fetchr highlights what it thinks the repeating element is. When it looks right, you click once to lock it in.

Fetchr then finds all similar elements on the page - typically the same type of list item or card - and highlights them. You can see immediately whether the selection is correct before defining any fields.

Step 2 - Pick the fields

With the repeating rows identified, you move to field selection. You hover inside one of the highlighted rows to see which specific sub-element gets highlighted - a name, a job title, a link, an image URL. Click to select it, give it a name (like “company_name” or “website”), and it is saved.

Fetchr determines whether the selected element is best captured as text, a link URL, an image URL, or raw HTML. You can override this choice if needed. Repeat for each field you want to collect.

Fetchr tracks how many of the highlighted rows successfully contain each field you define - so you know before running the full extraction whether your selectors will produce complete data.

Step 3 - Set up pagination (if needed)

If the site spreads data across multiple pages, you define the pagination method: a Next button to click, a Load More button, or infinite scroll. For Next and Load More buttons, you click the actual button on the page and Fetchr captures its selector. For infinite scroll, Fetchr handles the scrolling automatically.

Fetchr supports extracting detail pages as well - if each row links to a full profile page with additional fields, you can define a second set of fields to collect from those detail pages.

Once the template is set up, click Run. Fetchr works through the pages automatically and collects all the rows matching your template, across as many pages as needed.

What you get at the end

Fetchr produces a structured dataset - either CSV or JSON - containing one row per extracted record, with consistent column names across every row.

You do not need to clean field names. You do not need to re-run prompts to get missing values. You do not need to normalise formats across records. The data comes out the way you defined it.

That structured dataset is what you take into your next step - whether that is importing into a CRM, sending to a spreadsheet, passing to a data enrichment tool, or feeding into an AI model for analysis.

When the data reaches the AI at this point, it is clean. It is structured. It contains only the fields that are relevant. The AI can focus on what it is actually good at - analysis, scoring, synthesis, writing - rather than fighting with raw HTML to find a phone number.

If you are collecting LinkedIn data specifically, Fetchr includes built-in extractors for person profiles and company pages that work without any template setup. See what Fetchr extracts from LinkedIn and how the workflow compares to manual research

Common use cases for token-free scraping

Here are some of the workflows where browser-based scraping produces better results than AI-assisted extraction:

Sales prospecting. Collecting company names, websites, and contact information from industry directories, conference attendee lists, or job board company pages. Fetchr extracts the list structure directly; the data is ready for enrichment or outreach without any AI involvement. See how to build a prospecting list that converts

Recruiting research. Building candidate lists from professional directories, alumni pages, or event attendee lists. Extract names, titles, companies, and profile links without manually copying each record. See how browser-based scraping increases research output

Competitive analysis. Collecting pricing, feature names, or product listings from competitor websites. Define a template once and re-run it whenever you need a fresh snapshot.

Market research. Scraping directories, review sites, or listing pages to collect structured information about companies, products, or services in a given space.

Lead list building. Collecting structured data from company directories or event attendee pages for use in sales sequences. Learn what to check before importing a lead list into your CRM

For all of these, the extraction step requires no AI. The AI’s role - if any - comes after, when you want to score, categorise, enrich, or write based on the structured data you already have.

The right division of labour between scraping and AI

The most efficient research workflows keep these two steps clearly separated:

Scraping: read the page, find the data, structure it consistently, export it. No intelligence required - just reliable execution.

AI analysis: score records, identify patterns, write personalised content, classify entries, fill in reasoning gaps. Intelligence required - and worth the token cost.

Mixing the two - asking AI to do both at once - wastes what AI is good at and pays premium rates for what a scraper can do better.

Fetchr handles the scraping step. It works on most websites and pages you can access in your browser, requires no code, produces consistent structured output, and does not touch your AI token budget until you decide the data is ready for analysis.

For a complete guide to reducing AI token usage across every step of your data workflow, see AI token saving for data extraction.

Fetchr extracts structured data from LinkedIn and websites you can access in your browser - without sending raw pages to AI. Sign up to DataFixr above to access the extension.