How to Access Google Search Results Programmatically

Google search results are the starting point for almost every information-seeking task on the internet. For AI agents, search is foundational — it's how agents find answers, discover resources, and navigate the web. But accessing Google search results programmatically is expensive, limited, and increasingly difficult.

The official API costs $5 per 1,000 queries. Third-party SERP APIs charge $50-200/month. Scraping Google directly is a losing battle against the most sophisticated anti-bot system ever built. For agents that need to search frequently, the costs add up fast.

The Problem with Google Search Data Access

Google search is simultaneously the most useful and most locked-down data source on the web:

• Custom Search JSON API: Google's official API costs $5 per 1,000 queries after a 100-query/day free tier. Limited to 10 results per query. Doesn't include featured snippets, knowledge panels, People Also Ask, or other rich SERP features. Returns a stripped-down version of what you see on google.com.
• SerpApi / ScaleSerp / ValueSERP: Third-party SERP APIs range from $50-200/month. They run proxy networks to scrape Google and return structured results. Reliability varies. Some don't capture all SERP features. Data freshness depends on their scraping schedule.
• Direct scraping: Google's anti-bot is the gold standard. ReCAPTCHA, fingerprinting, IP reputation, behavioral analysis, and automatic blocking. Even premium proxy services (Bright Data, Oxylabs) with residential IPs get blocked at scale. Cost: $100-500/month in proxy fees alone.
• Programmable Search Engine: Free, but searches only your specified sites — not the full Google index. Useless for general web search.

For an AI agent that makes 100 searches a day, the official API costs $15/month. At 1,000 searches/day (a moderate workload for an active agent), it's $150/month just for basic search results.

Shadow APIs: The Alternative

Every time you search on google.com, your browser receives structured data behind the scenes. Google's search page isn't just HTML — it includes embedded JSON data structures, AJAX endpoints for dynamic features, and API calls for autocomplete, People Also Ask expansions, and result card interactions.

These internal data sources return the full richness of Google search — featured snippets, knowledge panels, People Also Ask, local results, shopping results, and more — in structured formats that the Custom Search API doesn't provide.

Unbrowse captures these internal data patterns automatically from real browsing sessions.

What Unbrowse Discovers on Google

Searching Google through Unbrowse reveals the data layer behind the SERP:

• Search Results: GET /search?q={query}&... — the primary search endpoint returns results with embedded structured data. Unbrowse extracts the JSON-LD and data attributes from the response that contain titles, URLs, descriptions, and SERP feature data in machine-readable format.
• Autocomplete/Suggestions: GET /complete/search?q={partial}&client=gws-wiz — real-time search suggestions with completion candidates, estimated result counts, and suggestion types (query, entity, calculator).
• People Also Ask: Dynamic expansion endpoints that load PAA question-answer pairs as you click them. Each expansion returns structured Q&A data with source attribution.
• Knowledge Panel: GET /search?kgmid={entity_id} — knowledge graph entity data including descriptions, facts, related entities, and image URLs. Returns structured knowledge graph data.
• Related Searches: Embedded in the search response as structured data — semantically related queries that Google's algorithms identify as relevant to your search.

The key insight is that Google's SERP embeds structured data (JSON-LD, microdata, data attributes) that's richer than what the Custom Search API returns. Unbrowse captures and structures this data.

How It Works

npm install -g unbrowse
unbrowse resolve "search for best MCP servers for AI agents" --url https://google.com

The process:

1. Cookie Injection: Unbrowse injects your Google session cookies from your local browser. Your logged-in session means results include personalization and your language/region preferences.
2. Browse: Kuri opens Google with your real session and anti-detection extensions. Google sees a legitimate Chrome browser with a real user session.
3. Capture: The search response is captured with its full data layer — embedded JSON structures, AJAX calls for dynamic features, and API calls for autocomplete and expansions.
4. Index: Search patterns are extracted: query templates, result parsing schemas, pagination patterns, and SERP feature endpoints. Each search type (web, images, news, shopping) is indexed separately.
5. Cache and Execute: The search route is cached. Future searches call Google directly with your session, returning full SERP data that Unbrowse structures into clean JSON.

Performance

Metric	Custom Search API	SerpApi	Unbrowse (cached)
Speed	~300ms	~2,000ms	<200ms
Cost per query	$0.005	$0.002-0.01	$0.005
Results per query	10 max	10-100	Full SERP
Featured snippets	No	Yes	Yes
Knowledge panels	No	Yes	Yes
People Also Ask	No	Yes	Yes
Monthly cost (1K queries/day)	$150	$50-200	Free (open source)

The richness advantage is the real differentiator. Google's Custom Search API returns basic blue links. Unbrowse captures the full SERP — featured snippets, knowledge panels, People Also Ask, local packs, shopping results, and more.

When to Use This Approach

AI agent web search: Your agent needs to search the web as part of a larger task — finding documentation, checking facts, researching topics. Instead of paying per query through official APIs, Unbrowse provides full SERP data through your existing Google session.

SEO monitoring: Track search rankings, featured snippet ownership, and SERP feature presence for target keywords. The full SERP capture includes all features that official APIs miss — PAA, knowledge panels, local packs.

Market research: Search for competitor products, pricing information, and market trends. The autocomplete and related searches endpoints reveal what people are searching for — valuable market intelligence.

Content gap analysis: Analyze SERP results for target keywords to identify content opportunities. People Also Ask data reveals questions your content should answer. Related searches show adjacent topics to cover.

Getting Started

# Install Unbrowse globally
npm install -g unbrowse

# Run initial setup
unbrowse setup

# Search Google with full SERP capture
unbrowse resolve "search for browser automation for AI agents" --url https://google.com

# Get autocomplete suggestions
unbrowse resolve "get suggestions for 'how to build'" --url https://google.com

Unbrowse is open source and published on arXiv. Works as an MCP server for Claude and other AI agents.

FAQ

Is this legal? Unbrowse uses your authenticated Google session and accesses the same endpoints your browser calls when you search on google.com. You're making the same requests Chrome makes when you type a query.

How is this different from scraping? Scraping parses Google's HTML, fighting against obfuscation, dynamic rendering, and anti-bot measures. Unbrowse captures the structured data Google embeds in the page — JSON-LD, data attributes, and AJAX responses. Structured data, not HTML parsing.

Will Google block this? Unbrowse runs a real browser with your real Google session. From Google's perspective, it's indistinguishable from you searching normally. There's no proxy network, no fake browser fingerprint, no anomalous behavior pattern. The requests are identical to what Chrome sends.

How many searches can I make? Google rate-limits aggressive search patterns (hundreds of queries in minutes), but normal search rates (a few queries per minute) work fine. Unbrowse makes requests at human-like rates using your real session.

Does this work with Google's AI Overviews? Yes. When Google shows AI Overviews (SGE) for a query, the response data is captured along with traditional results. This gives you access to Google's AI-generated summaries — data that no official API provides.