How to Access Twitter/X Data Without Scraping

Twitter's API used to be one of the most generous in tech. Researchers, developers, and startups built entire businesses on free tweet access. Then Elon Musk bought Twitter, rebranded it to X, and put the API behind a paywall: $100/month for Basic access, $5,000/month for Pro, and $42,000/month for Enterprise.

If you need tweet data for an AI agent, research project, or monitoring tool, you're looking at either paying up or fighting X's increasingly aggressive anti-scraping measures. But there's a path most developers overlook.

The Problem with X Data Access in 2026

The current landscape for accessing X/Twitter data is bleak:

• Official API (Basic): $100/month. Limited to 10,000 tweets/month read access. No search endpoint. No streaming.
• Official API (Pro): $5,000/month. Full search, 1M tweets/month. Overkill for most use cases, but the only option for search.
• HTML Scraping: X.com is a React SPA that loads content dynamically. Traditional HTTP-based scraping doesn't work. Headless browser scraping is blocked by sophisticated fingerprinting (Canvas, WebGL, font enumeration).
• Third-party services: Apify, ScrapFly, and similar services charge $49-299/month for X data, with no SLA on reliability.
• Nitter/RSS proxies: Mostly dead. X systematically shut down public Nitter instances in 2024.

For AI agents, the situation is worse. An agent that needs to check what people are saying about a topic on X has to either budget $100+/month per user or give up on X data entirely.

Shadow APIs: The Alternative

Every time you visit x.com, your browser makes GraphQL API calls behind the scenes. These internal endpoints return clean JSON data — the same data that renders your timeline, search results, and tweet threads.

X.com's frontend is built on a GraphQL API layer that handles everything: timeline fetching, search, user profiles, trending topics, and more. These endpoints are authenticated via your session cookies and return richly structured data.

Unbrowse captures these shadow APIs automatically from real browsing sessions.

What Unbrowse Discovers on X

When you browse X through Unbrowse, it captures the GraphQL endpoints powering the frontend:

• Search: GET /i/api/graphql/{hash}/SearchTimeline?variables={query} — full search results with tweets, users, and media. Supports "Latest" and "Top" sorting, media filters, and date ranges. Returns tweet text, engagement metrics, author info, and media URLs.
• Home Timeline: GET /i/api/graphql/{hash}/HomeTimeline — your personalized feed with full tweet objects, retweets, and quoted tweets.
• User Tweets: GET /i/api/graphql/{hash}/UserTweets?variables={userId} — all tweets from a specific user with pagination cursors.
• Tweet Detail: GET /i/api/graphql/{hash}/TweetDetail?variables={tweetId} — full tweet with complete reply thread, quote tweets, and engagement counts.
• Trending: GET /i/api/graphql/{hash}/ExploreTrending — current trending topics with tweet counts and categories.

These GraphQL endpoints return the same data structure X's frontend uses to render pages — complete with tweet text, author metadata, engagement counts, media URLs, and conversation threading.

How It Works

npm install -g unbrowse
unbrowse resolve "search for AI agent news" --url https://x.com

The process:

1. Browse: Unbrowse opens x.com in Kuri (a real Chromium browser) with your existing X session cookies extracted from your local Chrome or Firefox profile. No login needed — your existing session is reused.
2. Capture: As the page loads, Unbrowse's fetch interceptor captures every GraphQL call the frontend makes. The HAR recorder logs request/response pairs with full headers and bodies.
3. Index: Each captured endpoint is analyzed: URL templates are extracted, authentication headers are identified (X uses x-csrf-token + session cookies), query variables are mapped to semantic intents.
4. Cache: Indexed routes are stored locally with their auth patterns. The GraphQL operation names (SearchTimeline, UserTweets, etc.) become the route identifiers.
5. Execute: Future requests call X's GraphQL API directly with your session credentials. No browser launch needed.

Performance

Metric	Browser Automation (Playwright)	X Official API (Basic)	Unbrowse (cached)
Speed	~4,200ms	~300ms	<100ms
Tokens consumed	~12,000	~800	~250
Cost per action	$0.53	$0.01 + $100/mo fee	$0.005
Search access	Via browser	Not included (Pro only)	Included
Rate limit	Browser resources	10K tweets/month	Session-based
Monthly cost	Compute only	$100 minimum	Free (open source)

The biggest advantage is search access. X's Basic API tier ($100/month) doesn't include search at all — you need the $5,000/month Pro tier. Unbrowse gives you search via the same GraphQL endpoint x.com uses.

When to Use This Approach

AI agent social monitoring: Your agent needs to track what people say about your product, competitors, or industry topics on X. Instead of paying $5,000/month for API search access, Unbrowse discovers the SearchTimeline endpoint and calls it directly.

Trend detection: Pull trending topics and viral tweets for content strategy, news monitoring, or market research. The ExploreTrending endpoint gives you real-time trend data with engagement metrics.

Influencer analytics: Monitor specific accounts' tweet performance, follower engagement, and posting patterns. UserTweets and TweetDetail endpoints provide complete engagement data without API quotas.

Content curation: Build feeds that aggregate tweets about specific topics for newsletters, dashboards, or AI training data pipelines.

Getting Started

# Install Unbrowse globally
npm install -g unbrowse

# Run initial setup (configures browser, detects cookies)
unbrowse setup

# Make sure you're logged into x.com in Chrome first, then:
unbrowse resolve "search for latest AI developments" --url https://x.com

# Subsequent calls use cached GraphQL routes
unbrowse resolve "get trending topics" --url https://x.com

Unbrowse is open source, published on arXiv, and works as an MCP server for AI agents.

FAQ

Is this legal? Unbrowse uses your authenticated browser session and respects robots.txt. It accesses the same GraphQL APIs your browser calls when you scroll through x.com. You're browsing with your own account — Unbrowse just captures and reuses the API patterns.

How is this different from scraping? Scraping parses rendered HTML or simulates browser interactions. Unbrowse discovers and calls the actual GraphQL APIs — the same ones X's React frontend uses. The data comes back as structured JSON with typed fields, not HTML fragments.

What about X's anti-bot detection? Unbrowse runs a real browser with real user credentials and a real browser fingerprint. From X's perspective, it looks like a normal user browsing the site. There's no bot detection to trigger because there's no bot — it's your actual browser session.

Will the GraphQL hashes change? X rotates GraphQL operation hashes periodically (they're content hashes of the query). Unbrowse re-discovers endpoints automatically when hashes change. The semantic intent mapping means your code doesn't need to track hash changes.

Can I use this with Claude or other AI agents? Yes. Unbrowse runs as an MCP server. Any agent that speaks MCP can call unbrowse resolve to search X, fetch tweets, or get trending topics — all returning structured JSON.