CFAI Scrapper MCP

1What it is

cfai-scrapper-mcp is a single MCP server that wraps every social-platform scraping backend Creativefuel uses behind one consistent surface. It exposes 18 tools across 6 platforms — X (Twitter), Facebook, LinkedIn, YouTube, Threads, and Instagram — available both as MCP stdio (for Claude Code, Claude Desktop, Cursor) and as a HTTP REST API (for production at scrapper.cfai.in).

Built so consumer products (community.cfai.in, cfai-chat, cfai-comment-insights, future tools) hit ONE service instead of each integrating Apify, Jarvis, and scrapper.jarvis.work directly. The server preserves Naveen's sequential-1.5s lock for IG scrapper calls, caches Jarvis IG responses for 1h, normalizes every backend's payload into a unified Envelope schema, and tracks per-key daily $ caps via SQLite.

Backends wrapped

2High-level architecture

Cross-cutting features

• Unified Envelope — every tool returns the same shape: {platform, kind, identity, author, content, engagement, timestamps, flags, raw, source}. The raw field always preserves the unmodified backend payload so consumers never lose data.
• Per-key auth — X-API-Key required on every /v1/tools/* call. Master key has unlimited cap; consumer keys have configurable daily $ caps.
• SQLite-backed — api_keys (bcrypt-hashed), audit_log (every call logged with cost/latency/status/args), daily_spend (per-key per-day with hard cap → HTTP 429), cache (TTL-based for Jarvis IG).
• Admin SPA at /admin/ with HTTP-Basic auth (ADMIN_PASSWORD env). Tabs: Overview · Activity · Logs · API Keys · Health.

3Installation

Prerequisites

• Python 3.11+ (developed against 3.13)
• Node 20+ + npm (only if you want the admin dashboard SPA)
• An Apify token with a paid plan (Starter+) — the actors are pay-per-event; the free tier returns x402-payment-required
• (Optional, for IG) Jarvis & scrapper.jarvis.work API keys

Local install

git clone https://github.com/Bot222555/cfai-scrapper-mcp.git
cd cfai-scrapper-mcp

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .

cp .env.example .env
# Open .env and fill: APIFY_TOKEN, MCP_MASTER_KEY, ADMIN_PASSWORD (minimum)

Build admin dashboard SPA

cd admin-ui
npm install
npm run build       # writes admin-ui/dist/, picked up automatically by HTTP server

.env reference

# REQUIRED
APIFY_TOKEN=apify_api_...
MCP_MASTER_KEY=cfai_scr_master_...
ADMIN_PASSWORD=...

# OPTIONAL — Instagram backends
JARVIS_BASE_URL=https://jarvis.work:8080
JARVIS_API_KEY=svc_...
JARVIS_VERIFY_SSL=false

SCRAPPER_BASE_URL=https://scrapper.jarvis.work
SCRAPPER_API_KEY=scrapper_...
SCRAPPER_PHA_KEY_COMMENTS=pha_...
SCRAPPER_PHA_KEY_ANALYSE=pha_...
SCRAPPER_SEQUENTIAL_DELAY_MS=1500

INSIGHTS_BASE_URL=http://localhost:5700
INSIGHTS_API_KEY=cfai_ci_...

RESOLVER_BASE_URL=https://api.cfai.in
RESOLVER_API_KEY=...

# OPTIONAL — bootstrap consumer keys (label:key:daily_cap_usd, comma separated)
MCP_BOOTSTRAP_KEYS=community:cfai_scr_community_2026:5,chat:cfai_scr_chat_2026:5
DEFAULT_DAILY_USD_CAP=5

# OPTIONAL — server config
HTTP_HOST=0.0.0.0
HTTP_PORT=5800
DB_PATH=./scrapper.db
LOG_LEVEL=INFO
APIFY_RUN_TIMEOUT_SECS=600
APIFY_POLL_INTERVAL_SECS=4

Running

# stdio mode (Claude Code/Desktop/Cursor)
.venv/bin/cfai-scrapper-mcp

# HTTP mode (production)
.venv/bin/cfai-scrapper-http
# binds 0.0.0.0:5800 by default

Quick sanity check

curl http://localhost:5800/healthz
# → {"ok":true,"service":"cfai-scrapper-mcp","version":"0.1.0"}

curl -X POST http://localhost:5800/v1/tools/list_platforms \
  -H 'X-API-Key: <your master key>' -d '{}'
# → {"platforms":["x","facebook","linkedin","youtube","threads","instagram"],"count":6}

4Wiring into Claude Code / Desktop / Cursor

Claude Code — project-level (recommended)

The repo includes a .mcp.json at the project root. Open Claude Code in the cfai-scrapper-mcp/ directory and the MCP server auto-discovers; you'll be prompted to approve on first call.

Claude Code — workspace-level

Drop a .mcp.json at /Users/tushar/CF AI/.mcp.json:

{
  "mcpServers": {
    "cfai-scrapper": {
      "command": "/Users/tushar/CF AI/cfai-scrapper-mcp/.venv/bin/cfai-scrapper-mcp"
    }
  }
}

To skip per-call permission prompts, add to ~/.claude/settings.json under permissions.allow: "mcp__cfai-scrapper__*".

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "cfai-scrapper": {
      "command": "/Users/tushar/CF AI/cfai-scrapper-mcp/.venv/bin/cfai-scrapper-mcp"
    }
  }
}

Restart Claude Desktop. The 18 tools appear in the tool picker.

Cursor

Edit ~/.cursor/mcp.json with the same shape as Claude Desktop above.

5Authentication & rate limits

Every /v1/tools/* request needs an X-API-Key header. Keys are issued from the admin dashboard at /admin/ (or POST /api/admin/keys). Each key has:

• Daily $ cap — resets at UTC midnight. Exceeded calls return HTTP 429.
• Active flag — toggle without deleting.
• Admin flag — admin keys have unlimited cap and can call meta tools.

Cached responses (source.cached = true) don't count toward the cap.

Bootstrap keys

Set MCP_BOOTSTRAP_KEYS in .env (comma-separated label:key:cap tuples) to auto-create consumer keys on first start. Idempotent — existing labels are skipped.

Issuing keys via API

curl -X POST https://scrapper.cfai.in/api/admin/keys \
  -u admin:<ADMIN_PASSWORD> \
  -H 'Content-Type: application/json' \
  -d '{"label":"my-script","daily_cap_usd":5,"is_admin":false}'
# → {"id":"...","label":"my-script","key":"cfai_scr_my-script_xxx...","daily_cap_usd":5}
# (the full key is shown ONCE — copy immediately)

6Unified response envelope

Every successful tool call returns this shape:

{
  "platform": "instagram | x | facebook | linkedin | youtube | threads",
  "kind": "profile | post | comment | search_result | company |
            transcript | reaction | conversation | shares | analysis |
            shortcode_lookup",
  "identity": { "id": "...", "url": "...", "shortcode": "..." },
  "author": {
    "platform_id": "...", "username": "...", "display_name": "...",
    "verified": true, "follower_count": 12345,
    "profile_pic_url": "...", "bio": "...", "location": "...",
    "extra": { /* platform-specific extras */ }
  },
  "content": {
    "text": "...", "language": "en",
    "media": [{"type": "image", "url": "..."}],
    "hashtags": [], "mentions": [], "links": []
  },
  "engagement": {
    "likes": 0, "comments": 0, "shares": 0, "views": 0,
    "saves": 0, "bookmarks": 0, "quotes": 0, "reposts": 0,
    "reactions": { "like": 0, "love": 0 }
  },
  "timestamps": { "posted_at_utc": "...", "fetched_at_utc": "..." },
  "flags": {
    "is_paid_partnership": false, "is_reply": false,
    "is_repost": false, "is_quote": false, "is_monetized": false,
    "possibly_sensitive": false, "age_restricted": false
  },
  "raw": { /* full original backend response, untouched */ },
  "source": {
    "backend": "apify | jarvis | scrapper | insights",
    "actor_or_endpoint": "apidojo/tweet-scraper",
    "cached": false, "cost_usd": 0.0004, "latency_ms": 6500
  }
}

For tools that return many records (posts, comments, search), the wrapper is:

{
  "platform": "x", "kind": "post",
  "items": [ /* array of envelopes above */ ],
  "count": 30,
  "source": { /* aggregate */ },
  "cursor": null
}

7Universal tools (5)

POST /v1/tools/scrape_profile
{ "platform": "x", "identifier": "MissMalini" }

POST /v1/tools/scrape_posts
{ "platform": "x",
  "identifier": "MissMalini",
  "count": 30,
  "since_date": "2026-04-01" }

POST /v1/tools/scrape_post
{ "platform": "youtube",
  "post_url": "https://www.youtube.com/watch?v=zRtGL0-5rg4" }

POST /v1/tools/scrape_comments
{ "platform": "instagram",
  "post_url": "https://www.instagram.com/p/DX10_sjCEKZ/",
  "count": 50 }

POST /v1/tools/scrape_search
{ "platform": "x",
  "query": "#Bollywood",
  "count": 30,
  "filters": {
    "language": "en",
    "min_likes": 50,
    "min_retweets": 10,
    "only_verified": true,
    "start": "2026-04-01",
    "end": "2026-04-30"
  } }

8Platform-specific tools (9)

9Meta tools (4)

10Capability matrix

✓ = wired in v0.1 · ✗ = not exposed by upstream (or not yet wired) · n/a = doesn't apply on that platform · partial = some forms.

11Per-platform field reference

X / Twitter — apidojo/tweet-scraper

Profile (extracted from latest tweet's author): identity.url/id; author.username/display_name/verified/follower_count/profile_pic_url/bio/location; author.extra.{following, statuses_count, media_count, favourites_count, created_at (account birthday), is_blue_verified, twitter_url}.

Post: identity.id/url/shortcode (=conversationId); author (full profile snapshot); content.text/fullText/language/source ("Web"/"iPhone"/etc); media (from extendedEntities); hashtags / mentions / links; engagement.likes/comments(=replyCount)/views/bookmarks/quotes/reposts; flags.is_reply/is_repost/is_quote/possibly_sensitive.

Filters in scrape_search: language, min_likes, min_retweets, min_replies, only_verified, only_blue, only_image, only_video, only_quote, start, end, sort.

Facebook — apify/facebook-* family

Page metadata (scrape_profile / scrape_facebook_metadata): identity.id (=pageId)/url; author.username/display_name (=title)/profile_pic_url; author.extra.{categories, websites, info, messenger, cover_photo, personal_profile (if URL was a personal account)}.

Post: identity.postId/url; author.username (=pageName)/display_name; content.text/media; engagement.likes/shares; engagement.reactions (LIKE+LOVE+total).

Gaps: no comment count, no view count, no full reaction breakdown.

LinkedIn — harvestapi/* + pratikdani

Profile (48 fields — biggest payload): identity.id/url; author.username (=publicIdentifier)/display_name (=firstName+lastName)/verified/follower_count/profile_pic_url/bio (=headline)/location; author.extra.{about, connections_count, current_position, experience[], education[], skills[], certifications, honors_and_awards, languages, publications, patents, projects, services, volunteering, received_recommendations, registered_at, influencer, premium, open_to_work, hiring}.

Post / Comment (returned mixed, discriminated by raw.type): POST: author, content, engagement.{likes,comments,reposts}, media. COMMENT: author (commenter), content (=commentary), engagement.likes.

Reactions (unique): EnvelopeList(kind=reaction), each with author = the individual reactor (with their bio + photo).

Company (36 fields): identity.id (=company_id)/url; author.username (=universal_name_id)/display_name/bio (=tagline or description)/profile_pic_url/follower_count/location (=headquarters); author.extra.{industry, employee_count, company_size, company_type, founded, specialties, website, funding, investors, jobs, locations, affiliated_pages, similar_pages, crunchbase_url, stock_info}.

YouTube — streamers/youtube-scraper

Channel profile: identity.id (=channelId)/url; author.username (=channelUsername)/display_name (=channelName)/verified (=isChannelVerified)/follower_count (=numberOfSubscribers)/profile_pic_url (=channelAvatarUrl)/bio (=channelDescription)/location; author.extra.{channel_url, channel_banner_url, channel_total_videos, channel_total_views, channel_joined_date, about_channel_info}.

Video / post: identity.id/url; author = full channel object (every video record bundles channel data); content.text (=title or description), media, hashtags, descriptionLinks; engagement.likes/comments (=commentsCount)/views (=viewCount); flags.is_monetized/age_restricted/is_paid_partnership (=isPaidContent); duration in raw.

Filters in scrape_search: date_filter, video_type, length_filter, is_hd, has_subtitles, has_cc, is_live, sort. Transcript: kind=transcript with subtitles in content.text (SRT or whatever format subtitlesFormat was set to).

Threads — apify/threads-*

Profile (27 fields + bundled latest 4 posts): identity.id (=pk)/url; author.username/display_name (=full_name)/verified (=is_verified)/follower_count/profile_pic_url/bio (=biography); author.extra.{is_private, bio_links, profile_tags, transparency_label, hd_profile_pic_versions}.

Post (33 fields): identity.id (=pk)/url (=canonical_url)/shortcode (=code); author = the post user; content.text/media (image/video/carousel/audio); engagement.likes (=like_count); engagement.comments (=text_post_app_info.direct_reply_count); engagement.reposts (=repost_count); engagement.quotes (=quote_count); engagement.shares (=reshare_count); flags.is_paid_partnership.

No view count — Meta hasn't exposed views on Threads' API. engagement.views will always be None.

Instagram — Jarvis + scrapper composite

Profile (Jarvis primary, scrapper fallback on 404 or followers=0 anomaly): identity.id/url; author.username/display_name (=full_name)/verified (=is_verified)/follower_count/profile_pic_url/bio (=biography); author.extra.{following, post_count (=media_count), is_private, is_active_on_text_post_app}. When scrapper fallback fires, extra also includes engagement_rate, avg_likes, avg_comments, avg_views, recentPosts[].

Post: identity.id/url/shortcode; author.username; content.text (=caption); engagement.likes/comments/views/shares (only via scrapper /share-count fallback).

Comments (scrapper only, sequential 1.5s lock): identity.id; author.username/display_name/profile_pic_url; content.text; engagement.likes. Top-100-liked + 100-random-sample by default (PHA cost-control).

Shares (scrapper /share-count specifically): engagement.shares = reshareCount, plus likes/comments/views/playCount.

Analysis (cfai-comment-insights multimodal): kind=analysis; raw.data = full LLM verdict with verdict (viral/strong/avg/under), content_analysis (visual_summary from thumbnail, hooks, CTA, branding), audience (segments, geo, lang_mix), reaction (sentiment, themes, notable_quotes), signals, recommendations.

12Admin endpoints (/api/admin)

HTTP-Basic auth using ADMIN_PASSWORD env var (no username check). Used by the React SPA at /admin/.

13Cost model

Every successful call records cost_usd from the upstream backend.

• Apify: per-actor pricing (PPE), e.g. apidojo/tweet-scraper at $0.40/1k tweets.
• Jarvis: $0 (in-house infra).
• Scrapper: $0 to caller (in-house, but rate-limited — sequential 1.5s).
• Insights: $0 to caller (in-house, but consumes its own LLM credits).

Daily $ cap is enforced PER API key. Cap exceeded → HTTP 429 until UTC midnight. Cached responses (source.cached = true) don't count toward the cap.

Sample cost projection

For a 100-creator monthly sweep × 5 Apify-backed platforms × 30 posts each = 15,000 records:

14Production deployment

Live at https://scrapper.cfai.in on cfai-llm-gateway EC2 (15.206.98.152, Tailscale 100.86.128.89), co-located with cfai-chat-backend.

• Path: /opt/cfai-scrapper-mcp/
• Env: /etc/cfai-scrapper-mcp/.env (chmod 640, root:ubuntu)
• Data: /var/lib/cfai-scrapper-mcp/scrapper.db
• Service: systemd cfai-scrapper-mcp.service, port 5800
• nginx vhost: /etc/nginx/sites-available/scrapper.cfai.in.conf
• TLS: Let's Encrypt cert auto-renewed via certbot timer
• DNS: Route53 A record scrapper.cfai.in → 15.206.98.152 (TTL 300)

Deploy command

# From local repo, after committing changes
./deploy.sh ubuntu@100.86.128.89

# What it does:
#   1. rsync source (excluding .venv, __pycache__, .env, *.db, .git)
#   2. python venv + pip install -e .
#   3. systemd unit install + daemon-reload
#   4. nginx vhost install (idempotent)
#   5. systemctl restart + curl /healthz

One-time host setup

# SCP setup-host.sh to the EC2 + run it once
ssh ubuntu@<host> "sudo bash" < deploy/setup-host.sh

# Then EDIT the production .env that setup-host.sh created
sudo nano /etc/cfai-scrapper-mcp/.env

# Then run deploy.sh from local

15Consumer integrations

Production consumer keys (issued 2026-05-02)

JHS-affiliate + trend-radar deferred per Tushar's instruction — those services keep direct integrations.

cfai-chat agent tools

The chat agent (chat.cfai.in) now has 5 scraper tools registered in its ALL_TOOLS: scrape_profile, scrape_posts, scrape_post, scrape_comments, scrape_search. Each takes a platform arg and routes to scrapper.cfai.in via SCRAPPER_MCP_API_KEY env var. Errors map cleanly: 401 → "connector unavailable", 429 → "daily $ cap reached", 501 → "tool not supported for that platform".

community.cfai.in feature flag

Three wrapper files (jarvis_ig.py, scrapper_client.py, scrapper_post.py) check the USE_SCRAPPER_MCP env flag. When true, they delegate to scrapper_mcp_client.py which calls scrapper.cfai.in. Same canonical return shapes preserved so callers + tests don't change.

16Troubleshooting

17Examples

Daily creator health snapshot across 5 platforms

HANDLE=missmalini
KEY=cfai_scr_community_2026

for PLATFORM in x facebook linkedin youtube threads; do
  echo "=== $PLATFORM ==="
  curl -sS -X POST https://scrapper.cfai.in/v1/tools/scrape_profile \
    -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
    -d "{\"platform\":\"$PLATFORM\",\"identifier\":\"$HANDLE\"}" \
    | python3 -c "
import sys, json
d = json.load(sys.stdin)
a = d.get('author', {})
print(f'  {a.get(\"username\")} · {a.get(\"follower_count\"):,} followers · verified={a.get(\"verified\")}')"
done

Hashtag monitoring on X with engagement floor

curl -X POST https://scrapper.cfai.in/v1/tools/scrape_search \
  -H 'X-API-Key: cfai_scr_trendradar_2026' \
  -H 'Content-Type: application/json' \
  -d '{
    "platform": "x",
    "query": "#cfai",
    "count": 50,
    "filters": {
      "language": "en",
      "min_likes": 10,
      "start": "2026-05-01",
      "only_blue": true
    }
  }'

IG comment depth-dive on a high-engagement post

curl -X POST https://scrapper.cfai.in/v1/tools/scrape_comments \
  -H 'X-API-Key: cfai_scr_community_2026' \
  -H 'Content-Type: application/json' \
  -d '{
    "platform": "instagram",
    "post_url": "https://www.instagram.com/p/DX10_sjCEKZ/",
    "count": 200
  }'

LinkedIn company page

curl -X POST https://scrapper.cfai.in/v1/tools/scrape_linkedin_company \
  -H 'X-API-Key: cfai_scr_master_2026' \
  -H 'Content-Type: application/json' \
  -d '{ "company_url": "https://www.linkedin.com/company/creativefuel/" }'

YouTube video transcript

curl -X POST https://scrapper.cfai.in/v1/tools/scrape_youtube_transcript \
  -H 'X-API-Key: cfai_scr_master_2026' \
  -H 'Content-Type: application/json' \
  -d '{
    "video_url": "https://www.youtube.com/watch?v=zRtGL0-5rg4",
    "language": "en"
  }'

X conversation tree

curl -X POST https://scrapper.cfai.in/v1/tools/scrape_x_conversation \
  -H 'X-API-Key: cfai_scr_master_2026' \
  -H 'Content-Type: application/json' \
  -d '{ "conversation_id": "2050084119883964840", "depth": 50 }'