Claude Code Web Automation Toolkit

3 web scraping and automation tools built for Claude Code workflows

A practical Life of Arjav guide for AI builders, Claude Code users, developers, founders, operators, and automation creators who want to scrape websites, control browsers, and automate structured workflows.

Brand: Life of Arjav
Instagram: @lifeofarjav
Keyword: Comment GITHUB to get this guide
Topic: Claude Code web scraping, browser automation, and workflow automation

---

What this guide helps you do

This guide explains 3 useful open-source tools:

1. Scrapling
2. nanobrowser
3. career-ops

These tools are useful if you want Claude Code to do more than write code.

They help with:

• scraping websites
• handling dynamic pages
• controlling browsers
• filling forms
• extracting structured data
• monitoring websites
• processing job listings
• tailoring workflows around real data

The point is not to collect random GitHub repos.

The point is to understand when each tool is useful, what problem it solves, and how to test it with Claude Code.

---

Quick links

---

The simple mental model

Claude Code is strongest when it can work with real files, real code, and real workflows.

But web automation creates different types of problems.

You need the right tool for the job.

Do not use a browser automation tool when a clean scraper is enough.

Do not use a scraper when the website requires real browser interaction.

Do not copy career-ops blindly. Study it as a workflow pattern.

---

Tool 1: Scrapling

Link

GitHub: https://github.com/D4Vinci/Scrapling
Docs: https://scrapling.readthedocs.io/
MCP guide: https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html

---

What Scrapling is

Scrapling is an adaptive web scraping framework.

Use it when you need to extract data from websites more reliably than basic requests plus BeautifulSoup.

It is useful for:

• single page scraping
• larger crawls
• dynamic pages
• JavaScript-heavy sites
• pagination
• selectors
• scraping workflows that need to survive small site changes
• market research
• competitor monitoring
• website change tracking

The carousel framing:

Scrapling helps Claude Code scrape competitor data, monitor websites for changes, collect market research, handle dynamic pages, and work with pagination or JavaScript-heavy sites.

---

When to use Scrapling

Use Scrapling when the task looks like this:

Go to these 200 competitor product pages.
Extract product name, price, category, availability, description, and review count.
Save the output to a CSV.
Flag pages where the layout changed or extraction failed.

Good fits:

• competitor pricing pages
• product catalogs
• directory pages
• job boards
• review pages
• documentation pages
• public market research sources
• pages with repeated structures

Bad fits:

• workflows that require logging into personal accounts
• complex browser sessions with many clicks
• anything that violates a site's terms
• scraping private, sensitive, or restricted data
• aggressive scraping without rate limits

---

What to ask Claude Code to build with Scrapling

Start with a narrow task.

Bad prompt:

Scrape the internet for market research.

Better prompt:

Use Scrapling to build a scraper for this public website.

Goal:
Extract company names, website URLs, categories, and descriptions from the directory pages.

Requirements:
1. Inspect the site structure first.
2. Create a small scraper for one page.
3. Test it on 3 pages.
4. Handle pagination if it exists.
5. Save results to output.csv.
6. Log failed pages.
7. Keep requests polite with delays.
8. Summarize what worked and what failed.

Do not scrape private pages.
Do not bypass access controls.

---

Example workflow

Website
  ↓
Scrapling scraper
  ↓
Clean structured data
  ↓
CSV or JSON output
  ↓
Claude Code analyzes the results

---

Claude Code prompt: Scrap competitor pages

I want to use Scrapling for a competitor research workflow.

Task:
Build a scraper that visits the public competitor pages I provide and extracts:
- company name
- homepage URL
- pricing page URL if visible
- main offer
- target customer
- key claims
- obvious CTA
- any visible pricing text

Inputs:
I will provide a list of URLs in input_urls.txt.

Requirements:
1. Inspect one URL first and explain the page structure.
2. Write the scraper in Python.
3. Save results to competitor_research.csv.
4. Add error handling.
5. Add polite delays.
6. Log failed URLs to failed_urls.txt.
7. Do not attempt to bypass logins or restricted content.
8. Summarize the findings after the run.

Use Scrapling where appropriate.
Make the script easy to rerun.

---

Claude Code prompt: Monitor website changes

Create a website monitoring workflow using Scrapling.

Goal:
Track changes on a list of public URLs once per day.

For each URL:
- extract page title
- extract main headline
- extract pricing text if visible
- extract primary CTA
- save a content hash
- compare against the previous run
- flag meaningful changes

Output:
1. changes_report.md
2. current_snapshot.csv
3. changed_urls.csv

Rules:
- Use polite request timing.
- Do not scrape private pages.
- Do not bypass access controls.
- Keep the code modular.

---

Quality checklist for Scrapling workflows

Before trusting the scraper, check:

• Did it handle failed pages?
• Did it respect rate limits?
• Did it save structured output?
• Did it avoid restricted pages?
• Did it test on a small sample first?
• Did it log errors clearly?
• Did it produce reusable code?
• Did the extracted data match the actual page?

A scraper that silently fails is worse than no scraper.

---

Tool 2: nanobrowser

Link

GitHub: https://github.com/nanobrowser/nanobrowser
Docs: https://nanobrowser.ai/docs
GitHub organization: https://github.com/nanobrowser

---

What nanobrowser is

nanobrowser is an open-source browser automation tool that runs in your browser.

Use it when the workflow needs real browser interaction.

This matters because some tasks cannot be handled well with API calls or basic scraping.

Examples:

• logging into web apps
• clicking through multiple screens
• filling forms
• navigating dashboards
• extracting information from dynamic interfaces
• running browser-based workflows
• seeing what is happening on screen

The carousel framing:

nanobrowser gives Claude Code actual browser control instead of only API calls.

---

When to use nanobrowser

Use nanobrowser when the task looks like this:

Open this dashboard.
Log in manually if needed.
Go to the reports section.
Change the date filter.
Export the data.
Repeat for 5 accounts.

Good fits:

• browser-first workflows
• form filling
• multi-step logins
• browser dashboards
• manual admin tasks
• workflows where the model needs visual feedback
• tasks where no API is available

Bad fits:

• simple data extraction from static HTML
• high-volume scraping
• tasks better solved with an official API
• anything involving sensitive actions without review
• anything that violates a website's rules

---

Example workflow

Claude Code
  ↓
Browser session
  ↓
Click, fill, navigate, extract
  ↓
Structured output
  ↓
Human review

---

Claude Code prompt: Browser workflow plan

Use this before building any nanobrowser workflow.

I want to automate a browser-based workflow.

Before coding or controlling the browser, create a workflow plan.

Task:
[Describe the task]

Website or app:
[Name or URL]

What needs to happen:
1. [Step 1]
2. [Step 2]
3. [Step 3]

Rules:
- Do not submit forms without confirmation.
- Do not make purchases.
- Do not delete anything.
- Do not change account settings without confirmation.
- Ask before taking any external action.
- Prefer read-only actions first.

Output:
1. Workflow map
2. Required inputs
3. Risks
4. Safe test plan
5. What should require human approval

---

Claude Code prompt: Extract dashboard data

Use a browser automation approach for this workflow.

Goal:
Extract report data from a browser dashboard.

Steps:
1. Open the dashboard URL I provide.
2. Wait for me to complete login if required.
3. Navigate to the reports page.
4. Set the date range to the last 30 days.
5. Extract the visible metrics.
6. Export or save the data if the interface allows it.
7. Save results as dashboard_report.csv.
8. Write a short summary in dashboard_summary.md.

Rules:
- Do not change settings.
- Do not submit anything externally.
- Do not delete data.
- Ask for confirmation before any write action.

---

Claude Code prompt: Form fill assistant

Help me build a safe browser automation workflow for form filling.

Goal:
Fill repeated form fields from a CSV file.

Inputs:
- form_data.csv
- target form URL

Rules:
1. Open the form.
2. Fill one test record first.
3. Stop before submitting.
4. Show me the filled form for approval.
5. Only submit after I confirm.
6. Log every submitted record.
7. Save failed records separately.

Output:
- automation script or workflow
- test plan
- submission log format
- failure handling plan

---

Safety checklist for browser automation

Browser automation can go wrong fast.

Before using it, check:

• Does this workflow require login?
• Does it touch private data?
• Does it submit forms?
• Does it send messages?
• Does it change settings?
• Does it delete or modify data?
• Does it need human approval?
• Is there an official API that would be safer?

If a workflow affects real people, money, accounts, or customer data, keep approval steps.

---

Tool 3: career-ops

Link

GitHub: https://github.com/santifer/career-ops
Website: https://career-ops.org/
Docs: https://career-ops.org/docs
Introduction: https://career-ops.org/docs/introduction/what-is-career-ops
Setup docs: https://github.com/santifer/career-ops/blob/main/docs/SETUP.md
CLAUDE.md: https://github.com/santifer/career-ops/blob/main/CLAUDE.md

---

What career-ops is

career-ops is an open-source local-first AI job search automation system.

It is built to run inside AI coding CLIs, including Claude Code.

It can help with job search workflows like:

• scanning job portals
• evaluating listings
• scoring fit
• tailoring CVs
• tracking applications
• generating structured outputs
• running job search workflows locally

The carousel framing:

career-ops is best as an example of how Claude Code can automate a complete real-world workflow.

That distinction matters.

You may not need job search automation.

But you should study the pattern.

career-ops shows how Claude Code can combine:

• web automation
• structured data processing
• local files
• prompts
• scoring rubrics
• repeatable workflows
• tracking systems

---

When to use career-ops

Use career-ops directly if you are:

• actively job hunting
• applying to many roles
• evaluating listings manually
• tailoring CVs repeatedly
• tracking applications in spreadsheets
• trying to learn from a complete AI workflow repo

Study career-ops as a pattern if you are building:

• lead research systems
• grant search systems
• vendor evaluation systems
• investor list systems
• recruiting workflows
• procurement workflows
• market monitoring workflows
• any workflow that scans many external records and scores them

---

The pattern behind career-ops

The workflow is not only "AI applies to jobs."

The real pattern is:

Find records
  ↓
Extract details
  ↓
Score against criteria
  ↓
Generate tailored output
  ↓
Track status
  ↓
Review results

That pattern applies far beyond job search.

Examples:

---

Claude Code prompt: Study career-ops

Use this if you want Claude Code to inspect the repo and explain it.

I want to study the career-ops repository as a Claude Code workflow pattern.

Repository:
https://github.com/santifer/career-ops

Please inspect the repo and explain:
1. What problem it solves
2. How the folder structure works
3. What Claude Code is expected to do
4. What prompts or commands drive the workflow
5. How scoring or evaluation works
6. How outputs are saved
7. What parts are reusable for other business workflows
8. What safety or reliability checks exist
9. What I should copy as a pattern
10. What I should not copy blindly

Do not run anything yet.
Start with a repo map and explanation.

---

Claude Code prompt: Adapt the pattern to lead research

Use the career-ops pattern to design a lead research workflow.

Goal:
Scan a list of company websites and score them as potential leads.

Inputs:
- company_urls.csv
- ICP criteria
- offer description
- disqualification rules

Workflow:
1. Visit each company website.
2. Extract relevant company information.
3. Score fit against the ICP.
4. Identify likely pain points.
5. Draft a short personalization angle.
6. Save results to lead_scores.csv.
7. Create a summary report.

Rules:
- Do not send outreach.
- Do not scrape private pages.
- Do not invent facts.
- Separate observed evidence from assumptions.
- Include confidence levels for each score.

Output:
- folder structure
- scoring rubric
- script plan
- prompt files
- CSV schema
- review checklist

---

Claude Code prompt: Build a scoring rubric

Create a structured scoring rubric for this workflow.

Workflow:
[Describe workflow]

Goal:
[What you are trying to choose, rank, or evaluate]

Criteria:
[List criteria]

Create:
1. Score dimensions
2. 1 to 5 scoring scale for each dimension
3. Required evidence for each score
4. Red flags
5. Disqualification rules
6. CSV output schema
7. Review checklist

Make the rubric strict enough that Claude cannot score everything highly.

---

Quality checklist for career-ops style workflows

Before trusting a scoring workflow, check:

• Are the scoring criteria explicit?
• Is the output saved in a structured format?
• Are assumptions separated from evidence?
• Is there a review process?
• Are low-confidence results flagged?
• Are disqualification rules clear?
• Can the workflow be rerun?
• Are results logged?

If every item gets a high score, your rubric is useless.

---

How to choose the right tool

Use this decision tree

Do you need to extract public website data?
  ↓
Use Scrapling.

Do you need clicks, forms, logins, or visual browser interaction?
  ↓
Use nanobrowser.

Do you want a complete example of Claude Code automating a structured real-world workflow?
  ↓
Study career-ops.

---

Simple comparison table

---

Practical starter project

If you want to test all 3 ideas, build this:

Project: Competitor monitoring system

Goal:

Track competitor websites and produce a weekly summary.

Use Scrapling for:

• public pricing pages
• product pages
• changelog pages
• blog pages

Use nanobrowser for:

• dashboards that require visual interaction
• multi-step pages
• pages where API calls are not enough

Use career-ops as inspiration for:

• scoring competitors
• tracking changes
• generating summary reports
• saving evidence
• maintaining a local tracker

---

Folder structure

claude-code-web-automation/
  inputs/
    competitor_urls.csv
    scoring_criteria.md

  scrapling/
    scraper.py
    snapshots/
    failed_urls.txt

  browser_workflows/
    workflow_plan.md
    extracted_reports/

  scoring/
    competitor_score_rubric.md
    scored_competitors.csv

  reports/
    weekly_competitor_summary.md
    changes_report.md

  prompts/
    scrape_competitor_pages.md
    browser_extract_dashboard.md
    score_competitors.md
    weekly_summary.md

---

Master prompt for Claude Code

You are helping me build a Claude Code web automation workflow.

Project:
Competitor monitoring system

Goal:
Track public competitor pages, detect meaningful changes, score them against our criteria, and create a weekly summary.

Tools to consider:
- Scrapling for public website scraping
- nanobrowser for real browser interaction when needed
- career-ops as a workflow pattern for scoring, tracking, and structured outputs

Rules:
1. Start by creating a repo map and implementation plan.
2. Do not scrape private or restricted pages.
3. Use polite request timing.
4. Prefer official APIs when available.
5. Save outputs in CSV or Markdown.
6. Log failures.
7. Separate observed evidence from assumptions.
8. Ask before any action that changes external systems.
9. Keep the first version small and testable.

Definition of done:
- One sample scraper works
- One output CSV is created
- One scoring rubric exists
- One weekly summary is generated
- Failed URLs are logged
- Next steps are documented

---

Setup checklist

Use this before you build.

1. Define the job

What exact workflow am I automating?

Bad:

Automate research.

Good:

Extract pricing changes from 20 competitor pages every Monday.

---

2. Choose the tool

Is this scraping, browser control, or a full workflow pattern?

• Scraping: Scrapling
• Browser control: nanobrowser
• Full workflow pattern: career-ops

---

3. Define inputs

Examples:

• URL list
• target fields
• scoring rubric
• account permissions
• output format
• review rules

---

4. Define outputs

Examples:

• output.csv
• summary.md
• failed_urls.txt
• change_log.md
• application_tracker.csv

---

5. Define safety rules

Examples:

• no private pages
• no bypassing access controls
• no form submission without approval
• no deleting
• no external messages without review
• no aggressive scraping

---

6. Test small

Start with:

• 1 website
• 3 pages
• 1 output file
• 1 summary

Do not start with a 500-site crawl.

That is how you create a fragile mess.

---

Common mistakes

Mistake 1: Using browser automation when scraping is enough

Browser automation is heavier.

If the data is available in clean HTML, use a scraper.

Mistake 2: Scraping when an API exists

If a safe official API exists, use it.

It is usually more stable and less risky.

Mistake 3: No failure logging

Web automation fails often.

Pages change. Selectors break. Logins expire. Rate limits happen.

Always log failures.

Mistake 4: No review step

Do not trust automation blindly.

Especially when results affect outreach, hiring, customer support, finance, or public publishing.

Mistake 5: No boundaries

Claude Code can act quickly.

That is useful.

It also means you need rules.

Read before write. Draft before send. Ask before submit. Log every result.

---

Safety and ethics

Web automation can be useful.

It can also cross lines if you are careless.

Use this rule:

If the data is private, restricted, sensitive, or protected by access controls, do not scrape it without clear permission.

Basic rules:

• Respect robots.txt and website terms where applicable
• Use rate limits
• Prefer official APIs
• Do not bypass paywalls or authentication
• Do not collect sensitive personal data unnecessarily
• Do not submit forms or messages without approval
• Do not use automation to spam
• Keep logs of what your automation did

A smart automation builder does not just ask, "Can I automate this?"

They ask, "Should I automate this, and what are the boundaries?"

---

One-page quick start

If you want web scraping

Start with Scrapling.

Prompt:

Build a small Scrapling scraper for these 3 public URLs.
Extract the fields I list.
Save results to CSV.
Log failures.
Use polite delays.
Do not bypass access controls.

If you want browser control

Start with nanobrowser.

Prompt:

Create a browser automation workflow plan.
The task requires clicking, filling forms, and extracting visible data.
Stop before submitting anything.
Ask for approval before external actions.

If you want a full workflow pattern

Study career-ops.

Prompt:

Inspect career-ops and explain how its job search workflow is structured.
Then show me how to adapt the same pattern to my workflow.

---

Final takeaway

The real lesson is not "use these 3 tools."

The real lesson is that Claude Code becomes much more powerful when it is paired with the right workflow layer.

Use:

• Scrapling when you need reliable web data
• nanobrowser when the workflow needs a real browser
• career-ops when you want to study an end-to-end automation system

Start small.

Test one workflow.

Log everything.

Review before action.

Then scale.

---

Life of Arjav

Follow @lifeofarjav for practical AI, tech, and growth systems.

Comment GITHUB on the Instagram post to get this guide.