Skip to the content

Scraper API

The line-up

Every scraper you can call today, by family, with what goes in and what comes out. What is still being built is listed at the end, and labelled that way until it is real.

Careers and job boards

Job openings from where employers publish them: their own careers sites and their applicant tracking systems.

careers.companyAvailable

Company careers site

In
A company's domain, or its careers page.
Out
The job openings that company lists itself.

Starting from the domain, it finds the careers page. When the page is backed by an applicant tracking system it recognises, it reads that board's public feed; otherwise it reads the openings off the page itself. Every record says which of the two it was, in its extraction field: ats or page.

What it does not do

  • Pages that list their openings only after running JavaScript are not read yet. Those runs end as unsupported, with the reason render_required, and cost nothing.
  • Openings read off the company's own pages are unverified: some may not be job postings. A run that has only such records ends as partial, with the reason unverified_page_extraction, never as a plain ok.
  • It can pick the wrong page. Pass careers_url when you know it.
  • company_name is null unless you pass one or the board's feed carries it.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

domainOne of the two
The company's website domain, such as northwind.example.
careers_urlOne of the two
The company's careers page, when you already know it.
company_nameOptional
Helps tell the company's own job board from another company's.
max_detail_pagesOptional
How many posting pages are read for details, at most. 30 unless you say otherwise; up to 200.
The smallest request body
{
  "input": {
    "domain": "northwind.example"
  }
}

The fields of a record

  • title
  • company_name
  • company_domain
  • location
  • employment_type
  • posted_at
  • valid_through
  • salary
  • description
  • apply_url
  • page_url
  • ats
  • external_id
  • fetched_at
  • extraction
ats.boardAvailable

ATS job board

In
The address of one board on an applicant tracking system (Greenhouse, Lever, Ashby, Workday and others).
Out
That board's job openings.

It recognises which applicant tracking system the address belongs to and reads the board the way that system publishes it, a public feed where there is one.

What it does not do

  • A board it does not recognise ends as unsupported, with the reason unknown_board, and costs nothing.
  • Boards that need a JavaScript render are not read yet (Dayforce boards always need one). Those runs end as unsupported, with the reason render_required, and cost nothing.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

board_urlRequired
The board's address, such as https://boards.greenhouse.io/northwind.
company_nameOptional
The employer's name, for boards that do not give it.
max_detail_pagesOptional
How many posting pages are read for details, at most. 30 unless you say otherwise; up to 200.
The smallest request body
{
  "input": {
    "board_url": "https://boards.greenhouse.io/northwind"
  }
}

The fields of a record

  • title
  • company_name
  • company_domain
  • location
  • employment_type
  • posted_at
  • valid_through
  • salary
  • description
  • apply_url
  • page_url
  • ats
  • external_id
  • fetched_at
  • extraction

LinkedIn

LinkedIn's public pages, as a visitor who is not signed in sees them. Nothing behind a sign-in is read.

linkedin.jobAvailable

LinkedIn job posting

In
A LinkedIn job posting's address, or its id.
Out
The posting's public details: description, seniority, employment type, job function and industries.

It reads the posting's public page, the one a visitor who is not signed in sees.

What it does not do

  • No apply link: LinkedIn keeps it behind a sign-in, so apply_url is null.
  • No posting date: the page gives only a relative age, returned in posted_ago as LinkedIn wrote it.
  • Names of recruiters or hiring managers on the page are personal data and are not returned.
  • A posting LinkedIn says does not exist ends as failed, with the reason not_found, and costs nothing.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

urlOne of the two
The posting's address on linkedin.com.
job_idOne of the two
The posting's numeric id, when you know it.
The smallest request body
{
  "input": {
    "job_id": "4487824457"
  }
}

The fields of a record

  • title
  • company_name
  • company_domain
  • location
  • employment_type
  • posted_at
  • valid_through
  • salary
  • description
  • apply_url
  • page_url
  • ats
  • external_id
  • fetched_at
  • company_linkedin_url
  • company_linkedin_id
  • posted_ago
  • seniority_level
  • job_function
  • industries
  • applicants
  • accepting_applications
  • description_html
linkedin.companyAvailable

LinkedIn company page

In
A LinkedIn company page's address, or its slug.
Out
The company's public profile: website, industry, size, headquarters, founding year and followers.

It reads the company's public page, the one a visitor who is not signed in sees.

What it does not do

  • Only what the logged-out page shows. The company's description, its employees, its posts and its funding are not in the record.
  • Size, type and founding year are LinkedIn's own words, not parsed into numbers.
  • A page LinkedIn says does not exist ends as failed, with the reason not_found, and costs nothing.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

urlOne of the two
The company page's address on linkedin.com.
slugOne of the two
The page's name in its address: northwind for linkedin.com/company/northwind.
The smallest request body
{
  "input": {
    "slug": "northwind"
  }
}

The fields of a record

  • name
  • slug
  • linkedin_url
  • linkedin_id
  • website
  • industry
  • company_size
  • headquarters
  • company_type
  • founded
  • specialties
  • followers
  • logo_url
  • fetched_at

Web

A public web page, as its server sends it.

web.fetchAvailable

Web page

In
The address of a public web page.
Out
The status, headers and body its server sent.

It makes one GET request and returns what came back. A page the site itself answers with a 404 or a 500 is still the page you asked for: the run ends as ok and the record carries that status.

What it does not do

  • GET only: no request body, no cookies, no login.
  • No JavaScript is run. A page that builds its content in the browser comes back as the shell it starts from.
  • The body is cut at 2 MB unless you ask for another limit, and the record's truncated field says when it was.
  • A site that refuses the request, with a 403, a 429 or a CAPTCHA, ends the run as failed, with the reason blocked, rate_limited or captcha. It costs nothing.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

urlRequired
The page's address: http or https, on a public host name.
methodOptional
GET is the only method.
headersOptional
A few request headers, such as Accept-Language or User-Agent. No cookie and no Authorization header.
follow_redirectsOptional
Follow up to 5 redirects. Send false to get the redirect itself.
timeout_sOptional
Seconds for the whole fetch. 30 unless you say otherwise; up to 120.
max_bytesOptional
The body is cut after this many bytes. 2,000,000 (2 MB) unless you say otherwise.
The smallest request body
{
  "input": {
    "url": "https://northwind.example/pricing"
  }
}

The fields of a record

  • url
  • final_url
  • status
  • headers
  • content_type
  • charset
  • body
  • body_encoding
  • bytes
  • truncated
  • redirects
  • fetched_at
web.markdownAvailable

Web page to Markdown

In
The address of a public web page.
Out
The page as clean Markdown, with its title, description, links and metadata.

What it does not do

  • No JavaScript is run. A page that builds its content in the browser is not read.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

main_content_onlyOptional
Write the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.
include_linksOptional
Write links as [text](address); false keeps their text only.
include_imagesOptional
Write images as ![alt](address); false keeps their alt text only.
urlRequired
The page's address: http or https, on a public host name.
timeout_sOptional
Seconds for the whole fetch, redirects included.
The smallest request body
{
  "input": {
    "url": "https://northwind.example/pricing"
  }
}

The fields of a record

  • url
  • final_url
  • status
  • content_type
  • title
  • description
  • language
  • canonical_url
  • markdown
  • text_length
  • word_count
  • links
  • metadata
  • truncated
  • depth
  • fetched_at
web.crawlAvailable

Website to Markdown

In
A start page on a public website.
Out
The site's pages as clean Markdown, one record for each page.

What it does not do

  • No JavaScript is run. A page that builds its content in the browser is not read.
  • It reads the start page's site only, and it obeys the site's robots.txt.

Billed per record: a run that returns data costs 1 unit for each record it returns.

Input

main_content_onlyOptional
Write the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.
include_linksOptional
Write links as [text](address); false keeps their text only.
include_imagesOptional
Write images as ![alt](address); false keeps their alt text only.
urlRequired
The start page: http or https, on a public host name.
max_pagesOptional
Stop after this many pages (records).
max_depthOptional
How many links away from the start page a page may be; 0 reads the start page only. Pages from the sitemap count as 1.
include_pathsOptional
Read only pages whose path starts with one of these (`/docs/`), or matches one with `*` (`/blog/*/comments`). The start page is always read.
exclude_pathsOptional
Never read pages whose path starts with or matches one of these.
same_site_onlyOptional
Only the start page's site is crawled; this cannot be turned off.
max_secondsOptional
The time budget of the whole crawl, in seconds.
The smallest request body
{
  "input": {
    "url": "https://northwind.example/"
  }
}

The fields of a record

  • url
  • final_url
  • status
  • content_type
  • title
  • description
  • language
  • canonical_url
  • markdown
  • text_length
  • word_count
  • links
  • metadata
  • truncated
  • depth
  • fetched_at

YouTube

YouTube's public pages, as a visitor who is not signed in sees them.

youtube.videoAvailable

YouTube video

In
A YouTube video's address, or its id.
Out
Its title, description, channel, dates, length, views, likes, tags and caption languages.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

urlOne of the two
The video's address: a youtube.com/watch, youtu.be, /shorts/, /live/ or /embed/ link.
video_idOne of the two
The video's 11-character id.
The smallest request body
{
  "input": {
    "video_id": "dQw4w9WgXcQ"
  }
}

The fields of a record

  • id
  • url
  • title
  • description
  • channel_name
  • channel_id
  • channel_url
  • channel_handle
  • channel_verified
  • channel_subscriber_count_text
  • published_at
  • uploaded_at
  • published_text
  • duration_seconds
  • view_count
  • like_count
  • keywords
  • category
  • is_live
  • is_live_content
  • is_upcoming
  • live_started_at
  • live_ended_at
  • is_unlisted
  • is_family_safe
  • age_restricted
  • playable
  • playability_status
  • playability_reason
  • embeddable
  • thumbnails
  • caption_tracks
  • caption_languages
  • fetched_at
youtube.channel_videosAvailable

YouTube channel videos

In
A YouTube channel's address, handle or id.
Out
The latest videos on the channel's Videos tab, newest first.

What it does not do

  • Only the first page of the Videos tab is read: 30 videos at most.

Billed per record: a run that returns data costs 1 unit for each record it returns.

Input

channelRequired
The channel: its address (youtube.com/@name, /channel/UC…, /c/name, /user/name), its handle (@name) or its id (UC…).
max_resultsOptional
The most videos to return, newest first. Only the first page of the Videos tab is read (30 videos).
The smallest request body
{
  "input": {
    "channel": "@northwind"
  }
}

The fields of a record

  • position
  • id
  • url
  • title
  • channel_id
  • channel_name
  • published_text
  • duration_text
  • duration_seconds
  • view_count_text
  • view_count
  • view_count_exact
  • badges
  • thumbnails

Walmart

Walmart's public search results and product pages.

walmart.productAvailable

Walmart product

In
A Walmart product's address, or its item id.
Out
Its price, availability, seller, rating, specifications, images and variants.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

urlOne of the two
The product's address, e.g. https://www.walmart.com/ip/Some-Product/13802306327.
item_idOne of the two
Walmart's item id: the number at the end of the product's address.
The smallest request body
{
  "input": {
    "item_id": "13802306327"
  }
}

The fields of a record

  • item_id
  • product_id
  • upc
  • gtin
  • url
  • title
  • brand
  • model
  • short_description
  • description
  • price
  • was_price
  • availability
  • seller_name
  • seller_id
  • rating
  • review_count
  • category_path
  • specifications
  • images
  • variants
  • fetched_at

App stores

An app's public listing, on Apple's App Store or on Google Play.

app_store.appAvailable

App Store app

In
An app's id or address on Apple's App Store, and a country.
Out
Its listing there: developer, category, rating, price, description, version and images.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

app_idOne of the two
Apple's numeric app id, e.g. 310633997 (the digits after "id" in the app's address).
urlOne of the two
The app's address, e.g. https://apps.apple.com/us/app/whatsapp-messenger/id310633997.
countryOptional
The storefront, as a two-letter country code (default us). Price, rating and availability are per storefront. With `url`, the address's own storefront is used and `country` may only repeat it.
The smallest request body
{
  "input": {
    "app_id": "310633997"
  }
}

The fields of a record

  • app_id
  • bundle_id
  • url
  • title
  • subtitle
  • description
  • developer
  • category
  • genres
  • rating
  • rating_count
  • price
  • free
  • in_app_purchases
  • content_rating
  • version
  • updated_at
  • whats_new
  • icon
  • screenshots
  • privacy_policy_url
  • requirements
  • devices
  • country
  • language
  • fetched_at
google_play.appAvailable

Google Play app

In
An app's package name on Google Play.
Out
Its listing there: developer, category, rating, installs, price, description, last update and images.

Billed per request: a run that returns data costs 1 unit, however many records it returns.

Input

app_idRequired
The app's package name, e.g. com.whatsapp.
countryOptional
Two-letter country code: the store's `gl` parameter. Prices and availability follow it.
languageOptional
Language code: the store's `hl` parameter, e.g. en, de, pt-BR.
The smallest request body
{
  "input": {
    "app_id": "com.northwind.app"
  }
}

The fields of a record

  • app_id
  • url
  • title
  • summary
  • description
  • developer
  • category
  • category_id
  • rating
  • rating_count
  • review_count
  • installs
  • min_installs
  • price
  • free
  • in_app_purchases
  • in_app_price_range
  • contains_ads
  • content_rating
  • released_at
  • updated_at
  • version
  • whats_new
  • icon
  • header_image
  • screenshots
  • privacy_policy_url
  • country
  • language
  • fetched_at

The job record

Every scraper that returns job openings returns these fields. Three are always present. Anything else we could not read is null, never an empty string or a zero.

FieldTypeWhat it holds
titlestringAlways present.
company_namestring or nullThe employer, when known.
company_domainstring or nullThe employer's website domain.
locationobject or nullraw, city, region, country, remote.
employment_typestring or nullAs the posting states it.
posted_atdate or nullWhen the posting says it was published.
valid_throughdate or nullThe date the posting expires, if it gives one.
salaryobject or nullraw, min, max, currency, period.
descriptionstring or nullThe posting's text.
apply_urlstring or nullWhere to apply.
page_urlstringThe posting's page. Always present.
atsstring or nullThe applicant tracking system, or generic for a page read directly.
external_idstring or nullThe posting's id on its board.
fetched_atdatetimeWhen we read it, in UTC. Always present.

The careers scrapers add extraction, which says whether a record came from an applicant tracking system (ats) or was read off the company’s own pages (page). The LinkedIn scrapers add what only LinkedIn says, such as seniority_level and posted_ago. Each scraper above lists its own fields.

The overview shows two records in full, and the documentation has the complete schema.

In development

These are being built. They cannot be called yet, and we are not giving dates. They will read public, logged-out pages only, like everything else here.

  • Search results (Bing)

    A search query in. The organic results out, in order: title, address and snippet.

    In development

  • Website contacts

    A company's domain in. The contact details its website publishes out: email addresses, phone numbers and social profiles.

    In development

More scrapers are planned, and they will be listed here once they are decided. Need one of these, or a source that is not listed? Tell us. It helps us decide what to build next.

Want job openings as data, without running scrapes? Datasets is a separate product, in development.