Scraper API
The line-up
Every scraper you can call today, by family, with what goes in and what comes out. What is still being built is listed at the end, and labelled that way until it is real.
Careers and job boards
Job openings from where employers publish them: their own careers sites and their applicant tracking systems.
careers.companyAvailableCompany careers site
- In
- A company's domain, or its careers page.
- Out
- The job openings that company lists itself.
Starting from the domain, it finds the careers page. When the page is backed by an applicant tracking system it recognises, it reads that board's public feed; otherwise it reads the openings off the page itself. Every record says which of the two it was, in its extraction field: ats or page.
What it does not do
- Pages that list their openings only after running JavaScript are not read yet. Those runs end as unsupported, with the reason render_required, and cost nothing.
- Openings read off the company's own pages are unverified: some may not be job postings. A run that has only such records ends as partial, with the reason unverified_page_extraction, never as a plain ok.
- It can pick the wrong page. Pass careers_url when you know it.
- company_name is null unless you pass one or the board's feed carries it.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
domainOne of the two- The company's website domain, such as northwind.example.
careers_urlOne of the two- The company's careers page, when you already know it.
company_nameOptional- Helps tell the company's own job board from another company's.
max_detail_pagesOptional- How many posting pages are read for details, at most. 30 unless you say otherwise; up to 200.
{
"input": {
"domain": "northwind.example"
}
}The fields of a record
titlecompany_namecompany_domainlocationemployment_typeposted_atvalid_throughsalarydescriptionapply_urlpage_urlatsexternal_idfetched_atextraction
ats.boardAvailableATS job board
- In
- The address of one board on an applicant tracking system (Greenhouse, Lever, Ashby, Workday and others).
- Out
- That board's job openings.
It recognises which applicant tracking system the address belongs to and reads the board the way that system publishes it, a public feed where there is one.
What it does not do
- A board it does not recognise ends as unsupported, with the reason unknown_board, and costs nothing.
- Boards that need a JavaScript render are not read yet (Dayforce boards always need one). Those runs end as unsupported, with the reason render_required, and cost nothing.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
board_urlRequired- The board's address, such as https://boards.greenhouse.io/northwind.
company_nameOptional- The employer's name, for boards that do not give it.
max_detail_pagesOptional- How many posting pages are read for details, at most. 30 unless you say otherwise; up to 200.
{
"input": {
"board_url": "https://boards.greenhouse.io/northwind"
}
}The fields of a record
titlecompany_namecompany_domainlocationemployment_typeposted_atvalid_throughsalarydescriptionapply_urlpage_urlatsexternal_idfetched_atextraction
LinkedIn's public pages, as a visitor who is not signed in sees them. Nothing behind a sign-in is read.
linkedin.jobs_searchAvailableLinkedIn job search
- In
- Keywords, a place, and how recently the job was listed.
- Out
- The postings LinkedIn's public job search lists for them.
It pages through LinkedIn's public job search, the one a visitor who is not signed in sees, until it has the number of postings you asked for or LinkedIn has no more.
What it does not do
- A record holds what a search result shows: title, company, location, the date it was listed and the posting's address. The description and the other details come from the LinkedIn job posting scraper.
- LinkedIn's job-type, workplace and experience filters are not offered: its logged-out search ignores them.
- LinkedIn serves at most 1,000 results for one search.
- If LinkedIn refuses a page part-way through, the run ends as partial with the postings read before it.
Billed per record: a run that returns data costs 1 unit for each record it returns.
Input
keywordsAt least one of the three- Words to search for, such as a job title.
locationAt least one of the three- A place as text, such as Texas or United States.
geo_idAt least one of the three- LinkedIn's numeric id for the place, which is more exact than text.
posted_within_hoursOptional- Only postings listed in the last this many hours.
max_resultsOptional- How many postings are returned, at most. 25 unless you say otherwise; LinkedIn serves 1,000 a search.
{
"input": {
"keywords": "plant manager",
"location": "Texas"
}
}The fields of a record
titlecompany_namecompany_domainlocationemployment_typeposted_atvalid_throughsalarydescriptionapply_urlpage_urlatsexternal_idfetched_atcompany_linkedin_urlcompany_linkedin_idposted_agoseniority_leveljob_functionindustriesapplicantsaccepting_applicationsdescription_html
linkedin.jobAvailableLinkedIn job posting
- In
- A LinkedIn job posting's address, or its id.
- Out
- The posting's public details: description, seniority, employment type, job function and industries.
It reads the posting's public page, the one a visitor who is not signed in sees.
What it does not do
- No apply link: LinkedIn keeps it behind a sign-in, so apply_url is null.
- No posting date: the page gives only a relative age, returned in posted_ago as LinkedIn wrote it.
- Names of recruiters or hiring managers on the page are personal data and are not returned.
- A posting LinkedIn says does not exist ends as failed, with the reason not_found, and costs nothing.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
urlOne of the two- The posting's address on linkedin.com.
job_idOne of the two- The posting's numeric id, when you know it.
{
"input": {
"job_id": "4487824457"
}
}The fields of a record
titlecompany_namecompany_domainlocationemployment_typeposted_atvalid_throughsalarydescriptionapply_urlpage_urlatsexternal_idfetched_atcompany_linkedin_urlcompany_linkedin_idposted_agoseniority_leveljob_functionindustriesapplicantsaccepting_applicationsdescription_html
linkedin.companyAvailableLinkedIn company page
- In
- A LinkedIn company page's address, or its slug.
- Out
- The company's public profile: website, industry, size, headquarters, founding year and followers.
It reads the company's public page, the one a visitor who is not signed in sees.
What it does not do
- Only what the logged-out page shows. The company's description, its employees, its posts and its funding are not in the record.
- Size, type and founding year are LinkedIn's own words, not parsed into numbers.
- A page LinkedIn says does not exist ends as failed, with the reason not_found, and costs nothing.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
urlOne of the two- The company page's address on linkedin.com.
slugOne of the two- The page's name in its address: northwind for linkedin.com/company/northwind.
{
"input": {
"slug": "northwind"
}
}The fields of a record
namesluglinkedin_urllinkedin_idwebsiteindustrycompany_sizeheadquarterscompany_typefoundedspecialtiesfollowerslogo_urlfetched_at
Web
A public web page, as its server sends it.
web.fetchAvailableWeb page
- In
- The address of a public web page.
- Out
- The status, headers and body its server sent.
It makes one GET request and returns what came back. A page the site itself answers with a 404 or a 500 is still the page you asked for: the run ends as ok and the record carries that status.
What it does not do
- GET only: no request body, no cookies, no login.
- No JavaScript is run. A page that builds its content in the browser comes back as the shell it starts from.
- The body is cut at 2 MB unless you ask for another limit, and the record's truncated field says when it was.
- A site that refuses the request, with a 403, a 429 or a CAPTCHA, ends the run as failed, with the reason blocked, rate_limited or captcha. It costs nothing.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
urlRequired- The page's address: http or https, on a public host name.
methodOptional- GET is the only method.
headersOptional- A few request headers, such as Accept-Language or User-Agent. No cookie and no Authorization header.
follow_redirectsOptional- Follow up to 5 redirects. Send false to get the redirect itself.
timeout_sOptional- Seconds for the whole fetch. 30 unless you say otherwise; up to 120.
max_bytesOptional- The body is cut after this many bytes. 2,000,000 (2 MB) unless you say otherwise.
{
"input": {
"url": "https://northwind.example/pricing"
}
}The fields of a record
urlfinal_urlstatusheaderscontent_typecharsetbodybody_encodingbytestruncatedredirectsfetched_at
web.markdownAvailableWeb page to Markdown
- In
- The address of a public web page.
- Out
- The page as clean Markdown, with its title, description, links and metadata.
What it does not do
- No JavaScript is run. A page that builds its content in the browser is not read.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
main_content_onlyOptional- Write the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.
include_linksOptional- Write links as [text](address); false keeps their text only.
include_imagesOptional- Write images as ; false keeps their alt text only.
urlRequired- The page's address: http or https, on a public host name.
timeout_sOptional- Seconds for the whole fetch, redirects included.
{
"input": {
"url": "https://northwind.example/pricing"
}
}The fields of a record
urlfinal_urlstatuscontent_typetitledescriptionlanguagecanonical_urlmarkdowntext_lengthword_countlinksmetadatatruncateddepthfetched_at
web.crawlAvailableWebsite to Markdown
- In
- A start page on a public website.
- Out
- The site's pages as clean Markdown, one record for each page.
What it does not do
- No JavaScript is run. A page that builds its content in the browser is not read.
- It reads the start page's site only, and it obeys the site's robots.txt.
Billed per record: a run that returns data costs 1 unit for each record it returns.
Input
main_content_onlyOptional- Write the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.
include_linksOptional- Write links as [text](address); false keeps their text only.
include_imagesOptional- Write images as ; false keeps their alt text only.
urlRequired- The start page: http or https, on a public host name.
max_pagesOptional- Stop after this many pages (records).
max_depthOptional- How many links away from the start page a page may be; 0 reads the start page only. Pages from the sitemap count as 1.
include_pathsOptional- Read only pages whose path starts with one of these (`/docs/`), or matches one with `*` (`/blog/*/comments`). The start page is always read.
exclude_pathsOptional- Never read pages whose path starts with or matches one of these.
same_site_onlyOptional- Only the start page's site is crawled; this cannot be turned off.
max_secondsOptional- The time budget of the whole crawl, in seconds.
{
"input": {
"url": "https://northwind.example/"
}
}The fields of a record
urlfinal_urlstatuscontent_typetitledescriptionlanguagecanonical_urlmarkdowntext_lengthword_countlinksmetadatatruncateddepthfetched_at
YouTube
YouTube's public pages, as a visitor who is not signed in sees them.
youtube.searchAvailableYouTube search
- In
- A search query.
- Out
- The videos, Shorts, channels and playlists on the first page of YouTube's results.
What it does not do
- Only the first page of results is read.
Billed per record: a run that returns data costs 1 unit for each record it returns.
Input
queryRequired- What to search YouTube for.
max_resultsOptional- The most results to return. Only the first page of results is read (about 20, plus Shorts), so fewer may come back.
typeOptional- Only results of this kind (YouTube's own Type filter). Leave out for the mixed results YouTube shows by default.
{
"input": {
"query": "how to fix a flat bike tire"
}
}The fields of a record
typepositionidurltitlechannel_namechannel_idchannel_urlchannel_handlechannel_verifiedpublished_textduration_textduration_secondsview_count_textview_countview_count_exactis_livelive_viewerssubscriber_count_textsubscriber_countvideo_count_textvideo_countdescription_snippetbadgesthumbnails
youtube.videoAvailableYouTube video
- In
- A YouTube video's address, or its id.
- Out
- Its title, description, channel, dates, length, views, likes, tags and caption languages.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
urlOne of the two- The video's address: a youtube.com/watch, youtu.be, /shorts/, /live/ or /embed/ link.
video_idOne of the two- The video's 11-character id.
{
"input": {
"video_id": "dQw4w9WgXcQ"
}
}The fields of a record
idurltitledescriptionchannel_namechannel_idchannel_urlchannel_handlechannel_verifiedchannel_subscriber_count_textpublished_atuploaded_atpublished_textduration_secondsview_countlike_countkeywordscategoryis_liveis_live_contentis_upcominglive_started_atlive_ended_atis_unlistedis_family_safeage_restrictedplayableplayability_statusplayability_reasonembeddablethumbnailscaption_trackscaption_languagesfetched_at
youtube.channel_videosAvailableYouTube channel videos
- In
- A YouTube channel's address, handle or id.
- Out
- The latest videos on the channel's Videos tab, newest first.
What it does not do
- Only the first page of the Videos tab is read: 30 videos at most.
Billed per record: a run that returns data costs 1 unit for each record it returns.
Input
channelRequired- The channel: its address (youtube.com/@name, /channel/UC…, /c/name, /user/name), its handle (@name) or its id (UC…).
max_resultsOptional- The most videos to return, newest first. Only the first page of the Videos tab is read (30 videos).
{
"input": {
"channel": "@northwind"
}
}The fields of a record
positionidurltitlechannel_idchannel_namepublished_textduration_textduration_secondsview_count_textview_countview_count_exactbadgesthumbnails
Walmart
Walmart's public search results and product pages.
walmart.searchAvailableWalmart search
- In
- A search query.
- Out
- The products on a page of Walmart's search results: prices, ratings and sellers, with advertisements marked.
What it does not do
- One run reads one page of results.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
queryRequired- What to search for, e.g. "wireless mouse".
pageOptional- The results page, from 1 (Walmart serves at most 25).
max_resultsOptional- Return at most this many items of the page; all of them when not given.
sortOptional- Walmart's sort order.
include_sponsoredOptional- False leaves the advertisements (sponsored: true) out.
{
"input": {
"query": "wireless mouse"
}
}The fields of a record
item_idproduct_idurltitlepricewas_priceratingreview_countbrandseller_nameseller_idavailabilityin_stockfulfilment_typefulfilment_optionssponsoredbadgeimagematchpositionpagefetched_at
walmart.productAvailableWalmart product
- In
- A Walmart product's address, or its item id.
- Out
- Its price, availability, seller, rating, specifications, images and variants.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
urlOne of the two- The product's address, e.g. https://www.walmart.com/ip/Some-Product/13802306327.
item_idOne of the two- Walmart's item id: the number at the end of the product's address.
{
"input": {
"item_id": "13802306327"
}
}The fields of a record
item_idproduct_idupcgtinurltitlebrandmodelshort_descriptiondescriptionpricewas_priceavailabilityseller_nameseller_idratingreview_countcategory_pathspecificationsimagesvariantsfetched_at
App stores
An app's public listing, on Apple's App Store or on Google Play.
app_store.appAvailableApp Store app
- In
- An app's id or address on Apple's App Store, and a country.
- Out
- Its listing there: developer, category, rating, price, description, version and images.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
app_idOne of the two- Apple's numeric app id, e.g. 310633997 (the digits after "id" in the app's address).
urlOne of the two- The app's address, e.g. https://apps.apple.com/us/app/whatsapp-messenger/id310633997.
countryOptional- The storefront, as a two-letter country code (default us). Price, rating and availability are per storefront. With `url`, the address's own storefront is used and `country` may only repeat it.
{
"input": {
"app_id": "310633997"
}
}The fields of a record
app_idbundle_idurltitlesubtitledescriptiondevelopercategorygenresratingrating_countpricefreein_app_purchasescontent_ratingversionupdated_atwhats_newiconscreenshotsprivacy_policy_urlrequirementsdevicescountrylanguagefetched_at
google_play.appAvailableGoogle Play app
- In
- An app's package name on Google Play.
- Out
- Its listing there: developer, category, rating, installs, price, description, last update and images.
Billed per request: a run that returns data costs 1 unit, however many records it returns.
Input
app_idRequired- The app's package name, e.g. com.whatsapp.
countryOptional- Two-letter country code: the store's `gl` parameter. Prices and availability follow it.
languageOptional- Language code: the store's `hl` parameter, e.g. en, de, pt-BR.
{
"input": {
"app_id": "com.northwind.app"
}
}The fields of a record
app_idurltitlesummarydescriptiondevelopercategorycategory_idratingrating_countreview_countinstallsmin_installspricefreein_app_purchasesin_app_price_rangecontains_adscontent_ratingreleased_atupdated_atversionwhats_newiconheader_imagescreenshotsprivacy_policy_urlcountrylanguagefetched_at
The job record
Every scraper that returns job openings returns these fields. Three are always present. Anything else we could not read is null, never an empty string or a zero.
| Field | Type | What it holds |
|---|---|---|
title | string | Always present. |
company_name | string or null | The employer, when known. |
company_domain | string or null | The employer's website domain. |
location | object or null | raw, city, region, country, remote. |
employment_type | string or null | As the posting states it. |
posted_at | date or null | When the posting says it was published. |
valid_through | date or null | The date the posting expires, if it gives one. |
salary | object or null | raw, min, max, currency, period. |
description | string or null | The posting's text. |
apply_url | string or null | Where to apply. |
page_url | string | The posting's page. Always present. |
ats | string or null | The applicant tracking system, or generic for a page read directly. |
external_id | string or null | The posting's id on its board. |
fetched_at | datetime | When we read it, in UTC. Always present. |
The careers scrapers add extraction, which says whether a record came from an applicant tracking system (ats) or was read off the company’s own pages (page). The LinkedIn scrapers add what only LinkedIn says, such as seniority_level and posted_ago. Each scraper above lists its own fields.
The overview shows two records in full, and the documentation has the complete schema.
In development
These are being built. They cannot be called yet, and we are not giving dates. They will read public, logged-out pages only, like everything else here.
Search results (Bing)
A search query in. The organic results out, in order: title, address and snippet.
In development
Website contacts
A company's domain in. The contact details its website publishes out: email addresses, phone numbers and social profiles.
In development
More scrapers are planned, and they will be listed here once they are decided. Need one of these, or a source that is not listed? Tell us. It helps us decide what to build next.
Want job openings as data, without running scrapes? Datasets is a separate product, in development.