Scraping snippets that allow you to scrape data from a page using JavaScript.

Scrape data with JavaScript

---

Get header and row values from an HTML table.

const headers = Array.prototype.map.call(document.querySelectorAll('#org-insight__a11y-table tr'), function(tr){
  return Array.prototype.map.call(tr.querySelectorAll('th'), function(th){
    return th.textContent.trim()
    })
  })

const table = Array.prototype.map.call(document.querySelectorAll('#org-insight__a11y-table tr'), function(tr){
  return Array.prototype.map.call(tr.querySelectorAll('td'), function(td){
    return td.textContent.trim()
    })
  })

const data = headers.concat(table)

return data

Scrape element attribute.

let results = []
let els = document.querySelectorAll("div:nth-child(1) > input")

for (const el of els) {
  result.push([el.getAttribute("name")])
}

return results

Scrape body and return result.

return [[document.querySelector("body").innerHTML]]

Get meta attributes from a page

---

It's possible to get meta attributes from a page using Javascript, this can be used to get Open Graph Meta Tags and other data about the page. You'll first need to ensure that the bot is currently on the page you wish to scrape the meta attributes from.

To return a single meta attribute's content:

let ogTitle = document.querySelector("meta[property='og:title']").getAttribute("content");
return ogTitle;

To return the name and contents of all meta tags from your automation:

let metaTags = document.getElementsByTagName("meta");
let metaContent = [];

for (var i = 0; i < metaTags.length; i++) {
  metaContent.push([metaTags[i].getAttribute("name"), metaTags[i].getAttribute("content")]);
}

return metaContent;

Note, some meta tags may not have a "name", this should be taken into account when building your automation.