JAVASCRIPT - HOW TO Extract All Links from URL/Web Address - Javascript Source Code

JAVASCRIPT - HOW TO Extact All the Links (or any other html element you wish) from a Web Page URL/Web Address - NOT from HTML Code but an Actual Web Address/URL

Home Short:

In Javascript you can scan a live web page and extract elements from it, just like you can in php. The main difference is with Javascript, you are mostly restricted to being able to scan web pages that reside on the same url as where the Javascript code resides due to known as Access Control Allow Origin which is a security feature that prevents a website from crawling and fetching content from a different website. Why html standards allow this to occur with PHP code and not Javascript I am unsure of.

Here is how to take a web page url on your own website, grab the html code from that web page url, convert the html code into a document and then use Javascripts .querySelectorAll() function to extract any html elements you wish. In this particular example we are extracting the links from the document created from the html code grabbed from a web page url.

3 Steps Required to Get All Links from a Web Adress/URL

Ajax to fetch the html from the web address/url.
Convert the Ajax returned html code into a document which can be processed by Javascript code.
Extract whatever elements you wish using the Javascript .querySelectorAll() function.

// CREATE VARIABLE THE CALLS THE FUNCTION THAT RETURNS THE SUCCESSFULLY CREATED AJAX OBJECT
var ajaxRequest = funcCreatedAjaxObject();

// TEST FOR SUCCESSFUL CREATION OF AJAX OBJECT & CONTINUE WITH CODE
if(ajaxRequest !== null){

// REPLACE [] WITH THE ACTUAL URL/WEB ADDRESS OF THE WEB PAGE YOU WANT TO EXTRACT CONTENT FROM
     ajaxRequest.open("GET","url-of-page-to-extract-links-from.html", true);
     ajaxRequest.send(null);

// IN CASE ERROR CREATING AJAX OBJECT
}else{
     // CODE TO WRITE IF FOR SOME STRANGE REASON AN AJAX OBJECT FAILS AT BEING CREATED
}

// FUNCTION TO CONVERT HTML CODE TO A DOCUMENT THAT JAVASCRIPT CAN EXTRACT ELEMENTS FROM
function funcScanLinks(htmlcode){

// CONVERT HTML CODE TO AN ACTUAL HTML DOCUMENT
  var doc = document.createElement("html");
  doc.innerHTML = htmlcode;

// IN THIS EXAMPE WE ARE EXTRACTING LINKS. WE COULD EXTRACT ANY HTML ELEMENT
  // WE WANT. FOR REFERENCE REGARDING JAVASCRIPTS .querySelectorAll SEE 
  // https://www.geeksforgeeks.org/html-dom-queryselectorall-method/
  let arrLinks = doc.querySelectorAll('a');
 
  // WE COULD LOOP THROUGH THE LINKS AND DO WHAT WE WISH WITH THEM
  for(let i = 0; i < arrLinks.length; i++){
       console.log(arrLinks[i].href + '|' + arrLinks[i].textContent + '\n');
  }
}

///////////  AJAX RESPONSE RECEIVED & COMPLETE /////////////
ajaxRequest.onload = function() {

// SUCCESSFULLY RECEIVED AJAX RESPONSE & DATA
  if (ajaxRequest.status == 200) {
      
       var varResponse = ajaxRequest.responseText.trim();

// FUNCTION TO CALL TO SEND THE HTML CODE. THE FUNCTION CONVERTS HTML CODE TO A DOCUMENT
       funcScanLinks(varResponse);

// AJAX STATUS ERROR
  } else {  
      
      // ERROR PROCESSING CODE

}
};

/////////// CREATE AJAX OBJECT /////////////
function funcCreatedAjaxObject(){

var aj;

try{
      // IN MOST CASES, THIS WILL SUCCESSFULLY CREATE THE AJAX OBJECT
      aj = new XMLHttpRequest();
 }

catch (e){

try{
           // IF THE COMMON WAY TO CREATE AJAX OBJECT FAILS ...
           aj = new ActiveXObject("Msxml2.XMLHTTP");
       }

catch(e2){

try{
                // IF THE SECOND METHOD OF CREATING AJAX OBJECT FAILS ...
                aj = new ActiveXObject("Microsoft.XMLHTTP");
            }

catch(e3){
                 // IF ALL METHODS OF CREATING AJAX OBJECT FAILS
                 return false;
            }
       }
  }

// RETURN THE AJAX OBJECT TO THE VARIABLE THAT CALLED THIS FUNCTION
  return aj;
}

source code home