There are several explanations you could possibly need to search out the many URLs on an internet site, but your specific purpose will decide Anything you’re looking for. By way of example, you might want to:
Determine just about every indexed URL to analyze challenges like cannibalization or index bloat
Acquire current and historic URLs Google has viewed, especially for web page migrations
Locate all 404 URLs to Get well from article-migration faults
In Just about every scenario, one tool won’t Provide you anything you need. Sadly, Google Search Console isn’t exhaustive, and a “web page:instance.com” look for is restricted and difficult to extract facts from.
With this article, I’ll stroll you thru some tools to construct your URL list and in advance of deduplicating the info using a spreadsheet or Jupyter Notebook, determined by your website’s size.
Old sitemaps and crawl exports
In the event you’re seeking URLs that disappeared from your live web site not too long ago, there’s an opportunity another person on the crew could have saved a sitemap file or perhaps a crawl export prior to the improvements ended up created. In case you haven’t by now, check for these files; they're able to typically give what you require. But, for those who’re looking through this, you almost certainly didn't get so Blessed.
Archive.org
Archive.org
Archive.org is an invaluable Device for Search engine optimisation responsibilities, funded by donations. In the event you seek for a site and select the “URLs” choice, you could accessibility around 10,000 mentioned URLs.
Even so, There are many limits:
URL limit: You'll be able to only retrieve approximately web designer kuala lumpur ten,000 URLs, which is insufficient for greater web sites.
Top quality: Several URLs could be malformed or reference resource data files (e.g., illustrations or photos or scripts).
No export choice: There isn’t a crafted-in solution to export the listing.
To bypass The dearth of an export button, use a browser scraping plugin like Dataminer.io. However, these constraints imply Archive.org might not offer a whole solution for more substantial sites. Also, Archive.org doesn’t reveal regardless of whether Google indexed a URL—but when Archive.org found it, there’s a superb prospect Google did, too.
Moz Professional
When you might normally make use of a backlink index to search out exterior web-sites linking to you, these resources also learn URLs on your site in the procedure.
Tips on how to utilize it:
Export your inbound hyperlinks in Moz Professional to acquire a fast and straightforward listing of goal URLs from the site. When you’re dealing with a huge Web site, consider using the Moz API to export knowledge outside of what’s manageable in Excel or Google Sheets.
It’s imperative that you note that Moz Professional doesn’t verify if URLs are indexed or found by Google. Nevertheless, given that most websites implement the identical robots.txt guidelines to Moz’s bots as they do to Google’s, this process frequently works nicely like a proxy for Googlebot’s discoverability.
Google Search Console
Google Look for Console presents many worthwhile sources for developing your listing of URLs.
One-way links studies:
Similar to Moz Pro, the One-way links area provides exportable lists of focus on URLs. Sadly, these exports are capped at one,000 URLs Just about every. You are able to apply filters for distinct webpages, but considering that filters don’t use to the export, you might really need to rely on browser scraping applications—limited to five hundred filtered URLs at any given time. Not best.
Overall performance → Search Results:
This export provides an index of pages receiving search impressions. Even though the export is restricted, you can use Google Search Console API for bigger datasets. You will also find totally free Google Sheets plugins that simplify pulling more extensive data.
Indexing → Web pages report:
This section supplies exports filtered by concern variety, though these are definitely also limited in scope.
Google Analytics
Google Analytics
The Engagement → Pages and Screens default report in GA4 is an excellent resource for amassing URLs, that has a generous limit of 100,000 URLs.
Better yet, you may use filters to generate distinct URL lists, successfully surpassing the 100k limit. For example, if you would like export only website URLs, observe these actions:
Stage one: Add a phase to the report
Step 2: Click on “Produce a new phase.”
Move three: Determine the section by using a narrower URL sample, which include URLs containing /web site/
Be aware: URLs located in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they offer important insights.
Server log files
Server or CDN log files are Probably the last word Software at your disposal. These logs seize an exhaustive list of each URL path queried by consumers, Googlebot, or other bots during the recorded period.
Things to consider:
Facts dimensions: Log information can be significant, a great number of web sites only keep the last two weeks of data.
Complexity: Analyzing log information is often hard, but numerous equipment can be obtained to simplify the method.
Mix, and great luck
After you’ve collected URLs from these sources, it’s time to mix them. If your website is sufficiently small, use Excel or, for larger sized datasets, instruments like Google Sheets or Jupyter Notebook. Make certain all URLs are consistently formatted, then deduplicate the listing.
And voilà—you now have an extensive listing of current, previous, and archived URLs. Fantastic luck!