eu.dnetlib.bioschemas.api

Class ServiceScrapeDriver

java.lang.Object

eu.dnetlib.bioschemas.api.ServiceScrapeDriver

public class ServiceScrapeDriver
extends Object

Runs the scrape. Collect a list of URLs (in the form of CrawlRecords) to scrape.

Constructor Summary

Constructors

Constructor and Description
ServiceScrapeDriver(String sitemapUrl, String sitemapURLKey, String maxScrapedPages, String outputFilename, String outputFolder)

Method Summary

Modifier and Type	Method and Description
String	getSitemapUrl()
String	getSitemapURLKey()
void	runScrape() Fires off threads Originally designed as a multi-threaded process; now reduced to a single thread as the selenium webdriver is too expensive to run multi-threaded.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ServiceScrapeDriver

public ServiceScrapeDriver(String sitemapUrl,
                           String sitemapURLKey,
                           String maxScrapedPages,
                           String outputFilename,
                           String outputFolder)

Method Detail

runScrape

public void runScrape()
               throws IOException

Fires off threads Originally designed as a multi-threaded process; now reduced to a single thread as the selenium webdriver is too expensive to run multi-threaded. However, the threading as been left in situ in case it is useful in the future.

Throws:

IOException

getSitemapUrl

public String getSitemapUrl()

getSitemapURLKey

public String getSitemapURLKey()