eu.dnetlib.bioschemas.api.scraper

Class ScrapeState

java.lang.Object

eu.dnetlib.bioschemas.api.scraper.ScrapeState

public class ScrapeState
extends Object

Constructor Summary

Constructors

Constructor and Description
ScrapeState(List<CrawlRecord> pagesToBeScraped)

Method Summary

Modifier and Type	Method and Description
void	addFailedToScrapeURL(CrawlRecord record) Adds the given CrawlRecord to the list of CrawlRecords NOT successfully scraped.
void	addNquads(String key, String nquads)
void	addSuccessfulScrapedURL(CrawlRecord record) Adds the given CrawlRecord to the list of CrawlRecords successfully scraped.
Map<String,Object>	getNquadsConcurrentHashMap()
int	getNumberPagesLeftToScrape() Returns the number of URLs that are still to be scraped in this cycle.
List<CrawlRecord>	getPagesProcessed() Gets the full list of URLs that have been processed in this cycle.
List<CrawlRecord>	getPagesProcessedAndUnprocessed() Gets the full list of URLs/CrawlRecords regardless of whether scraped or not in the current cycle.
CrawlRecord	getURLToProcess() Returns the next URL/CrawlRecord to be scraped
boolean	pagesLeftToScrape() Any pages/URLs left to scrape?
void	setStatusTo404(CrawlRecord record) Changes the status of the CrawlRecord to DOES_NOT_EXIST.
void	setStatusToHumanInspection(CrawlRecord record) Changes the status of the CrawlRecord to HUMAN_INSPECTION.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ScrapeState

public ScrapeState(List<CrawlRecord> pagesToBeScraped)

Parameters:

pagesToBeScraped - The list of sites to be scraped

See Also:

ScrapeThread, CrawlRecord

Method Detail

pagesLeftToScrape

public boolean pagesLeftToScrape()

Any pages/URLs left to scrape?

Returns:

True for yes & false for no

See Also:

CrawlRecord

getURLToProcess

public CrawlRecord getURLToProcess()

Returns the next URL/CrawlRecord to be scraped

Returns:

First page/URL that needs to be scraped next

See Also:

CrawlRecord

addSuccessfulScrapedURL

public void addSuccessfulScrapedURL(CrawlRecord record)

Adds the given CrawlRecord to the list of CrawlRecords successfully scraped. Updates the status of the CrawlRecord to SUCCESS.

Parameters:

url - The latest URL/page that has been successfully scraped

See Also:

CrawlRecord

addFailedToScrapeURL

public void addFailedToScrapeURL(CrawlRecord record)

Adds the given CrawlRecord to the list of CrawlRecords NOT successfully scraped. Updates the status of the CrawlRecord; if first failure the status is FAILED. If status is already FAILED it is changed to GIVEN_UP. If the status is FAILED, another try will be made in a future run.

Parameters:

url - The latest URL/page that has been unsuccessfully scraped

See Also:

CrawlRecord

setStatusTo404

public void setStatusTo404(CrawlRecord record)

Changes the status of the CrawlRecord to DOES_NOT_EXIST. As Selenium does not return the HTTP codes, it is questionable how useful this is.

Parameters:

url - The latest URL/page that has been 404'd

See Also:

CrawlRecord

setStatusToHumanInspection

public void setStatusToHumanInspection(CrawlRecord record)

Changes the status of the CrawlRecord to HUMAN_INSPECTION. This captures the idea that the URLs may contain unexpected markup that needs a human to review and possibly update the scraper.

Parameters:

url - The latest URL/page that needs human inspection

See Also:

CrawlRecord

getNumberPagesLeftToScrape

public int getNumberPagesLeftToScrape()

Returns the number of URLs that are still to be scraped in this cycle. This does not return the number of URLs left to scrape in the DBMS, just in the current cycle.

Returns:

Number of URLs left to scrape in this cycle

See Also:

CrawlRecord

getPagesProcessed

public List<CrawlRecord> getPagesProcessed()

Gets the full list of URLs that have been processed in this cycle. This does not return the number of URLs that have been scraped in total across all cycles.

Returns:

See Also:

CrawlRecord

getPagesProcessedAndUnprocessed

public List<CrawlRecord> getPagesProcessedAndUnprocessed()

Gets the full list of URLs/CrawlRecords regardless of whether scraped or not in the current cycle.

Returns:

List of all CrawlRecords in this cycle.

See Also:

CrawlRecord

addNquads

public void addNquads(String key,
                      String nquads)

getNquadsConcurrentHashMap

public Map<String,Object> getNquadsConcurrentHashMap()