A B C D E F G H I M P R S U V W

A

addFailedToScrapeURL(CrawlRecord) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Adds the given CrawlRecord to the list of CrawlRecords NOT successfully scraped.

addNquads(String, String) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

addSuccessfulScrapedURL(CrawlRecord) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Adds the given CrawlRecord to the list of CrawlRecords successfully scraped.

apiDoc() - Method in class eu.dnetlib.bioschemas.api.controller.HomeController

AppConfigGarr - Class in eu.dnetlib.bioschemas.api

AppConfigGarr() - Constructor for class eu.dnetlib.bioschemas.api.AppConfigGarr

B

BIOSCHEMAS_APIS - Static variable in class eu.dnetlib.bioschemas.api.MainApplication

BioschemasAPIController - Class in eu.dnetlib.bioschemas.api.controller

BioschemasAPIController() - Constructor for class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

BioschemasException - Exception in eu.dnetlib.bioschemas.api.utils

BioschemasException() - Constructor for exception eu.dnetlib.bioschemas.api.utils.BioschemasException

BioschemasException(String) - Constructor for exception eu.dnetlib.bioschemas.api.utils.BioschemasException

BioschemasException(String, Throwable) - Constructor for exception eu.dnetlib.bioschemas.api.utils.BioschemasException

BioschemasException(Throwable) - Constructor for exception eu.dnetlib.bioschemas.api.utils.BioschemasException

BioschemasException(String, Throwable, boolean, boolean) - Constructor for exception eu.dnetlib.bioschemas.api.utils.BioschemasException

BMUSEScraper - Class in eu.dnetlib.bioschemas.api.scraper

BMUSEScraper() - Constructor for class eu.dnetlib.bioschemas.api.scraper.BMUSEScraper

C

complete() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

CompressorUtil - Class in eu.dnetlib.bioschemas.api.utils

CompressorUtil() - Constructor for class eu.dnetlib.bioschemas.api.utils.CompressorUtil

compressValue(String) - Static method in class eu.dnetlib.bioschemas.api.utils.CompressorUtil

CrawlRecord - Class in eu.dnetlib.bioschemas.api.crawl

Store the current status of a single URL in the scrape service.

CrawlRecord() - Constructor for class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

CrawlRecord(String) - Constructor for class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

D

decompressValue(String) - Static method in class eu.dnetlib.bioschemas.api.utils.CompressorUtil

E

equals(Object) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

eu.dnetlib.bioschemas.api - package eu.dnetlib.bioschemas.api

eu.dnetlib.bioschemas.api.controller - package eu.dnetlib.bioschemas.api.controller

eu.dnetlib.bioschemas.api.crawl - package eu.dnetlib.bioschemas.api.crawl

eu.dnetlib.bioschemas.api.scraper - package eu.dnetlib.bioschemas.api.scraper

eu.dnetlib.bioschemas.api.utils - package eu.dnetlib.bioschemas.api.utils

F

fail(Throwable) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

G

getContext() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getDateEnd() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

getDateScraped() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getDateStart() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

getId() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getId() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

getLastScrapingExecution() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecutor

getMessage() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

getName() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getNQuads(String, HttpServletResponse) - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

getNquads() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getNquadsConcurrentHashMap() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

getNQUADSFromUrl(String, Boolean) - Method in class eu.dnetlib.bioschemas.api.scraper.BMUSEScraper

getNumberPagesLeftToScrape() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Returns the number of URLs that are still to be scraped in this cycle.

getOutputDataPattern() - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

getOutputFolder() - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

getPagesProcessed() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Gets the full list of URLs that have been processed in this cycle.

getPagesProcessedAndUnprocessed() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Gets the full list of URLs/CrawlRecords regardless of whether scraped or not in the current cycle.

getSitemapList(String, String) - Static method in class eu.dnetlib.bioschemas.api.utils.UrlParser

getSitemapUrl() - Method in class eu.dnetlib.bioschemas.api.ServiceScrapeDriver

getSitemapURLKey() - Method in class eu.dnetlib.bioschemas.api.ServiceScrapeDriver

getStatus() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getStatus() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

getUrl() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

getURLToProcess() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Returns the next URL/CrawlRecord to be scraped

H

hashCode() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

HomeController - Class in eu.dnetlib.bioschemas.api.controller

HomeController() - Constructor for class eu.dnetlib.bioschemas.api.controller.HomeController

I

isBeingScraped() - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

isFileWritten() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeThread

M

main(String[]) - Static method in class eu.dnetlib.bioschemas.api.MainApplication

MainApplication - Class in eu.dnetlib.bioschemas.api

MainApplication() - Constructor for class eu.dnetlib.bioschemas.api.MainApplication

P

pagesLeftToScrape() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Any pages/URLs left to scrape?

publicApi() - Method in class eu.dnetlib.bioschemas.api.MainApplication

R

run() - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeThread

runScrape() - Method in class eu.dnetlib.bioschemas.api.ServiceScrapeDriver

Fires off threads Originally designed as a multi-threaded process; now reduced to a single thread as the selenium webdriver is too expensive to run multi-threaded.

S

scrape(String, Long, String, String, StatusOfScrape) - Method in class eu.dnetlib.bioschemas.api.scraper.ServiceScraper

Orchestrates the process of scraping a site before converting the extracted triples to NQuads and writing to a file.

ScrapeState - Class in eu.dnetlib.bioschemas.api.scraper

ScrapeState(List<CrawlRecord>) - Constructor for class eu.dnetlib.bioschemas.api.scraper.ScrapeState

ScrapeThread - Class in eu.dnetlib.bioschemas.api.scraper

ScrapeThread(BMUSEScraper, ScrapeState, int, int) - Constructor for class eu.dnetlib.bioschemas.api.scraper.ScrapeThread

Sets up a thread for actually scrapping.

ScrapingExecution - Class in eu.dnetlib.bioschemas.api.scraper

ScrapingExecution() - Constructor for class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

ScrapingExecution(String, Long, Long, ScrapingStatus, String) - Constructor for class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

ScrapingExecutor - Class in eu.dnetlib.bioschemas.api.scraper

ScrapingExecutor() - Constructor for class eu.dnetlib.bioschemas.api.scraper.ScrapingExecutor

ScrapingStatus - Enum in eu.dnetlib.bioschemas.api.scraper

ServiceScrapeDriver - Class in eu.dnetlib.bioschemas.api

Runs the scrape.

ServiceScrapeDriver(String, String, String, String, String) - Constructor for class eu.dnetlib.bioschemas.api.ServiceScrapeDriver

ServiceScraper - Class in eu.dnetlib.bioschemas.api.scraper

Provides the actual scraping functionality.

ServiceScraper() - Constructor for class eu.dnetlib.bioschemas.api.scraper.ServiceScraper

setBeingScraped(boolean) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setContext(String) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setDateEnd(Long) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

setDateScraped(Date) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setDateStart(Long) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

setId(Long) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setId(String) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

setMessage(String) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

setName(String) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setNquads(String) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setOutputDataPattern(String) - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

setOutputFolder(String) - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

setStatus(StatusOfScrape) - Method in class eu.dnetlib.bioschemas.api.crawl.CrawlRecord

setStatus(ScrapingStatus) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

setStatusTo404(CrawlRecord) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Changes the status of the CrawlRecord to DOES_NOT_EXIST.

setStatusToHumanInspection(CrawlRecord) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapeState

Changes the status of the CrawlRecord to HUMAN_INSPECTION.

startNew(String) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecution

startScraping(String, String, HttpServletRequest) - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

startScraping(String, String, String, String, String) - Method in class eu.dnetlib.bioschemas.api.scraper.ScrapingExecutor

StatusOfScrape - Enum in eu.dnetlib.bioschemas.api.crawl

StatusOfScrape describes the possible status levels the scrape for each URL/CrawlRecord.

statusScraping() - Method in class eu.dnetlib.bioschemas.api.controller.BioschemasAPIController

swaggerDesc() - Method in class eu.dnetlib.bioschemas.api.MainApplication

swaggerTags() - Method in class eu.dnetlib.bioschemas.api.MainApplication

swaggerTitle() - Method in class eu.dnetlib.bioschemas.api.MainApplication

U

UrlParser - Class in eu.dnetlib.bioschemas.api.utils

UrlParser() - Constructor for class eu.dnetlib.bioschemas.api.utils.UrlParser

V

valueOf(String) - Static method in enum eu.dnetlib.bioschemas.api.crawl.StatusOfScrape

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum eu.dnetlib.bioschemas.api.scraper.ScrapingStatus

Returns the enum constant of this type with the specified name.

values() - Static method in enum eu.dnetlib.bioschemas.api.crawl.StatusOfScrape

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum eu.dnetlib.bioschemas.api.scraper.ScrapingStatus

Returns an array containing the constants of this enum type, in the order they are declared.

W

wrapHTMLExtraction(String) - Method in class eu.dnetlib.bioschemas.api.scraper.ServiceScraper

A B C D E F G H I M P R S U V W