eu.dnetlib.functionality.index.solr.feed

Class StreamingInputDocumentFactory

java.lang.Object

eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory

eu.dnetlib.functionality.index.solr.feed.StreamingInputDocumentFactory

public class StreamingInputDocumentFactory
extends InputDocumentFactory

Optimized version of the document parser, drop in replacement of InputDocumentFactory.

Faster because:

• Doesn't create a DOM for the full document
• Doesn't execute xpaths agains the DOM
• Quickly serialize the 'result' element directly in a string.
• Uses less memory: less pressure on GC and allows more threads to process this in parallel

This class is fully reentrant and can be invoked in parallel.

Author:

marko

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	DEFAULTDNETRESULT
protected ThreadLocal<XMLEventFactory>	eventFactory
protected static String	INDEX_RECORD_ID_ELEMENT
protected ThreadLocal<XMLInputFactory>	inputFactory
protected static int	MAX_FIELD_LENGTH
protected ThreadLocal<XMLOutputFactory>	outputFactory
protected static String	ROOT_ELEMENT
protected static String	TARGETFIELDS

Fields inherited from class eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory

DS_ID, DS_VERSION, INDEX_FIELD_PREFIX, INDEX_RECORD_ID, INDEX_RESULT, RESULT

Constructor Summary

Constructors

Constructor and Description
StreamingInputDocumentFactory()

Method Summary

Modifier and Type	Method and Description
protected void	copyResult(org.apache.solr.common.SolrInputDocument indexDocument, StringWriter results, XMLEventReader parser, List<Namespace> nsList, String dnetResult, ResultTransformer resultTransformer) Copy the /indexRecord/result element and children, preserving namespace declarations etc.
protected String	getText(XMLEvent text) Helper used to get the string from a text element.
org.apache.solr.common.SolrInputDocument	parseDocument(String version, String inputDocument, String dsId, String resultName)
org.apache.solr.common.SolrInputDocument	parseDocument(String version, String inputDocument, String dsId, String resultName, ResultTransformer resultTransformer)
protected void	parseTargetFields(org.apache.solr.common.SolrInputDocument indexDocument, XMLEventReader parser) Parse the targetFields block and add fields to the solr document.

Methods inherited from class eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory

getParsedDateField, parseDate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULTDNETRESULT

protected static final String DEFAULTDNETRESULT

See Also:

Constant Field Values

TARGETFIELDS

protected static final String TARGETFIELDS

See Also:

Constant Field Values

INDEX_RECORD_ID_ELEMENT

protected static final String INDEX_RECORD_ID_ELEMENT

See Also:

Constant Field Values

ROOT_ELEMENT

protected static final String ROOT_ELEMENT

See Also:

Constant Field Values

MAX_FIELD_LENGTH

protected static final int MAX_FIELD_LENGTH

See Also:

Constant Field Values

inputFactory

protected ThreadLocal<XMLInputFactory> inputFactory

outputFactory

protected ThreadLocal<XMLOutputFactory> outputFactory

eventFactory

protected ThreadLocal<XMLEventFactory> eventFactory

Constructor Detail

StreamingInputDocumentFactory

public StreamingInputDocumentFactory()

Method Detail

parseDocument

public org.apache.solr.common.SolrInputDocument parseDocument(String version,
                                                              String inputDocument,
                                                              String dsId,
                                                              String resultName)
                                                       throws XMLStreamException

Specified by:

parseDocument in class InputDocumentFactory

Throws:

XMLStreamException

See Also:

eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory#parseDocument(eu.dnetlib.functionality.index.solr.feed.IndexDocument, java.lang.String)

parseDocument

public org.apache.solr.common.SolrInputDocument parseDocument(String version,
                                                              String inputDocument,
                                                              String dsId,
                                                              String resultName,
                                                              ResultTransformer resultTransformer)

Specified by:

parseDocument in class InputDocumentFactory

See Also:

eu.dnetlib.functionality.index.solr.feed.InputDocumentFactory#parseDocument(eu.dnetlib.functionality.index.solr.feed.IndexDocument, java.lang.String, com.google.common.base.Function)

parseTargetFields

protected void parseTargetFields(org.apache.solr.common.SolrInputDocument indexDocument,
                                 XMLEventReader parser)
                          throws XMLStreamException

Parse the targetFields block and add fields to the solr document.

Parameters:

indexDocument -

parser -

Throws:

XMLStreamException

copyResult

protected void copyResult(org.apache.solr.common.SolrInputDocument indexDocument,
                          StringWriter results,
                          XMLEventReader parser,
                          List<Namespace> nsList,
                          String dnetResult,
                          ResultTransformer resultTransformer)
                   throws XMLStreamException

Copy the /indexRecord/result element and children, preserving namespace declarations etc.

Parameters:

indexDocument -

results -

parser -

nsList -

Throws:

XMLStreamException

getText

protected final String getText(XMLEvent text)

Helper used to get the string from a text element.

Parameters:

text -

Returns: