 |
Qore DataProvider Module Reference
1.0.2
|
199 hash<string, PipelineQueue>
pmap;
const DPE_SKIP_DATA
Skips processing a record.
Definition: DataProviderPipeline.qc.dox.h:53
Mutex lck
Parent lock.
Definition: DataProviderPipeline.qc.dox.h:125
logError(string fmt)
Logs to the error log, if set.
date start_time
run start time
Definition: DataProviderPipeline.qc.dox.h:214
*string name
the name of the pipeline for logging purposes; if this key is not included, a generic name will be ge...
Definition: DataProviderPipeline.qc.dox.h:114
*code debug_log
a closure or call reference for debug logging
Definition: DataProviderPipeline.qc.dox.h:96
hash< string, PipelineQueue > pmap
Hash of queues keyed by queue ID.
Definition: DataProviderPipeline.qc.dox.h:199
bool locked
Locked flag.
Definition: DataProviderPipeline.qc.dox.h:205
string name
A descriptive name for logging purposes.
Definition: DataProviderPipeline.qc.dox.h:196
PipelineQueue copyPipeline(PipelineQueue old_queue)
Called by the copy constructor to copy the queues.
append(int id, AbstractDataProcessor processor)
Appends a data processor to a pipeline.
submit(auto _data)
Submits data for processing.
bool stop_flag
Stop flag.
Definition: DataProviderPipeline.qc.dox.h:240
submit(auto record)
Submits data for processing.
reset()
Resets the pipeline.
int getId()
Returns the pipeline ID.
Pipeline info.
Definition: DataProviderPipeline.qc.dox.h:57
run(Counter run_cnt)
Processing thread.
int data_waiting
Number of threads waiting on data.
Definition: DataProviderPipeline.qc.dox.h:137
*date start_time
Start of processing.
Definition: DataProviderPipeline.qc.dox.h:62
*code error_log
Error log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:226
*code debug_log
Debug log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:229
string status
Pipeline status.
Definition: DataProviderPipeline.qc.dox.h:70
append(AbstractDataProcessor processor)
Appends a data processor to the default pipeline.
string getName()
Returns the pipeline name.
*code error_log
a closure or call reference for error logging
Definition: DataProviderPipeline.qc.dox.h:101
Condition cond()
Queue condition variable.
int appendQueue(int id)
Appends a new queue to an existing pipeline and returns the new queue ID.
Qore AbstractDataField class definition.
Definition: AbstractDataField.qc.dox.h:32
Defines an abstract class for accepting data and outputting optionally transformed or filtered data.
Definition: AbstractDataProcessor.qc.dox.h:33
const PS_RUNNING
Pipeline status: RUNNING.
Definition: DataProviderPipeline.qc.dox.h:40
bool isProcessing()
Returns True if the pipeline is processing data.
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline threads
Definition: DataProviderPipeline.qc.dox.h:111
bool do_bulk
Bulk flag.
Definition: DataProviderPipeline.qc.dox.h:202
int record_count
Number of input records submitted.
Definition: DataProviderPipeline.qc.dox.h:73
bool aborting()
Returns True if the object is aborting.
*code info_log
a closure or call reference for info logging
Definition: DataProviderPipeline.qc.dox.h:106
constructor(DataProviderPipeline parent, Mutex lck, Counter cnt, int id, int size)
Creates the object.
int num_queues
Number of pipeline queues.
Definition: DataProviderPipeline.qc.dox.h:76
submit(AbstractDataProviderBulkRecordInterface i)
Submits data for processing.
DataProviderPipeline parent
Parent object.
Definition: DataProviderPipeline.qc.dox.h:156
date stop_time
run stop time (set in waitDone())
Definition: DataProviderPipeline.qc.dox.h:217
abort(*bool ignore_exceptions)
Aborts execution of a pipeline in progress.
list< auto > elems()
Pipeline elements.
Defines the abstract class for data provider iterators; the destructor releases the iterator.
Definition: AbstractDataProviderRecordIterator.qc.dox.h:33
bool abort_flag
Abort flag.
Definition: DataProviderPipeline.qc.dox.h:243
Pipeline option info.
Definition: DataProviderPipeline.qc.dox.h:92
copy()
Copy constructor; creates an empty pipeline with the same configuration as the original.
Pipeline element.
Definition: DataProviderPipeline.qc.dox.h:118
list< hash< ExceptionInfo > > error_list
list of exceptions in pipelines
Definition: DataProviderPipeline.qc.dox.h:211
destructor()
Destroys the object.
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline threads
Definition: DataProviderPipeline.qc.dox.h:234
int queue_waiting
Number of threads waiting data to be removed from the queue.
Definition: DataProviderPipeline.qc.dox.h:134
waitDone()
Waits for all queues in all pipelines to have processed remaining data.
Defines the abstract class for data provider iterators; the destructor releases the iterator.
Definition: AbstractDataProviderBulkRecordInterface.qc.dox.h:33
submitData(AbstractIterator i)
Submits data for processing.
submitIntern(auto _data)
Submits data for processing.
*date stop_time
Stop time for processing.
Definition: DataProviderPipeline.qc.dox.h:65
waitDone()
Wait for the queue to be empty, then wait for all terminating pipelines to be empty.
const PS_IDLE
Pipeline status: IDLE.
Definition: DataProviderPipeline.qc.dox.h:43
int size
Maximum queue size.
Definition: DataProviderPipeline.qc.dox.h:143
float duration_secs
Total time processing end to end as a floating-point value in durationSecondsFloat.
Definition: DataProviderPipeline.qc.dox.h:85
reportError(PipelineQueue queue, hash< ExceptionInfo > ex)
Called from a pipeline queue object to report a fatal error durring processing.
checkLockedIntern()
Throws an exception if the pipeline is locked.
int id
Pipeline ID.
Definition: DataProviderPipeline.qc.dox.h:122
*code info_log
Info log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:223
float recs_per_sec
Records processed per second end to end.
Definition: DataProviderPipeline.qc.dox.h:88
throwPipelineException()
Throws an exception if errors occured in background pipeline processing.
checkUpdatePipelineIntern(int id)
Check if the given queue exists.
stopInternUnlocked()
Stops all background pipeline queues; lock must be held.
int tid
TID of the background thread.
Definition: DataProviderPipeline.qc.dox.h:146
submit(AbstractDataProviderRecordIterator i)
Submits data for processing.
logDebug(string fmt)
Logs to the debug log, if set.
checkSubmitIntern()
Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise.
int record_count
Record count.
Definition: DataProviderPipeline.qc.dox.h:220
logInfo(string fmt)
Logs to the info log, if set.
constructor(*hash< PipelineOptionInfo > opts)
Creates the object with the given options.
bool hasQueue(int id)
Returns True if the given queue exists, False if not.
hash< PipelineInfo > getInfo()
Returns pipeline info.
Sequence seq(1)
Pipeline ID sequence generator.
list< auto > queue
Data queue.
Definition: DataProviderPipeline.qc.dox.h:140
Counter cnt()
Thread counter.
date duration
Total time processing end to end.
Definition: DataProviderPipeline.qc.dox.h:82
submitDataIntern(auto _data)
Submits a single record for processing.
Defines a class for passing data through record processors.
Definition: DataProviderPipeline.qc.dox.h:189
string name
The name of the pipeline.
Definition: DataProviderPipeline.qc.dox.h:59
submitBulkIntern(AbstractDataProviderBulkRecordInterface i)
Submits bulk data for processing.
Counter cnt
Parent counter.
Definition: DataProviderPipeline.qc.dox.h:128
const PS_ABORTED
Pipeline status: ABORTED.
Definition: DataProviderPipeline.qc.dox.h:37
stopIntern()
Stops all background pipeline queues.
bool stopping()
Returns True if the object is stopping.
registerThread(PipelineQueue queue)
Registers a new thread.
bool bulk
Flag that indicates if the pipeline is capable of bulk record processing.
Definition: DataProviderPipeline.qc.dox.h:79
resetIntern()
Resets the pipeline.