Common Settings for DataLoggers
Different pervasive service plugins exists for data logging (publishers) and data consuming (subscribers):
| Name | Description | Publisher | Subscriber |
|---|---|---|---|
| HumanOS.PeSeL.AzureIoTClient | Access to Microsoft Azure cloud | X | X |
| HumanOS.PeSeL.CsvDataLogger | Logger to simple csv file | X | - |
| HumanOS.PeSeL.InfluxDbLogger | Logger to the influx database | X | - |
| HumanOS.PeSeL.MQTTClient | Access to a MQTT Broker | X | X |
| HumanOS.PeSeL.NodeSpaceDataLogger | Access to a NodeSpace commands | X | - |
| HumanOS.PeSeL.RabbitMqClient | Access to RabbitMessageQueue protocol | X | X |
| HumanOS.PeSeL.RestDataLogger | Logger to any web endpoint | X | - |
| HumanOS.PeSeL.SqlDataLogger | Logger to SQL Database | X | - |
Plugin Configuration
The plugin configuration for publisher and subscriber contains various settings.
The next picture shows the class model of the configuration.
Each plugin is configured by a DataLoggerPluginConfiguration. This Configuration contains following Parameters:
| Parameter | Description | Data Type |
|---|---|---|
| Disabled | Flag if the plugin is disabled. Set to false for plugin activation. | System.Boolean |
| Publishers | List of data publishers | TAbstractDataLoggerConfiguration |
| Subscribers | List of data subscribers | TAbstractDataPublisherConfiguration |
DataConsumer Configuration (Subscriber)
Each subscriber instance represents a client subscribing data from a data source. Each subscriber has its own configuration.
| Parameter | Description | Data Type |
|---|---|---|
| Id | Id of the data consumer | System.Guid |
| Name | Name of the data consumer. Used in log files | System.String |
| Connection | Connection string of the data consumer | System.String |
| NetworkTimeout | [OPT] Timeout in milliseconds [ms] of the network call when sending or receiving data from the platform. | System.Double |
| PayloadScriptFile | C# Script file to parse the subscribed data. Further information see the specific plugin documentation. | System.String |
| Properties | [OPT] array of key-value pairs containing custom specific data. Properties are accessible for payload processing. | [String, String] |
| SamplingRate | Time interval in seconds [s] in which data are polled. | System.Double |
| ServiceRule | The service rule enables and binds the client to the HumanOS® kernel. | TServiceRuleConfig |
DataLogger Configuration (Publisher)
Each publisher instance represents a client publishing data to a destination (data sink). Each publisher has its own configuration.
| Parameter | Description | Data Type |
|---|---|---|
| Id | Id of the publisher | System.Guid |
| Name | Name of the publisher. Used in log files | System.String |
| Connection | Connection string of the data logger | System.String |
| DataSets | Array of data sets see DataSet Config. | |
| NetworkTimeout | [OPT] Timeout in milliseconds [ms] of the network call when sending or receiving data from the platform. Careful this impacts the buffering of data directly. Default is 1000ms | System.Double |
| PayloadDumpEnabled | [OPT] Flag if payload dump is enabled. | System.Boolean |
| PayloadDumpFileName | [OPT] Filename of the log file to dump the payload. Payload is logged to a text file when a send of the payload is successful | System.String |
| PayloadDumpRetentionTime | [OPT] Retention time in days [d] to keep the log files. | System.Int32 |
| PayloadScriptFile | C# Script file to process datasets and generate the customized publish payload (message). | System.String |
| Properties | [OPT] array of key-value pairs containing custom specific data. Properties are accessible for payload processing. | [String, String] |
| SamplerType | Type of the data sampler. | EDataSamplerType |
| SamplingRate | Time interval in seconds [s] in which log data is sent. If set to 0, then only data is sent on changes (event-based logging) | System.Double |
| StartingSecond | Defines the a starting second, e.g. 10 will trigger the logging on the next 10 second mark | System.Int32 |
| MinimalInterruptionTime | Minimum time which the logger must rest between sending | System.Double |
| ProcessOnlyValidData | If this flag is set, values are reported only if the data state is EDataState.Good | System.Boolean |
| DisgardPersistedData | If this flag is set and the logger was buffering during shutdown, it continues in the same state after reboot. Careful Values set in the PayloadScript are still lost. | System.Boolean |
| ServiceRule | The service rule enables and binds the client to the HumanOS® kernel. | TServiceRuleConfig |
Well known Properties
Here are some properties that are always available on a logger:
| Parameter | Description | Data Type |
|---|---|---|
| LastTimeStamp | Timestamp of the last successful send | System.DateTime |
| Connected | Data logger is running and not buffering | System.Boolean |
| Function | Node indicator, always 'DataLogger' | System.String |
| SamplingRate | The sampling rate in which the data logger sends data | System.Double |
| Buffering | Indicates wether the logger is buffering | System.Boolean |
Sampler Type
The following values are possible for the property SamplerType:
| Value | Description |
|---|---|
| OnlyLastValue | Sends only the last value in the buffer. (last data received on interval end) |
| AllData | Sends all data recorded between two sampling cycles as time series. (Send all data received during interval) |
Buffering
Whenever a data logger cannot send its data to the endpoint, it enters the buffering mode.
| SamplingRate | SamplerType | Description |
|---|---|---|
| = 0ms | OnlyLastValue | In case of buffering the complete history is processed at once. |
| = 0ms | AllData | In case of alarm and event buffering, all alarms are processed |
| > 0ms | OnlyLastValue | Returns the history in sampling cycles, but only the last value within this cycle. If no values are present in current cycle, the last value of the prior cycle is returned. If history is empty, no data is returned. |
| > 0ms | AllData | Returns all history data within a sampling cycle. If no values are present in current cycle, no data is returned. |
NOTE
For buffering, the history mode is required for data nodes and alarm event pools. See historization.
DataSet Configuration
The DataSet-element specifies the log entry data set. The data set contains fields.
info
The Dataset Name must be UNIQUE.
The dataset is specified by following elements (fields, more fields are possible):
| Parameter | Description | Data Type |
|---|---|---|
| Name | Name of the field | System.String |
| NodeFilter | Node filter expression, e.g. node.hasProperty<string>("EnableMqtt", "1") | System.String |
| Type | Type of data provider. Possible values are: "AlarmEvent" (Alarm event objects), "DataNode" (data nodes) | EDataSetType |
| Tag | Tag data of the dataset | System.String |
| Properties | [OPT] array of key-value pairs containing custom specific data. Properties are accessible for payload processing. | [String, String] |
| Fields | Field data of the dataset. See Field Configuration. |
DataField Configuration
| Parameter | Description | Data Type |
|---|---|---|
| Name | Name of the field | System.String |
| NodeFilter | Node filter expression | System.String |
| DataType | Datatype | System.Type |
| Query | Expression selecting the data from a node. Data from a node is always expressed in the following format: item.{DataField}. See Available Query Information. ** | System.String |
Following objects can be used to query information:
| Query | Value |
|---|---|
item.GlobalId | Global node id |
item.Node | Data node object |
item.TimeStamp | Source timestamp |
item.Value | Value |
item.DataState | DataState |
item.getValue<T>(name) | Gets the sub-value. Only available for TGenericEntity data types |
logger.TimeStamp | Sampler timestamp, takes into account the historization |
Service Rules
Service rules are used to bind services and devices. More information about Service Rules.
Important:
- Each client (publisher or subscriber) needs its own rule to bind and unbind nodes from devices. This allows to pipe each device to another Data Logger Client (broker, payload, etc.)
Understanding the Sampling Based Logger
If a sampling rate greater than zero is set, a sampling behavior applies. There are three main cases which can occur while sampling:
- Case 1: Sample duration not exceeded and minimal interruption can be applied
- Each sample is executed as scheduled
- Case 2: Sample duration exceeded
- Reschedules to the next possible execution based on the scheduling timeline
- Will then add any missed data to the actual data
- Case 3: Minimal interruption time cannot be applied
- Minimal interruption time is applied and the scheduler will then behave as described in case 2
Below is an example with a sample rate of 2s and a minimal interruption time of 1s:
