Skip to content

Buffer Threshold Seconds

BUFFER_THRESHOLD_SECONDS

Config Name

BUFFER_THRESHOLD_SECONDS

Description

The configuration that defines the maximum lifetime in seconds of a buffer before it's considered for data ingestion.

Default value

300 seconds

Example values

  • 60 - Sets the buffer lifetime to 1 minute.
  • 600 - Sets the buffer lifetime to 10 minutes.

Detailed Description

The BUFFER_THRESHOLD_SECONDS configuration sets the maximum duration a buffer can hold data before it's considered ready for data ingestion by Skippr. Once a buffer's lifetime, as determined by the updated_at timestamp, exceeds the value set for BUFFER_THRESHOLD_SECONDS, the data contained in that buffer will be ingested. This mechanism allows for a buffer to collect a reasonable amount of data before processing, potentially improving ingestion efficiency and reducing system resource usage.

The value of BUFFER_THRESHOLD_SECONDS is expressed in seconds. This means a setting of 300 (the default) corresponds to a 5-minute buffer lifetime, while a setting of 3600 would correspond to an hour.

Considerations

  1. Data Ingestion Frequency: The BUFFER_THRESHOLD_SECONDS value affects how often data ingestion operations occur. Lower values will lead to more frequent data ingestions, while higher values will result in less frequent ingestions. Adjust this configuration to match your data ingestion needs and system resources.

  2. System Resources: Depending on the volume and velocity of incoming data, increasing the BUFFER_THRESHOLD_SECONDS value might lead to more data being buffered before ingestion. This could increase memory usage. Ensure your system has enough resources to handle this increased demand.

  3. Data Latency: A higher BUFFER_THRESHOLD_SECONDS value might introduce data latency as data waits in the buffer for a longer time before being ingested. If your use case requires real-time or near-real-time data processing, consider setting a lower BUFFER_THRESHOLD_SECONDS value.