[Cloudflare Logpull][Meta] invalid time range: too early: logs older than 168h0m0s are not available

[andrewkroh]

The cloudflare.logpull integration is failing due to an invalid time range.

Error while processing http request: failed to execute rf.collectResponse: failed to execute http client.Do: server responded with status code 400: bad query: error parsing time: invalid time range: too early: logs older than 168h0m0s are not available

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

@andrewkroh I've seen this come up a few times recently. Do we know what it will take to fix or what do we need from customers to allow us to make progress?

[andrewkroh]

From https://developers.cloudflare.com/logs/logpull/requesting-logs/, some of the highlights are

start Must be no more than 7 days earlier than now
end Must be at least 1 minute earlier than now and later than start
Note: The maximum time range from start to end cannot exceed 1 hour.

This is our current config:

integrations/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs

Lines 31 to 38 in 9c9f1f3

	- set:
	target: url.params.start
	value: "[[.cursor.last_execution_datetime]]"
	default: '[[formatDate (((now).Add (parseDuration "-1m")).Add (parseDuration "-{{interval}}"))]]'
	- set:
	target: url.params.end
	value: '[[formatDate ((parseDate .cursor.last_execution_datetime).Add (parseDuration "{{interval}}"))]]'
	default: '[[formatDate ((now).Add (parseDuration "-1m"))]]'

integrations/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs

Lines 45 to 47 in 9c9f1f3

	cursor:
	last_execution_datetime:
	value: '[[.last_response.url.params.Get "end"]]'

---

So there are four constraints that we must ensure are always true.

• now - start <= 168h
• now - end > 1m
• start < end
• start - end <= 1h

If we used exactly start = now - 168h, then by the time the request was processed the start time would greater than 7 days old and in theory would be invalid. So some tolerance should be built in.

Here's a simplified summary of the current logic. There are no protections to guard against some of the constraints. You can imagine hitting some of those constraints if the input is stopped for a week, uses an interval greater than 1h, and probably more.

interval = 5m
start = now - 1m - interval
end = now - 1m

do
  response = do_request()
  start = response.time_of_last_event
  end = start + interval
while sleep interval

[andrewkroh]

I think we would need to add some protections like

start = max(time_of_last_event, now - 168h) # Always satisfy the 168h requirement.
end = min(start + 1h, now - 1m) # Always satisfy the 1h window requirement. And never request too new of data.

We need to investigate if this can be achieved in httpjson. I don't see any helper functions that would allow min or max.

[vulnerabivoro]

Hey @andrewkroh , using latest cloudflare integration we are getting this:

Error while processing http request: failed to execute rf.collectResponse: failed to execute http client.Do: server responded with status code 400: bad query: error parsing time: invalid time range: too recent: minimum delay in serving logs is 1m0s

[andrewkroh]

@vulnerabivoro What collection interval do you have the integration configured for?

[vulnerabivoro]

10m in the Cloudflare logs settings option.

[ShourieG]

The integration PR will be put up soon since we cannot merge it before 8.10 public release, as it will have dependency on the new min/max funcs added to httpjson in 8.10