[integrations][Cloudflare Logpull] - Fixed invalid time range issue

[ShourieG]

Type of change

• Bug

What does this PR do?

This PR uses a certain time range constraint discussed here to address the issue. The main concept lies in the following time constraints :

now - start <= 168h
now - end > 1m
start < end
end - stard <= 1h
 
start = max(time_of_last_event, now - 168h)  # This Always satisfies the 168h requirement.
end = min(start + 1h, now - 1m)  # Always satisfies the 1h window requirement, and never requests earlier data.

The integration uses the above constraints while calculating the value of the start & end query params.

NOTE:

1. HTTPJSON does not have any mechanism to store values internally while setting parameters via the config. It also does not have the capability to pull values from the .url.* objects, only .last_response.url.* works. This creates a problem we need to work around. For the calculation of end using the logic end = min(start + 1h, now - 1m), the value of start needs to be calculated here since we cannot use [[.url.params.Get "start"]]. Similarly we are unable to store a singular value for now and it gets computed everywhere it's used. This creates very minor deviations in the timestamps of a few milliseconds to maybe a max of 1 second. We can probably live by this. This re-calculation of values effectively transforms the internal logic for computing the value of end to:

end = min(max(time_of_last_event, now - 168h) + 1h, now - 1m)

2. This was tested manually with a local server and all of the constraints passed.
3. A mock service has been integrated via docker-compose which simulates the exact api constraints during the system test
   with a hit_count of 3 responses. The service sends a rolling response similar to elastic stream http servers.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [Cloudflare Logpull][Meta] invalid time range: too early: logs older than 168h0m0s are not available #4414

Screenshots

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-12T09:52:18.492+0000

• Duration: 19 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	21
Skipped	0
Total	21

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (3/3)	💚
Classes	100.0% (3/3)	💚
Methods	100.0% (37/37)	💚 4.762
Lines	94.772% (707/746)	👍 17.64
Conditionals	100.0% (0/0)	💚

[efd6]

You'll need to bump the stack version to pick up the max and min functions.

[ShourieG]

@andrewkroh I've added a mock service that simulates the api constraints during system tests and resolved rest of the suggestions.

[andrewkroh]

The mock server you added is helpful, but it's hard to prove the httpjson config is working correctly because it does not appear that the min/max range protections get exercised. So I think it would be good if we can find a way to test these cases.

[ShourieG]

@andrewkroh resolved all the suggested changes, introduced a new test that works over 2 iterations so cursor value is used. Modified the server code to match a specific criteria for the new test case to validate cursor value.

[elasticmachine]

Package cloudflare - 2.20.0 containing this change is available at https://epr.elastic.co/search?package=cloudflare