[salesforce] Add `salesforce` package foundation along with `login_rest` data stream #4261

yug-rajani · 2022-09-21T19:39:22Z

Enhancement

What does this PR do?

Generated the skeleton of Salesforce integration package.
Added 1 data stream ( login_rest )
Added data collection logic.
Added the ingest pipelines.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added system test cases.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/salesforce directory.
Run the following command to run tests.

elastic-package test

elasticmachine · 2022-09-21T21:30:04Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2022-09-21T21:31:36Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-05T07:15:58.067+0000
Duration: 15 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6
Skipped	0
Total	6

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-09-21T21:31:49Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 3.0
Classes	100.0% (`1/1`)	💚 3.0
Methods	100.0% (`14/14`)	💚 12.903
Lines	95.339% (`225/236`)	👍 3.407
Conditionals	100.0% (`0/0`)	💚

kush-elastic

Looks good to me.

packages/salesforce/_dev/deploy/docker/docker-compose.yml

agithomas · 2022-10-13T03:45:54Z

If specifying the version is important, then below questions apply

Will the salesforce version that supports /services/data/v54.0/query endpoint support /services/data/v53.0/query
What is the difference between v54 and v53? Are these endpoints not available in v53. If these endpoints are available in v53, integration can be said it supports v53 as well. Isn’t it so?
What is the relationship between the salesforce product version and API version?
Does API version increment with minor version upgrade or major version upgrade?
What is the step a user of salesforce must follow to find out the version of salesforce they are using?
What is the step a user of salesforce must follow to find out the API VERSION of salesforce they are using?
Will the users of Salesforce always be using the latest version of Salesforce?

packages/salesforce/_dev/build/docs/README.md

packages/salesforce/data_stream/login_rest/agent/stream/httpjson.yml.hbs

yug-rajani · 2022-10-13T11:49:04Z

If specifying the version is important, then below questions apply

Will the salesforce version that supports /services/data/v54.0/query endpoint support /services/data/v53.0/query

Yes, it should. The deployment logic associated with the old version is still available for use as long as you force the API to use the old version (reference).

What is the difference between v54 and v53? Are these endpoints not available in v53. If these endpoints are available in v53, integration can be said it supports v53 as well. Isn’t it so?

Technically, it must be supported in 53.0. We tested the version 52.0 when we developed the Filebeat Module (which was the latest stable version then), while we tested it with the latest stable version now (based on Workbench) i.e. 54.0 when developing the current integration for Salesforce. However, we are hard-coding the version and hence using the API v53.0 wouldn't be possible here.

What is the relationship between the salesforce product version and API version?

Whenever Salesforce releases a new application version, Salesforce internally updates the API version (reference).

Does API version increment with minor version upgrade or major version upgrade?

Based on the release notes here, I think the API version increments with the major version upgrade.

What is the step a user of salesforce must follow to find out the version of salesforce they are using?

The steps can be found here.

What is the step a user of salesforce must follow to find out the API VERSION of salesforce they are using?

The steps can be found here.

Will the users of Salesforce always be using the latest version of Salesforce?

As it is a cloud platform, I think the users will always be using the latest version of Salesforce. Also, the test instance we use was created approximately a year ago, when the latest version was v52.0. However, when I checked the version using the steps mentioned in the above question, the version was v55.0. Based on this, I believe that the answer to this question should be a 'Yes'.

…to package_salesforce_login_rest

packages/salesforce/data_stream/login_rest/fields/fields.yml

packages/salesforce/_dev/build/docs/README.md

packages/salesforce/data_stream/login_rest/fields/fields.yml

kush-elastic · 2022-11-25T08:37:55Z

Here in the system test event contains the TIMESTAMP_DERIVED which is in 2021 so the @timestamp is in 2021 and as the system test was run in 2022 so the event.created is in 2022.

Technically correct, but logically incorrect, i fear In the manifest.yml, the max log collection frequency is 1hr. Considering this, a timestamp of creation and TIMESTAMP_DERIVED date must max be separated by 1hr, if my understanding is correct and 1 year is too much of a difference to mention in the documentation.

This is system test event only that's why you are seeing this difference. For customer there won't be a scenario in which they'll face this huge difference. Let me update current test event with the live response so you won't see this difference in the system test as well.

packages/salesforce/_dev/build/docs/README.md

agithomas · 2022-11-28T08:52:07Z

Can we also attempt to do a Maps visualization in the dashboard, making use of the source.geo.location towards the end of the dashboard to show the distribution of geo invocation?

This is the requirement i mentioned in the TDD document

agithomas · 2022-11-28T09:02:10Z

If the user has multiple salesforce URLs (spanned across multiple policy), how would the user select a specific salesforce instance in kibana? Shouldn't HostControl be used to help user?

agithomas · 2022-11-28T10:34:29Z

Can we also attempt to do a Maps visualization in the dashboard, making use of the source.geo.location towards the end of the dashboard to show the distribution of geo invocation?

This is the requirement i mentioned in the TDD document

Please hold this requirement until more clarity is available on elastic/elastic-package#1053 (comment)

agithomas · 2022-11-28T11:02:12Z

Can we also attempt to do a Maps visualization in the dashboard, making use of the source.geo.location towards the end of the dashboard to show the distribution of geo invocation?

This is the requirement i mentioned in the TDD document

Reference : elastic/elastic-package#1053 (comment)

@ruflin , @jsoriano If the geoIP processor might not always give the latitute and longitude values, is it a good approach to go with a kibana map for geo distribution of users (in this case salesforce users).

Three approaches I can think of are :

Approach 1 :
Since certain events (events failing geoIP translation) would miss the map representation, indicate in the title of the dashboard panel - limited events ?

Approach 2 :
Geo Map representation with no indication that certain IPs (or events) are filtered.

Approach 3:
Only tabular representation of top 10 IPs and no geo maps used.

This is a challenge in every dashboard where geomap is used. What can be the best approach? Please note, Geo Map representation is an additional representation over the table representation displaying top 10 source IPs.

I find that GeoMap representation would be a convenient representation for security audit users.

ruflin · 2022-11-29T07:54:38Z

I think we use "Approach 2" already in quite a few dashboards and did not have complaints yet. I would expect that in general it is understood that not 100% of ip addresses can be translated to geo locations so that some ip addresses are missing is expected. I personally always like to see the map visualisation as it is eye catching.

agithomas · 2022-11-30T06:27:01Z

packages/salesforce/_dev/build/docs/README.md

+
+- In case of data ingestion if the user finds following type of error logs:
+```
+{"log.level":"error","@timestamp":"2022-11-24T12:59:36.835+0530","log.logger":"input.httpjson-cursor","log.origin":{"[file.name](http://file.name/)":"compat/compat.go","file.line":124},"message":"Input 'httpjson-cursor' failed with: input.go:130: input 8A049E17A5CA661D failed (id=8A049E17A5CA661D)\n\toauth2 client: error loading credentials using user and password: oauth2: cannot fetch token: 400 Bad Request\n\tResponse: {\"error\":\"invalid_grant\",\"error_description\":\"authentication failure\"}","[service.name](http://service.name/)":"filebeat","id":"8A049E17A5CA661D","ecs.version":"1.6.0"}


Please do the formatting.

agithomas · 2022-11-30T06:27:55Z

packages/salesforce/_dev/build/docs/README.md

+
+Example:
+
+Address Bar Content: https://elastic1234-dev-ed.my.salesforce.com/home/home.jsp?source=lex


Please elaborate as

If Address bar content is Salesforce Instance URL is

agithomas · 2022-11-30T07:28:07Z

packages/salesforce/_dev/build/docs/README.md

+
+### Steps to find out the version of Salesforce
+
+On the Home tab in Salesforce Classic, on the top right of the screen is a link to release like `Summer '22`. This indicates your release.


What about for Salesforce Lightening?

We could not find the exact way for the Salesforce Lightning, However User will still have two options

Switch his view to Classic and can find the same, they just need one click to perform this activity.

To find out the version of Salesforce by hitting the following URL: (Salesforce Instance URL)/services/data
The option 2 is already mentioned in the Readme

agithomas · 2022-11-30T08:17:57Z

@SubhrataK , kindly have a look at the README and share your feedback.

SubhrataK · 2022-12-02T10:01:21Z

Feedback shared via the Google Doc. @kush-elastic kindly update the readme based on my comments provided in the doc.

agithomas · 2022-12-05T12:08:31Z

@SubhrataK , @kush-elastic has requested for the final approval of the PR. Can you please check if it looks good to you?

SubhrataK · 2022-12-06T04:42:11Z

@agithomas Looks good to me.

agithomas · 2022-12-06T08:51:19Z

LGTM

agithomas

LGTM

elasticmachine · 2022-12-06T11:09:01Z

Package salesforce - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=salesforce

Add 'salesforce' package foundation along with 'login_rest' data stream

defea0f

yug-rajani added enhancement New feature or request Team:Service-Integrations Label for the Observability Service Integrations team Integration:salesforce Salesforce labels Sep 21, 2022

yug-rajani self-assigned this Sep 21, 2022

yug-rajani linked an issue Sep 21, 2022 that may be closed by this pull request

Create package for Salesforce integration #3940

Closed

8 tasks

Add changelog.yml entry

9d63d03

yug-rajani requested review from kush-elastic, lalit-satapathy and a team September 21, 2022 19:48

Merge branch 'elastic:main' into package_salesforce_login_rest

6f4571e

yug-rajani changed the title ~~Add salesforce package foundation along with login_rest data stream~~ [salesforce] Add salesforce package foundation along with login_rest data stream Sep 27, 2022

yug-rajani added 2 commits October 1, 2022 21:51

Update mechanism to avoid duplication, pipeline and links in README

81d52e6

Remove entries from 'links_table.yml'

d727374

yug-rajani marked this pull request as ready for review October 3, 2022 05:49

kush-elastic approved these changes Oct 7, 2022

View reviewed changes

yug-rajani and others added 2 commits October 7, 2022 11:31

Minor updates to the pipeline

8008b02

Merge branch 'elastic:main' into package_salesforce_login_rest

9758621

agithomas reviewed Oct 12, 2022

View reviewed changes

packages/salesforce/_dev/deploy/docker/docker-compose.yml Show resolved Hide resolved

agithomas reviewed Oct 13, 2022

View reviewed changes

packages/salesforce/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/salesforce/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/salesforce/data_stream/login_rest/agent/stream/httpjson.yml.hbs Outdated Show resolved Hide resolved

yug-rajani added 3 commits October 17, 2022 15:03

Add missing processors and update README as per review comments

a380c66

Minor change to period description and update event.original testing

cbd8834

Merge branch 'main' of https://github.com/yug-elastic/integrations in…

982850d

…to package_salesforce_login_rest

yug-rajani requested a review from agithomas October 17, 2022 13:42

agithomas reviewed Oct 18, 2022

View reviewed changes

agithomas mentioned this pull request Nov 28, 2022

Discuss : Allowed IPs for system testing elastic/elastic-package#1053

Closed

Address Review Comments

db72510

agithomas reviewed Nov 28, 2022

View reviewed changes

packages/salesforce/_dev/build/docs/README.md Outdated Show resolved Hide resolved

kush-elastic added 2 commits November 29, 2022 18:44

Address Review Comments

5b90278

Update Readme

9fb2275

kush-elastic requested a review from agithomas November 29, 2022 13:47

agithomas reviewed Nov 30, 2022

View reviewed changes

agithomas requested a review from SubhrataK November 30, 2022 08:18

Update README

60f3191

kush-elastic requested a review from agithomas December 5, 2022 07:38

SubhrataK approved these changes Dec 6, 2022

View reviewed changes

agithomas approved these changes Dec 6, 2022

View reviewed changes

kush-elastic merged commit 5e3cac3 into elastic:main Dec 6, 2022

kush-elastic mentioned this pull request Dec 19, 2022

[Salesforce] Add pagination support for login_rest and logout_rest data streams #4866

Merged

4 tasks

This was referenced Jan 5, 2023

[salesforce] Implement login_stream data stream for Salesforce #4941

Merged

[salesforce] Implement logout_stream data stream for Salesforce #4942

Closed

kush-elastic mentioned this pull request Apr 25, 2023

[salesforce] Implement logout_stream data stream for Salesforce #5990

Merged

4 tasks

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024


		Example:

		Address Bar Content: https://elastic1234-dev-ed.my.salesforce.com/home/home.jsp?source=lex


		### Steps to find out the version of Salesforce

		On the Home tab in Salesforce Classic, on the top right of the screen is a link to release like `Summer '22`. This indicates your release.

[salesforce] Add salesforce package foundation along with login_rest data stream #4261

[salesforce] Add salesforce package foundation along with login_rest data stream #4261

Uh oh!

Conversation

yug-rajani commented Sep 21, 2022 • edited by kush-elastic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

kush-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agithomas commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yug-rajani commented Oct 13, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kush-elastic commented Nov 25, 2022

Uh oh!

Uh oh!

agithomas commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agithomas commented Nov 28, 2022

Uh oh!

agithomas commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agithomas commented Nov 28, 2022

Uh oh!

ruflin commented Nov 29, 2022

Uh oh!

agithomas Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

agithomas Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

agithomas Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

kush-elastic Nov 30, 2022

Choose a reason for hiding this comment

Uh oh!

agithomas commented Nov 30, 2022

Uh oh!

SubhrataK commented Dec 2, 2022

Uh oh!

agithomas commented Dec 5, 2022

Uh oh!

SubhrataK commented Dec 6, 2022

Uh oh!

agithomas commented Dec 6, 2022

Uh oh!

agithomas left a comment

[salesforce] Add `salesforce` package foundation along with `login_rest` data stream #4261

[salesforce] Add `salesforce` package foundation along with `login_rest` data stream #4261

yug-rajani commented Sep 21, 2022 •

edited by kush-elastic

Loading

elasticmachine commented Sep 21, 2022 •

edited

Loading

elasticmachine commented Sep 21, 2022 •

edited

Loading

elasticmachine commented Sep 21, 2022 •

edited

Loading

agithomas commented Oct 13, 2022 •

edited

Loading

agithomas commented Nov 28, 2022 •

edited

Loading

agithomas commented Nov 28, 2022 •

edited

Loading