aws_securityhub: add field support for Vulnerability, Data Security, and Detection OCSF finding types - #20843
Conversation
efd6
added
enhancement
23dbbe7 to
3f5a588
Compare
Elastic Docs Style Checker (Vale)Summary: 2 warnings found
|
| File | Line | Rule | Message |
|---|---|---|---|
| packages/aws_securityhub/data_stream/finding/fields/fields.yml | 969 | Elastic.BritishSpellings | Use American English spelling 'toward' instead of British English 'towards'. |
| packages/aws_securityhub/data_stream/finding/fields/fields.yml | 4250 | Elastic.BritishSpellings | Use American English spelling 'toward' instead of British English 'towards'. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| } | ||
| resource.tags = tags; | ||
| } | ||
| if (resource.databucket instanceof Map && resource.databucket.tags instanceof List) { |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml:1478
The tag-to-object conversion is applied to resources[].databucket.tags but not to the top-level finding.databucket.tags, so the same OCSF object is stored in two different shapes in one document; convert the top-level databucket tags too (and change its fields.yml mapping to tags.* object).
Details
A Data Security Finding carries the same databucket object twice: once at aws_securityhub.finding.databucket and once at aws_securityhub.finding.resources[].databucket. This loop only normalises the copy nested under resources, so in the committed expected output the two copies come out differently for identical input: resources[].databucket.tags becomes {"division":"acme",...} (test-findings.log-expected.json line 3160) while finding.databucket.tags stays [{"name":"division","value":"acme"},...] (line 2962). fields.yml matches that split, declaring resources.databucket.tags.* as an object but finding.databucket.tags as a group of name/value keywords. A user filtering on bucket tags therefore needs two different queries depending on which copy they hit, and the searchability benefit the changelog claims for databucket tags is only delivered on one of the two paths. Note the enclosing script is gated on resources being a non-empty list, so the top-level conversion is better placed in its own processor rather than inside this loop.
Recommendation:
Normalise the top-level databucket tags as well, in a dedicated processor so it does not depend on resources being present:
- script:
description: Convert key:value databucket tags into an object for better searchability.
tag: script_convert_databucket_tags_to_object
lang: painless
if: ctx.aws_securityhub?.finding?.databucket?.tags instanceof List
source: |-
def tags = [:];
for (def tag: ctx.aws_securityhub.finding.databucket.tags) {
tags[tag.name] = tag.value;
}
ctx.aws_securityhub.finding.databucket.tags = tags;
on_failure:
- append:
tag: append_error_message_databucket_tags
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'and align the mapping with the one already used for the resources copy:
- name: tags.*
type: object
description: The list of tags.
object_type: keyword
subobjects: false
object_type_mapping_type: '*'🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| } | ||
| } | ||
| } | ||
| on_failure: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/pipeline_object_evidence.yml:196
The new date-normalising scripts only append error.message on failure, dropping the remove of the unparseable value that the replaced date processors performed, so a bad timestamp now stays in a date-mapped field and the document is rejected at index time; remove or null the field inside the script when parsing fails.
Details
Each date processor this script replaces had an on_failure that first removed the offending field and only then appended error.message, so an unparseable timestamp cost one field but the document still indexed. This on_failure only appends error.message: the raw value is left in place at aws_securityhub.finding.evidences[].actor.process.terminated_time_dt (and the session fields), all of which fields.yml maps as date (fields.yml lines 1778-1783, 1817-1822). A value that neither ZonedDateTime.parse nor the space-format parser accepts therefore reaches Elasticsearch as a non-date string in a date field and the whole document fails to index. Two related regressions come from the same shape: the cast ((Number)proc.terminated_time).longValue() throws ClassCastException if the epoch value arrives as a string, which the previous UNIX_MS date processor accepted; and because the failure escapes the for loop, every evidence entry after the failing one is left unnormalised. The same pattern is in this file at line 691 and in pipeline_object_resources.yml at lines 418 and 454.
Recommendation:
Handle the parse failure inside the script so the document stays indexable, e.g. with a shared helper that removes the field it cannot normalise:
source: |
void normalize(def owner, def key, DateTimeFormatter spaceFmt) {
def v = owner[key];
if (v == null || (v instanceof String && v == '')) {
return;
}
try {
def zdt;
if (v instanceof Number) {
zdt = ZonedDateTime.ofInstant(Instant.ofEpochMilli(((Number)v).longValue()), ZoneOffset.UTC);
} else {
try { zdt = ZonedDateTime.parse((String)v); }
catch (Exception e) { zdt = ZonedDateTime.parse((String)v, spaceFmt); }
}
owner[key] = zdt.format(DateTimeFormatter.ISO_OFFSET_DATE_TIME);
} catch (Exception e) {
owner.remove(key);
}
}
def spaceFmt = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss[.SSSSSSSSS][.SSSSSSSS][.SSSSSSS][.SSSSSS][.SSSSS][.SSSS][.SSS][.SS][.S]X");
for (def ev : ctx.aws_securityhub.finding.evidences) {
if (ev == null || !(ev.actor instanceof Map)) {
continue;
}
if (ev.actor.process instanceof Map) {
normalize(ev.actor.process, 'terminated_time_dt', spaceFmt);
normalize(ev.actor.process, 'terminated_time', spaceFmt);
}
if (ev.actor.session instanceof Map) {
normalize(ev.actor.session, 'created_time_dt', spaceFmt);
normalize(ev.actor.session, 'created_time', spaceFmt);
normalize(ev.actor.session, 'expiration_time_dt', spaceFmt);
normalize(ev.actor.session, 'expiration_time', spaceFmt);
}
}🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| fields: | ||
| tags: | ||
| - preserve_duplicate_custom_fields | ||
| numeric_keyword_fields: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/aws_securityhub/data_stream/finding/_dev/test/pipeline/test-common-config.yml:4
The new normalized *_id fields are declared keyword but left numeric and silenced with numeric_keyword_fields, unlike every other *_id in this package which gets a convert ... type: string processor; add the convert processors instead of the test suppressions.
Details
This package normalises OCSF identifier fields to strings in the pipeline - default.yml alone carries convert-to-string processors for activity_id, category_uid, class_uid, cloud.account.type_id, compliance.status_id, observables.type_id and many more, and the expected output shows them as "1", "2003" and so on. The fields added here (resources.role_id, resources.user.type_id, resources.image.hash.algorithm_id and the databucket type_id/algorithm_id/category_id/status_id set) are declared as keyword in fields.yml but get no convert processor, so they stay JSON numbers in _source and the test-time type check has to be suppressed with numeric_keyword_fields. The result is that semantically identical fields render as 1 in some places and "1" in others within the same document, which is a trap for anyone building queries or dashboards off _source.
Recommendation:
Convert the new identifier fields like the existing ones and drop the corresponding numeric_keyword_fields entries, e.g. for the resource-level ones:
- foreach:
tag: foreach_aws_securityhub_finding_resources_role_id
field: aws_securityhub.finding.resources
if: ctx.aws_securityhub?.finding?.resources instanceof List
processor:
convert:
field: _ingest._value.role_id
tag: convert_resources_role_id_to_string
type: string
ignore_missing: true
- foreach:
tag: foreach_aws_securityhub_finding_resources_user_type_id
field: aws_securityhub.finding.resources
if: ctx.aws_securityhub?.finding?.resources instanceof List
processor:
convert:
field: _ingest._value.user.type_id
tag: convert_resources_user_type_id_to_string
type: string
ignore_missing: true🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - yyyy-MM-dd HH:mm:ss[.SSSSSSSSS][.SSSSSSSS][.SSSSSSS][.SSSSSS][.SSSSS][.SSSS][.SSS][.SS][.S]X | ||
| if: ctx.aws_securityhub?.finding?.databucket?.created_time_dt != null && ctx.aws_securityhub.finding.databucket.created_time_dt != '' | ||
| on_failure: | ||
| - remove: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml:1217
The on_failure sub-processors added by this PR have no tag, unlike every other processor in these pipelines; add tags so failures remain traceable and the package stays consistent ahead of the format_version 3.6.0 tag requirement.
Details
Every pre-existing processor in default.yml, pipeline_object_evidence.yml and pipeline_object_resources.yml carries a tag, including the nested remove/append pairs inside on_failure (for example remove_aws_securityhub_finding_end_time_dt_fe41467c and append_error_message_df5f1975 at default.yml lines 579-586). The four date processors added here (default.yml lines 1207-1267) tag the date processor but leave their on_failure remove and append untagged, and the four new script processors do the same for their on_failure append (pipeline_object_evidence.yml lines 197 and 692, pipeline_object_resources.yml lines 419 and 455). Because the error.message template interpolates _ingest.on_failure_processor_tag, an untagged failing processor produces an error message with an empty tag, which is exactly the information needed to locate it. The package is on format_version 3.5.0 so elastic-package does not enforce this yet.
Recommendation:
Tag the on_failure sub-processors the same way the surrounding ones are tagged:
on_failure:
- remove:
tag: remove_aws_securityhub_finding_databucket_created_time_dt
field: aws_securityhub.finding.databucket.created_time_dt
ignore_missing: true
- append:
tag: append_error_message_databucket_created_time_dt
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
e5e1f4c to
9b95ddd
Compare
9b95ddd to
080fb14
Compare
|
🟢 No issues across the latest commits 6300d33…080fb14 (62 commits). Review summaryIssues found across earlier commits 3f5a588 — 2 medium, 2 low
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
|
/test |
brijesh-elastic
left a comment
brijesh-elastic
left a comment
There was a problem hiding this comment.
LGTM once the CI issue is resolved.
…and Detection OCSF finding types
Add field definitions and date processing for three OCSF finding
types not previously covered:
- Vulnerability Finding: resources.image (architecture, hash,
timestamps, registry and repository identifiers).
- Data Security Finding (class_uid 2006): finding.databucket and
resources.databucket with encryption details, file
classifications, and tag conversion.
- Detection Finding (class_uid 2004): anomaly_analyses.baselines.
observed_pattern, resources.role, resources.role_id, and
resources.user.
Date fields in the new objects are normalised to ISO 8601 using
Painless script processors; foreach+date cannot guard against
absent intermediate path elements within a list.
Fix the resource tags conversion script to execute before the
early-return path taken when no primary resource is found among
multiple resources, so all resources get their tags converted.
Extend the conversion to also handle resources.databucket.tags,
turning the array-of-objects form into a searchable key-value
object.
Tests for date processing with space-separated format derived from
existing tests. Tests for new databucket and resource fields from
live instance with sanitisation.
080fb14 to
5a180a9
Compare
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
cc @efd6 |
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package aws_securityhub - 2.1.0 containing this change is available at https://epr.elastic.co/package/aws_securityhub/2.1.0/ |
2 participants
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots