Modified Streaming and Generating Custom Search Command #407
Conversation
akaila-splunk
commented
Oct 27, 2021
- Fix field dropping issue for Streaming and Generating CSC
Description added for how to use add_field method to preserve conditionally added new fields and values.
| def add_field(self, current_record, field_name, field_value): | ||
| self._record_writer.custom_fields.add(field_name) | ||
| current_record[field_name] = field_value |
There was a problem hiding this comment.
This function should return current_record or else the documentation is incorrect.
| def add_field(self, current_record, field_name, field_value): | |
| self._record_writer.custom_fields.add(field_name) | |
| current_record[field_name] = field_value | |
| def add_field(self, current_record, field_name, field_value): | |
| self._record_writer.custom_fields.add(field_name) | |
| current_record[field_name] = field_value | |
| return current_record |
There was a problem hiding this comment.
Good catch @Bre77 - I think in this case the documentation needs to be updated, the current_record should be updated with the new field without the need to return since it's essentially pass-by-reference for this usage (I always go back to this article when I need to refresh my memory: https://robertheaton.com/2014/02/09/pythons-pass-by-object-reference-as-explained-by-philip-k-dick/).
So if you all agree can we fix the README.md above to the following?
class CustomStreamingCommand(StreamingCommand):
def stream(self, records):
for index, record in enumerate(records):
if index % 1 == 0:
self.add_field(record, "odd_record", "true")
yield record
There was a problem hiding this comment.
And to make sure that example we provide is working ^ that's probably a good implementation to add when adding tests for the StreamingCommand / add_field case
There was a problem hiding this comment.
I've changed my implementation to using pass by reference (as per the suggested doco update) and everything works as expected.
There was a problem hiding this comment.
Good catch @Bre77
I've updated the README accordingly.
| output_records = [i for i in output_iter.next().data] | ||
|
|
||
| # Assert that count of records having "odd_field" is 0 | ||
| assert len(list(filter(lambda r: "odd_field" in r, output_records))) == 0 |
There was a problem hiding this comment.
Really nice demonstration of the issue!