Adding support for custom reporters#154
Conversation
|
This looks good, but by adding this does we give up the ability to get line numbers and user-defined custom message? Ideally, it will be super useful if we construct the output with the following properties: Example rule: KMS key rotation |
136d38c to
049a711
Compare
|
This looks good. Just one more thing: is it also possible to show line numbers (line and column) and/or CDK metadata (if present)? |
gandhek
left a comment
There was a problem hiding this comment.
Overall LGTM! Couple of minor things -
| .arg(Arg::with_name("type").long("type").short("t").takes_value(true).possible_values(&["CFNTemplate"]) | ||
| .help("Specify the type of data file used for improved messaging")) |
There was a problem hiding this comment.
Can we infer the type since we only need to check whether or not its a CloudFormation template?
There was a problem hiding this comment.
We can add auto-detection as a follow-up to the explicit type. Currently, CFN Template is not structurally unique and does not need to include any identifying information like TemplateVersion as required fields.
|
|
||
| let mut overall = Status::PASS; | ||
| for (each, data_file_name) in iterator? { | ||
| let mut reporters= match data_type { |
There was a problem hiding this comment.
Nit: Space after "reporters"
|
Having an interesting issue. When more than one of the same resource fails the same rule, I only see the first resource printed. For instance: my_ruleset.guard template.json Running the following: Yields this output: When I'd expect it to be: |
|
@jstmcneil there is an internal issue we are tracking to get it fixed. Currently, the evaluation engine short-circuits when it encounters the first one that fails. We are amidst resolving that issue. For now, you can change the rule as which would produce the right output for-all volumes. $ cfn-guard validate -r /tmp/sample.guard -d /tmp/sample.json --type CFNTemplate
FAILED rules
/tmp/sample.guard/migrated_rules FAIL
Evaluation against template /tmp/sample.json, number of resource failures = 2
-
Resource NewVolume2 failed due to the following checks
/tmp/sample.guard/migrated_rules EQUALS failed at property path Properties.Encrypted because provided value [false] did not match with expected value [true]. Error Message [[EC2-008] : EC2 volumes should be encrypted]
-
Resource NewVolume failed due to the following checks
/tmp/sample.guard/migrated_rules EQUALS failed at property path Properties.Encrypted because provided value [false] did not match with expected value [true]. Error Message [[EC2-008] : EC2 volumes should be encrypted]
-as expected. Please pick up the latest changes (or wait for release coming soon) that contains fixes for migrate tool that generated these incorrectly as well. |
|
So I translated the rules to that format, but whenever I have a rule that can't find a corresponding resource, the whole thing seems to break and output nothing. For instance: my_ruleset.guard template.json Running the following: cfn-guard validate --type CFNTemplate --show-summary none -r my_ruleset.guard -d template.json Yields no output at all, despite there being policy infringements. Perhaps I'm not translating my rules correctly? |
|
Scratch that. Played around with some of the filtering rules and got it to work. |
|
@jstmcneil have updated #156. Let us continue the conversation there. thanks |
| let stacker = StackTracker::new(&root_context); | ||
| let reporter = ConsoleReporter::new(stacker, "lambda-function", "input-payload", true, true, false); | ||
| let reporters = vec![]; | ||
| let reporter = ConsoleReporter::new(stacker, &reporters, "lambda-function", "input-payload", true, true, false); |
There was a problem hiding this comment.
Is validate_and_return_json function required in validate.rs? Because for guard-lambda, it is using validate_and_return_json function in helper.rs?
There was a problem hiding this comment.
@shreyasdamle yes this is part that we will re-visit post PR merge to expose this functionality via Lambda as well. This is in addition to the multiple rule functionality that you added.
d91a4b1 to
0c7beca
Compare
| if !failed_rules.is_empty() { | ||
| let mut by_resource_name = HashMap::new(); | ||
| for each_failed_rule in failed_rules { | ||
| let failed = find_all_failing_clauses(each_failed_rule); |
There was a problem hiding this comment.
It looks like output is not working for non-boolean clauses.
Command:
cargo run --bin cfn-guard -- validate --data test.json --rules rule.guard --output-format yaml
Result:
FAILED rules
rule.guard/aws_ddb_sse_customer_managed_cmk FAIL
---
data_from: test.json
rules_from: rule.guard
failed: {}
However, it displays the context correctly when we run the following command:
cargo run --bin cfn-guard -- validate --data test.json --rules rule.guard --show-clause-failures
Result:
FAILED rules
rule.guard/aws_ddb_sse_customer_managed_cmk FAIL
Evaluation of rules rule.guard against data test.json
-
Clause Failure Summary
rule.guard/aws_ddb_sse_customer_managed_cmk Clause #1 FAIL(Clause(Location[file:rule.guard, line:19, column:5], Check: %aws_dynamodb_table_resources.Properties.SSESpecification.SSEEnabled EQUALS Bool(true)))
Attempting to retrieve array index or key from map at Path = /Resources/Mytable3CF392D5/Properties, Type was not an array/object map, Remaining Query = SSESpecification.SSEEnabled
Clause #2 FAIL(Clause(Location[file:rule.guard, line:22, column:5], Check: %aws_dynamodb_table_resources.Properties.SSESpecification.SSEType EQUALS %aws_dynamodb_table_resources_allowed_algorithms))
Attempting to retrieve array index or key from map at Path = /Resources/LalTable85A8FD0C/Properties/SSESpecification, Type was not an array/object map, Remaining Query = SSEType
There was a problem hiding this comment.
@shreyasdamle can you attach the same rule and template here?
There was a problem hiding this comment.
Sorry, I misinterpreted this error initially. It has nothing to do with boolean values. It's a user experience related issue. So in a given template if a particular property is not explicitly defined in a CloudFormation Resource, we get the following msg:
Attempting to retrieve array index or key from map at Path...
Hence, is it possible to make message more clear and intuitive when context (comparison, expected) is null?
- Support for "CFNTemplate" data type added with reporting that is similar to Guard 1.0 output format - Refactored all console rendering of summaries with variable format support - Added convertion back into serde::Value for easy output - Added the ability to suppress summary table completely or just show what is needed - Provided a single line format compatible with 1.0. There is some discrepency as 2.0 supports multiple files, 1.0 did not - Provided support for YAML/JSON output formats - Provided support for specializing for data file type that alters the output for specificity
- refactored common code across CFN specific and generic
- Fixed PR comments
- Sending in overall status for data file
- Introduced block summary in addition to single line
- Added compliant and not applicable rule names for YAML/JSON formats
c4d3209 to
fee5e4b
Compare
TODO: Retrieval errors are still fast fail, need to merge changes to query to return partial and complete traversals and report out errors consistently
| if self.summary_type.contains(SummaryType::SKIP) && !skipped.is_empty() { | ||
| writeln!(writer, "{}", "SKIP rules".bold()); | ||
| print_partition(writer, self.rules_file_name, &skipped, longest_rule_name)?; | ||
|
|
| super::common::print_name_info( | ||
| writer, &info, longest_rule_len, rules_file_name, data_file_name, | ||
| |_, _, info| { | ||
| Ok(format!("Resource [{}] traversed until [{}] for template [{}] wasn't compliant with [{}/{}] due to retrieval error. Error Message [{}]", |
There was a problem hiding this comment.
The functions retrieval_error_message, unary_error_message, binary_error_message are abstracted out pretty nicely in the generic summary, but not here. Is there a reason for that? IMO, its just easier to read what is getting passed on to print_name_info.
There was a problem hiding this comment.
This is because the closures are capturing the resource variable from the loop. See
for (resource, info) in by_resource_name.iter() {
super::common::print_name_info(
writer, &info, longest_rule_len, rules_file_name, data_file_name,all other functionality that is common between these reporters is abstracted out into common functions. Some were made redundant as we will revisit these when appropriate for messaging changes
| if !passed.is_empty() { | ||
| writeln!(writer, "--"); | ||
| } | ||
| for pass in passed { | ||
| writeln!(writer, "Rule [{}/{}] is compliant for data [{}]", rules_file_name, pass, data_file_name); | ||
| } | ||
|
|
||
| if !skipped.is_empty() { | ||
| writeln!(writer, "--"); | ||
| } | ||
| for skip in skipped { | ||
| writeln!(writer, "Rule [{}/{}] is not applicable for data [{}]", rules_file_name, skip, data_file_name); | ||
| } | ||
| writeln!(writer, "--"); | ||
| Ok(()) |
There was a problem hiding this comment.
This is duplicated in both cfn reporter and here, any way we can pull this out into a common construct? Not a 100% sure what the rust best practices are, but something like an abstract class maybe?
There was a problem hiding this comment.
This was deliberate, as messaging could potentially differ in the future for these reporters.
There was a problem hiding this comment.
Can we use string constants for capturing the boiler plate text of the error messages?
| U: Fn(&str, &str, &str, &NameInfo<'_>) -> crate::rules::Result<String>, | ||
| B: Fn(&str, &str, &str, &NameInfo<'_>) -> crate::rules::Result<String> |
There was a problem hiding this comment.
unary and binary have the same api, any reason to declare them separately? Can we have a type "error" as Fn(&str, &str, &NameInfo<'_> -> crate::rules::Result<String>) and another type "message" as
Fn(&str, &str, &str, &NameInfo<'_>) -> crate::rules::Result<String> instead of three types?
There was a problem hiding this comment.
| binary_message( | ||
| rules_file_name, | ||
| data_file_name, | ||
| if not { "did" } else { "did not" }, |
There was a problem hiding this comment.
This is rather confusing. If not then did, else did not? Can we use variable names that go with the flow?
It would have been easier to read had it been
if not
{"did not"}
else
{"did"}
There was a problem hiding this comment.
@gandhek, not here is a boolean variable. So the variable name can be changed to improve readability.
There was a problem hiding this comment.
not means exactly that, NOT or negation of the operator. !=, NOT EXISTS, NOT EMPTY. What alternative names would you suggest?
There was a problem hiding this comment.
May be not can be named negation_present?
| B: Fn(&str, &str, &str, &NameInfo<'_>) -> crate::rules::Result<String> | ||
| { | ||
| for each in info { | ||
| let (cmp, not) = match &each.comparison { |
There was a problem hiding this comment.
The Comparison struct is defined as,
pub(super) struct Comparison {
operator: CmpOperator,
not: bool,
}
The members are operator and not. Using cmp to represent operator is confusing at first sight as it confuses the reader if it is representing an object of the Comparison struct or is representing a comparison operator. The use of not as variable name in this class is also confusing, may be we can use something like not_operator_present where we can clearly tell it is a variable.
There was a problem hiding this comment.
Operator is ==, >, <, <= etc. and not is negation of that operation. So, != or not Key == "Key" is represented as { operator: CmpOperation::Eq, not: true }. Comparison operator not being present is what Option indicates. That comparison operation is None for block query semantics that fail retrieval errors.
There was a problem hiding this comment.
May be not can be named negation_present?
| fn end_evaluation(&self, eval_type: EvaluationType, context: &str, msg: String, from: Option<PathAwareValue>, to: Option<PathAwareValue>, status: Option<Status>) { | ||
| fn end_evaluation(&self, eval_type: EvaluationType, context: &str, msg: String, from: Option<PathAwareValue>, to: Option<PathAwareValue>, status: Option<Status>, _cmp: Option<(CmpOperator, bool)>) { | ||
| assert_ne!(msg.as_str(), ""); | ||
| assert_eq!(msg.starts_with("FIRST PART"), true); |
There was a problem hiding this comment.
Can we declare "FIRST PART" as a constant and reuse in this test class?
There was a problem hiding this comment.
This is unrelated to this change. Can we postpone these as it will confuse the PR?
- Fix evaluate to remove accrued errors for IN comparison if any one succeeds
Description of changes:
Adding the ability to provide the single-line-summary format from 1.0 with custom messages. This PR adds support for the CFN Templates. Will follow with additional PRs to support other formats
Sample run
Update on 2021-06-04
Support for multiple output formats is now added along with Summary table display selection or no-display. Updating PR text to show-case these
By default, no options, single-line-summary
No summary table option, single-line-summary
Summary Table only PASS,FAIL rules, single-line-summary
Summary Table PASS, FAIL, output-format YAML
No summary table, ouput-format yaml
No summary table, outut-format yaml, multiple data files
Output has a document per data file and rule combination for failed resources
$ cfn-guard validate -r migrated-3.guard -d /tmp/data/ --show-summary none --output-format yaml --- data_from: /tmp/data/sample-template-2.yaml rules_from: migrated-3.guard failed: aws_ec2_volume_checks: - rule: aws_ec2_volume_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: - Eq - false message: "" mixed_types_checks: - rule: mixed_types_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: - Eq - false message: "" --- data_from: /tmp/data/sample-template.yaml rules_from: migrated-3.guard failed: mixed_types_checks: - rule: mixed_types_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: - Eq - false message: "" aws_ec2_volume_checks: - rule: aws_ec2_volume_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: - Eq - false message: ""No Summary table, output-format json multiple data files
Each line JSON document for each data-file/rule-file combination
$ cfn-guard validate -r migrated-3.guard -d /tmp/data/ --show-summary none --output-format json {"data_from":"/tmp/data/sample-template-2.yaml","rules_from":"migrated-3.guard","failed":{"aws_ec2_volume_checks":[{"rule":"aws_ec2_volume_checks","path":"/Resources/vol2/Properties/Encrypted","provided":false,"expected":true,"comparison":["Eq",false],"message":""}],"mixed_types_checks":[{"rule":"mixed_types_checks","path":"/Resources/vol2/Properties/Encrypted","provided":false,"expected":true,"comparison":["Eq",false],"message":""}]}} {"data_from":"/tmp/data/sample-template.yaml","rules_from":"migrated-3.guard","failed":{"mixed_types_checks":[{"rule":"mixed_types_checks","path":"/Resources/vol2/Properties/Encrypted","provided":false,"expected":true,"comparison":["Eq",false],"message":""}],"aws_ec2_volume_checks":[{"rule":"aws_ec2_volume_checks","path":"/Resources/vol2/Properties/Encrypted","provided":false,"expected":true,"comparison":["Eq",false],"message":""}]}}No Summary, multiple rule files and multiple data files
$ cfn-guard validate -r /tmp/rules/ -d /tmp/data/ --output-format yaml --show-summary none --- data_from: sample-template.yaml rules_from: cluster.guard not_compliant: {} not_applicable: - test compliant: [] --- data_from: sample-template.yaml rules_from: migrated-3.guard not_compliant: aws_ec2_volume_checks: - rule: aws_ec2_volume_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: operator: Eq not: false message: "" - rule: aws_ec2_volume_checks path: /Resources/vol2/Properties provided: ~ expected: ~ comparison: ~ message: "Attempting to retrieve array index or key from map at path = /Resources/vol2/Properties , Type was not an array/object map, Remaining Query = Size" mixed_types_checks: - rule: mixed_types_checks path: /Resources/vol2/Properties/Encrypted provided: false expected: true comparison: operator: Eq not: false message: "" not_applicable: - aws_apigateway_deployment_checks - aws_apigateway_stage_checks - aws_dynamodb_table_checks compliant: - aws_events_rule_checks - aws_iam_role_checksBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.