Improve user experience of usage of custom rules

[dymurray]

Today, the usage of custom rules within the project is inconsistent and confusing for a few reasons. At runtime within the CLI or within the UI workflow, the selection of rules to process for violations always depends upon a label selector. This label selector is built up a few different ways across these 2 workflows, but changing that is not relevant nor in scope for this RFE. It it sometimes possible for a user to specify a custom rule or ruleset and for that rule to never be processed at analysis time if the corresponding label selector does not match the labels on the provided rule. Additionally, if a user supplies a custom rule with some labels that match other rules in the default ruleset, it is possible for those rules to be included in the analysis.

I propose that we conform to two axioms:

• If a user supplies a custom rule, ALWAYS process that rule at analysis time.
• If a user supplies a custom rule, this should have zero impact on the inclusion of other rules at analysis time

In the CLI, this means if a user runs:

kantra analyze --input <app> --source=eap6 --target eap8 --rules ./custom-rules.yaml

The rules in custom-rules.yaml will always be processed, even if the rule does not have labels that match the source/target label selector that is built up. The same should apply to the UI workflow, but in the UI it also means that we do zero processing of the labels on that rule so that analysis does not include additional rules based on those labels.

[konveyor-ci-bot]

This issue is currently awaiting triage.
If contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members.

[shawn-hurley]

Why pass in --source and/or --target?

What happens when someone has a shared folder of rulesets for themselves and wants to use a label selector?

I personally liked the behavior when, if you passed in a custom rule/ruleset, we did not run the default rulesets, IIRC.

I feel like the --source, --target, and label-selector flags have a very specific meaning and are scoped well. They can also be turned off (in fact, that is the default), and all rules will be run.

I am not suggesting that we don't have an issue in the UX, but I don't know if this won't introduce a new set of issues.

I would personally like to get rid of source/target; a user should be able to select rulesets and optionally filter them. Then a custom rule or ruleset is just another option.

In other words, I want to expose the rulesets and make them the first-class way to choose which rules to run.