[FEATURE] Reduce Mandatory arguments from code and use rule run settings from rules table

[asingamaneni]

Is your feature request related to a problem? Please describe.
As we have started to develop Spark-Expectations as a decorator pattern approach, we took all the options into the decorator and made some mandatory options for user to be provided in code, It has come to lot of options to maintain in code now, which can be avoided by maintaining the same settings from the rules table.

Below are the options that have to be provided right now

@se.with_expectations(
    se.reader.get_rules_from_table(
        product_rules_table="dq_spark_local.dq_rules",
        target_table_name="dq_spark_local.customer_order",
        dq_stats_table_name="dq_spark_local.dq_stats",
    ),
    write_to_table=True,
    row_dq=True,
    agg_dq={
        user_config.se_agg_dq: True,
        user_config.se_source_agg_dq: True,
        user_config.se_final_agg_dq: True,
    },
    query_dq={
        user_config.se_query_dq: True,
        user_config.se_source_query_dq: True,
        user_config.se_final_query_dq: True,
        user_config.se_target_table_view: "order",
    },
    spark_conf=global_spark_Conf,
)

Describe the solution you'd like
We could remove most of the mandatory arguments from the decorator and source them from the rule_table. The options can still exist for debugging and feature development and testing purposes for the engineers but first preference would be taking from the rules_table

@se.with_expectations(
    se.reader.get_rules_from_table(
        product_rules_table="dq_spark_local.dq_rules",
        target_table_name="dq_spark_local.customer_order",
        dq_stats_table_name="dq_spark_local.dq_stats",
    ),
    write_to_table=True,
    write_to_temp_table=True,
    spark_conf=global_spark_Conf
)

row_dq, agg_dq, query_dq arguments which were removed from the decorators arguments can be validated from the rules table columns rule_type, is_active, enable_for_source_dq_validation, enable_for_target_dq_validation

If the user want to override the configuration provided in the rule_table, then should be able to provide them as a single dict kwargs. This essentially will be helpful for engineers testing feature branches and new rules that needs to be tested.

Example:

override_confg: {
    "row_dq": False,
    "agg_dq": False,
    "target_dq": False,
}

override_confg: {
    "row_dq": False,
    "agg_dq": False,
    "source_dq": True,
    "target_dq": False,
}

Below could be the implementation strategy

• fetch rules which are active

  • check if atleast one rule has source_agg enabled
    • check if atleast one rule has agg_dq type
      • run for each rule
    • check if atleast one rule has query_dq type
      • run for each rule
  • check if atleast one rule had row_dq enable
    * run row_dq and return row_dq_df
    • check if atleast one rule has target_agg enabled
      • check if atleast one rule has agg_dq type
        • run for each rule on row_dq_df
      • check if atleast one rule has query_dq type
        • run for each rule on row_dq_df

• Send stats by default irrespective of success or failure

Describe alternatives you've considered
NA

Additional context
As we are maturing the product, it is time to probably time to rethink the strategy to make things easy on implementation