Skip to content

DataFactoryTestingFrameworkExpressionsEvaluator adds status property even if None is set. #164

Description

We have scenarios where certain Activity outputs (like status as described in #162 ) are not present on the Activity Outputs.

Currently neither functional tests nor activity tests can capture these scenarios because we add the status always to the activity results (see https://github.com/microsoft/data-factory-testing-framework/blob/main/src/data_factory_testing_framework/_pythonnet/data_factory_testing_framework_expressions_evaluator.py#L40).

The expression evaluator returns a error we can handle to mimic the behaviour:

>           raise Exception(f"Unknown error: {e}") from e
E           Exception: Unknown error: The template language expression 'contains(
E             createArray(
E               activity('Fail').status,
E               activity('Notebook').status
E             ),
E             'Failed'
E           )' cannot be evaluated because property 'status' doesn't exist, available properties are 'output, error'. Please see https://aka.ms/logicexpressions for usage details.
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.SelectObjectProperty(JObject root, JToken property, Boolean safeDereference)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.SelectProperty(JToken root, TemplateDereferenceExpression[] dereferences, TemplateExpressionEvaluationContext context, Boolean failOnError)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.LazyParameterEnumerator.MoveNext(Int32 skip)
E              at Microsoft.Azure.Workflows.Templates.ISkippableEnumeratorExtensions.ToEnumerable[T](ISkippableEnumerator`1 enumerator)+MoveNext()
E              at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
E              at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionBuiltInFunctions.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount, Boolean preserveAnnotations)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionEvaluationHelper.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.LazyParameterEnumerator.MoveNext(Int32 skip)
E              at Microsoft.Azure.Workflows.Templates.ISkippableEnumeratorExtensions.ToEnumerable[T](ISkippableEnumerator`1 enumerator)+MoveNext()
E              at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
E              at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionBuiltInFunctions.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount, Boolean preserveAnnotations)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionEvaluationHelper.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E              at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E              at Microsoft.Azure.Workflows.Templates.Helpers.TemplateExpressionsHelper.EvaluateTemplateLanguageExpression(String expression, TemplateExpressionEvaluationContext evaluationContext)
E              at Microsoft.Azure.DataFactoryTestingFramework.Expressions.Evaluator.EvaluateExpression(String expression, String parametersJson, String variablesJson, String itemValueJson, String activityValuesJson)
E              at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
E              at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Proposed solution:

Only add the add properties if they are set to anything other than None.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions