output does not return the expected type 'bool' from CreateUIDefinition

[v-mosh21]

Issue:
Got "output cosmosFreeApply does not return the expected type 'bool'" error when using test toolkit to check the template: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/test-toolkit#test-parameters.

Investigation:

1. mainTemplate.json contains a bool parameter:
   "parameters": {
   "cosmosFreeApply": {
   "type": "bool",
   "defaultValue": true,
   "metadata": {
   "description": "Apply Free Level Discount."
   }
   }
2. createUIDefinition.json contains an output to the same parameter:
   {
   "$schema": "https://schema.management.azure.com/schemas/0.1.2-preview/CreateUIDefinition.MultiVm.json#",
   "handler": "Microsoft.Azure.CreateUIDef",
   "version": "0.1.2-preview",
   "parameters": {
   "basics": [],
   "steps": [
   {
   "name": "otherSettings",
   "label": "Details",
   "elements": [
   {
   "name": "sectionCosmos",
   "type": "Microsoft.Common.Section",
   "label": "Cosmos DB",
   "elements": [
   {
   "name": "cosmosTier",
   "type": "Microsoft.Common.OptionsGroup",
   "label": "Apply Free Plan",
   "defaultValue": "No",
   "toolTip": "Select whether to use Free Plan for Cosmos DB",
   "constraints": {
   "allowedValues": [
   {
   "label": "Yes",
   "value": "[bool('true')]"
   },
   {
   "label": "No",
   "value": "[bool('false')]"
   }
   ],
   "required": true
   }
   }
   ]
   }
   ]
   }
   ],
   "outputs": {
   "cosmosFreeApply": "[steps('otherSettings').sectionCosmos.cosmosTier]",
   "location": "[location()]"
   }
   }
   }

3.When using the Test-AzTemplate to check template, I got below error:
Validating customertest\createUiDefinition.json
JSONFiles Should Be Valid
[+] JSONFiles Should Be Valid (7 ms)

Allowed Values Should Actually Be Allowed
[+] Allowed Values Should Actually Be Allowed (21 ms)
Controls In Outputs Must Exist
[+] Controls In Outputs Must Exist (4 ms)
CreateUIDefinition Must Not Have Blanks
[+] CreateUIDefinition Must Not Have Blanks (5 ms)
CreateUIDefinition Should Have Schema
[+] CreateUIDefinition Should Have Schema (5 ms)
Credential Confirmation Should Not Be Hidden
[+] Credential Confirmation Should Not Be Hidden (16 ms)
Handler Must Be Correct
[+] Handler Must Be Correct (4 ms)
HideExisting Must Be Correctly Handled
[+] HideExisting Must Be Correctly Handled (15 ms)
Location Should Be In Outputs
[+] Location Should Be In Outputs (3 ms)
CreateUIDefinition
[-] Outputs Must Be Present In Template Parameters (28 ms)
output cosmosFreeApply does not return the expected type 'bool'

Parameters Without Default Must Exist In CreateUIDefinition
[+] Parameters Without Default Must Exist In CreateUIDefinition (4 ms)
Password Textboxes Must Be Used For Password Parameters
[+] Password Textboxes Must Be Used For Password Parameters (8 ms)
PasswordBoxes Must Have Min Length
[+] PasswordBoxes Must Have Min Length (8 ms)

.......

Pass : 47
Fail : 1
Total : 48

4.Navigate to the source script which is used to check output parameters: https://github.com/Azure/arm-ttk/blob/master/arm-ttk/testcases/CreateUIDefinition/Outputs-Must-Be-Present-In-Template-Parameters.test.ps1

It is evaluating two values: outputParameterType and firstOutputFunction:

if ($outputParameterType -eq $af.Key -and $firstOutputFunction -notin $af.Value) {
                    Write-Error "output $outputName does not return the expected type '$outputParameterType'" -ErrorId CreateUIDefinition.Output.Incorrect -TargetObject $parameterInfo.outputs
}

The $af.Key and $af.Value are retrieved from $AllowedFunctionInOutput as below:

$AllowedFunctionInOutput = $(@{
    int = 'int', 'min', 'max', 'div', 'add', 'mod', 'mul', 'sub', 'copyIndex','length', 'coalesce'
    bool = 'equals', 'less', 'lessOrEquals', 'greater', 'greaterOrEquals', 'and', 'or','not', 'true', 'false', 'contains','empty','coalesce','if'
})
)

Note the outputParameterType is generated from template parameter type (from mainTemplate) and firstOutputFunction is generated from the function name of the output (from CreateUiDefinition):

$outputParameterType = $TemplateObject.parameters.$outputName.type
$firstOutputFunction = $output.Value | ?<ARM_Template_Function> -Extract | Select-Object -ExpandProperty FunctionName

The outputParameterType is bool which is expected, however, the firstOutputFunction in this case gets a value as steps which is not in above $AllowedFunctionInOutput values {'equals', 'less', 'lessOrEquals', 'greater', 'greaterOrEquals', 'and', 'or','not', 'true', 'false', 'contains','empty','coalesce','if'}. Therefore, it results in an error.

"outputs": {
            "cosmosFreeApply": "[steps('otherSettings').sectionCosmos.cosmosTier]",
            "location": "[location()]"
        }

It looks like logic is not suitable in CreateUiDefinition because this file is made up of different elements instead of normal function used in ARM template.

May I ask if you could help to look at and fix the issue?

[bmoore-msft]

I remove this particular test/error in the 0.8 release.

I'm going to leave this active as a task to fix the test properly.

TLDR; there are a few controls that allow specific setting of the value property - in that property the proper functions may be used and we don't detect them. As well, those controls have multiple values so in theory we need to check all of the values for each element of the array to ensure they match.

TODO is in the code with more detail.

Also, this should be caught in pre-flight - so may cross the bar for TTK.