AAX SDK  2.4.1
Avid Audio Extensions Development Kit

How to automate different test scenarios for DSH.

Contents

What is DTT

DishTestTool (DTT) is a Ruby-based command line tool, which provides the ability to script and thus automate DSH test scenarios. It is included in the DigiShell Tools package in the /DTT directory.
Note
Ruby is installed by default on OS X. On Windows, you will need to install Ruby and add it to your PATH variable manually. For information regarding Ruby version compatibility with a specific build of DTT, see the ReadMe.txt file in /DTT/sources.
The DTT folder consists of:

How to run tests and suites in DTT

run_test is the main DTT execution program. run_test is able to execute Ruby scripts which have been placed in the scripts folder within the DTT directory.
Running DTT tests
Figure 1: running DTT tests.
For more information about script arguments, see Describing and using input arguments of the script

Writing DTT scripts

Most of the DTT scripts require DigiShell, which allows them to run dsh and execute different dsh commands. Each script should be represented in the form of class, which inherits Script class, and also each script must have at least two elements: self.inputs section, where all the input arguments of the test should be described, and run method, which is the main body of your script.
require 'DigiShell'
class ScriptSample < Script
def self.inputs
return {}
end
def run
return pass("Well, it didn't explode. So that's something.");
end
end
Listing 1: Skeleton of the script

Describing and using input arguments of the script

The available parameters and their values for a script are listed in the static self.inputs routine. Input arguments must be organized in the form of a hash map which is returned from this routine.
def self.inputs
return {
:sample_rate => [44100,[44100,48000,88200]],
:path_to_tfx => ['none'],
:threshold => [-96],
}
end
@dsh.init_dae(sample_rate)
Listing 2: Desribing input arguments for the script and using them
Hash entries should be in the following format:
:arg_name => [default_value, [range of allowed values]]
These arguments can be used then by just calling them by name, like in the example above with sample_rate argument.

Writing body of the script

The body of the script must be enclosed in the body of the run method of the script class. As far as most DTT tests need DSH, in the example below it's shows how to create a DSH instance in the script and how to use it then. DSH instance can be creted with DigiDhell.new method, which requites DigiShell module, as has already been said. Then all the DSH commands become available as methods of the DSH instance, and input arguments can be passed to these command as input arguments for the methods, i.e. in parentheses dsh.load_dish("DAE"). Also it's recommended to handle possible exceptions that may occure during the execution of the code, and to make sure that DSH has been closed, if it was instantiated on the moment of the failure.
def run
begin
dsh = DigiShell.new(target)
dsh.load_dish("DAE")
dsh.init_dae(sample_rate)
dsh.close
return pass("Well, it didn't explode. So that's something.")
rescue Exception => e
# make sure to close down dsh before returning
if (dsh)
dsh.close
end
return fail(e)
end
end
Listing 3: Example of the body of the script.

Logging in DTT and debugging DTT scripts

DTT tool has logging, and all the logs collected in the Logs folder, which is located in root directory of the DTT. DTT creates a separate folder for each test and names these folders with the corresponding names + the time when the particular test has been executed. For example:
DSH_SigCancellation_20131225_185146_0001
Inside each folder there are several log files:

Interactive mode

There is an option to run DTT in interactive mode using interactive ruby shell (irb). When running in this mode, DTT creates a shell which is an extended version of the standard Ruby interpreter. Besides the standard functionality, it knows how to work with DTT classes and can give hints on their methods. In particular, the DTT interactive mode shell knows how to work with DigiShell.
To run DTT in interactive mode, go to the DTT folder and launch the run_irb program. At this point you will send ruby commands to dsh through the pipe in YAML format:
t = Target.new # creates an instance of Target. In this case target is a local machine, though we have a possibility to run test on remote machine.
dsh = DigiShell.new(t) # creates an instance of DigiShell(aka launching dsh binary)
dsh.load_dish('DAE') #Loads 'DAE' dish
dsh.help('init_dae') # Requests help from dsh for 'init_dae' command
dsh.init_dae
plugins = dsh.run #returns an array of plug-ins and writes them to plugins var.
plug-ins[0] #reaching first plug-in from the list
#... whatever you want to do
Listing 4: Running DTT in interactive mode

Working with DTT test suites

Suites are files which contain the list of DTT scripts that should be run, and parameteres for these tests. These files should be created in YAML format. The list of the tests should be preceeded by tests: line. Then tests to be run should be described as a map with the following members:
The input parameters of the test should be orginized as a hash map. That means that all keys should start with ":" and look like ":plugin_spec: ".
Also suite may contain a section with the general parameters like:
Example of the suite:
suitesettings:
verbose: false
tests:
#
# Cycle counting test
#
- name: DSH_TI_CycleCounts
enabled: true
args:
:plugin_spec: 'Digi,Pich,Psmm'
:sample_rate: 48000
Listing 5: Example of the DTT test suite
Note
All the suites files should have an extension .gss

Autogeneration of the suites

Sometimes it is necessary to generate the suites for the particular script for all the plug-ins from the bundle and/or for different sample rates. In this case instead of the copy-paste, which may lead to some mistakes and erratums, suitegenerator can be used. This is a special script, which takes as arguments the name of the script(s), for which the suites should be generated, and the list of their input parameters. Strange as it may sound, this data should be formed as a suite. Script itself is available as SuiteGenerator.rb along with other scripts in the DTT.
Note
SuiteGenerator.rb can generate suites only for the two tests: DSH_SigCancellation test and DSH_TI_CycleCounts test
Here is the example of how to use this script to generate the suites for all the plug-ins from the 'D-Verb' bundle for all the sample rates the cancellation test AAX Native vs AAX DSP, and for the cycle counts test:
tests:
# Generate suite for Cancellation test: AAX Native vs AAX DSP
- name: SuiteGenerator
args:
:plugin_name: 'D-Verb'
:path_to_audio_files: /Volumes/G_Audio/GS_Test_Resources/audio/
:path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
:test_script: DSH_SigCancellation
enabled: true
# Generate suite for Instance Count test
- name: SuiteGenerator
args:
:plugin_name: 'D-Verb'
:path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
:test_script: DSH_TI_CycleCounts_se
enabled: true
suitesettings:
verbose: false
timeoutFactor: "16.0"
Listing 6: Example of how to use SuiteGenerator script for generating suites for all the plug-ins from the 'D-Verb' bundle for the all sample rates for the cancellation and for the cycle counts test.
To generate the suites, one should run this suite for the SuiteGenerator as an ordinary suite by executing the run_test.command <the name of the suite for the SuiteGenerator> command. All the suites will be generated into the single file, which will be located inside the suites folder, and will be called like:
dspVSnative(optional)_<the name of the plug-in>_<the name of the test>.gss
Examples:
Collaboration diagram for DTT Guide: