How to automate different test scenarios for DSH.
Contents
What is DTT
DishTestTool (DTT) is a Ruby-based command line tool, which provides the ability to script and thus automate DSH test scenarios. It is included in the DigiShell Tools package in the /DTT directory.
- Note
- Ruby is installed by default on OS X. On Windows, you will need to install Ruby and add it to your PATH variable manually. For information regarding Ruby version compatibility with a specific build of DTT, see the ReadMe.txt file in /DTT/sources.
The DTT folder consists of:
-
Sources
-
DTT core
-
scripts folder - place all DTT script files here
By default, this folder includes a few example scripts demonstrating basic DTT operations and plug-in testing steps:
-
DSH_SigCancellation.rb - script for the cancellation test
-
DSH_TI_CycleCounts - script for performing cycle count test
-
SuiteGenerator.rb - generates suites for the DSH_SigCancellation and DSH_TI_CycleCounts tests
-
suites folder - place your DTT suite files here
-
run_test.command (on Mac) or run_test.bat (on Windows) - command file for runnung tests
-
run_irb.command (on Mac) or run_irb.bat (on Windows) - command-line interpreter
How to run tests and suites in DTT
run_test
is the main DTT execution program. run_test
is able to execute Ruby scripts which have been placed in the scripts folder within the DTT directory.
-
run_test.command -l
- lists all the available scripts and suites
-
run_test.command 1
- runs test by number
-
run_test.command DSH_SigCancellation
- runs a test by name. Pay attensions that the name of test should be without (!) extension.
-
run_test.command –script DSH_SigCancellation -a sample_rate=48000 -a threshold=-80
- runs a test with test script arguments, which are specified using the -a
option
Figure 1: running DTT tests.
Writing DTT scripts
Most of the DTT scripts require DigiShell
, which allows them to run dsh and execute different dsh commands. Each script should be represented in the form of class, which inherits Script class, and also each script must have at least two elements: self.inputs section, where all the input arguments of the test should be described, and run method, which is the main body of your script.
require 'DigiShell'
class ScriptSample < Script
def self.inputs
return {}
end
def run
return pass("Well, it didn't explode. So that's something.");
end
end
Listing 1: Skeleton of the script
Describing and using input arguments of the script
The available parameters and their values for a script are listed in the static self.inputs
routine. Input arguments must be organized in the form of a hash map which is returned from this routine.
def self.inputs
return {
:sample_rate => [44100,[44100,48000,88200]],
:path_to_tfx => ['none'],
:threshold => [-96],
}
end
@dsh.init_dae(sample_rate)
Listing 2: Desribing input arguments for the script and using them
Hash entries should be in the following format:
:arg_name => [default_value, [range of allowed values]]
These arguments can be used then by just calling them by name, like in the example above with sample_rate
argument.
Writing body of the script
The body of the script must be enclosed in the body of the run
method of the script class. As far as most DTT tests need DSH, in the example below it's shows how to create a DSH instance in the script and how to use it then. DSH instance can be creted with DigiDhell.new
method, which requites DigiShell module, as has already been said. Then all the DSH commands become available as methods of the DSH instance, and input arguments can be passed to these command as input arguments for the methods, i.e. in parentheses dsh.load_dish("DAE")
. Also it's recommended to handle possible exceptions that may occure during the execution of the code, and to make sure that DSH has been closed, if it was instantiated on the moment of the failure.
def run
begin
dsh = DigiShell.new(target)
dsh.load_dish("DAE")
dsh.init_dae(sample_rate)
dsh.close
return pass("Well, it didn't explode. So that's something.")
rescue Exception => e
# make sure to close down dsh before returning
if (dsh)
dsh.close
end
return fail(e)
end
end
Listing 3: Example of the body of the script.
Logging in DTT and debugging DTT scripts
DTT tool has logging, and all the logs collected in the Logs folder, which is located in root directory of the DTT. DTT creates a separate folder for each test and names these folders with the corresponding names + the time when the particular test has been executed. For example:
DSH_SigCancellation_20131225_185146_0001
Inside each folder there are several log files:
-
xxx.html – contains info about input & output of the test in a fancy form (tables)
-
xxx_c.txt – contains the list of the DSH commands that have been executed
-
xxx_d.txt – DSH output
-
xxx_i.txt – info about your system
-
xxx_l.txt – standard output
-
xxx_v.txt – verbose output
Interactive mode
There is an option to run DTT in interactive mode using interactive ruby shell (irb). When running in this mode, DTT creates a shell which is an extended version of the standard Ruby interpreter. Besides the standard functionality, it knows how to work with DTT classes and can give hints on their methods. In particular, the DTT interactive mode shell knows how to work with DigiShell.
To run DTT in interactive mode, go to the DTT folder and launch the run_irb
program. At this point you will send ruby commands to dsh through the pipe in YAML format:
t = Target.new # creates an instance of Target. In this case target is a local machine, though we have a possibility to run test on remote machine.
dsh = DigiShell.new(t) # creates an instance of DigiShell(aka launching dsh binary)
dsh.load_dish('DAE') #Loads 'DAE' dish
dsh.help('init_dae') # Requests help from dsh for 'init_dae' command
dsh.init_dae
plugins = dsh.run #returns an array of plug-ins and writes them to plugins var.
plug-ins[0] #reaching first plug-in from the list
#... whatever you want to do
Listing 4: Running DTT in interactive mode
Working with DTT test suites
Suites are files which contain the list of DTT scripts that should be run, and parameteres for these tests. These files should be created in YAML format. The list of the tests should be preceeded by tests:
line. Then tests to be run should be described as a map with the following members:
-
name:
- name of the test
-
enabled:
- determines whether test will be run or skipped
-
args:
- input parameters of the test
The input parameters of the test should be orginized as a hash map. That means that all keys should start with ":" and look like ":plugin_spec: ".
Also suite may contain a section with the general parameters like:
-
verbose: false
which determines whether the output of the test in the console will be verbose or not.
-
timeoutFactor: "16.0"
which defines the time period, after which test will exit in case it stuck on the execution of the certain peice of code.
Example of the suite:
suitesettings:
verbose: false
tests:
#
# Cycle counting test
#
- name: DSH_TI_CycleCounts
enabled: true
args:
:plugin_spec: 'Digi,Pich,Psmm'
:sample_rate: 48000
Listing 5: Example of the DTT test suite
- Note
- All the suites files should have an extension .gss
Autogeneration of the suites
Sometimes it is necessary to generate the suites for the particular script for all the plug-ins from the bundle and/or for different sample rates. In this case instead of the copy-paste, which may lead to some mistakes and erratums, suitegenerator can be used. This is a special script, which takes as arguments the name of the script(s), for which the suites should be generated, and the list of their input parameters. Strange as it may sound, this data should be formed as a suite. Script itself is available as SuiteGenerator.rb along with other scripts in the DTT.
- Note
- SuiteGenerator.rb can generate suites only for the two tests: DSH_SigCancellation test and DSH_TI_CycleCounts test
Here is the example of how to use this script to generate the suites for all the plug-ins from the 'D-Verb' bundle for all the sample rates the cancellation test AAX Native vs AAX DSP, and for the cycle counts test:
tests:
# Generate suite for Cancellation test: AAX Native vs AAX DSP
- name: SuiteGenerator
args:
:plugin_name: 'D-Verb'
:path_to_audio_files: /Volumes/G_Audio/GS_Test_Resources/audio/
:path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
:test_script: DSH_SigCancellation
enabled: true
# Generate suite for Instance Count test
- name: SuiteGenerator
args:
:plugin_name: 'D-Verb'
:path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
:test_script: DSH_TI_CycleCounts_se
enabled: true
suitesettings:
verbose: false
timeoutFactor: "16.0"
Listing 6: Example of how to use SuiteGenerator script for generating suites for all the plug-ins from the 'D-Verb' bundle for the all sample rates for the cancellation and for the cycle counts test.
To generate the suites, one should run this suite for the SuiteGenerator as an ordinary suite by executing the run_test.command <the name of the suite for the SuiteGenerator>
command. All the suites will be generated into the single file, which will be located inside the suites folder, and will be called like:
dspVSnative(optional)_<the name of the plug-in>_<the name of the test>.gss
Examples:
-
TRIM_DSH_TI_CycleCounts.gss
-
dspVSnative_TRIM_DSH_SigCancellation.gss