c@0: classdef MASSEFresults < handle c@0: %MASSEFRESULTS A class for storing MASSEF results c@0: % c@0: % The multichannel audio source separation evaluation framework uses this c@0: % class to store results. You can use its methods to recall and filter c@0: % the data. c@0: % c@0: % MASSEFRESULTS properties: c@0: % Read-only properties: c@0: % data - The full results set returned as a Table object c@0: % (read only). The Table has the following columns: c@0: % - algorithmNum - the algorithm number; c@0: % - channel - the audio channel c@0: % - estimateNum - the estimate number; c@0: % - estTag - a tag for each estimate from a c@0: % given algorithm; c@0: % - metric - the performance metric; c@0: % - mixNum - the mixture number; and c@0: % - value - the value of the performance metric c@0: % Additional mixture and algorithm information is c@0: % also returned in the results set. c@0: % estTags - A list of the estTags, taken from the separators, c@0: % that feature in the results set (read only). c@0: % metrics - The metrics that feature in the results set (read c@0: % only). c@0: % numAlgorithms - The number of algorithms in the results set (read c@0: % only). c@0: % numMixtures - The number of mixtures in the results set (read c@0: % only). c@0: % numEstimates - The (maximum) number of estimates in the results c@0: % set (read only). c@0: % numChannels - The (maximum) number of channels in the results c@0: % set (read only). c@0: % c@0: % MASSEFRESULTS methods: c@0: % MASSEFresults - Create a MASSEFresults object. c@0: % algorithmInfo - Add algorithm information. c@27: % boxPlot - Analyse the results data by plotting notched c@27: % box plots. c@0: % filter - Filter the results set. c@0: % input - Add performance data. c@0: % merge - Combine results array into singular object. c@0: % mixtureInfo - Add mixture information. c@0: % removeDuplicates - Remove duplicate data from the results c@0: % object. c@0: % c@0: % See also MASSEFFW. c@0: c@0: % Copyright 2016 University of Surrey. c@0: c@0: properties (SetAccess = private, Dependent) c@0: data % The full results set. c@0: estTags % A tag assigned to each estimate. c@0: metrics % The metrics that feature in the results set. c@0: numAlgorithms % The number of algorithms in the results set. c@0: numMixtures % The number of mixtures in the results set. c@0: numEstimates % The (maximum) number of estimates in the results set. c@0: numChannels % The (maximum) number of channels in the results set. c@0: end c@0: c@0: properties (Access = private) c@0: mixtures % mixtures table c@0: algorithms % algorithms table c@0: performance % main performance table c@0: end c@0: c@0: methods c@0: c@0: function obj = MASSEFresults() c@0: %MASSEFRESULTS Create the results set. c@0: % c@0: % R = MASSEFRESULTS creates an empty MASSEFRESULTS object. Use the c@0: % MASSEFRESULTS.INPUT method to add data. c@0: c@0: obj.mixtures = cell2table(cell(0,9), 'VariableNames',{'mixNum','azi_sep','elevation','filename_t','filename_i','sofa_path','target_azi','target_ele','tir'}); c@0: obj.algorithms = cell2table(cell(0,2), 'VariableNames',{'algorithmNum','algorithmLabel'}); c@0: obj.performance = cell2table(cell(0,7), 'VariableNames',{'mixNum','algorithmNum','estimateNum','channel','metric','estTag','value'}); c@0: c@0: end c@0: c@0: function data = get.data(obj) c@0: % GET.DATA get data set c@0: c@0: data = obj.joinLookupTables(obj.performance); c@0: c@0: end c@0: c@0: function metrics = get.metrics(obj) c@0: % GET.METRICS get metrics c@0: metrics = unique(obj.performance.metric); c@0: end c@0: c@0: function estTags = get.estTags(obj) c@0: % GET.METRICS get metrics c@0: estTags = unique(obj.performance.estTag); c@0: end c@0: c@0: function numAlgorithms = get.numAlgorithms(obj) c@0: % GET.NUMALGORITHMS get number of algorithms c@0: numAlgorithms = length(unique(obj.performance.algorithmNum)); c@0: end c@0: c@0: function numMixtures = get.numMixtures(obj) c@0: % GET.NUMMIXTURES get number of mixtures c@0: numMixtures = length(unique(obj.performance.mixNum)); c@0: end c@0: c@0: function numEstimates = get.numEstimates(obj) c@0: % GET.NUMESTIMATES get number of estimates c@0: numEstimates = length(unique(obj.performance.estimateNum)); c@0: end c@0: c@0: function numChannels = get.numChannels(obj) c@0: % GET.NUMCHANNEL get number of channels c@0: numChannels = length(unique(obj.performance.channel)); c@0: end c@0: c@0: function newObj = merge(obj) c@0: % MERGE Combine MASSEFresults array into singular object c@0: % c@0: % NEWOBJ = R.MERGE() combines the elements of the MASSEFRESULTS array c@0: % R into a singular MASSEFRESULTS object NEWOBJ. c@0: c@0: newObj = obj(1); c@0: if length(obj) > 1 c@0: for n = 2:length(obj) c@0: % combine lookup table and eliminate duplicate rows c@0: newObj.mixtures = vertcat(newObj.mixtures,obj(n).mixtures); c@0: newObj.algorithms = vertcat(newObj.algorithms,obj(n).algorithms); c@0: % combine all rows of performance tables c@0: newObj.performance = vertcat(newObj.performance,obj(n).performance); c@0: end c@0: end c@0: newObj.removeDuplicates(); c@0: c@0: end c@0: c@0: function removeDuplicates(obj) c@0: % REMOVEDUPLICATES Remove duplicate data from MASSEFresults object c@0: % c@0: % R.REMOVEDUPLICATES() removes duplicate results from the c@0: % MASSEFRESULTS object R. c@0: c@0: obj.mixtures = unique(obj.mixtures); c@0: obj.algorithms = unique(obj.algorithms); c@0: obj.performance = unique(obj.performance); c@0: c@0: end c@0: c@0: function data = filter(obj,varargin) c@0: %FILTER Filter the results data set c@0: % c@0: % R.FILTER(NAME,VALUE) filters the results set contained in the c@0: % MASSEFRESULTS object R using the variable names and values c@0: % contained in the NAME / VALUE pair arguments. The parameters c@0: % are: c@0: % c@0: % - 'algorithmnum' - Filters the data according to the c@0: % algorithm number. The parameter should be a function c@0: % handle that takes the mixture number as its input, and c@0: % returns a logical value. c@0: % - 'channel' - Filters the data according to channel c@0: % information. The parameter can be a function handle that c@0: % takes the channel number as its input, and returns a c@0: % logical value. Alternatively, the parameter can be 'max' c@0: % or 'mean', which calculates the the maximum or mean c@0: % respectively for every combination of the other c@0: % variables. c@0: % - 'estimate' - Filters the data according to estimate c@0: % information. The specification is identical to 'channel'. c@0: % - 'estTag' - Filters the data according to the estmate tag. c@0: % The parameter should be a function handle that takes the c@0: % tag string as its input, and returns a logical value. c@0: % - 'metric' - Filters the data according to the metric. The c@0: % parameter should be a function handle that takes the c@0: % metric name as its input, and returns a logical value. c@0: % - 'mixnum' - Filters the data according to the mixture c@0: % number. The parameter should be a function handle that c@0: % takes the mixture number as its input, and returns a c@0: % logical value. c@0: % - 'value' - Filters the data according to the value. The c@0: % parameter should be a function handle that takes the c@0: % value as its input, and returns a logical value. c@33: % c@33: % If using the 'mean' or 'max' option, the respective variable is c@33: % removed from the output. c@0: c@23: assert(mod(length(varargin),2)==0,'MASSEFresults:filter:invalidArgs','input must contain parameter/value pairs') c@0: data = obj.data; c@0: c@0: % work through varargin c@0: for n = 1:2:length(varargin) c@0: filtername = varargin{n}; % column to filter on c@0: filterval = varargin{n+1}; % value used to filter c@0: switch lower(filtername) % do filtering c@0: case 'channel' c@0: data = obj.filterRowOrAggregate(data,filterval,'channel',... c@0: {'mixNum','algorithmNum','estimateNum','metric'},... c@0: {'mixNum','algorithmNum','metric'}); c@0: case 'estimate' c@0: data = obj.filterRowOrAggregate(data,filterval,'estimateNum',... c@0: {'mixNum','algorithmNum','channel','metric'},... c@0: {'mixNum','algorithmNum','metric'}); c@0: case 'mixnum' c@0: data = obj.filterRows(data,filterval,'mixNum'); c@0: case 'algorithmnum' c@0: data = obj.filterRows(data,filterval,'algorithmNum'); c@0: case 'metric' c@0: data = obj.filterRows(data,filterval,'metric'); c@0: case 'esttag' c@0: data = obj.filterRows(data,filterval,'estTag'); c@0: case 'value' c@0: data = obj.filterRows(data,filterval,'value'); c@0: otherwise c@0: c@0: end c@0: c@0: end c@0: c@0: end c@0: c@0: function mixtureInfo(obj,mixtureNumber,varargin) c@0: %MIXTUREINFO Add mixture information. c@0: % c@0: % R.MIXTUREINFO(MIXTURENUM,NAME,VALUE) adds algorithm information c@0: % for the mixture with number MIXTURENUM to the results set c@0: % contained in the MASSEFRESULTS object R using the variable names c@0: % and values contained in the NAME / VALUE pair arguments. The c@0: % following information can be stored about each mixture: c@0: % c@0: % - 'azi_sep' - azimuthal separation of widest sources c@0: % (numeric); c@0: % - 'elevation' - median elevation of sources (numeric); c@0: % - 'filename_t' - target filename (char array); c@0: % - 'filename_i' - interferer filename (char array); c@0: % - 'sofa_path' - SOFA filename (char array); c@0: % - 'target_azi' - the target azimuth (numeric); c@0: % - 'target_ele' - the target elevation (numeric); and c@0: % - 'tir' - target-to-interferer ratio (dB) (numeric). c@0: c@0: % ensure some inputs are strings c@0: varargin = obj.ensureKeyValParamStrs({'sofa_path','filename_t','filename_i'},varargin); c@0: c@0: % add data c@0: obj.mixtures = obj.addData(obj.mixtures,'mixNum',mixtureNumber,varargin{:}); c@0: c@0: end c@0: c@0: function algorithmInfo(obj,algorithmNumber,varargin) c@0: %ALGORITHMINFO Add algorithm information. c@0: % c@0: % R.ALGORITHMINFO(ALGORITHMNUM,NAME,VALUE)| adds algorithm c@0: % information for the algorithm with number ALGORITHMNUM to the c@0: % results set contained in the MASSEFRESULTS object R using the c@0: % variable names and values contained in the NAME / VALUE pair c@0: % arguments. The following information can be stored about each c@0: % algorithm: c@0: % c@0: % - 'algorithmLabel' - a label for the algorithm (char c@0: % array). c@0: c@0: % ensure some inputs are strings c@0: varargin = obj.ensureKeyValParamStrs('algorithmLabel',varargin); c@0: c@0: obj.algorithms = obj.addData(obj.algorithms,'algorithmNum',algorithmNumber,varargin{:}); c@0: c@0: end c@0: c@0: function input(obj,mixtureNum,algorithmNum,estimateNum,metric,channel,estTag,value) c@0: %INPUT Input performance data. c@0: % c@0: % R.INPUT(MIXTURENUM,ALGORITHMNUM,... c@0: % ESTIMATENUM,METRIC,CHANNEL,ESTTAG,VALUE) c@0: % inputs the performance data for mixture number MIXTURENUM, c@0: % algorithm number ALGORITHMNUM, estimate number ESTIMATENUM, c@0: % metric METRIC, channel number CHANNEL, estimate tag ESTTAG, and c@0: % value VALUE to the MASSEFRESULTS instance R. c@0: c@0: if ~ischar(estTag) c@0: estTag = char(estTag); c@0: end c@0: c@0: rownames = {'mixNum','algorithmNum','estimateNum','metric','channel','estTag','value'}; c@0: values = {mixtureNum,algorithmNum,estimateNum,metric,channel,estTag,value}; c@0: row = cell2table(values, 'VariableNames',rownames); c@0: c@0: try c@0: % find existing row c@0: match = obj.performance.mixNum==mixtureNum && ... c@0: obj.performance.algorithmNum==algorithmNum && ... c@0: obj.performance.estimateNum==estimateNum && ... c@0: strcmp(metric,obj.performance.metric) && ... c@0: obj.performance.channel==channel && ... c@0: strcmp(estTag,obj.performance.estTag); c@0: c@0: if any(match) c@0: % replace c@0: obj.performance(find(match,1,'first'),:) = row; c@0: else c@0: % append c@0: obj.performance = [obj.performance; row]; c@0: end c@0: catch % there is no data c@0: % append c@0: obj.performance = [obj.performance; row]; c@0: end c@0: c@0: end c@0: c@27: function bph = boxPlot(obj) c@27: %BOXPLOT Analyse the results data by plotting notched box plots c@27: % c@27: % R.BOXPLOT() produces a series of box plots, one for each c@27: % metric, plotting the performance of each algorithms/estimate c@27: % aggregated across all mixtures. c@27: % c@27: % BPH = R.BOXPLOT() returns an array of IOSR.STATISTICS.BOXPLOT c@27: % objects BPH for the plots. c@27: c@27: for m = numel(obj.metrics):-1:1 c@27: c@27: tabData = obj.filter('metric', @(x) strcmp(x, obj.metrics{m}), 'channel', 'max'); c@27: tabData.name = strcat(tabData.algorithmLabel, ':', {' '}, tabData.estTag); c@27: c@27: [y, x] = iosr.statistics.tab2box(tabData.name, tabData.value); c@27: c@27: figure c@27: bph(m) = iosr.statistics.boxPlot(x, y, 'notch', true); c@27: ylabel(obj.metrics{m}) c@27: xlabel('Algorithm') c@27: box on c@27: c@27: end c@27: c@27: end c@27: c@0: end % public methods c@0: c@0: methods (Hidden) c@0: c@0: function debug(obj) %#ok c@0: keyboard; c@0: end c@0: c@0: end % hidden methods c@0: c@0: methods (Access = private) c@0: c@0: function dataTable = filterRowOrAggregate(obj,dataTable,filterval,col,group,altgroup) c@0: %FILTERROWORAGGREGATE filter data based on aggregate function. c@0: c@0: if ischar(filterval) c@0: % special aggregate function c@0: switch lower(filterval) c@0: case 'max' c@0: fhandle = @max; c@0: case 'mean' c@0: fhandle = @mean; c@0: otherwise c@23: error('MASSEFresults:filterRowOrAggregate:unknownOption','Unknown filter parameter ''%s''.',filterval) c@0: end c@0: % do stats c@0: try c@27: filteredTable = varfun(fhandle,dataTable,'InputVariables','value',... c@0: 'GroupingVariables',group); c@31: keys = group; c@0: catch c@27: filteredTable = varfun(fhandle,dataTable,'InputVariables','value',... c@0: 'GroupingVariables',altgroup); c@31: keys = altgroup; c@0: end c@0: % rename value column and delete GroupCount column c@27: filteredTable = obj.findRenameVar(filteredTable,'value','value'); c@27: filteredTable.GroupCount = []; c@31: [~, ia, ib] = intersect(filteredTable(:,keys), dataTable(:,keys)); c@31: dataTable.value(ib) = filteredTable.value(ia); c@31: dataTable = dataTable(ib, :); c@33: dataTable(:, col) = []; c@0: else c@0: % normal filter function c@0: dataTable = obj.filterRows(dataTable,filterval,col); c@0: end c@0: c@0: end c@0: c@0: function dataTable = joinLookupTables(obj,dataTable) c@0: %JOINLOOKUPTABLES join a table to the lookup tables c@0: c@0: if ~isempty(obj.mixtures) c@0: dataTable = outerjoin(dataTable,obj.mixtures,'Type','left','MergeKeys',true,'keys','mixNum'); c@0: end c@0: if ~isempty(obj.algorithms) c@0: dataTable = outerjoin(dataTable,obj.algorithms,'Type','left','MergeKeys',true); c@0: end c@0: c@0: end c@0: c@0: end % private methods c@0: c@0: methods (Static, Access = private) c@0: c@0: function dataTable = addData(dataTable,key,keyVal,varargin) c@0: %ADDDATA add data to lookup tables. c@0: c@23: assert(mod(length(varargin),2)==0,'MASSEFresults:addData:invalidArgs','input must contain parameter/value pairs') c@0: rI = find(ismember(dataTable.(key),keyVal),1,'first'); c@0: if isempty(rI) % add a new row c@0: % get data from varargin c@0: varnames = cell(0); c@0: vals = cell(0); c@0: for n = 1:2:length(varargin) c@0: varnames{(n+1)/2} = varargin{n}; c@0: vals{(n+1)/2} = varargin{n+1}; c@0: end c@0: % make new row c@0: vars = dataTable.Properties.VariableNames; c@0: newrow = cell(1,length(vars)); c@0: newrow{strcmp(key,vars)} = keyVal; c@0: for r = 1:length(varnames) c@0: if ismember(varnames{r},vars); c@0: newrow{strcmp(varnames{r},vars)} = vals{r}; c@0: end c@0: end c@0: newrow = cell2table(newrow,'VariableNames',vars); c@0: % append c@0: dataTable = [dataTable; newrow]; c@0: else % update row c@0: for n = 1:2:length(varargin) c@0: try c@0: dataTable(rI,varargin(n)) = varargin(n+1); c@0: catch c@0: dataTable.(varargin{n})(rI) = varargin(n+1); c@0: end c@0: end c@0: end c@0: c@0: end c@0: c@0: function dataTable = findRenameVar(dataTable,old,new) c@0: %FINDRENAMEVAR Find and rename a variable. c@0: c@0: varnames = dataTable.Properties.VariableNames; c@0: k = find(cellfun(@(x) ~isempty(strfind(x,old)),varnames),1,'first'); c@0: dataTable.Properties.VariableNames{varnames{k}} = new; c@0: c@0: end c@0: c@0: function dataTable = filterRows(dataTable,fhandle,col) c@0: %FILTERROWS filter the rows in a table. c@0: c@23: assert(isa(fhandle,'function_handle'),'MASSEFresults:filterRows:invalidFhandle','Parameter must be a function handle') c@0: dataTable = dataTable(fhandle(dataTable.(col)),:); c@0: c@0: end c@0: c@0: function C = ensureKeyValParamStrs(keys,keyValArray) c@0: keys = cellstr(keys); c@0: C = keyValArray; c@0: keyValArrayStr = cellfun(@char,keyValArray,'UniformOutput',false); c@0: [~,~,ib] = intersect(keys,keyValArrayStr); c@0: C(ib+1) = cellfun(@char,C(ib+1),'UniformOutput',false); c@0: end c@0: c@0: end % private static methods c@0: c@0: end