# HG changeset patch # User Lucas Thompson # Date 1481503297 0 # Node ID 270f59ef3b83ebc2979dd9ebbab164b5cd755520 # Parent dc07ec24349129b163a2c7bc00f359cfdb1d7052 Incorporate recent piper changes and use output descriptor for some basic feature discrimination to render appropriate layers. diff -r dc07ec243491 -r 270f59ef3b83 src/app/services/feature-extraction/FeatureExtractionWorker.ts --- a/src/app/services/feature-extraction/FeatureExtractionWorker.ts Fri Dec 09 14:06:44 2016 +0000 +++ b/src/app/services/feature-extraction/FeatureExtractionWorker.ts Mon Dec 12 00:41:37 2016 +0000 @@ -2,11 +2,9 @@ * Created by lucas on 01/12/2016. */ -import {ListResponse, EmscriptenProxy} from 'piper'; -import {PiperSimpleClient} from 'piper/HigherLevelUtilities'; +import { EmscriptenProxy } from 'piper'; +import { PiperSimpleClient } from 'piper/HigherLevelUtilities'; import { VampExamplePlugins } from 'piper/ext/VampExamplePluginsModule'; -import { VampTestPlugin } from 'piper/ext/VampTestPluginModule'; - // TODO TypeScript has a .d.ts file for webworkers, but for some reason it clashes with the typings for dom and causes compiler errors interface WorkerGlobalScope { @@ -24,12 +22,14 @@ constructor(workerScope: WorkerGlobalScope) { this.workerScope = workerScope; - this.piperClient = new PiperSimpleClient(new EmscriptenProxy(VampTestPlugin())); + this.piperClient = new PiperSimpleClient(new EmscriptenProxy(VampExamplePlugins())); this.workerScope.onmessage = (ev: MessageEvent) => { - const sendResponse = (result) => this.workerScope.postMessage({ - method: ev.data.method, - result: result - }); + const sendResponse = (result) => { + this.workerScope.postMessage({ + method: ev.data.method, + result: result + }); + }; switch (ev.data.method) { case 'list': this.piperClient.list({}).then(sendResponse); @@ -38,7 +38,7 @@ this.piperClient.process(ev.data.params).then(sendResponse); break; case 'collect': - this.piperClient.collect(ev.data.params).then(sendResponse); + this.piperClient.collect(ev.data.params).then(sendResponse).catch(err => console.error(err)); } }; } diff -r dc07ec243491 -r 270f59ef3b83 src/app/services/feature-extraction/feature-extraction.service.ts --- a/src/app/services/feature-extraction/feature-extraction.service.ts Fri Dec 09 14:06:44 2016 +0000 +++ b/src/app/services/feature-extraction/feature-extraction.service.ts Mon Dec 12 00:41:37 2016 +0000 @@ -2,8 +2,9 @@ import { ListResponse, ListRequest } from "piper"; -import {SimpleRequest, FeatureCollection} from "piper/HigherLevelUtilities"; -import {FeatureList} from "piper/Feature"; +import { + SimpleRequest, SimpleResponse +} from "piper/HigherLevelUtilities"; import {Subject} from "rxjs/Subject"; import {Observable} from "rxjs"; @@ -17,18 +18,16 @@ result: ResponseType; } -export type Extracted = FeatureList | FeatureCollection; - @Injectable() export class FeatureExtractionService { private worker: Worker; - private featuresExtracted: Subject; - featuresExtracted$: Observable; + private featuresExtracted: Subject; + featuresExtracted$: Observable; constructor() { this.worker = new Worker('bootstrap-feature-extraction-worker.js'); - this.featuresExtracted = new Subject(); + this.featuresExtracted = new Subject(); this.featuresExtracted$ = this.featuresExtracted.asObservable(); } @@ -39,8 +38,8 @@ ).then(msg => msg.result); } - process(request: SimpleRequest): Promise { - return this.request( + process(request: SimpleRequest): Promise { + return this.request( {method: 'process', params: request}, (ev: MessageEvent) => ev.data.method === 'process' ).then(msg => { @@ -49,8 +48,8 @@ }); } - collect(request: SimpleRequest): Promise { - return this.request( + collect(request: SimpleRequest): Promise { + return this.request( {method: 'collect', params: request}, (ev: MessageEvent) => ev.data.method === 'collect' ).then(msg => { diff -r dc07ec243491 -r 270f59ef3b83 src/app/waveform/waveform.component.ts --- a/src/app/waveform/waveform.component.ts Fri Dec 09 14:06:44 2016 +0000 +++ b/src/app/waveform/waveform.component.ts Mon Dec 12 00:41:37 2016 +0000 @@ -5,15 +5,15 @@ import {AudioPlayerService} from "../services/audio-player/audio-player.service"; import wavesUI from 'waves-ui'; import { - FeatureExtractionService, - Extracted + FeatureExtractionService } from "../services/feature-extraction/feature-extraction.service"; import {Subscription} from "rxjs"; import { FeatureCollection, - FixedSpacedFeatures + FixedSpacedFeatures, SimpleResponse } from "piper/HigherLevelUtilities"; import {toSeconds} from "piper"; +import {FeatureList} from "piper/Feature"; type Timeline = any; // TODO what type actually is it.. start a .d.ts for waves-ui? type Layer = any; @@ -163,11 +163,17 @@ } // TODO refactor - this doesn't belong here - private renderFeatures(extracted: Extracted, colour: Colour): void { - if (!extracted.hasOwnProperty('shape') || !extracted.hasOwnProperty('data')) return; - const features: FeatureCollection = (extracted as FeatureCollection); + private renderFeatures(extracted: SimpleResponse, colour: Colour): void { + if (!extracted.hasOwnProperty('features') || !extracted.hasOwnProperty('outputDescriptor')) return; + if (!extracted.features.hasOwnProperty('shape') || !extracted.features.hasOwnProperty('data')) return; + const features: FeatureCollection = (extracted.features as FeatureCollection); + const outputDescriptor = extracted.outputDescriptor; + const height = this.trackDiv.nativeElement.getBoundingClientRect().height; + const mainTrack = this.timeline.getTrackById('main'); + + // TODO refactor all of this switch (features.shape) { - case 'vector': + case 'vector': { const stepDuration = (features as FixedSpacedFeatures).stepDuration; const featureData = (features.data as Float32Array); const normalisationFactor = 1.0 / @@ -183,14 +189,90 @@ }); let breakpointLayer = new wavesUI.helpers.BreakpointLayer(plotData, { color: colour, - height: this.trackDiv.nativeElement.getBoundingClientRect().height + height: height }); this.colouredLayers.set(this.addLayer( breakpointLayer, - this.timeline.getTrackById('main'), + mainTrack, this.timeline.timeContext ), colour); break; + } + case 'list': { + const featureData = (features.data as FeatureList); + // TODO look at output descriptor instead of directly inspecting features + const hasDuration = outputDescriptor.configured.hasDuration; + const isMarker = !hasDuration + && outputDescriptor.configured.binCount === 0 + && featureData[0].featureValues == null; + const isRegion = hasDuration + && featureData[0].timestamp != null; + + // TODO refactor, this is incomprehensible + if (isMarker) { + const plotData = featureData.map(feature => { + return {x: toSeconds(feature.timestamp)} + }); + let markerLayer = new wavesUI.helpers.MarkerLayer(plotData, { + height: height, + color: colour, + }); + this.colouredLayers.set(this.addLayer( + markerLayer, + mainTrack, + this.timeline.timeContext + ), colour); + } else if (isRegion) { + const isBarRegion = featureData[0].featureValues.length === 1; + const getSegmentArgs = () => { + if (isBarRegion) { + const min = featureData.reduce((min, feature) => + Math.min(min, feature.featureValues[0]), + Infinity + ); + + const max = featureData.reduce((max, feature) => + Math.max(max, feature.featureValues[0]), + -Infinity + ); + + return [ + featureData.map(feature => { + return { + x: toSeconds(feature.timestamp), + y: feature.featureValues[0], + width: toSeconds(feature.duration), + height: 0.05 * max, + color: colour, + opacity: 0.8 + } + }), + {yDomain: [min, max + 0.05 * max], height: height} as any + ] + } else { + return [featureData.map(feature => { + return { + x: toSeconds(feature.timestamp), + width: toSeconds(feature.duration), + color: colour, + opacity: 0.8 + } + }), {height: height}]; + } + }; + + let segmentLayer = new wavesUI.helpers.SegmentLayer( + ...getSegmentArgs() + ); + this.colouredLayers.set(this.addLayer( + segmentLayer, + mainTrack, + this.timeline.timeContext + ), colour); + } + + break; + } } this.timeline.tracks.update();