amine@32: `auditok` API Tutorial amine@32: ====================== amine@32: amine@32: .. contents:: `Contents` amine@32: :depth: 3 amine@32: amine@32: amine@32: **auditok** is a module that can be used as a generic tool for data amine@32: tokenization. Although its core motivation is **Acoustic Activity amine@32: Detection** (AAD) and extraction from audio streams (i.e. detect amine@32: where a noise/an acoustic activity occurs within an audio stream and amine@32: extract the corresponding portion of signal), it can easily be amine@32: adapted to other tasks. amine@32: amine@32: Globally speaking, it can be used to extract, from a sequence of amine@32: observations, all sub-sequences that meet a certain number of amine@32: criteria in terms of: amine@32: amine@32: 1. Minimum length of a **valid** token (i.e. sub-sequence) amine@35: 2. Maximum length of a **valid** token amine@32: 3. Maximum tolerated consecutive **non-valid** observations within amine@32: a valid token amine@32: amine@32: Examples of a non-valid observation are: a non-numeric ascii symbol amine@32: if you are interested in sub-sequences of numeric symbols, or a silent amine@32: audio window (of 10, 20 or 100 milliseconds for instance) if what amine@32: interests you are audio regions made up of a sequence of "noisy" amine@32: windows (whatever kind of noise: speech, baby cry, laughter, etc.). amine@32: amine@32: The most important component of `auditok` is the :class:`auditok.core.StreamTokenizer` amine@32: class. An instance of this class encapsulates a :class:`auditok.util.DataValidator` and can be amine@32: configured to detect the desired regions from a stream. amine@32: The :func:`auditok.core.StreamTokenizer.tokenize` method accepts a :class:`auditok.util.DataSource` amine@32: object that has a `read` method. Read data can be of any type accepted amine@32: by the `validator`. amine@32: amine@32: amine@32: As the main aim of this module is **Audio Activity Detection**, amine@32: it provides the :class:`auditok.util.ADSFactory` factory class that makes amine@32: it very easy to create an :class:`auditok.util.ADSFactory.AudioDataSource` amine@32: (a class that implements :class:`auditok.util.DataSource`) object, be that from: amine@32: amine@32: - A file on the disk amine@32: - A buffer of data amine@32: - The built-in microphone (requires PyAudio) amine@32: amine@32: amine@32: The :class:`auditok.util.ADSFactory.AudioDataSource` class inherits from amine@32: :class:`auditok.util.DataSource` and supplies a higher abstraction level amine@32: than :class:`auditok.io.AudioSource` thanks to a bunch of handy features: amine@32: amine@32: - Define a fixed-length `block_size` (alias `bs`, i.e. analysis window) amine@32: - Alternatively, use `block_dur` (duration in seconds, alias `bd`) amine@32: - Allow overlap between two consecutive analysis windows amine@32: (if one of `hop_size` , `hs` or `hop_dur` , `hd` keywords is used and is > 0 and < `block_size`). amine@32: This can be very important if your validator use the **spectral** information of audio data amine@32: instead of raw audio samples. amine@32: - Limit the amount (i.e. duration) of read data (if keyword `max_time` or `mt` is used, very useful when reading data from the microphone) amine@32: - Record all read data and rewind if necessary (if keyword `record` or `rec` , also useful if you read data from the microphone and amine@32: you want to process it many times off-line and/or save it) amine@32: amine@32: See :class:`auditok.util.ADSFactory` documentation for more information. amine@32: amine@32: Last but not least, the current version has only one audio window validator based on amine@32: signal energy (:class:`auditok.util.AudioEnergyValidator). amine@32: amine@32: ********************************** amine@32: Illustrative examples with strings amine@32: ********************************** amine@32: amine@32: Let us look at some examples using the :class:`auditok.util.StringDataSource` class amine@32: created for test and illustration purposes. Imagine that each character of amine@33: :class:`auditok.util.StringDataSource` data represents an audio slice of 100 ms for amine@32: example. In the following examples we will use upper case letters to represent amine@32: noisy audio slices (i.e. analysis windows or frames) and lower case letter for amine@32: silent frames. amine@32: amine@32: amine@32: Extract sub-sequences of consecutive upper case letters amine@32: ####################################################### amine@32: amine@32: amine@32: We want to extract sub-sequences of characters that have: amine@32: amine@32: - A minimum length of 1 (`min_length` = 1) amine@32: - A maximum length of 9999 (`max_length` = 9999) amine@32: - Zero consecutive lower case characters within them (`max_continuous_silence` = 0) amine@32: amine@32: We also create the `UpperCaseChecker` with a `read` method that returns `True` if the amine@32: checked character is in upper case and `False` otherwise. amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import StreamTokenizer, StringDataSource, DataValidator amine@32: amine@32: class UpperCaseChecker(DataValidator): amine@32: def is_valid(self, frame): amine@32: return frame.isupper() amine@32: amine@32: dsource = StringDataSource("aaaABCDEFbbGHIJKccc") amine@32: tokenizer = StreamTokenizer(validator=UpperCaseChecker(), amine@32: min_length=1, max_length=9999, max_continuous_silence=0) amine@32: amine@32: tokenizer.tokenize(dsource) amine@32: amine@32: The output is a list of two tuples, each contains the extracted sub-sequence and its amine@32: start and end position in the original sequence respectively: amine@32: amine@32: amine@32: .. code:: python amine@32: amine@32: amine@32: [(['A', 'B', 'C', 'D', 'E', 'F'], 3, 8), (['G', 'H', 'I', 'J', 'K'], 11, 15)] amine@32: amine@32: amine@32: Tolerate up to two non-valid (lower case) letters within an extracted sequence amine@32: ############################################################################## amine@32: amine@32: To do so, we set `max_continuous_silence` =2: amine@32: amine@32: .. code:: python amine@32: amine@32: amine@32: from auditok import StreamTokenizer, StringDataSource, DataValidator amine@32: amine@32: class UpperCaseChecker(DataValidator): amine@32: def is_valid(self, frame): amine@32: return frame.isupper() amine@32: amine@32: dsource = StringDataSource("aaaABCDbbEFcGHIdddJKee") amine@32: tokenizer = StreamTokenizer(validator=UpperCaseChecker(), amine@32: min_length=1, max_length=9999, max_continuous_silence=2) amine@32: amine@32: tokenizer.tokenize(dsource) amine@32: amine@32: amine@32: output: amine@32: amine@32: .. code:: python amine@32: amine@32: [(['A', 'B', 'C', 'D', 'b', 'b', 'E', 'F', 'c', 'G', 'H', 'I', 'd', 'd'], 3, 16), (['J', 'K', 'e', 'e'], 18, 21)] amine@32: amine@32: Notice the trailing lower case letters "dd" and "ee" at the end of the two amine@32: tokens. The default behavior of :class:`auditok.core.StreamTokenizer` is to keep the *trailing amine@35: silence* if it does not exceed `max_continuous_silence`. This can be changed amine@32: using the `StreamTokenizer.DROP_TRAILING_SILENCE` mode (see next example). amine@32: amine@32: Remove trailing silence amine@32: ####################### amine@32: amine@32: Trailing silence can be useful for many sound recognition applications, including amine@32: speech recognition. Moreover, from the human auditory system point of view, trailing amine@32: low energy signal helps removing abrupt signal cuts. amine@32: amine@32: If you want to remove it anyway, you can do it by setting `mode` to `StreamTokenizer.DROP_TRAILING_SILENCE`: amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import StreamTokenizer, StringDataSource, DataValidator amine@32: amine@32: class UpperCaseChecker(DataValidator): amine@32: def is_valid(self, frame): amine@32: return frame.isupper() amine@32: amine@32: dsource = StringDataSource("aaaABCDbbEFcGHIdddJKee") amine@32: tokenizer = StreamTokenizer(validator=UpperCaseChecker(), amine@32: min_length=1, max_length=9999, max_continuous_silence=2, amine@32: mode=StreamTokenizer.DROP_TRAILING_SILENCE) amine@32: amine@32: tokenizer.tokenize(dsource) amine@32: amine@32: output: amine@32: amine@32: .. code:: python amine@32: amine@32: [(['A', 'B', 'C', 'D', 'b', 'b', 'E', 'F', 'c', 'G', 'H', 'I'], 3, 14), (['J', 'K'], 18, 19)] amine@32: amine@32: amine@32: amine@32: Limit the length of detected tokens amine@32: ################################### amine@32: amine@32: amine@32: Imagine that you just want to detect and recognize a small part of a long amine@32: acoustic event (e.g. engine noise, water flow, etc.) and avoid that that amine@32: event hogs the tokenizer and prevent it from feeding the event to the next amine@32: processing step (i.e. a sound recognizer). You can do this by: amine@32: amine@32: - limiting the length of a detected token. amine@32: amine@32: and amine@32: amine@32: - using a callback function as an argument to :class:`auditok.core.StreamTokenizer.tokenize` amine@32: so that the tokenizer delivers a token as soon as it is detected. amine@32: amine@32: The following code limits the length of a token to 5: amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import StreamTokenizer, StringDataSource, DataValidator amine@32: amine@32: class UpperCaseChecker(DataValidator): amine@32: def is_valid(self, frame): amine@32: return frame.isupper() amine@32: amine@32: dsource = StringDataSource("aaaABCDEFGHIJKbbb") amine@32: tokenizer = StreamTokenizer(validator=UpperCaseChecker(), amine@32: min_length=1, max_length=5, max_continuous_silence=0) amine@32: amine@32: def print_token(data, start, end): amine@32: print("token = '{0}', starts at {1}, ends at {2}".format(''.join(data), start, end)) amine@32: amine@32: tokenizer.tokenize(dsource, callback=print_token) amine@32: amine@32: amine@32: output: amine@32: amine@32: .. code:: python amine@32: amine@32: "token = 'ABCDE', starts at 3, ends at 7" amine@32: "token = 'FGHIJ', starts at 8, ends at 12" amine@32: "token = 'K', starts at 13, ends at 13" amine@32: amine@32: amine@32: ************************ amine@32: `auditok` and Audio Data amine@32: ************************ amine@32: amine@35: In the rest of this document we will use :class:`auditok.util.ADSFactory`, :class:`auditok.util.AudioEnergyValidator` amine@35: and :class:`auditok.core.StreamTokenizer` for Audio Activity Detection demos using audio data. Before we get any amine@32: further it is worth, explaining a certain number of points. amine@32: amine@35: :func:`auditok.util.ADSFactory.ads` method is used to create an :class:`auditok.util.ADSFactory.AudioDataSource` amine@35: object either from a wave file, the built-in microphone or a user-supplied data buffer. Refer to the API reference amine@35: for more information and examples on :func:`ADSFactory.ads` and :class:`AudioDataSource`. amine@35: amine@35: The created :class:`AudioDataSource` object is then passed to :func:`StreamTokenizer.tokenize` for tokenization. amine@35: amine@35: :func:`auditok.util.ADSFactory.ads` accepts a number of keyword arguments, of which none is mandatory. amine@35: The returned :class:`AudioDataSource` object's features and behavior can however greatly differ amine@35: depending on the passed arguments. Further details can be found in the respective method documentation. amine@35: amine@35: Note however the following two calls that will create an :class:`AudioDataSource` amine@35: that reads data from an audio file and from the built-in microphone respectively. amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import ADSFactory amine@32: amine@32: # Get an AudioDataSource from a file amine@35: # use 'filename', alias 'fn' keyword argument amine@32: file_ads = ADSFactory.ads(filename = "path/to/file/") amine@32: amine@32: # Get an AudioDataSource from the built-in microphone amine@32: # The returned object has the default values for sampling amine@32: # rate, sample width an number of channels. see method's amine@32: # documentation for customized values amine@32: mic_ads = ADSFactory.ads() amine@32: amine@35: For :class:`StreamTkenizer`, parameters `min_length`, `max_length` and `max_continuous_silence` amine@35: are expressed in terms of number of frames. Each call to :func:`AudioDataSource.read` returns amine@35: one frame of data or None. amine@32: amine@35: If you want a `max_length` of 2 seconds for your detected sound events and your *analysis window* amine@35: is *10 ms* long, you have to specify a `max_length` of 200 (`int(2. / (10. / 1000)) == 200`). amine@35: For a `max_continuous_silence` of *300 ms* for instance, the value to pass to StreamTokenizer is 30 amine@35: (`int(0.3 / (10. / 1000)) == 30`). amine@32: amine@35: Each time :class:`StreamTkenizer` calls the :func:`read` (has no argument) method of an amine@35: :class:`AudioDataSource` object, it returns the same amount of data, except if there are no more amine@35: data (returns what's left in stream or None). amine@32: amine@35: This fixed-length amount of data is referred here to as **analysis window** and is a parameter of amine@35: :func:`ADSFactory.ads` method. By default :func:`ADSFactory.ads` uses an analysis window of 10 ms. amine@32: amine@35: The number of samples that 10 ms of audio data contain will vary, depending on the sampling amine@35: rate of your audio source/data (file, microphone, etc.). amine@32: For a sampling rate of 16KHz (16000 samples per second), we have 160 samples for 10 ms. amine@35: amine@35: You can use the `block_size` keyword (alias `bs`) to define your analysis window: amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import ADSFactory amine@32: amine@35: ''' amine@35: Assume you have an audio file with a sampling rate of 16000 amine@35: ''' amine@35: amine@35: # file_ads.read() will return blocks of 160 sample amine@32: file_ads = ADSFactory.ads(filename = "path/to/file/", block_size = 160) amine@32: amine@35: # file_ads.read() will return blocks of 320 sample amine@35: file_ads = ADSFactory.ads(filename = "path/to/file/", bs = 320) amine@32: amine@35: amine@35: Fortunately, you can specify the size of your analysis window in seconds, thanks to keyword `block_dur` amine@35: (alias `bd`): amine@32: amine@32: .. code:: python amine@32: amine@35: from auditok import ADSFactory amine@35: # use an analysis window of 20 ms amine@35: file_ads = ADSFactory.ads(filename = "path/to/file/", bd = 0.02) amine@35: amine@35: For :class:`StreamTkenizer`, each :func:`read` call that does not return `None` is treated as a processing amine@35: frame. :class:`StreamTkenizer` has no way to figure out the temporal length of that frame (why sould it?). So to amine@35: correctly initialize your :class:`StreamTokenizer`, based on your analysis window duration, use something like: amine@35: amine@35: amine@35: .. code:: python amine@35: amine@35: analysis_win_seconds = 0.01 # 10 ms amine@35: my_ads = ADSFactory.ads(block_dur = analysis_win_seconds) amine@32: analysis_window_ms = analysis_win_seconds * 1000 amine@32: amine@35: # If you want your maximum continuous silence to be 300 ms use: amine@32: max_continuous_silence = int(300. / analysis_window_ms) amine@32: amine@35: # which is the same as: amine@32: max_continuous_silence = int(0.3 / (analysis_window_ms / 1000)) amine@32: amine@35: # or simply: amine@35: max_continuous_silence = 30 amine@35: amine@32: amine@32: ****************************** amine@32: Examples using real audio data amine@32: ****************************** amine@32: amine@32: amine@32: Extract isolated phrases from an utterance amine@32: ########################################## amine@32: amine@32: We will build an :class:`auditok.util.ADSFactory.AudioDataSource` using a wave file from amine@32: the database. The file contains of isolated pronunciation of digits from 1 to 1 amine@32: in Arabic as well as breath-in/out between 2 and 3. The code will play the amine@32: original file then the detected sounds separately. Note that we use an amine@32: `energy_threshold` of 65, this parameter should be carefully chosen. It depends amine@32: on microphone quality, background noise and the amplitude of events you want to amine@32: detect. amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import ADSFactory, AudioEnergyValidator, StreamTokenizer, player_for, dataset amine@32: amine@32: # We set the `record` argument to True so that we can rewind the source amine@32: asource = ADSFactory.ads(filename=dataset.one_to_six_arabic_16000_mono_bc_noise, record=True) amine@32: amine@32: validator = AudioEnergyValidator(sample_width=asource.get_sample_width(), energy_threshold=65) amine@32: hoelzl@61: # Default analysis window is 10 ms (float(asource.get_block_size()) / asource.get_sampling_rate()) amine@32: # min_length=20 : minimum length of a valid audio activity is 20 * 10 == 200 ms amine@32: # max_length=4000 : maximum length of a valid audio activity is 400 * 10 == 4000 ms == 4 seconds amine@32: # max_continuous_silence=30 : maximum length of a tolerated silence within a valid audio activity is 30 * 30 == 300 ms amine@32: tokenizer = StreamTokenizer(validator=validator, min_length=20, max_length=400, max_continuous_silence=30) amine@32: amine@32: asource.open() amine@32: tokens = tokenizer.tokenize(asource) amine@32: amine@32: # Play detected regions back amine@32: amine@32: player = player_for(asource) amine@32: amine@32: # Rewind and read the whole signal amine@32: asource.rewind() amine@32: original_signal = [] amine@32: amine@32: while True: amine@32: w = asource.read() amine@32: if w is None: amine@32: break amine@32: original_signal.append(w) amine@32: amine@32: original_signal = ''.join(original_signal) amine@32: amine@32: print("Playing the original file...") amine@32: player.play(original_signal) amine@32: amine@32: print("playing detected regions...") amine@32: for t in tokens: amine@32: print("Token starts at {0} and ends at {1}".format(t[1], t[2])) amine@32: data = ''.join(t[0]) amine@32: player.play(data) amine@32: amine@32: assert len(tokens) == 8 amine@32: amine@32: amine@32: The tokenizer extracts 8 audio regions from the signal, including all isolated digits amine@32: (from 1 to 6) as well as the 2-phase respiration of the subject. You might have noticed amine@32: that, in the original file, the last three digit are closer to each other than the amine@32: previous ones. If you wan them to be extracted as one single phrase, you can do so amine@32: by tolerating a larger continuous silence within a detection: amine@32: amine@32: .. code:: python amine@32: amine@32: tokenizer.max_continuous_silence = 50 amine@32: asource.rewind() amine@32: tokens = tokenizer.tokenize(asource) amine@32: amine@32: for t in tokens: amine@32: print("Token starts at {0} and ends at {1}".format(t[1], t[2])) amine@32: data = ''.join(t[0]) amine@32: player.play(data) amine@32: amine@32: assert len(tokens) == 6 amine@32: amine@32: amine@32: Trim leading and trailing silence amine@32: ################################# amine@32: amine@32: The tokenizer in the following example is set up to remove the silence amine@32: that precedes the first acoustic activity or follows the last activity amine@32: in a record. It preserves whatever it founds between the two activities. amine@32: In other words, it removes the leading and trailing silence. amine@32: amine@32: Sampling rate is 44100 sample per second, we'll use an analysis window of 100 ms amine@32: (i.e. block_size == 4410) amine@32: amine@32: Energy threshold is 50. amine@32: amine@32: The tokenizer will start accumulating windows up from the moment it encounters amine@32: the first analysis window of an energy >= 50. ALL the following windows will be amine@32: kept regardless of their energy. At the end of the analysis, it will drop trailing amine@32: windows with an energy below 50. amine@32: amine@32: This is an interesting example because the audio file we're analyzing contains a very amine@32: brief noise that occurs within the leading silence. We certainly do want our tokenizer amine@32: to stop at this point and considers whatever it comes after as a useful signal. amine@32: To force the tokenizer to ignore that brief event we use two other parameters `init_min` amine@48: and `init_max_silence`. By `init_min` = 3 and `init_max_silence` = 1 we tell the tokenizer amine@32: that a valid event must start with at least 3 noisy windows, between which there amine@32: is at most 1 silent window. amine@32: amine@32: Still with this configuration we can get the tokenizer detect that noise as a valid event amine@32: (if it actually contains 3 consecutive noisy frames). To circumvent this we use an enough amine@32: large analysis window (here of 100 ms) to ensure that the brief noise be surrounded by a much amine@32: longer silence and hence the energy of the overall analysis window will be below 50. amine@32: amine@35: When using a shorter analysis window (of 10 ms for instance, block_size == 441), the brief amine@32: noise contributes more to energy calculation which yields an energy of over 50 for the window. amine@32: Again we can deal with this situation by using a higher energy threshold (55 for example). amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import ADSFactory, AudioEnergyValidator, StreamTokenizer, player_for, dataset amine@32: amine@32: # record = True so that we'll be able to rewind the source. amine@32: asource = ADSFactory.ads(filename=dataset.was_der_mensch_saet_mono_44100_lead_trail_silence, amine@32: record=True, block_size=4410) amine@32: asource.open() amine@32: amine@32: original_signal = [] amine@32: # Read the whole signal amine@32: while True: amine@32: w = asource.read() amine@32: if w is None: amine@32: break amine@32: original_signal.append(w) amine@32: amine@32: original_signal = ''.join(original_signal) amine@32: amine@32: # rewind source amine@32: asource.rewind() amine@32: amine@32: # Create a validator with an energy threshold of 50 amine@32: validator = AudioEnergyValidator(sample_width=asource.get_sample_width(), energy_threshold=50) amine@32: amine@32: # Create a tokenizer with an unlimited token length and continuous silence within a token amine@32: # Note the DROP_TRAILING_SILENCE mode that will ensure removing trailing silence amine@32: trimmer = StreamTokenizer(validator, min_length = 20, max_length=99999999, init_min=3, init_max_silence=1, max_continuous_silence=9999999, mode=StreamTokenizer.DROP_TRAILING_SILENCE) amine@32: amine@32: tokens = trimmer.tokenize(asource) amine@32: amine@32: # Make sure we only have one token amine@32: assert len(tokens) == 1, "Should have detected one single token" amine@32: amine@32: trimmed_signal = ''.join(tokens[0][0]) amine@32: amine@32: player = player_for(asource) amine@32: amine@32: print("Playing original signal (with leading and trailing silence)...") amine@32: player.play(original_signal) amine@32: print("Playing trimmed signal...") amine@32: player.play(trimmed_signal) amine@32: amine@32: amine@32: Online audio signal processing amine@32: ############################## amine@32: amine@32: In the next example, audio data is directly acquired from the built-in microphone. amine@32: The :func:`auditok.core.StreamTokenizer.tokenize` method is passed a callback function amine@32: so that audio activities are delivered as soon as they are detected. Each detected amine@32: activity is played back using the build-in audio output device. amine@32: amine@32: As mentioned before , Signal energy is strongly related to many factors such amine@32: microphone sensitivity, background noise (including noise inherent to the hardware), amine@32: distance and your operating system sound settings. Try a lower `energy_threshold` amine@32: if your noise does not seem to be detected and a higher threshold if you notice amine@32: an over detection (echo method prints a detection where you have made no noise). amine@32: amine@32: .. code:: python amine@32: amine@32: from auditok import ADSFactory, AudioEnergyValidator, StreamTokenizer, player_for amine@32: amine@32: # record = True so that we'll be able to rewind the source. amine@32: # max_time = 10: read 10 seconds from the microphone amine@32: asource = ADSFactory.ads(record=True, max_time=10) amine@32: amine@32: validator = AudioEnergyValidator(sample_width=asource.get_sample_width(), energy_threshold=50) amine@32: tokenizer = StreamTokenizer(validator=validator, min_length=20, max_length=250, max_continuous_silence=30) amine@32: amine@32: player = player_for(asource) amine@32: amine@32: def echo(data, start, end): amine@32: print("Acoustic activity at: {0}--{1}".format(start, end)) amine@32: player.play(''.join(data)) amine@32: amine@32: asource.open() amine@32: amine@32: tokenizer.tokenize(asource, callback=echo) amine@32: amine@32: If you want to re-run the tokenizer after changing of one or many parameters, use the following code: amine@32: amine@32: .. code:: python amine@32: amine@32: asource.rewind() amine@32: # change energy threshold for example amine@32: tokenizer.validator.set_energy_threshold(55) amine@32: tokenizer.tokenize(asource, callback=echo) amine@32: amine@32: In case you want to play the whole recorded signal back use: amine@32: amine@32: .. code:: python amine@32: amine@32: player.play(asource.get_audio_source().get_data_buffer()) amine@32: amine@32: amine@32: ************ amine@32: Contributing amine@32: ************ amine@32: amine@32: **auditok** is on `GitHub `_. You're welcome to fork it and contribute. amine@32: amine@32: amine@32: Amine SEHILI amine@32: September 2015 amine@32: amine@32: ******* amine@32: License amine@32: ******* amine@32: amine@32: This package is published under GNU GPL Version 3.