Projects
JSAP
JSAP units are HTML5 audio processors for Web Audio API enabled sites. These allow the designing and deployment of advanced audio units with similar functionality to desktop VST and AU processors.
Read more ->Open Multitrack Testbed
The Open Multitrack Testbed is an online repository of multitrack audio, mixes or processed versions thereof, and corresponding mix settings or process parameters such as DAW files.
Read more ->SAFE Plugins
The SAFE Project is motivated by the lack of statistically-defined transferable semantic terms in music production and the requirement for more intuitive control of low-level parameters.
Read more ->Web Audio Evaluation Tool
Perceptual listening tests are commonplace in audio research and a vital form of evaluation.
Read more ->Data Sets
SAFE-DB
A Dataset of Music Production Metadata
The Semantic Audio Feature Extraction Dataset (SAFE-DB) is a continually updating database of semantically annotated music production metadata, taken from an international user group of sound engineers.
Read more ->SAFE-RDF
A triplestore representation of the SAFE dataset
To make the SAFE dataset of music production metadata confluent with semantic web technologies, we have produced a dataset of RDF triples representing the SAFE database.
Read more ->SAFE-API
A Web API for access to the SAFE dataset
For software integration, we have released an open source web API to access the SAFE dataset.
Read more ->Get in touch with us
Email: info@semanticaudio.co.uk
JSAP
JavaScript Audio Plugins
JSAP units are HTML5 audio processors for Web Audio API enabled sites. These allow the designing and deployment of advanced audio units with similar functionality to desktop VST and AU processors. Processors can all of the browser defined AudioUnits to create their DSP, including the script processors, allowing for any current DSP process to be converted to JavaScript compatible systems.
The project defines both the host and process frameworks to ease deployment. On the host side, we define the PluginFactory to hold the JSAP prototypes. The Factory can generate the plugins and manage them from one central resource. The host also defines SubFactory units which define a chain of plugins in the traditional form. These SubFactory units handle:
- Plugin Order
- Plugin construction and destruction in the chain
- Track / Channel information (track name, instrument etc.)
The standard also specifies a new method for cross-adaptive processing by sharing features rather than routing audio. If several plugins require the same feature from the same audio stream, traditional systems would require the audio to be routed between all the nodes and then process the audio multiple times to generate the same result. The JSAP system attaches a JS-Xtract node to each plugin output allowing for advanced querying of the features. These features are then sent to the requesting plugins, saving on processing power as well as improved memory management by sharing objects
Documentation
The documentation is currently under development, as the standard itself is being processed. Documentation can be found here
Get the code!
The code can be accessed on GitHub
Papers
This work was presented at the AES 141st Convention in Los Angeles, USA and at the 2nd Workshop on Intelligent Music Production. Please cite this paper in any academic works: Jillings et al. “JSAP: A Plugin Standard for the Web Audio API with Intelligent Functionality”. Audio Engineering Society Convention 141, Los Angeles, CA. September, 2016. Available at http://www.aes.org/e-lib/browse.cfm?elib=18397. bibtex
Related Projects
Open Multitrack Testbed
The Open Multitrack Testbed is an online repository of multitrack audio, mixes or processed versions thereof, and corresponding mix settings or process parameters such as DAW files.
Multitrack audio is a much sought after resource for audio researchers, students, and content producers, and while some online resources exist, few are large and reusable and none allow querying audio fulfilling specific criteria.
This Testbed, however, present contains a semantic database of metadata corresponding with the songs and individual tracks, enabling users to retrieve “all recordings of an accordion from Canada”, or “all songs with at least 10 different mixes”.
Many entries have a Creative Commons license, allowing us to host the material and allowing you to use, edit and/or redistribute it for your own purposes. Some audio is hosted on other websites, but can be found using our multitrack browser or search engine because of the rich metadata we attach to it.
Publication
When using the Open Multitrack Testbed for academic purposes, please cite Brecht De Man, Mariano Mora-Mcginity, György Fazekas and Joshua D. Reiss, “The Open Multitrack Testbed,” 137th Convention of the Audio Engineering Society, October 2014. [pdf | BibTeX | poster | website]
Related Projects
Safe plugins
The SAFE Project is motivated by the lack of statistically-defined transferable semantic terms in music production and the requirement for more intuitive control of low-level parameters. One of the main outcomes of the project is a suite of DAW plug-ins that allow you to both save and load semantic terms from within the audio production workflow. This means you can now control your audio effect plug-ins using terms that normal human beings can understand! Just type a term (such as warm, bright, etc…) into the text-box, and if we have it on our server, your parameters will be moved around to represent this. Similarly, if you achieve an effect that can be explained in semantic terms, input it into the box and hit save. As the plug-ins are based on the music production community, the more users that input terms, the more representative the effects become for everyone else!
The song used in the video is The Alchemist by Little Tybee.
All the stems and other things can be found here:
Most Recent Version [v1.31]
- Mac OS X (10.6 and onwards) [32 & 64-Bit]
- Windows VST [32-Bit, 64-Bit]
- Linux VST [32-bit, 64-Bit]
Referencing
If you would like to cite the systems or data used in the EPSRC-funded Semantic Audio project, part of the Semantic Media Network, please use the following citation [pdf | bib]:
R. Stables, S. Enderby, B. De Man, G. Fazekas, and J. D. Reiss, “SAFE: A system for the extraction and retrieval of semantic audio descriptors,”, The International Society for Music Information Retrieval (ISMIR), 2014.
Source Code
You can access the source code to the plug-ins: here. Alternatively, you can clone the repo using the following link:
git clone https://github.com/semanticaudio/SAFE.git
Manual
You can get hold of the manual in PDF form here, or read it on this website.
The plug-ins will run on Windows and Mac across a large number of DAWs, providing they support either VST or Audio Units.
Related Projects
JS-XTRACT
JS-Xtract: A Javascript Library for Audio Feature Extraction
Due to the increasing capability of web platforms and the introduction of the Web Audio API, high resolution audio processing can now be implemented in the web browser, with processing loads managed client-side. This allows for the development of tools such as listening test environments and online audio processing tools, which can be developed natively using JavaScript and HTML 5. These tools often require audio feature extraction in order to perform analysis on time-series data, however incorporating libraries that are developed for compiled languages such as C, or Java can be difficult to implement due to… In this paper we present JS-Xtract, a light-weight JavaScript library for feature extraction, which is agnostic of the data’s type or origin. This allows users to apply feature extraction in realtime by incorporating it into a callback function, or to apply it to offline data.
Realtime example
To demonstrate this, the following demonstration uses your microphone and processes the data in real time on your device. No data is transmitted or captured in any way as all the processing is completely done in your browser. We give you an option of several functions to calculate with already and the results are shown in the graph.
Documentation
Full documentation on the library is available here. This covers the features available, using the object-oriented or procedural functions and guides on using it with a Web Audio environment.
Download
Source code for the library can be found on Github: https://github.com/nickjillings/js-xtract
Publication
N. Jillings, J. Bullock, R. Stables “JS-Xtract: A Realtime Audio Feature Extraction Library for the Web”, International Society for Music Information Retrieval Conference (ISMIR 2016), August 2016. [PDF] [bibtex]
Related Projects
Web Audio Evaluation Tool
Perceptual listening tests are commonplace in audio research and a vital form of evaluation. Many tools exist to run such tests, however many operate one test type and are therefore limited whilst most require proprietary software. Using Web Audio, the Web Audio Evaluation Tool (WAET) addresses these concerns by having one toolbox which can be configured to run many different tests, perform it through a web browser and without needing proprietary software or computer programming knowledge. In this paper the role of the Web Audio API in giving WAET key functionalities are shown. The paper also highlights less common features, available to web based tools, such as easy remote testing environment and in-browser analytics.
MUSHRA test on the WAET platform
APE on the WAET platform
Interfaces
The Web Audio Evaluation Toolbox supports building tests complying to the following known standards:
- Audio Perceptual Evaluation – APE [ref]
- ITU-R BS.1534-1 (MUSHRA) [ref]
- ITU-R BS.1116-1 (ABC / Hidden Reference) [ref]
- ITU-R BS. 562-3 (Continuous Impairment Scale) [ref]
- ITU-T P. 800 [ref]
- Absolute Category Rating (ACR)
- Comparison Category Rating (CCR)
- Degredation Category Rating (DCR)
- A/B Testing and Pairwise [David, Herbert Aron. The method of paired comparisons. Vol. 12. London, 1963.]
- -50 to 50 / Bipolar
- ABX [ref]
- Likert Scale [ref]
- Rank Scale [ref]
- 9 Point Hedonic Scale [ref]
Download
Source code and examples are available at: https://github.com/BrechtDeMan/WebAudioEvaluationTool
Publications
Jillings, N., Moffat, D., De Man, B., Reiss, JD., and Stables, R. “Web Audio Evaluation Tool: A framework for subjective assessment of audio”. In Proceedings of the 2nd Web Audio Conference (WAC-2016), Atlanta, 2016. [PDF]
Jillings, N., De Man, B., Moffat, D., & Reiss, JD. “Web Audio Evaluation Tool: A Browser-Based Listening Test Environment”, in Proceedings of the 12th Sound and Music Conference, Maynooth, 2015. [PDF]
Related Projects
SAFE-DB
A Dataset of Music Production Metadata
The Semantic Audio Feature Extraction Dataset (SAFE-DB) is a continually updating database of semantically annotated music production metadata, taken from an international user group of sound engineers. The data is taken from 4 audio effects: a dynamic range compressor, an overdrive distortion, a parametric equaliser and an algorithmic reverb. Each entry into the dataset contains the following items:
Descriptor
A string representing a user’s description of the audio transformation (e.g. warm, bright, fluffly).
Audio Features
An array of audio features extracted from the audio file, taken before and after processing is applied to the signal. Feature list: Mean, Variance, StdDev, RMS_Amplitude, Zero_Crossing_Rate, Spectral_Centroid, Spectral_Variance, Spectral_Standard_Deviation, Spectral_Skewness, Spectral_Kurtosis, Irregularity_J, Irregularity_K, F0 (autocorrelation), Smoothness, Spectral_Roll_Off, Spectral_Flatness, Spectral_Crest, Spectral_Slope, Peak_Spectral_Centroid, Peak_Spectral_Variance, Peak_Spectral_Standard_Deviation, Peak_Spectral_Skewness, Peak_Spectral_Kurtosis, Peak_Irregularity_J, Peak_Irregularity_K, Peak_Tristimulus (x3), Inharmonicity, Harmonic_Spectral_Centroid, Harmonic_Spectral_Variance, Harmonic_Spectral_Standard_Deviation, Harmonic_Spectral_Skewness, Harmonic_Spectral_Kurtosis, Harmonic_Irregularity_J, Harmonic_Irregularity_K, Harmonic_Tristimulus (x3) Noisiness, Parity_Ratio, Bark_Coefficients (x25), MFCCs (x13)
Audio Effect Parameters
A variable sized array of plug-in parameters used to apply processing to the signal.
User Data
A series of strings representing the user’s age, location, language and experience, along with the genre of the music and the instrument being processed. These fields can be left blank.
Note: To limit the file-size of the download, feature-sets are averaged. For data from a specific date, or for multi-channel time series features please get in touch
Download
- Dynamic Range Compressor – User Data | Feature Data
- Overdrive Distortion - User Data | Feature Data
- Parametric Equaliser – User Data | Feature Data
- Algorithmic Reverb - User Data | Feature Data
- Download All(.zip)
Last Updated: Monday 11th July 2016
Publication
R. Stables, S. Enderby, B. De Man, G. Fazekas, and J. D. Reiss, “SAFE: A system for the extraction and retrieval of semantic audio descriptors”, The International Society for Music Information Retrieval (ISMIR), 2014. [pdf | bib]
Data Visualisation
Using multidimensional scaling, we can map the descriptive terms from the dataset into a more intuitive space. The interactive visualisation below shows the feature space of the SAFE-DB in 3-dimensions.
Related Data Sets
SAFE-RDF
A triplestore representation of the SAFE dataset
SPARQL Querying Area
To make the SAFE dataset of music production metadata confluent with semantic web technologies, we have produced a dataset of RDF triples representing the SAFE database. This can be access here through an endpoint, or via a semantic web API.
API
Coming soon…
Publication
Coming soon…
Related Data Sets
SAFE-API
A Web API for access to the SAFE dataset
Web API
For software integration, we have released an open source web API to access the SAFE dataset. Here, you can learn how to use the API and download the framework that was developed to host and distribute the dataset…
Source
You can access the web API developed around the SAFE data using the following link: https://github.com/semanticaudio/safe-api
Examples
An example of it being used to retrieve feature data can be found here. Please get in contact if you would like to use the API, or the SAFE data in your own project.