Antoine Caillon
Research scientist - deep generative modeling applied to musical signals.
Currently
Research Scientist at Google DeepMind.
PhD Subject
Hierachical temporal learning for multi-instrument and orchestral audio synthesis
Research interests
Neural audio synthesis, real-time implementation of deep generative models, artistic use of artificial intelligence
Education
2020-2023
PhD: Hierachical temporal learning for multi-instrument and orchestral audio synthesis
- ircam (paris)
- Directed by Philippe Esling
- Obtained with High Honours
- PhD manuscript
- Defense
- Actor project
2018-2019
Master Degree 2 ATIAM
- ircam (paris)
- Obtained with High Honours
- Signal filtering, source separation, machine learning, Reactive programming, modelisation and synthesis of different instruments
2017-2018
Master Degree 1 Engineering Sciences
- Sorbonne Université (Paris IV)
- Signal processing, acoustic, mecanic, numeric methods (finite element method, finite difference method)
2015-2017
Mathematic degree
- Sorbonne Université (Paris IV)
2012-2014
Sound engineering degree
- ITEMM, Le Mans
Previous positions
2022
Google Brain, invited student researcher
- Neil Zeghidour, Jesse Engel
- 8-months project, multimodal modeling of raw audio signals
- Speech processing, music processing, large scale modeling
- Work on MusicLM and SingSong
2020-2023
IRCAM, PhD student
2019
IRCAM, Software engineer
- 2-months project, building the models behind Alexander Schubert’s convergence piece
2019
Technicolor, intern
- Alexey Ozerov, Ngoc Q. K. Duong
- 6-month internship on the use of deep learning models to age or de-age speech.
- Deep Learning, Audio processing
Publications
2022
SingSong: Generating musical accompaniments from singing
- Chris Donahue, Antoine Caillon, Adam Roberts et al.
- Preprint
2022
MusicLM: Generating Music From Text
- Preprint
2022
Streamable Neural Audio Synthesis With Non-Causal Convolutions
- Antoine Caillon, Philippe Esling
- DAFx 2020in22
2021
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
- Antoine Caillon, Philippe Esling
- Preprint
2020
Timbre latent space: exploration and creative aspects
- Antoine Caillon, Adrien Bitton, Brice Gatinet, Philippe Esling
- TIMBRE 2020
2020
Diet deep generative audio models with structured lottery
- Philippe Esling, Ninon Devis, Adrien Bitton, Antoine Caillon, Axel Chemla–Romeu-Santos, Constance Douwes
- DaFX 2020
2019
Assisted Sound Sample Generation with Musical Conditioning in Adversarial Auto-Encoders
- Adrien Bitton, Philippe Esling, Antoine Caillon, Martin Fouilleul
- DaFX 2019
Projects
2024
MusicFX DJ v2
- Work done as part of the Magenta team (Google DeepMind)
- Designed, built, trained and deployed the new version of the MusicFX DJ model
- Google DeepMind blogpost
- jax / c++ / proto
2022
nn~ : a Max/MSP external for real-time ai audio processing
- main researcher / developer Antoine Caillon
- open source code for the nn~ external
- torch / c++ / c
2021-2022
RAVE: Official implementation
- main researcher / developer Antoine Caillon
- open source code for the RAVE model
- torch
2020
ddsp_pytorch
- main researcher / developer Antoine Caillon
- Based on the work from magenta
- Real-time implementation of the DDSP model
- torch / c++ / c
Artistic collaborations
2024
Jacob Collier
- Artist consultancy on the development of the MusicFX DJ model
2021-2022
ANIMA TM
- Alexander Schubert, in collaboration with Antoine Caillon
- Performed at Centre Georges Pompidou (see description here)
2021
Improvisation, apprentissage profond et fusion d’espace latent
- Maxime Mantovani
- Latent exploration with custom controllers and prior matching techniques
2019-2020
Convergence
- Alexander Schubert, in collaboration with Antoine Caillon, Philippe Esling and Jorge Davila-Chacon
- Awarded with a Golden Nica (Ars Electronica 2020) in Digital arts (see here)
2019-2020
unknown title
- Brice Gatinet, in collaboration with Antoine Caillon
- Creation of wavae, initial work on real-time use of deep learning models
Master class
2022
Neural Audio Synthesis
- Université de Cergy
Neural Audio Synthesis
- Hochschule für Musik und Theater Hamburg
RAVE + nn~
Teaching
2020-now
Internship supervision
2020-2023
Machine learning project
- Master 2 ATIAM
- 3 months project
2020-now
React Native course
- L3 Sorbonne Université