Authors

Shoutong Wang (Zhejiang University)

Jinglin Liu (Zhejiang University)

Yi Ren (Zhejiang University)

Zhen Wang (State Key Laboratory of Media Convergence Production Technology and Systems)

Changliang Xu (State Key Laboratory of Media Convergence Production Technology and Systems)

Zhou Zhao (Zhejiang University)

Abstract

Multi-speaker singing voice synthesis is to generate the singing voice sung by different speakers. To generalize to new speakers, previous zero-shot singing adaptation methods obtain the timbre of the target speaker with a fixed-size embedding from single reference audio. However, they face several challenges: 1) the fixed-size speaker embedding is not powerful enough to capture full details of the target timbre; 2) single reference audio does not contain sufficient timbre information of the target speaker; 3) the pitch inconsistency between different speakers also leads to a degradation in the generated voice. In this paper, we propose a new model called MR-SVS to tackle these problems. Specifically, we employ both a multi-reference encoder and a fixed-size encoder to encode the timbre of the target speaker from multiple reference audios. The Multi-reference encoder can capture more details and variations of the target timbre. Besides, we propose a well-designed pitch shift method to address the pitch inconsistency problem. Experiments indicate that our method outperforms the baseline method both in naturalness and similarity.

Synthesis Results

Speaker 1 reference:

GT mel + Vocoder	MR-SVS	Baseline	Multi-ref-only	Single-head

Speaker 2 reference:

GT mel + Vocoder	MR-SVS	Baseline	Multi-ref-only	Single-head

Conversion w/o Pitch Shift

GT mel + Vocoder	Speaker 1	Speaker 2

Controlled Singing Voice Synthesis

Low Pitch	Normal	High Pitch