English's Github Trend's slide
Trending Slides
Japanese Github Trend's slide
Trending Slides
@isletennos
AIã䜿ã£ããªã¢ã«ã¿ã€ã ãã€ã¹ãã§ã³ãžã£ãŒã®ã¢ãã«åŠç¿çšããŒã«
Description
AIã䜿ã£ããªã¢ã«ã¿ã€ã ãã€ã¹ãã§ã³ãžã£ãŒãMMVC(RealTime-Many to Many Voice Conversion)ã ã§äœ¿çšããã¢ãã«ãåŠç¿ããããã®ãªããžããªã§ãã
Google Colaboratory (Google Colab) ãçšããããšã§ãå人ã®ç°å¢ã«äŸåããããã€ç°¡åã«æ©æ¢°åŠç¿ã®åŠç¿ãã§ãŒãºãå®è¡å¯èœã§ãã
Concept
ãç°¡åããã ãã§ããã奜ããªå£°ã«ãããªã¢ã«ã¿ã€ã ã§ã
Demo
å¶äœäž (v1.3.0.0)
https://www.nicovideo.jp/watch/sm40386035 (v1.2.0.0)
MMVCã®å©çšèŠçŽ åã³ MMVCçšé³æºã®é åžå (2022/08/10)
readme
MMVCã®å©çšèŠçŽ
MMVC(ä»¥äžæ¬ãœãããŠã§ã¢)ã®å©çšèŠçŽã¯ãåºæ¬çã«MITã©ã€ã»ã³ã¹ã«æºæ ããŸãã
æ¬ãœãããŠã§ã¢ã¯ãã³ããŒå©çšãé åžã倿Žã®è¿œå ã倿Žãå ãããã®ã®åé åžãåçšå©çšãææè²©å£²ãªã©ãã©ãªãã§ãèªç±ã«ã䜿ãããã ãããšãã§ããŸãã
ã©ã€ã»ã³ã¹ã®èšèŒãå¯èœãªãã©ãããã©ãŒã ã§äœ¿çšãããå Žåãäžèšã«ç€ºãã©ã€ã»ã³ã¹ãã¿ãŒã³ã®ã©ã¡ããã®èšèŒããé¡ãããããŸãã
VRChatã§ã®äœ¿çšãªã©ãã©ã€ã»ã³ã¹èšèŒãå°é£ãªå ŽåãèšèŒã¯äžèŠã§ãã
æ¬ãœãããŠã§ã¢ã«ã€ããŠãéçºè ã¯ãããªãä¿èšŒãããããŸããã
ãŸããæ¬ãœãããŠã§ã¢ãå©çšããããšã«ãã£ãŠçããåé¡ã«ã€ããŠãéçºè ã¯äžåã®è²¬ä»»ãè² ããããŸãã
åŠç¿å ãšããŠçšããé³å£°ããŒã¿ã«ã€ããŠã¯ãå¿ ã䜿çšåã«ããŒã¿ã®æš©å©è ãã蚱諟ãåŸãŠãã ããã
ãŸããé³å£°ããŒã¿ã®é åžå ã®å©çšèŠçŽå ã§å©çšããŠãã ããã
MMVCå ¬åŒé åžã®é³å£°ããŒã¿ã®å©çšèŠçŽãšããŠã³ããŒãå ã«ã€ããŠ
æ¬ãœãããŠã§ã¢ã®å©çšèŠçŽãšã¯å¥ã«ãäžèšé³å£°ããŒã¿ãå©çšããå Žåãããããã®é³å£°ã©ã€ãã©ãªæäŸè æ§ã®å©çšèŠçŽã«åæããå¿ èŠããããŸãã
â»æ¬ãœãããŠã§ã¢ã§ã¯äžèšäŒæ¥æ§ã»å£äœæ§ã«ç¹å¥ã«èš±å¯ãé ããé³å£°ããŒã¿ãæ¬ãœãããŠã§ã¢çšã«æ¹å€ãåé åžãè¡ã£ãŠãããŸãã
SSS LLC.
[å©çšèŠçŽ][ããã ãã é³å£°ããŒã¿]ãâ»æ¬ãœãããŠã§ã¢ã«å梱ããŠãããã®ãšåæ§ã®é³å£°ããŒã¿ã«ãªããŸã
[å©çšèŠçŽ][ä¹å·ãã é³å£°ããŒã¿]
æ¥æ¥éšã€ãããããžã§ã¯ãæ§
ã©ã€ã»ã³ã¹è¡šèšã«ã€ããŠ
ããã ãã/ååœããã/ä¹å·ãã/æ¥æ¥éšã€ãã
ã®3ãã£ã©ã¯ã¿ãŒãå©çšããå Žåã«éããäžèšã©ã€ã»ã³ã¹ãã¿ãŒã³ã«å ããŠãã©ã®ããŒã«ã§äœãããé³å£°ããããããã«
çã®èšèŒãäžèšã©ã€ã»ã³ã¹ãã¿ãŒã³ãšäžç·ã«ãèšèŒãã ããã
ãã¡ãã«ã€ããŸããŠããVRChatã§ã®äœ¿çšãªã©ãã©ã€ã»ã³ã¹èšèŒãå°é£ãªå ŽåãèšèŒã¯äžèŠã§ãã
ã©ã€ã»ã³ã¹ãã¿ãŒã³ 1
ã©ã€ã»ã³ã¹ãã¿ãŒã³ 2 (éçºè æšå¥š)
Requirement
ã»Google ã¢ã«ãŠã³ã
Install
äžèšãã¿ã³ããColabäžã§èªåã®Google Driveäžã«ã€ã³ã¹ããŒã«ããŠãã ããã
ãã®åŸãGoogle DriveãéããŠããã€ãã©ã€ã > MMVC_Trainer > notebookãã®äžã®åããŒããéããŠå®è¡ããŠãã ããã
Usage
ãã¥ãŒããªã¢ã« : ããã ããã«ãªã
æ¬ãã¥ãŒããªã¢ã«ã§ã¯ããã ãã(SSS LLC.)ã®é³å£°ããŒã¿ãå©çšããŸãã
ãã®ãããMMVCã®å©çšèŠçŽãšã¯å¥ã«[ããã ãã å©çšèŠçŽ]ãéµå®ããå¿ èŠããããŸãã
Ph1. èªåã®é³å£°ã®é²é³ãšé³å£°ããŒã¿ã®é 眮
èªåã®å£°ã®é³å£°ããŒã¿ãé²é³ããŸãã
notebookãã£ã¬ã¯ããªã«ããã00_Rec_Voice.ipynbãããèªåã®é³å£°ãé²é³ããŠãã ããã
ãããã¯ãèªåã®PCã§é²é³ããŠGoogle Driveäžã«é 眮ããŠãã ããã
JVSã³ãŒãã¹ãITAã³ãŒãã¹çãå°æ¬ã«ãã100æçšåºŠèªã¿äžããŸãã
ãŸããé²é³ããé³å£°ã¯24000Hz 16bit 1chã§ããå¿ èŠããããŸãã
â»MMVCçšã«ããã¹ããåå²ããITAã³ãŒãã¹ã§ãããå©çšãã ããã
https://drive.google.com/file/d/14oXoQqLxRkP8NJK8qMYGee1_q2uEED1z/view?usp=sharing
dataset/textful/000_myvoice ã«é³å£°ããŒã¿ãšããã¹ãããŒã¿ãé 眮ããŸãã
00_Rec_Voice.ipynbãå©çšããŠé²é³ããå Žåã¯ãã®ããã«é 眮ãããŠããŸãã®ã§ãã®ãŸãŸã§å€§äžå€«ã§ãã
æçµçã«äžèšã®ãããªãã£ã¬ã¯ããªæ§æã«ãªãããã«ãã¡ã€ã«ãé 眮ããŠãã ããã
Ph2. ã¢ãã«ã®åŠç¿æ¹æ³
äºååŠç¿æžã¿ããŒã¿ãé 眮ããŸãã
00_Clone_Repo.ipynbã䜿ã£ãŠã€ã³ã¹ããŒã«ããå Žåã¯æ¢ã«äºååŠç¿æžã¿ããŒã¿ãé 眮æžã¿ãªã®ã§ãã®ãŸãŸã§å€§äžå€«ã§ãã
00_Clone_Repo.ipynbãå©çšããªãã£ãå Žåããfine_modelããã£ã¬ã¯ããªã«äžèšãã¡ã€ã«ãååšããªãã£ãå Žåã以äžã®æé ã§ãã¡ã€ã«ãé 眮ããŠãã ããã
äžèšãªã³ã¯ããããG_180000.pthããD_180000.pthããããŠã³ããŒãã
https://drive.google.com/drive/folders/1vXdL1zSrgsuyACMkiTUtVbHgpMSA1Y5I?usp=sharing
ãG_180000.pthããD_180000.pthãããfine_modelããã£ã¬ã¯ããªã«é 眮ããŸãã(è¯ãå¿ãããã€ã³ããªã®ã§èŠæ³šæïŒ)
notebookãã£ã¬ã¯ããªã«ããã01_Create_Configfile.ipynbããGoogle Colab äžã§å®è¡ãåŠç¿ã«å¿ èŠãªconfigãã¡ã€ã«ãäœæããŸã
configsã«äœæãããtrain_config.jsonã®
"eval_interval"
modelãä¿åããééã§ãã
"batch_size"
colabã§å²ãåœãŠãGPUã«åãããŠèª¿æŽããŠãã ããã
äžèš2é ç®ãç°å¢ã«å¿ããŠæé©åããŠãã ãããããããªãæ¹ã¯ãã®ãŸãŸã§å€§äžå€«ã§ãã
notebookãã£ã¬ã¯ããªã«ããã02_Train_MMVC.ipynbããgoogle colab äžã§å®è¡ããŠãã ããã
logs/ã«ã¢ãã«ãçæãããŸãã
Ph3. åŠç¿ããã¢ãã«ã®æ§èœæ€èšŒ
notebookãã£ã¬ã¯ããªã«ããã03_MMVC_Interface.ipynbããgoogle colab äžã§å®è¡ããŠãã ããã
=== 奜ããªãã£ã©ã¯ã¿ãŒã®å£°ã«ãªã
==== Ph1. èªåã®é³å£°ã®é²é³ãšé³å£°ããŒã¿ã®é 眮 åã³ã¿ãŒã²ããé³å£°ããŒã¿ã®é 眮
èªåã®å£°ã®é³å£°ããŒã¿ãšãã®é³å£°ããŒã¿ã«å¯Ÿå¿ããããã¹ãã倿ããã声ã®é³å£°ããŒã¿ãšãã®é³å£°ããŒã¿ã«å¯Ÿå¿ããããã¹ããçšæããŸãã
ãã®æãçšæããé³å£°(èªåã®å£°ã®é³å£°ããŒã¿/倿ããã声ã®é³å£°ããŒã¿å ±ã«)ã¯24000Hz 16bit 1chãåŒ·ãæšå¥šããŠãããŸãã
äžèšã®ãããªãã£ã¬ã¯ããªæ§æã«ãªãããã«é³å£°ããŒã¿ãšããã¹ãããŒã¿ãé 眮ããŸãã
textfulã®çŽäžã«ã¯2ãã£ã¬ã¯ããªã«ãªããŸãã
(1205_zundamonãã£ã¬ã¯ããªã¯ç¡ããŠãåé¡ãããŸãã)
Ph2. ã¢ãã«ã®åŠç¿æ¹æ³
以éãããã¥ãŒããªã¢ã« : ããã ããã«ãªã Ph2.ããšåæ§ã®ãã岿
Ph3. åŠç¿ããã¢ãã«ã®æ§èœæ€èšŒ
以éãããã¥ãŒããªã¢ã« : ããã ããã«ãªã Ph3.ããšåæ§ã®ãã岿
MMVC_Client
ãããžã§ã¯ãã«ããMMVC Client
MMVCãå®éã«åããClient software
https://raw.githubusercontent.com/isletennos/MMVC_Client
æå¿ã«ããMMVC Client
(1) Voice Changer Trainer and Player
æ§ã ãªç°å¢ã§MMVCãåããããã«äœæãããClient softwareã
åäœç¢ºèªç¶æ³
=
os
middle
ãã¬ãŒãã³ã°ã¢ããª
ãã€ã¹ãã§ã³ãžã£ãŒ
1
Windows
Anaconda
æª
æª
2
Windows(WSL2)
Docker
wsl2+ubuntuã§ç¢ºèª
wsl2+ubuntuã§ç¢ºèª
3
Windows(WSL2)
Anaconda
æª
ubuntuã§ç¢ºèª
4
Mac(Intel)
Anaconda
æª
åäœãããæ¿éã(2019, corei5)
5
Mac(M1)
Anaconda
æª
M1 MBA, M1 MBPã§ç¢ºèª
6
Linux
Docker
debianã§ç¢ºèª
debianã§ç¢ºèª
7
Linux
Anaconda
æª
æª
8
Colab
Notebook
ç¢ºèªæžã¿
ç¢ºèªæžã¿
ããçšåºŠæè¿ã®ãã®ã§ããã°CPUã§ã®çšŒåãå¯èœã§ã(i7-9700Kã§å®çžŸãããäžèšãã¢åç §)ã
ãã¢åç» Dockerç°å¢(GPU) ãã¢, Dockerç°å¢(CPU)ãã¢, Colabç°å¢ãã¢
æå¿ã«ãããã¥ãŒããªã¢ã«åç»
v1.2.1.x
| åæºåç·š | ãã³ãã³åç» | YouTube | | :---------- | :-------------------------------------- | :---------------------------------------------- | | èŠä¿®æ£é³å£° | ãã³ãã³åç» | YouTube | | åæºåç·š2 | ãã³ãã³åç» | YouTube | | åŠç¿ç·š_å1 | ãã³ãã³åç» | YouTube | | åŠç¿ç·š_å2 | ãã³ãã³åç» | YouTube | | åŠç¿ç·š_åŸ | ãã³ãã³åç» | YouTube | | ãªã¢ã«ã¿ã€ã ç·š | ãã³ãã³åç» | YouTube | | 質åç·š | ãã³ãã³åç» | YouTube | | å¿çšç·š_ä¹å·ãã | ãã³ãã³åç» | YouTube | | å¿çšç·š_é³è¡ãŠã | ãã³ãã³åç» | YouTube |
Q&A
äžèšãµã€ãããåèãã ããã
äžæãªç¹ãããããŸãããã以äžã®discordãŸã§ãåãåãããã ããã
https://raw.githubusercontent.com/isletennos/MMVC_Trainer/wiki/FAQ
https://mmvc.readthedocs.io/ja/latest/index.html
MMVCã³ãã¥ããã£ãµãŒã(discord)
éçºã®ææ°æ å ±ããäžæç¹ã®ãååããMMVCã®æŽ»çšæ³ãªã©MMVCã«é¢ããã³ãã¥ããã£ãµãŒãã§ãã
https://discord.gg/2MGysH3QpD
Special thanks
JVS (Japanese versatile speech) corpus
contributors : é«é æ ä¹ä»æ§/äžäº å¥å€ªéæ§/éœè€ äœæš¹æ§/é¡å±± ç¥æš¹æ§/䞹治 å°åæ§/ç¿æž¡ æŽæ§
https://sites.google.com/site/shinnosuketakamichi/research-topics/jvs_corpus
ITAã³ãŒãã¹ ãã«ãã¢ãŒãã«ããŒã¿ããŒã¹
contributors : éäºé乿§/åèé壱æ§/éœè€å岿§/森å¢å°é æ§/å°å£çŽç¢æ§/èœå¢éæ§/å°Ÿäžçæåæ§/å°ç°æå€®æ§
CharacterVoice : æ±åã€ã¿ã³(æšæžè¡£å¹æ§)/ããã ãã(äŒè€ãããªæ§)/ååœããã(ç°äžå°éªæ§)/ä¹å·ãã(è¥¿ç°æèŠ)
https://zunko.jp/multimodal_dev/login.php
ã€ããã¿ã¡ããã³ãŒãã¹
contributor : 倢å黿§
CharacterVoice : ã€ããã¿ã¡ãã(倢å黿§)
https://tyc.rei-yumesaki.net/material/corpus/
Reference
https://arxiv.org/abs/2106.06103
https://raw.githubusercontent.com/jaywalnut310/vits
Author
Isle Tennos
Twitter : https://twitter.com/IsleTennos