[ASR] Support Hubert, fintuned on the librispeech dataset #3088
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
|
This pull request is now in conflict :( |
zxcd
reviewed
Mar 27, 2023
| @@ -55,6 +58,8 @@ def __init__(self, config: dict): | |||
| reduction='mean') | |||
|
|
|||
| def forward(self, wav, wavs_lens_rate, target, target_lens): | |||
| # import pdb | |||
| # pdb.set_trace() | |||
zxcd
reviewed
Mar 27, 2023
examples/librispeech/asr3/run.sh
Outdated
| @@ -19,6 +20,7 @@ audio_file=data/demo_002_en.wav | |||
|
|
|||
| avg_ckpt=avg_${avg_num} | |||
| ckpt=$(basename ${conf_path} | awk -F'.' '{print $1}') | |||
| ckpt=test6 | |||
zxcd
reviewed
Mar 27, 2023
dataset/librispeech/librispeech.py
Outdated
| @@ -133,7 +133,7 @@ def create_manifest(data_dir, manifest_path): | |||
| def prepare_dataset(url, md5sum, target_dir, manifest_path): | |||
| """Download, unpack and create summmary manifest file. | |||
| """ | |||
| if not os.path.exists(os.path.join(target_dir, "LibriSpeech")): | |||
There was a problem hiding this comment.
原先的代码似乎和librispeech解压出的结果不太一致，本地已有librispeech数据集的情况下不太方便
zxcd
reviewed
Mar 27, 2023
|
|
||
|
|
||
| task_cfg: | ||
| sample_rate: 16000 |
zxcd
reviewed
Mar 27, 2023
| # Data Augmentation # | ||
| ############################################ | ||
| audio_augment: # for raw audio | ||
| sample_rate: 16000 |
zxcd
reviewed
Mar 27, 2023
| self.mask_emb = paddle.create_parameter( | ||
| shape=[cfg.encoder_embed_dim], | ||
| dtype='float32', | ||
| default_initializer=paddle.nn.initializer.Uniform(), |
There was a problem hiding this comment.
torch和paddle对于uniform的初始化范围不一致，torch为(0,1)，paddle为(-1,1)，可以确定下是否会对训练产生影响，或者直接加上low和high参数
zxcd
reviewed
Mar 27, 2023
| self.label_embs_concat = paddle.create_parameter( | ||
| shape=[sum(self.num_classes), final_dim], | ||
| dtype='float32', | ||
| default_initializer=paddle.nn.initializer.Uniform(), |
zxcd
reviewed
Mar 27, 2023
|
|
||
| return x, mask_indices | ||
|
|
||
| def compute_nce(x, pos, negs): |
zxcd
reviewed
Mar 27, 2023
| from dataclasses import dataclass, field, is_dataclass | ||
| from copy import deepcopy | ||
|
|
||
| from omegaconf import II, MISSING, open_dict |
zxcd
reviewed
Mar 27, 2023
|
|
||
|
|
||
| class HubertBase(nn.Layer): | ||
| """Wav2vec2 model""" |
zxcd
reviewed
Mar 30, 2023
| enc_n_units: 1024 | ||
| blank_id: 0 | ||
| dropout_rate: 0.0 | ||
| hubert_params_path: "exp/hubert/pd_hubert.pdparams" |
zh794390558
reviewed
Apr 11, 2023
| fp16: True | ||
| label_rate: 50 | ||
| extractor_mode: layer_norm | ||
| encoder_layers: 24 |
examples/librispeech/asr3/path.sh
Outdated
| @@ -10,6 +10,5 @@ export PYTHONPATH=${MAIN_ROOT}:${PYTHONPATH} | |||
|
|
|||
| export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib/ | |||
|
|
|||
|
|
|||
| MODEL=wav2vec2 | |||
| MODEL=$1 | |||
There was a problem hiding this comment.
这个不需要固定，不能用传参的方式。如果是和wav2vec一个asr目录的话就单开个吧。
examples/librispeech/asr3/run.sh
Outdated
| stage=0 | ||
| stop_stage=0 | ||
| conf_path=conf/wav2vec2ASR.yaml | ||
| gpus=2 |
| logger = Log(__name__).getlog() | ||
|
|
||
|
|
||
| def clip_grad_norm_( |
There was a problem hiding this comment.
我看wav2vec2目前也用的这个接口？paddle的对应api是哪个？可以用了吗？
There was a problem hiding this comment.
这里我注释了todo，后面paddle依赖改为2.5后再改这里吧
| self.feat2tar_ratio = cfg.label_rate * feature_ds_rate / task_cfg.sample_rate | ||
|
|
||
| self.post_extract_proj = ( | ||
| nn.Linear(self.embed, cfg.encoder_embed_dim) |
There was a problem hiding this comment.
需要替换成align.Linaer，相关的都需要操作下。
| self.target_glu = None | ||
| if cfg.target_glu: | ||
| self.target_glu = nn.Sequential( | ||
| nn.Linear(final_dim, final_dim * 2), GLU()) |
| self.target_glu = nn.Sequential( | ||
| nn.Linear(final_dim, final_dim * 2), GLU()) | ||
|
|
||
| self.final_proj = nn.Linear(cfg.encoder_embed_dim, final_dim) |
| is_group_norm=False, | ||
| conv_bias=False, ): | ||
| def make_conv(): | ||
| conv = nn.Conv1D( |
| def make_conv_block(e, k, g, l): | ||
| return nn.Sequential(*[ | ||
| nn.Sequential( | ||
| nn.Conv1D( |
|
|
||
| # layer norm associated with the self attention layer | ||
| self.self_attn_layer_norm = LayerNorm(self.embedding_dim) | ||
| self.fc1 = nn.Linear(self.embedding_dim, ffn_embedding_dim) |
zxcd
reviewed
Apr 25, 2023
| @@ -0,0 +1,586 @@ | |||
| # Copyright (c) Facebook, Inc. and its affiliates. | |||
luotao1
pushed a commit
to luotao1/PaddleSpeech
that referenced
this pull request
Jun 11, 2024
…le#3088) * librispeech hubert, test=asr * librispeech hubert, test=asr * hubert decode * review * copyright, notes, example related * hubert cli * pre-commit format * fix conflicts * fix conflicts * doc related * doc and train config * librispeech.py * support hubert cli
zxcd
added a commit
that referenced
this pull request
Jun 13, 2024
* [TTS]add Diffsinger with opencpop dataset (#3005) * Update requirements.txt * fix vits reduce_sum's input/output dtype, test=tts (#3028) * [TTS] add opencpop PWGAN example (#3031) * add opencpop voc, test=tts * soft link * Update textnorm_test_cases.txt * [TTS] add opencpop HIFIGAN example (#3038) * add opencpop voc, test=tts * soft link * add opencpop hifigan, test=tts * update * fix dtype diff of last expand_v2 op of VITS (#3041) * [ASR]add squeezeformer model (#2755) * add squeezeformer model * change CodeStyle, test=asr * change CodeStyle, test=asr * fix subsample rate error, test=asr * merge classes as required, test=asr * change CodeStyle, test=asr * fix missing code, test=asr * split code to new file, test=asr * remove rel_shift, test=asr * Update README.md * Update README_cn.md * Update README.md * Update README_cn.md * Update README.md * fix input dtype of elementwise_mul op from bool to int64 (#3054) * [TTS] add svs frontend (#3062) * [TTS]clean starganv2 vc model code and add docstring (#2987) * clean code * add docstring * [Doc] change define asr server config to chunk asr config, test=doc (#3067) * Update README.md * Update README_cn.md * get music score, test=doc (#3070) * [TTS]fix elementwise_floordiv's fill_constant (#3075) * fix elementwise_floordiv's fill_constant * add float converter for min_value in attention * fix paddle2onnx's install version, install the newest paddle2onnx in run.sh (#3084) * [TTS] update svs_music_score.md (#3085) * rm unused dep, test=tts (#3097) * Update bug-report-tts.md (#3120) * [TTS]Fix VITS lite infer (#3098) * [TTS]add starganv2 vc trainer (#3143) * add starganv2 vc trainer * fix StarGANv2VCUpdater and losses * fix StarGANv2VCEvaluator * add some typehint * [TTS]【Hackathon + No.190】 + 模型复现：iSTFTNet (#3006) * iSTFTNet implementation based on hifigan, not affect the function and execution of HIFIGAN * modify the comment in iSTFT.yaml * add the comments in hifigan * iSTFTNet implementation based on hifigan, not affect the function and execution of HIFIGAN * modify the comment in iSTFT.yaml * add the comments in hifigan * add iSTFTNet.md * modify the format of iSTFTNet.md * modify iSTFT.yaml and hifigan.py * Format code using pre-commit * modify hifigan.py,delete the unused self.istft_layer_id , move the self.output_conv behind else, change conv_post to output_conv * update iSTFTNet_csmsc_ckpt.zip download link * modify iSTFTNet.md * modify hifigan.py and iSTFT.yaml * modify iSTFTNet.md * add function for generating srt file (#3123) * add function for generating srt file 在原来websocket_client.py的基础上，增加了由wav或mp3格式的音频文件生成对应srt格式字幕文件的功能 * add function for generating srt file 在原来websocket_client.py的基础上，增加了由wav或mp3格式的音频文件生成对应srt格式字幕文件的功能 * keep origin websocket_client.py 恢复原本的websocket_client.py文件 * add generating subtitle function into README * add generate subtitle funciton into README * add subtitle generation function * add subtitle generation function * fix example/aishell local/train.sh if condition bug, test=asr (#3146) * fix some preprocess bugs (#3155) * add amp for U2 conformer. * fix scaler save * fix scaler save and load. * mv scaler.unscale_ blow grad_clip. * [TTS]add StarGANv2VC preprocess (#3163) * [TTS] [黑客松]Add JETS (#3109) * Update quick_start.md (#3175) * [BUG] Fix progress bar unit. (#3177) * Update quick_start_cn.md (#3176) * [TTS]StarGANv2 VC fix some trainer bugs, add add reset_parameters (#3182) * VITS learning rate revised, test=tts * VITS learning rate revised, test=tts * [s2t] mv dataset into paddlespeech.dataset (#3183) * mv dataset into paddlespeech.dataset * add aidatatang * fix import * Fix some typos. (#3178) * [s2t] move s2t data preprocess into paddlespeech.dataset (#3189) * move s2t data preprocess into paddlespeech.dataset * avg model, compute wer, format rsl into paddlespeech.dataset * fix format rsl * fix avg ckpts * Update pretrained model in README (#3193) * [TTS]Fix losses of StarGAN v2 VC (#3184) * VITS learning rate revised, test=tts * VITS learning rate revised, test=tts * add new aishell model for better CER. * add readme * [s2t] fix cli args to config (#3194) * fix cli args to config * fix train cli * Update README.md * [ASR] Support Hubert, fintuned on the librispeech dataset (#3088) * librispeech hubert, test=asr * librispeech hubert, test=asr * hubert decode * review * copyright, notes, example related * hubert cli * pre-commit format * fix conflicts * fix conflicts * doc related * doc and train config * librispeech.py * support hubert cli * [ASR] fix asr 0-d tensor. (#3214) * Update README.md * Update README.md * fix: 🐛 修复服务端 python ASREngine 无法使用conformer_talcs模型 (#3230) * fix: 🐛 fix python ASREngine not pass codeswitch * docs: 📝 Update Docs * 修改模型判断方式 * Adding WavLM implementation * fix model m5s * Code clean up according to comments in #3242 * fix error in tts/st * Changed the path for the uploaded weight * Update phonecode.py # 固话的正则 错误修改 参考https://github.com/speechio/chinese_text_normalization/blob/master/python/cn_tn.py 固化的正则为： pattern = re.compile(r"\D((0(10|2[1-3]|[3-9]\d{2})-?)?[1-9]\d{6,7})\D") * Adapted wavlmASR model to pretrained weights and CLI * Changed the MD5 of the pretrained tar file due to bug fixes * Deleted examples/librispeech/asr5/format_rsl.py * Update released_model.md * Code clean up for CIs * Fixed the transpose usages ignored before * Update setup.py * refactor mfa scripts * Final cleaning; Modified SSL/infer.py and README for wavlm inclusion in model options * updating readme and readme_cn * remove tsinghua pypi * Update setup.py (#3294) * Update setup.py * refactor rhy * fix ckpt * add dtype param for arange API. (#3302) * add scripts for tts code switch * add t2s assets * more comment on tts frontend * fix librosa==0.8.1 numpy==1.23.5 for paddleaudio align with this version * move ssl into t2s.frontend; fix spk_id for 0-D tensor; * add ssml unit test * add en_frontend file * add mix frontend test * fix long text oom using ssml; filter comma; update polyphonic * remove print * hotfix english G2P * en frontend unit text * fix profiler (#3323) * old grad clip has 0d tensor problem, fix it (#3334) * update to py3.8 * remove fluid. * add roformer * fix bugs * add roformer result * support position interpolation for langer attention context windown length. * RoPE with position interpolation * rope for streaming decoding * update result * fix rotary embeding * Update README.md * fix weight decay * fix develop view confict with model's * Add XPU support for SpeedySpeech (#3502) * Add XPU support for SpeedySpeech * fix typos * update description of nxpu * Add XPU support for FastSpeech2 (#3514) * Add XPU support for FastSpeech2 * optimize * Update ge2e_clone.py (#3517) 修复在windows上的多空格错误 * Fix Readme. (#3527) * Update README.md * Update README_cn.md * Update README_cn.md * Update README.md * FIX: Added missing imports * FIX: Fixed the implementation of a special method * 【benchmark】add max_mem_reserved for benchmark (#3604) * fix profiler * add max_mem_reserved for benchmark * fix develop bug function:view to reshape (#3633) * 【benchmark】fix gpu_mem unit (#3634) * fix profiler * add max_mem_reserved for benchmark * fix benchmark * 增加文件编码读取 (#3606) Fixed #3605 * bugfix: audio_len should be 1D, no 0D, which will raise list index out (#3490) of range error in the following decode process Co-authored-by: Luzhenhui <[email protected]> * Update README.md (#3532) Fixed a typo * fixed version for paddlepaddle. (#3701) * fixed version for paddlepaddle. * fix code style * 【Fix Speech Issue No.5】issue 3444 transformation import error (#3779) * fix paddlespeech.s2t.transform.transformation import error * fix paddlespeech.s2t.transform import error * 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug (#3786) * 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug * 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug * 【test】add cli test readme (#3784) * add cli test readme * fix code style * 【test】fix test cli bug (#3793) * add cli test readme * fix code style * fix bug * Update setup.py (#3795) * adapt view behavior change, fix KeyError. (#3794) * adapt view behavior change, fix KeyError. * fix readme demo run error. * fixed opencc version --------- Co-authored-by: liangym <[email protected]> Co-authored-by: TianYuan <[email protected]> Co-authored-by: 夜雨飘零 <[email protected]> Co-authored-by: zxcd <[email protected]> Co-authored-by: longRookie <[email protected]> Co-authored-by: twoDogy <[email protected]> Co-authored-by: lemondy <[email protected]> Co-authored-by: ljhzxc <[email protected]> Co-authored-by: PiaoYang <[email protected]> Co-authored-by: WongLaw <[email protected]> Co-authored-by: Hui Zhang <[email protected]> Co-authored-by: Shuangchi He <[email protected]> Co-authored-by: TianHao Zhang <[email protected]> Co-authored-by: guanyc <[email protected]> Co-authored-by: jiamingkong <[email protected]> Co-authored-by: zoooo0820 <[email protected]> Co-authored-by: shuishu <[email protected]> Co-authored-by: LixinGuo <[email protected]> Co-authored-by: gmm <[email protected]> Co-authored-by: Wang Huan <[email protected]> Co-authored-by: Kai Song <[email protected]> Co-authored-by: skyboooox <[email protected]> Co-authored-by: fazledyn-or <[email protected]> Co-authored-by: luyao-cv <[email protected]> Co-authored-by: Color_yr <[email protected]> Co-authored-by: JeffLu <[email protected]> Co-authored-by: Luzhenhui <[email protected]> Co-authored-by: satani99 <[email protected]> Co-authored-by: mjxs <[email protected]> Co-authored-by: Mattheliu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Models
Describe
support ASR Hubert