About lr schedule: how to apply different learning rate to different parameter of the network?

[rorschach-xiao]

I want to apply different learning rate to backbone parameters and non-backbone parameters, can below configs work?

optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0005,
                paramwise_cfg = dict(
                    custom_keys={
                        '.backbone': dict(
                            lr_mult=0.1, decay_mult=0.9)
                        }
                    )
                 )

[xvjiarui]

Hi @rorschach-xiao
The config you provided will change the learning rate and weight decay of parameters that contain '.backbone' in the name.

You may refer to training tricks for changing head only.

[rorschach-xiao]

thanks for your reply! I noticed that in training tricks doc，you add paramwise_cfg in optimizer_config.

optimizer_config=dict(
    paramwise_cfg = dict(
        custom_keys={
            'head': dict(lr_mult=10.)}))

But in training code , mmseg use cfg.optimizer to build optimizer ,

optimizer = build_optimizer(model, cfg.optimizer)

and in mmcv/runner/optimizer/builder.py , the key paramwise_cfg will be popped from cfg.

def build_optimizer(model, cfg):
    optimizer_cfg = copy.deepcopy(cfg)
    constructor_type = optimizer_cfg.pop('constructor',
                                         'DefaultOptimizerConstructor')
    paramwise_cfg = optimizer_cfg.pop('paramwise_cfg', None)
    optim_constructor = build_optimizer_constructor(
        dict(
            type=constructor_type,
            optimizer_cfg=optimizer_cfg,
            paramwise_cfg=paramwise_cfg))
    optimizer = optim_constructor(model)
    return optimizer

I wonder the paramwise_cfg should be added into optimizer or optimizer_config?

[xvjiarui]

Hi @rorschach-xiao
Thanks for pointing out. paramwise_cfg should be added into optimizer instead of optimizer_config. I will create a PR to fix the doc.