WeightDecayScheduler

class WeightDecayScheduler(base_value=0.04, final_value=0.2, warmup_ratio=1.0)

Bases: HookBase

Apply a cosine schedule to optimizer parameter-group weight decay.

In before_train it builds a CosineScheduler spanning cfg.scheduler.total_steps * warmup_ratio iterations (advanced to the current step so resumes continue smoothly). In before_step it steps the schedule and writes the resulting weight decay onto every optimizer group flagged apply_wd=True (groups flagged False keep their original, e.g. 0.0), then logs it to the writer as params/wd. Registered as WeightDecayScheduler.

Parameters:

• base_value (float) – Starting weight decay. Defaults to 0.04.

• final_value (float) – Ending weight decay after the schedule. Defaults to 0.2.

• warmup_ratio (float) – Fraction of cfg.scheduler.total_steps over which the cosine schedule runs (scales the schedule length). Defaults to 1.0.

Note

The apply_wd flags are produced by WeightDecayExclusion; without it every group defaults to apply_wd=True and is scheduled. Place this hook after WeightDecayExclusion.

Example

Add to cfg.hooks after WeightDecayExclusion; it cosine-schedules weight decay each step:

hooks = [
    dict(type="WeightDecayExclusion"),
    dict(type="WeightDecayScheduler", base_value=0.04,
         final_value=0.2, warmup_ratio=1.0),
]
# → every step (before_step) sets param_group["weight_decay"] (cosine
#   0.04 -> 0.2) on every apply_wd=True group and writes the scalar
#   "params/wd" to the writer; apply_wd=False groups keep weight_decay=0.0

before_step()

before_train()