[adm5120] fix usb mis-cleanup after r27162 (#9581)