realtek: pcs: rtl931x: improve mode setting functions

realtek: pcs: rtl931x: improve mode setting functions

Improve the current mode setting functions to address a few issues:
- add missing mode from SDK reference (instead of using the
  corresponding value as a default value)
- use and return error values
- give internal variable a more meaningful name
- consistently use lowercase letters in hex values

Signed-off-by: Jonas Jelonek <jelonek.jonas@gmail.com>
Link: https://github.com/openwrt/openwrt/pull/20736
Signed-off-by: Robert Marko <robimarko@gmail.com>