You can disable unicode support in samba and manually specify charsets (dos and unix). These can be done via auxiliary parameters under services->smb and the settings are described in the smb.conf manpage. https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html
As far as max filename length goes, there's not a good way to enforce that. No knobs that I know of at the protocol level, and so you'd be speaking something that isn't SMB.