Important Observations and Facts

Python and Locale Settings

See also

For locale settings:

Python:

A locale name generally has the form:

language[_territory][.codeset][@modifier]

where language is an ISO 639 language code, territory is an ISO 3166 country code, and codeset is a character set or encoding identifier like ISO8859-1 or UTF-8. For a list of all supported locales, try locale -a.

  • The relevant locale setting for the FSENCODING is LC_CTYPE.

  • For the (legacy) plain C locale the behaviour in Python has changed over time:

    • Older versions use US-ASCII or ASCII as the effective filesystem encoding

    • With PEP 538 (Coercing the legacy C locale to a UTF-8 based locale) Python 3.7+ uses the UTF-8 encoding instead.

  • For Python to recognize a codeset a really existing locale must be used: for a list see the output of locale -a. For example; en_US.ISO8859-1 or C.UTF-8 are existing locales on FreeBSD, but C.ISO8859-1 is not a valid locale.

    In this case sys.getfilesystemencoding() and other locale dependent things behave as configured and (hopefully) intended by the user.

    Non-existing locales behave differently: for example, on FreeBSD a locale like C.ISO8859-1 does not exist; it behaves just like plain C in Python.

  • Existing locales with a codeset that is not available in Python may lead to core dumps or other initialization errors.

  • Typically, the POSIX or C locales do not contain as many codeset variants as other locales. On FreeBSD there are just C and C.UTF-8.

Todo

Windows: Is there a command to list all available locales? Registry keys? “NLS” support?

Open Questions

VFAT and Encodings

The handling of mountpoint-specific encoding settings (typical when mounting (Windows) media with VFAT/FAT/exFAT filesystems and others on POSIX systems)

Todo

Mountpoint-specific encodings seem too complicated—maybe later?

On FreeBSD for mount_msdosfs:

Options:

  -L locale
     Specify locale name used for file name conversions
     for DOS and Win'95 names.
     By default ISO 8859-1 assumed as local character
     set.

  -D DOS_codepage
     Specify the MS-DOS code page (aka IBM/OEM code page)
     name used for file name conversions for DOS names.

Examples:

  To mount a Russian MS-DOS file system located in /dev/ada1s1:

    mount_msdosfs -L ru_RU.KOI8-R -D CP866 /dev/ada1s1 /mnt

  To mount a Japanese MS-DOS file system located in /dev/ada1s1:

    mount_msdosfs -L ja_JP.eucJP -D CP932 /dev/ada1s1 /mnt