Message Store requirements for Internationalized Email

Internationalized email includes UTF-8 characters on email headers . Internationalized email is incompatible with EAI unaware Mail Transport and User Agents. Therefore it is required that Internationalized email are separated from EAI unaware Mail Agents. UTF8SMTP extension of ESMTP takes care that mails with UTF-8 characters in SMTP envelope and mail headers are not delivered to EAI non-compliant SMTP servers. EAI compliant Message Store need take care that mails with UTF-8 characters in mail headers are not seen by EAI non-compliant Mail User Agents. IMAP protocol extension and POP protocol extension takes care that mails with UTF-8 characters in mail headers are not seen by EAI non-compliant Mail User Agents when served via IMAP and POP protocol. Therefore this this document focus more traditional Unix mailboxes, where Mail User Agents access mail via file system access. Focus of this document is on requirements of Message Stores and not actual implementation choice although some suggestions are given. IMAP and POP protocol assumes existence of Message Store. Requirements given on this document apply also Message Stores used by POP and IMAP servers. Mail submission is not discussed on this document.

Terms "Mail Transfer Agents" (MTA), "Mail User Agent" (MUA), "Message Delivery Agent" (MDA) and "Message Store" (MS) are used according of . Term "final delivery MTA" is used according of . Term "UTF8SMTP message" are used for messages which follow . Term "ASCII header message" are used for messages which follow .

This document assumes following model for mail architecture. +----------+ +---------+ -- SMTP ---> | final | -----> | | (a) | delivery | (b) | MDA | | MTA | | | +----------+ +---------+ | | write \ / (c) | (---------) ( ) ( MS ) ( ) (---------) | (d) / \ access | | | | +---------+ | | | MUA | | | +---------+ Mail arrives via SMTP . Final MTA passes mail to MDA with proprietary method or via LMTP . MDA writes message to MS. MUA access messages on MS with some means. This model is divided to two special model: Case where MUA access MS with POP and IMAP protocol, and Case where MUA access MS via file system access.

This model assumes that MUAs do not access MS via file system access. +----------+ -- SMTP ---> | final | (a) | delivery | | MTA | +----------+ | | (b) | v (----) +-----------------------+ (c) ( ) | MDA | ----- ( ) |.......................| ( ) | | ( MS ) | mail server | (d) ( ) | | ----- ( ) | | ( ) +-----------------------+ (----) | | | (e) IMAP (f)POP (g)HTTP | | | | v | +---------+ +-------+ +--------+ | | | | | WWW | | MUA | | MUA | | browser| | | | | +--------+ +---------+ +-------+ | (-------) ( MUA's ) (folders) (-------) As on general model. Mail server access messages on MS with some means. MUA access mail server (IMAP server) with IMAP . MUA access mail server (POP server) with POP . WWW browser access mail server with HTTP .

This model assumes that MUA access Message Store with file system access. +----------+ +---------+ -- SMTP ---> | final | -----> | | (a) | delivery | (b) | MDA | | MTA | | | +----------+ +---------+ || (c) write || vv (----------------------------) ( MS [incoming mailboxes] ) (----------------------------) | | | | (d) read,write | | +-------+ | | | MUA | | | +-------+ | (-------) ( MUA's ) (folders) (-------) As on general model. MUA access MS via file system access. Traditionally Unix incoming mailboxes are files /var/mail/ or /var/spool/mail/ directory or similar places. User's incoming mail is located on file which name correspond to username of user. These files are on "mbox" format . Also other arrangements and formats exists.

Requirements for UTF8SMTP aware Message Store are following: UTF8SMTP messages must be accessible to UTF8SMTP aware MUAs on UTF8SMTP form without information lost. UTF8SMTP ignorant MUAs must not see UTF8SMTP messages (messages can be hidden or downgraded for UTF8SMTP ignorant MUAs). All messages should be accessible to UTF8SMTP aware MUAs on that form which MS received them. This requirements is needed that possible signatures and hashes calculated for message can be verified. There is two mail reason why UTF8SMTP ignorant MUAs must not see UTF8SMTP messages: Earlier standards allows only ASCII header fields. Therefore UTF-8 may cause malfunction for MUA. Address syntax on UTF8SMTP header fields includes fallback address. Therefore UTF8SMTP ignorant MUAs are not able to parse these header fields. Notes: It is not required UTF8SMTP ignorant MUA can see UTF8SMTP message on downgraded form. When MUAs access MS via file system this is seen to difficult (or wasteful) requirement. Although this document suggest that messages with ASCII header fields and UTF8SMTP messages are handled separately it is allowed that UTF8SMTP aware MS handles all messages as UTF8SMTP messages. ASCII header message can be treated as subset of UTF8SMTP messages. This implies that UTF8SMTP ignorant MUA does not see any message (if downgrading is not provided). On some environment it is common that Subject: and some other header fields are not encoded sometimes, but use some random 8-bit (presumably system local) character set. If used character set is not UTF-8, these are not UTF8SMTP messages. Therefore these messages can not be treated as UTF8SMTP messages as ASCII header messages can be treated. First requirement (messages accessible on UTF8SMTP form) practically forbids storing messages only on downgraded form.

Requirements for UTF8SMTP aware Mail Delivery Agent are following: MDA must not deliver UTF8SMTP messages to UTF8SMTP unaware Message Store. Notes: In practice this means that if Mail Delivery Agent is UTF8SMTP, MS must be constructed that way that requirements previous section are fulfilled. UTF8SMTP messages does not include label, which which tells which messages are UTF8SMTP messages and which messages are ASCII header messages. To discover message is UTF8SMTP message may require that all message header fields (including header fields from MIME body parts) are parsed. Because MDAs are not expected to parse messages, it is suggested that final delivery MTA pass that information to MDA. On some legacy environments it is common that Subject: and some other header fields are not encoded. Therefore presence of 8-bit bytes itself does not indicate that message is UTF8SMTP message. Test is also needed that sequence of 8-bit bytes forms UTF-8 characters.

UTF8SMTP aware MTAs must fill requirements of . For UTF8SMTP aware final delivery MTA there is following additional requirements: Final delivery MTA must not deliver UTF8SMTP messages to UTF8SMTP unaware mail delivery agent. Notes: If final delivery MTA is UTF8SMTP aware, it is recommended that MDA is arranged that way that it is UTF8SMTP aware. If communication between final delivery MTA and MDA use LMTP, UTF8SMTP response to LHLO command tells that MDA is UTF8SMTP aware. In general final delivery MTA can be expected to parse messages, so it knows when them are UTF8SMTP messages. Passing that information to MDA may require that LMTP is extended.

On this section suggestion to how handle traditional Unix mailboxes is given. Let's assume that incoming mail for user is stored to /var/mail/{username} file on UTF8SMTP unaware MS by using "mbox" format , where {username} is user's account name. To make MS UTF8SMTP aware, MS is modified following way: UTF8SMTP messages are stored to /var/mail/{username}:UTF8 file by MDA (for UTF8SMTP messages ':UTF8' is appended to name of file.) If environment is fully UTF8SMTP aware, all messages are stored to /var/mail/{username}:UTF8 file by MDA. ASCII header messages can be still stored to /var/mail/{username} file. MDA is allowed (but not required) to store copy of UTF8SMTP messages on downgraded form to /var/mail/{username} file. This means that if downgrading is provided, storage space requirements are doubled. If downgrading fails, MDA must not reject message, but instead just store original UTF8SMTP message to /var/mail/{username}:UTF8 file. On some cases it is common that Subject: and some other header fields are not encoded and use some random 8-bit character set (presumably local character set of system). If UTF-8 is not used on header fields, these messages are not UTF8SMTP messages. Therefore they are not stored to /var/mail/{username}:UTF8 file. Handling of these messages is not changed. It is very unlikely that Unix account names includes ':' characters, therefore it is expected that ':UTF8' suffix does not conflict with user's account names. UTF8SMTP aware MUA needs to do following: When incoming mailbox is opened, both files /var/mail/{username} and /var/mail/{username}:UTF8 are parsed. If message with same Message-ID header field is presented both on /var/mail/{username} and /var/mail/{username}:UTF8 file, only message from /var/mail/{username}:UTF8 file is presented. UTF8SMTP aware MUA must not write UTF8SMTP messages to /var/mail/{username} file. It is recommended that configuration for MDA is provided, which allows specify which users accepts only non-UTF8SMTP messages, and which users accepts also UTF8SMTP messages. In lack of configuration, following heuristic is suggested: Presence of /var/mail/{username}:UTF8 file indicates that user accepts UTF8SMTP messages. If file /var/mail/{username} exists and /var/mail/{username}:UTF8 file do not exists, that can be treated that user accepts only non-UTF8SMTP messages. Presence of /var/mail/{username}:UTF8 file without /var/mail/{username} file can be treated that all messages should be treated as UTF8SMTP. If either /var/mail/{username} or /var/mail/{username}:UTF8 file exists, that can be treated as temporary error condition.

There is no IANA considerations in this document.

If user uses UTF8SMTP unaware MUA, UTF8SMTP messages may look for his/her that they go to bit bucket although they appear to be delivered as far sender is considered. See "Security considerations" section in for more discussion.

Various requirements and ideas are suggested on IMA mailing list discussions.