Remote IMAP

Since the Remote command have such a huge number of parameters, we have to split the documentation into several documents. This document describes the IMAP supplementation of remote lookup.

IMAP background

Since POP3 is such a simple protocol and only meant to download and delete the mail form the server, a new protocol was created to handle sharing mail between users and keep the mail on the server. To do this, a much richer protocol was created. This is IMAP. The current version is IMAP4R1, but you usually just see it as just IMAP. This is also what we will use as the name of the command. In the command we will use this syntax: Remote("@imap:command parameter1 parameter2 etc."). Due to the nature of the client server architecture of IMAP, it is domed to be more complex to use. We want to show that this not has to be the case if you just want to use it the same way as POP3. It is when you want to manage folders, store data on the server, update read and reply flags, things might start to look complicated. It is not to bad as you start to understand how it is done. We will start with examples of  simple us and come back to the more complex uses at the end of the document. 

A few things that you have to know before you start is: since you are meant to organize folders on the server, IMAP adds one more level to get started. You have to select the folder before you start read mails. To make this confusing, the documentation insists on calling the mail folders for mail boxes. Just ignore that and use the term folder and much of the confusion disappears. You are usually good as soon as you have selected the folder called INBOX that most servers support. That done, you can search for mails and download then to your computer much like you did using POP3.

Reference

Remote("@imap:command")

Remote("@imap:command param1")

Remote("@imap:command param1 param2 etc.")

All imap commands use a setup for mail server and username/password. Then you do the actions using sub commands that are added after the initial @imap command using a : and the command. If the command have more parameters like search and fetch, then you add this to the end of the sub command using a space, : or , as the separator. You can set the security on the protocol by use the :securitytype command. The default is any, that will test from securest to pain text and use the one that is excepted. Make sure that the last command you use are a :logout command as the line to the server will be open until it times out and possible locked at the server side. Any command to delete folders or messages are not run until the logout or expunge command are sent.

Commands

capability	None	Return what capabilities the server has. All the capabilities are stored as an array of strings to the capabilities array..
close	None	Close the currently selected folder.
create	folder	Create a new mail folder.
delete	folder	Delete a folder.
expunge	None	Commits changes to the server.
fetch	id, [section] and [partial]	This command is the one used for reading the mail message and the headers. Since we are using the imap protocol, we can specify what sections and parts we want back from the server. This is described in full in RFC 3501 page 54, but what you usually want to use is: Remote("@imap:fetch 1"), Remote("@imap:fetch 1 HEADER")  or Remote("@imap:fetch 1 TEXT") where the 1 indicates the message id found by search. The first example reads all of the message including headers and all body parts. The second only reads the headers and the third reads only the text part of the message. After you have fetched the message, you can read any parts of it using the Remote("mails[0].subject") where 0 is the posistion in the mails array and subject are representing a field name that can be found in the message or header. To read a part of the body if you have read the full message use Remote("mails[0].body|html") to read the html part of the body using the html filter on the body.
get	id and any RFC 3591 field	Get metadata from the server like flags, uid, internal date and body size. Ex.  Remote("@imap:get 1 (UID FLAGS INTERNALDATE)") . This will return the listed parameters inside the parentheses with lowercase names in the mails array with id=1. This commad supports all RFC 3501 parameters, but do not return other than metadata. To get the body info, you must use the fetch command.
list	None or folder	The list command lists the folders. By no parameter you get all the folders and adding a folder name, you get the sub folders for that folder. Mind that if you read the imap specification, they call folders mailboxes even thou nobody calls them other than folders in any mail program. List is the default command if you omit any parameter.
logout		Close connection to server and do expunge on all changes
mode	None, id or uid	Set the operation mode. IMAP can use sequence numbers or uids. The uid never changes between session as long as the folder is not deleted or the mailserver is changed. The ids are sequence numbers. You can use this to set the operation mode to either id or uid mode. The uid mode is default. This means that all search, fetch and store uses uid as first parameter. Command without parameter returns current mode.
noop	None	Used to keep the session alive and reset the timeout counter. It does nothing and returns nothing.
rename	folder newname	Rename a folder.
select	folder	Selects the folder you specify as a parameter. This will be used for all subsequent commands until you select another. The command returns status for the selected folder. It will take each response from the server and store the values in lower case with values as a array under. There are some special return values that will be added as know values at the root. These are mailcount, exists, mailnew and recent. Other values normally found that are stored as array or return values and lowercase are: FLAGS to flags with values as array of string, [PERMANENTFLAGS to permanentflags as array of strings, [UNSEEN to unseen with string value... The conversion routines follows simple rules a NAME (val val2 val3) is returned as lowername with an array of string, [NAME (va1 val2 val3...)] as a lowername without [ and an array of strings, [NAME value] become name with a value.
search	what	Search using any valid imap search command.
status	folder returntypes	Returns the status of a none selected folder. Do not use this to get the status of the selected folder as you already get theses data when you select the folder. The returntypes give you the elements and the names that are return from the function. You can add as few or many as you like as long as you have at least one. This are the valid returntype: MESSAGES, RECENT, UIDNEXT, UIDVALIDITY and UNSEEN. You can also use extension values as return types like gmails HIGHESTMODSEQ. You can find out supported extensions using the capability command. The values are set as lower case values at the root level and can be read with ex.: Remote("messages").
store	what and changes	This command are used to update the flags of an mail. You can overwrite, add or remove flags by using: Remote("@imap:store flag (\flag1 \flag2 etc)" to replace, Remote("@imap:store +flag (\flag1 \flag2 etc)" to add and Remote("@imap:store -flag (\flag1 \flag2 etc)" to remove. Valid flag can be found in RFC3501, but the most used are \Seen, \Answered, \Flagged, \Deleted, \Draft and \Recent. The recent flag can not be stored back to the server as it is a special case only shown when no other has read that mail.

Return values

capabilities	A string array of the returned capabilities
folders	An array of available folder returned by the list command. The array contains object with flags, sep and name values. The one you probably will use is only name that is the folder name used by select. The other elements are internal values returned by the imap protocol.
mailcount	The number of mails in the remote mailbox. This is returned by the stat command.
mailnew	New mails in mailbox that not has been read by any mail program yet.
emails	An array of mail data from the server. The fields populated depends on the commands that is used to fetch mail. The array will contain field data like id, uid, flags etc and a body if fetch or get is use. -search add id/uid -fetch add MIME data with any filed it contains for one mail if called without parameters other than id/uid. MIME data contains fields like: To, From, Subject, Body, Date and a lot of others based on what is set in the mail header by the server. It also add capability to read Body with filters for attachments and type data like text, html and multipart. You can also read the data as mime using raw. Section let you only read parts of the message and partial let you cut down on the size you read. -get can read the same as fetch, but you can also give your command more direct special function like Remote("@imap:get 1:* (FLAGS)") that will return and update all flags into the array.
.attachment	Save attachment to disk. Return the file path to the files that is saved to disk Parameters  - [attachment number] default is first, attachment number is starting at 1  - [directory to store attacment to] default is app folder  - [file name to use] default is the file name given in the mail Ex. Remote(concat("email[0].Body.attachment 1 "C:\Documents" myatt.txt") will store the first attachment to a file called myatt.txt in C:\Documents
.attachmentcount	Returns the number of attachments in the email at index. Must be read first with fetch or get. Ex. Remote("emails[3].body.attachmentcount") will return the number of attachments in the 4th email
.attachmentext	Return the extention of the file name given by the email. Parameter  - attachment number - if there are more than one attachment in the email you must add witch attachment Ex. Remote("emails[5].body.attachmentext 1") will return the 6th (0 based index) emails first attachment extension of the file name
.attachmentname	Return the extention of the file name given by the email. Parameter  - attachment number - if there are more than one attachment in the email you must add witch attachment Ex. Remote("emails[0].body.attachmentname") will return the first (0 based index) emails first attachment file name
emails[idx].body emails[idx].body.html .charset .transferencoding .text .text|raw	Return the body part of a email. Default is to read the first MIME part in the mail (usually text), but you can ask for specifics like html, attacment or multipart. Ex. Remote("emails[5].body.html") will return the 6th (0 based index) body in html format if it exists. Ex. Remote("emails[5].body.text") will return the 6th (0 based index) body in text format if it exists. Ex. Remote("emails[5].body") will return the 6th (0 based index) body in the first format it finds. This is usually text format. Ex. Remote("emails[5].body|html") will return the 6th (0 based index) in html format if it exists using old style filters. Ex. Remote("emails[5].body.html.charset") will return the 6th (0 based index) bodys charset if it exists.
emails[idx].raw	Return the raw MIME data of the complete email
emails[idx].AnyHeader	Return data from any header fields the mail has like Subject, To, From, Date and Return-Path.The trigger to read the header field is first character in name is UPPERCASE.
<imapval>	Any value returned by the imap protocol in lower case. The best way of figuring out what is returned is to use the MemoField := Remote("@json") command to store the JSON to a Memo field for reading.

Mail values

The values that can be found in each email object that is fetched by the server can vary based on what is returned by the different server. A few of the values will always be available. 

These are id, To, From, Subject and Body. The id comes from the mail number used to fetch the mail. If you listed mails using uid command, you will also have the uid value. The rest might vary, but all header values starts with an uppercase character. Here are a list of some that we have seen in use:

To, From, Subject, Body, Date, Return-Path, Delivered-To, Received, X-HalOne-ID, DKIM-Signature, X-Received, Return-Path, Message-ID, User-Agent, MIME-Version, Content-Type and Content-Transfer-Encoding. The easiest way to see what's in the buffers is simply to use @json command and return all fields set.

Ex. Remote("@json:emails[0]") to return all fields for the first mail in the mails array as a json string.

Examples

Setting up an email imap with tls or without since the default is to try the strongest login methods first and then less and less secure methods. Depending on if the server uses port 993 (secure imap with SSL) or 143 (plain text imap with or without tsl) you use the imaps:// or imap:// for the url. If the server uses none standard ports, just add that to the end of the server part of the url prefixed by a colon Ex. imaps://imapspesialpot.example.com:999. The function used in the example returns the statistics of the selected inbox. This contains the number of emails on the server and how many new mails you have and an array of other server specific variables like the next uid or extensions like the maxmodseq that we will use in a later example to speed up the update of mail changes.

define temp "Dummy" text 255 .
define temp "mailcount" text .
define temp "mailnew" text .
Dummy := Remote("@current:maildata").
Dummy := Remote("@cleanup").
Dummy := Remote("@connect:imap://imap.example.com") .
-- with ssl (port 995)Dummy := Remote("@connect:imaps://imap.example.com") .
Dummy := Remote("@username:myusername") .
Dummy := Remote("@password:mysecretpassword") .
Dummy := Remote("@imap:select INBOX") .
Dummy := Remote("@imap:logout") .
mailcount := Remote("mailcount") .
mailnew := Remote("mailnew") .

Listing all folders on a server. This example works for a gmail account if you have activated imap in the settings. You usually can assume that a server have a folder called INBOX, but where the sent mail is stored differs, so it is always smart to do a folder list on the server the first time to see the structure and then use this information to do the select for the folder you want to work on.

define temp "Dummy" text 255 .
define temp "telle" number .
define temp "folder" text 255 .
define temp "luuid" text .
Dummy := Remote("@current:maildata").
Dummy := Remote("@cleanup").
Dummy := Remote("@connect:imaps://imap.gmail.com") .
Dummy := Remote("@username:myusername@gmail.com") .
Dummy := Remote("@password:mysecretpassword") .
Dummy := Remote("@imap:list") .
telle := 0 .
folder := Remote(concat("folders[", telle, "].name")) .
while folder not = "" do
  list records
    telle ;
    folder .
  telle := telle + 1 .
  folder := Remote(concat("folders[", telle, "].name")) .
end
Dummy := Remote("@imap:logout") .
-- The body --
.items
Folder: [{subject}]
.end
-- Example output from gmail --
Folder: Holyday 2015
Folder: INBOX
Folder: Sent
Folder: Trash
Folder: [Gmail]
Folder: [Gmail]/All Mail
Folder: [Gmail]/Drafts
Folder: [Gmail]/Important
Folder: [Gmail]/Sent Mail
Folder: [Gmail]/Spam
Folder: [Gmail]/Starred
Folder: [Gmail]/Trash
Folder: projects
Folder: projects/DataEase
Folder: projects/WebEasy

Showing some fields of a message when you know the id. In this case we use stat to fetch the numbers of mails in the mail box and if the number is not "0" we fetch the mail number 1. This will usually be the first mail stored on the imap server but not always, so if you do not get anything back by testing this example , just skip to the next more advanced example that search for mails newer than a certain date and list all of them in the next chapter.

define temp "Dummy" text 255 .
define temp "mailcount" text .
define temp "mailsize" text .
define temp "to" text 60 .
define temp "subject" text 255 .
Dummy := Remote("@current:maildata").
Dummy := Remote("@cleanup").
Dummy := Remote("@connect:imaps://imap.example.com") .
Dummy := Remote("@username:myusername") .
Dummy := Remote("@password:mysecretpassword") .
Dummy := Remote("@imap:select inbox") .
Dummy := Remote("@imap:mode id") .
mailcount := Remote("mailcount") .
mailsize := Remote("mailsize") .
if mailcount not = "0" then
  Dummy := Remote("@imap:fetch 1") .
  to := Remote("emails[0].To") .
  subject := Remote("emails[0].Subject") .
end
Dummy := Remote("@imap:quit") .

In this example we will fetch the headers of all mail newer than a date and list the date, to and from and the subject. This can be done by fetching partials mails from the server. This works much the same as the top command in pop3, just that you use the same command as for downloading mail instead. I have prefixed all temp variables with a t to avoid conflict with the fields in the table. If you do not do this, you must add the temp keyword before the variable name on any instance where it is used. If you happens to get mails with other dates than what you have searched for, do not despair because we search for the internal date, and lists the mail header dates. This means that we search for when the server got the mail and lists when it was sent from the other server, these two dates might be different.

-- The DQL --
define temp "newmails" number .
define temp "existingmails" number .
define temp "tid" text .
define temp "sid" text 255 .
define temp "tdate" text .
define temp "tsubject" text 255 .
define temp "tto" text 60 .
define temp "tfrom" text 60 .
define temp "ctype" text 160 .
define temp "ctte" text 160 .
define temp "mver" text 60 .
Dummy := Remote("@current:maildata").
Dummy := Remote("@cleanup").
Dummy := Remote("@connect:imaps://imap.example.com") .
Dummy := Remote("@username:myusername") .
Dummy := Remote("@password:mysecretpassword") .
Dummy := Remote("@imap:select inbox") .
Dummy := Remote("@imap:mode id") .
Dummy := Remote("@imap:search SINCE 18-Sep-2015") .
telle := 0 .
temp tid := Remote(concat("emails[", telle, "].id")) .
while tid not = "" do
  Dummy := Remote(concat("@imap:fetch ",tid," HEADER")) .
  tto := Remote(concat("emails[", telle, "].To")) .
  tfrom := Remote(concat("emails[", telle, "].From")) .
  tsubject := Remote(concat("emails[", telle, "].Subject")) .
  ctype := Remote(concat("emails[", telle, "].Content-Type")) .
  ctte := Remote(concat("emails[", telle, "].Content-Transfer-Encoding")) .
  mver := Remote(concat("emails[", telle, "].MIME-Version")) .
  tdate := Remote(concat("emails[", telle, "].Date")) .
  list records
    telle ;
    tto ;
    tfrom ;
    tsubject ;
    ctype ;
    ctte ;
    mver ;
    tdate ;
    tid .
  telle := telle + 1 .
  temp tid := Remote(concat("emails[", telle, "].id")) .
end
Dummy := Remote("@imap:logout") .
-- The body --
.items
==========================
[{telle:R2}]:[{tid:R5}]
From: [{tfrom}]
Subject: [{tsubject}]
.end
-- Example output --
==========================
 0: 5133
From: First Last <mail@example.com>
Subject :=?utf-8?B?UkFEIFN0dWRpbyAxMCDigJMgaGFyIGR1IG9wcGxldmQgZGV0IGVuZGE/?=
==========================
 1: 5134
From: "Orderno: 335222545" <order@my.example.com>
Subject: Thank you for your order 335222545
==========================
 2: 5135
From: noreply@example.com
Subject: Newsletter for software

In this example we will calculate a unigue id based on the uid for the mail and the folder and only download new mails and store them to t mail table. We use the same table as for the POP3 example.