## Larceny Note #9: The I/O system

Lars T Hansen / November 14, 1997
William D Clinger / 18 July 2008

This note is being written

## The I/O architecture

Larceny's I/O system is strongly influenced by the Modula-3 I/O system presented in Greg Nelson's book, Systems Programming in Modula-3. The I/O system is designed with performance and user-extensibity in mind.

Larceny's I/O system is composed of several layers:

- I/O facilities provided by the underlying operating system
- syscalls that make a slightly OS-dependent abstraction of certain low-level I/O operations available to Scheme
- Scheme code that constructs an OS-independent abstraction of certain low-level I/O operations
- the `ioproc` layer, which implements an abstraction of binary input and output
- the `iosys` layer, which implements buffering, transcoding, and end-of-line handling
- specialized and custom ports of the R6RS io system
- R5RS and R6RS io systems

## Low-level operations

Larceny translates some of the basic I/O operations provided by the underlying operating system into a small set of OS-independent operations that open, close, read, and write a slight abstraction of low-level file and console ports. This abstraction is implemented by the following files:

- `src/Rts/Sys/syscall.c`
- `src/Rts/Sys/osdep-generic.c`
- `src/Rts/Sys/osdep-unix.c`
- `src/Rts/Sys/osdep-macos.c`
- `src/Rts/Sys/osdep-win32.c`
- `src/Lib/Common/sys-unix.sch`
- `src/Lib/Common/sys-macos.sch`
- `src/Lib/Common/sys-win32.sch`

## The `ioproc` layer

At the `ioproc` layer, a port is implemented using a Scheme procedure of one argument: a symbol that indicates the operation to perform. After dispatching on that symbol, the ioproc procedure returns a port-specific procedure that, when called, performs the operation. The operations are:

- `read : iodata * buffer → { fixnum, 'eof, 'error }`
- `write : iodata * buffer * count → { 'ok, 'error }`
- `close : iodata → { 'ok, 'error }`
- `name : iodata → string`
- `set-position! : iodata * posn → { 'ok, 'error }`

The `read` operation, which attempts to fill the given input buffer, is supported only for input ports; the `write` operation, which empties the given output buffer, is supported only for output ports; and the `set-position!` operation is supported only for ports that support `set-port-position!`.

The `iodata` argument to an `ioproc` represents any additional state that may be associated with a port but encoded separately from the port's ioproc and its methods. There is no standard representation for the `iodata` object; its nature and interpretation vary from one kind of port to another.

## The `iosys` layer

The `iosys` layer implements Scheme ports. Each port is an object that encapsulates:

- whether the port is binary or textual
- the port's direction: one of
  `closed input output input/output`
- an `ioproc`
- any `iodata` needed by the `ioproc`
- port state: one of
  `closed error eof binary textual auxstart auxend input/output`
- reader mode (for textual ports only)
  such as whether the port is case-sensitive
- transcoder
buffer mode: one of
`none line datum block`
- buffers
- whether the port supports `set-port-position!`
- port position
- whether the last end-of-line was represented by a return

### The port state

Once a port enters the `closed`, `error`, or `eof` states, it remains in that state.

> On Unix systems, where some programmers (and the R6RS, alas) like to think an end-of-file on a console port can be followed by more input, what actually happens in Larceny is that the end-of-file causes the console port to be abandoned by the `(current-input-port)` procedure, which opens a fresh console port that becomes the current input port.

An open binary port is normally in the `binary` state.

A textual input port is normally in the `textual` state, but often enters the `auxstart` state when the main buffer is filled and often enters the `auxend` state when the main buffer has almost been emptied. These two states use a small auxiliary buffer to deal with multi-byte encodings that span a buffer boundary. They also support the use of a sentinel byte that simplifies the inline code for `get-char`.

Textual output ports make similar use of the auxiliary buffer.

Combined input/output ports are normally in the `input/output` state. They buffer at most one character, for a maximum of four bytes.

### Port position

All of Larceny's textual input ports keep track of their current port position in units of characters, lines, and offsets within a line. Larceny's end-of-line processing depends upon the offset within a line together with a boolean that indicates whether the previous end-of-line was represented by a lone return character.

The R6RS end-of-line semantics has an interesting interaction with `set-port-position!`. Unless the end-of-line style of a textual input port is `none`, setting a new port position sometimes requires R6RS-conforming implementations to examine the character that precedes the new position in order to implement the mandated end-of-line semantics.

## Specialized ports

## Custom ports

## Thread-aware I/O

To support thread-aware I/O, we need two things:

- Mutual exclusion in access to the I/O structures.
- Blocking I/O that is compatible with the threads system.

Mutual exclusion is not hard; the procedures in `Lib/stdio.sch` can be wrapped in a `without-interrupts` form. The lock should probably be a public part of the port structure so that it's possible for (system) code to acquire it once and then call low-level primitives for better performance.

Since the threads system is (currently) written in Scheme on top of continuation, blocking system calls are no good. Instead, I/O system calls that may block indefinitely must be avoided.

The right thing to do seems to consider two subtypes of I/O ports, along the lines of the Modula-3 I/O system. A port is classified either as intermittent or not. Intermittent ports may have to wait an unbounded amount of time before input is available or output is accepted. Currently, the only intermittent ports are console I/O ports, but when the extensible I/O system goes public, we'll have sockets pretty quickly.

Intermittent ports havs the following unique attribute: the underlying read and write methods return `would-block` if no work was accomplished (no input was ready or no output would be accepted). If that token is returned to the fill or flush methods, then the operation on the port must block until the port is ready. This blocking can be done either by polling or by interrupts. If I/O interrupts are available, then the I/O system must enable them and set up an I/O event handler. If not, the I/O system must register an I/O poll procedure for the port as a periodic system task. In either event, the I/O system will then block the thread on a condition variable that will be signalled by the ready handler, whichever method is used.

I think that the actual underlying mechanism chosen for unblocking threads can and should be independent of the Scheme I/O system. This is possible if the I/O system supports an installable "ioblock" handler that it will call to wait for I/O on a port. System code will then install the correct ioblock handler for the I/O event system chosen on the particular platform.

$Id: note9-iosys.html 87 1998-11-25 14:38:41Z lth $
larceny@ccs.neu.edu