Streaming: How to Get Bytes From Here to There in Smalltalk

Craig Latta
26 May 1998

I've written a framework for communicating bytes in Smalltalk with a simple transport-independent interface. It's based on the stream, which is essentially a sequence of elements that remembers a position. I use streams to access files, network connections, serial ports, and other external resources. I also use them to manipulate "internal" collections in local memory.

In this tutorial we'll see how to create and use streams on various resources.

creating and using simple streams

We'll start by creating a couple of internal streams and manipulating them in simple ways. First let's make a global variable for our examples:


	"Make a global variable for referring to an example stream."
	Smalltalk at: #ExampleStream put: nil

reading

There are two ways to create a stream on a collection. One is to send a message to the collection. The other is to send a message to one of the stream classes, with a collection as a parameter. For example, the following creates a stream on a collection of characters:


	"Create a stream on a collection of characters."
	ExampleStream := 'this is merely a test' stream

Now let's read a character from it:


	"Read an element from ExampleStream."
	ExampleStream next

The answer is the Character $t. You can also read several characters at once:


	"Read several elements from the example stream."
	ExampleStream next: 3

The answer is a String, 'his'. Note that the stream remembered its position within the sequence after reading the first character. This is why the messages are based around the word next; you're reading the next elements from the current position.

You can also ask a stream for its entire contents:


	"Answer the entire contents of the example stream."
	ExampleStream contents

Sending contents doesn't affect the stream's position. Try sending next again to see this. You can, however, change the stream's position without reading per se:


	"Reposition the example stream two elements forward."
	ExampleStream skip: 2.

	"Reset the example stream's position to the beginning."
	ExampleStream reset

Finally, you can read elements from the collection without changing the stream's position:


	"Read an element from the example stream without changing
the stream's position."
	ExampleStream peek.

	"Read several elements from the example stream without
changing the stream's position."
	ExampleStream peek: 3

Reading from streams is most commonly used in the system for parsing. For an example, see NetMessage>>initializeFrom:withSize:.

writing

The stream we just created can read, but it can't write. Writeable streams are instances of a different Class which understands more messages and whose instances take up slightly more memory. They're created with a different message, and are typically created on empty collections:


	"Create a stream on an empty String."
	ExampleStream := (String new: 16) writeableStream

The messages for writing are similar to those for reading:


	"Write a character with the ExampleStream."
	ExampleStream nextPut: $h.

	"Write several characters with the ExampleStream."
	ExampleStream nextPutAll: 'ello world!'

As before, you can ask a stream for its contents. Since we're writing to the example stream, it will answer differently as elements are written.

Writeable streams have more diverse positioning behavior, to avoid ambiguity when positioning backward. You can simply move a stream's position backward, or you can also retract elements which have been written. For example, after this:


	"Reposition the example stream backward two elements."
	ExampleStream skip: -1

The stream's contents is still 'hello world!'. But after this:


	"Retract the example stream's previous element."
	ExampleStream retract

The stream's contents is 'hello word!' (assuming you performed the skip: in the previous example).

      This touches on a subtle point. A stream's position is a pointer into sequence, between two elements, before the first element, or after the last element. For example, if you reset the example stream, its position will be zero. At the end of 'hello world!', the position is twelve. After writing the first element, the position is one.

      Writeable streams also understand the reading messages. In addition to writing after positioning backward, several places in the system actually perform both reading and writing with the same writeable streams. This makes careful use of positioning behavior very important.

      Streams understand several more messages that we didn't explore here. For example, you can ask a stream if it's positioned at the end (with atEnd), and you can search for patterns of elements (with throughAll:). Browse the Stream and PositionableStream classes to see them.

streaming over diverse collections

For simplicity, our examples so far have involved collections of characters. However, you can use streams to manipulate collections of any objects you like. For example, you could create streams on collections of colors, musical notes, or instances of any new class you create. Perhaps the most popular kind of stream content currently is the integer, specifically those integers between 0 and 255...

streaming bytes to and from the system

Another simplifying characteristic of our examples has been the internal nature of the collections. We used collections whose elements were all stored within local memory. Often it's useful to perform streaming operations with collections located elsewhere. The very use of the words "collection" and "located" may be an abstraction in some cases. All that matters is that the elements are sequenceable. Streams that perform these external exchanges are called external streams.

The "collection" an external stream manipulates is called an external resource. The most common external resources are disk files, serial ports, and sockets. We'll look at creating and using external streams on these resources.

file streams

External resources are identified by name. In turn, external streams are typically created from a resource name, or "universal resource locator" (URL). To create a file stream, you first must create a filename URL, then send a stream-creation message to the filename.


	"Create a stream on a named file."
	ExampleStream := 'file:///c:/autoexec.bat' asFilename stream

Filename URLs understand the same stream-creation messages that SequenceableCollections do, so you can create writeable streams in the usual way. Once you have a stream, you can use the usual messages for reading, writing, positioning, etc.

A new concept here is that of closing. In creating a stream, you are implicitly making a connection between the stream and the thing it will manipulate. That connection needs to be severed gracefully. For an internal stream, this connection is a memory reference, and is automatically retired by the memory management system. Therefore, when you're finished using an internal stream you may simply ignore it. But an external resource's shutdown behavior often involves action outside the system. External streams need to be closed explicitly:


	"Close an external stream."
	ExampleStream close

All external streams are closed this way, regardless of the kind of external resources to which they have been connected.

It's possible that an error or other unexpected event may occur between the time an external stream is created and when it should be closed. The system has a facility for ensuring that actions occur despite such interruptions:


	"Ensure that a file stream is closed, despite the occurrence
of an error just after it is opened."
	[
	        ExampleStream := 'file:///c:/autoexec.bat'
asFilename stream.
	        3 zork
	]
	        valueEnsuring: [ExampleStream close]

This facility is part of the exception-handling system. For more information about exception-handling, see the examples in BlockContext class. Another way of ensuring special terminating actions is by performing them as part of automatic memory reclamation. This is commonly known as finalization.

Most underlying filesystems have a permissions system. Note that you can create a writeable file stream on a "read-only" file without complaint from the system. It's only when you actually try to write with such a stream that you'll encounter an error. We've left it to the underlying filesystem to keep track of permissions and to report errors (which are then relayed by Smalltalk).

net streams

External streams on resources other than files are called net streams. Conceptually, you can think of a net stream as connected to a sequence of bytes which is undefined at creation time. This is in contrast to streams created on internal collections or files, which may be regarded as a predefined sequences of bytes. Indeed, what you'll read from a typical net stream depends strongly on what you write to it.

Net streams are created by sending messages to the NetStream class.

serial port streams

As we've seen, creating an external stream is mainly a matter of referring to the desired resource appropriately. A serial port stream is created like this:


	"Create a stream on a serial port."
	ExampleStream := NetStream onSerialPortAt: 1

Once created, a serial port stream is not yet active. This is because there are typically several additional parameters which need to be set to non-default values before actually opening the resource. For example, you can change the baud rate from the default of 9600 baud:


	"Change the baud rate for a serial port stream."
	ExampleStream baudRate: 38400

For the other such messages you can send, see the control-serial protocol of NetStream. After these parameters have been set, you can open the stream:


	"Open a serial port stream."
	ExampleStream open

As with other external streams, don't forget to close serial port streams after you're finished with them.

socket streams

IP sockets are identified by family (TCP or UDP), port, hostname, and role (client or server). The NetStream class provides several messages requiring varying levels of detail (and providing various defaults). Here's an example TCP client:


	"Create a TCP client to the echo port (7) at the host named
netjam.org."
	ExampleStream := NetStream tcpClientToPort: 7
	atHostNamed: 'netjam.org'

Unlike serial port streams, socket streams are active as soon as they are created.

other NetStream features

Particular numeric positions are no longer meaningful when dealing with net streams. This is because the byte sequences being manipulated usually have no well-defined beginning or end at stream-creation time. Attempting to get or set the numeric position of a net stream results in an error. You can, however, peek forward and skip backward. You can also keep track of the number of bytes that have been read, and you can reopen a stream that has been closed.

correspondents

There is also support in the system for modeling the local side of a conversation involving various net streams. It is provided by the Correspondent hierarchy. In particular, the Client class is useful for defining socket-based clients (e.g., POPClient), the Server class is useful for defining socket-based servers (e.g., HTTPServer), and the SerialCorrespondent class is useful for defining correspondents which communicate via serial ports. The correspondent framework helps to organize various details about initializing external resources (e.g., remembering that 'echo' protocol conversations happen on remote port 7), and provide a place to put other protocol-specific behavior.

explore and enjoy

We've looked at the basic ideas involved in using streams in flow. Have fun!

concept hierarchy

stream
        sequence of elements
                internal resources (collections)
                external resources
        position
creation
        stream
        writeableStream
reading
        next
        next:
        peek
        peek:
writing
        nextPut:
        nextPutAll:
positioning
        skip
        skip:
        retract
        retract:
        position
        position:
        reset
        resetRetainingContents        
position indices point between elements
utilities
        atEnd
        throughAll:
        (etc.)
external streams
        external resources
                files
                net resources
                        serial ports
                        sockets
                                IP
                                        TCP
                                        UDP
                        (numeric positions no longer apply)
                        (correspondents)
        closing
                ensured behavior
                        exception-handling
                        finalization
        reopening
        permissions