On this page:
5.1 The Program:   Echo
5.1.1 Untyped
5.1.2 Typed
5.2 Running the Benchmark
6.2.900.17

5 Walkthrough: Creating and Analyzing a Benchmark Program

To bring it all together we provide a walkthrough of creating a benchmark program, running all configurations and analyzing the data.

5.1 The Program: Echo

First we make an untyped program and add types to it. The program is intentionally small and over-modularized for demonstration purposes. The program is a simple echo server with a client to send it some data.

In the benchmark/ directory, create a echo/ directory with untyped/ and typed/ subdirectories.

5.1.1 Untyped

We start with the untyped/ files. A file constants.rkt containing global constants:

#lang racket/base
 
(provide
 ; Natural number port number to run the echo server on
 PORT
 ; String message to send over the tcp connection
 DATA)
 
(define PORT 8887)
(define DATA "Hello there sailor\n")

server.rkt which contains the actual echo server:

#lang racket/base
 
; TCP server: read from a buffer until end of file.
 
(provide server)
 
(require "constants.rkt"
         (only-in racket/tcp tcp-accept tcp-listen))
 
; ---------------------------------------------------------------------------------------------------
 
(define (server)
 (define-values (in out) (tcp-accept (tcp-listen PORT 5 #t)))
 (define buffer (make-string (string-length DATA)))
 (file-stream-buffer-mode out 'none)
 (let loop ([i (read-string! buffer in)]
            [bytes 0])
   (cond [(not (eof-object? i))
          (display buffer out)
          (loop (read-string! buffer in)
            (+ bytes (string-length buffer)))]
         [else (printf "server processed ~a bytes\n" bytes)])))

client.rkt which is a simple client that repeatedly sends the same message to the server:
#lang racket/base
 
; TCP client bot: loop for a fixed number of iterations
; sending a message over a port.
; The message and port are defined in constants.rkt
 
(provide client)
 
(require "constants.rkt"
         (only-in racket/tcp tcp-connect))
; ---------------------------------------------------------------------------------------------------
 
; `client n` loop for `n` iterations, sending a constant message on a constant port.
(define (client num-iters)
  (define-values (in out) (tcp-connect "127.0.0.1" PORT))
  (define buffer (make-string (string-length DATA)))
  (file-stream-buffer-mode out 'none)
  (for ([n num-iters])
    (display DATA out)
    (read-string! buffer in)
    (unless (equal? DATA buffer)
        (error (format "Unexpected data ~e in buffer" DATA))))
  (close-output-port out))

and finally main.rkt, which hooks the client up to the server and runs for a while. Importantly, the file includes a usage of racket’s time form, which prints out the time that the block inside it takes to execute. This is what the benchmarking script will parse as the runtime.

#lang racket/base
 
(require (only-in "client.rkt" client)
         (only-in "server.rkt" server))
 
; ---------------------------------------------------------------------------------------------------
 
(define (main arg)
        (thread (lambda () (client arg)))
        (server))
 
(time (main 200000))
5.1.2 Typed

Fortunately, it is not difficult to add types to the code above. Put all of the following files in the benchmarks/echo/typed/ directory. The only changes we need to make are to use typed racket and add a few annotations.

First, we annotate the constants in constants.rkt:
#lang typed/racket/base
 
(provide
  ; Natural number port number to run the echo server on
  PORT
  ; String message to send over the tcp connection
  DATA)
 
(: PORT Natural)
(define PORT 8887)
 
(: DATA String)
(define DATA "Hello there sailor\n")

For server.rkt we add an annotation, and we annotate our import of constants.rkt for the cases where server.rkt is typed and constants.rkt is untyped:
#lang typed/racket/base
 
; TCP server: read from a buffer until end of file.
 
(provide server)
 
(require benchmark-util
         (only-in racket/tcp tcp-accept tcp-listen))
 
(require/typed/check "constants.rkt"
                     [PORT Natural]
                     [DATA String])
 
; ---------------------------------------------------------------------------------------------------
(: server (-> Void))
(define (server)
  (define-values (in out) (tcp-accept (tcp-listen PORT 5 #t)))
  (define buffer (make-string (string-length DATA)))
  (file-stream-buffer-mode out 'none)
  (let loop ([i (read-string! buffer in)]
             [bytes 0])
    (cond [(not (eof-object? i))
           (display buffer out)
           (loop (read-string! buffer in)
             (+ bytes (string-length buffer)))]
          [else (printf "server processed ~a bytes\n" bytes)])))

client.rkt is similar:
#lang typed/racket/base
 
; TCP client bot: loop for a fixed number of iterations
; sending a message over a port.
; The message and port are defined in constants.rkt
 
(provide client)
 
(require benchmark-util
         (only-in racket/tcp tcp-connect))
 
(require/typed/check "constants.rkt"
  [PORT Natural]
  [DATA String])
 
; ---------------------------------------------------------------------------------------------------
 
; `client n` loop for `n` iterations, sending a constant message on a constant port.
(: client (-> Natural Void))
(define (client num-iters)
  (define-values (in out) (tcp-connect "127.0.0.1" PORT))
  (define buffer (make-string (string-length DATA)))
  (file-stream-buffer-mode out 'none)
  (for ([n num-iters])
    (display DATA out)
    (read-string! buffer in)
    (unless (equal? DATA buffer)
        (error (format "Unexpected data ~e in buffer" DATA))))
  (close-output-port out))

As is main.rkt:
#lang typed/racket/base
 
(require benchmark-util)
 
(require/typed/check "client.rkt"
  [client (-> Natural Void)])
 
(require/typed/check "server.rkt"
  [server (-> Void)])
 
; ---------------------------------------------------------------------------------------------------
 
(: main (-> Natural Void))
(define (main arg)
  (thread (lambda () (client arg)))
  (server))
 
(time (main 200000))

5.2 Running the Benchmark

As before we can use ./run.sh benchmarks/echo to run our benchmark. The raw data is now in benchmarks/echo.rktd, if you read it you see the numbers are basically all the same, so there’s very little overhead from typing.

You can see the LNM graph at benchmarks/echo.png. You’ll just see a blue line at the top since all configurations work here.

For another analysis, we can run racket tools/data-lattice.rkt benchmarks/echo.rktd, which will show that no configurations have high overhead (all the numbers should be close to 1).