RetroNews

Parsing Prefixes

2010-05-16T14:59:00.002-04:00

Luke (docl in the #retro irc channel) has contributed an update to the prefix handling code that adds support for parsing prefixes. This allows for some interesting stuff we couldn't do before:

: __$ ( $-n ) hex >number decimal ;
: __% ( $-n ) binary >number decimal ;

These allow for parsing numbers as hex or binary, based on a prefix. Luke has gone much further though:

{
variable end
: pad 1024 here + ;
: keep ( - ) @compiler 0; drop keepString literal, ;
: >pad ( $-$ ) pad over getLength 1+ copy pad keep ;
: >$ ( n- ) !@end end ++ ;
: "? ( -f ) @end @1- char: " = ;
: chop ( $-$ ) end -- 0 !@end ;
: "; ( $-$ ) "? if chop >pad rdrop ;then ;
: __" ( "-a )
dup dup getLength + !end
"; 32 >$ repeat "; key dup emit >$ again ; parsing prefix
}

With this, we can do strings without a leading space:

: hello ( - ) "hello, world!" type cr ;

Luke's __" is now part of RetroForth 10.6. I've also added __$, __#, __%, and __' (parsing hex, decimal, binary, and single characters). This restores more of the functionality dropped between 9.3 and 10.0, and adds some new features to the table.

Prefixes

2010-05-02T18:12:00.003-04:00

The Rx Core now has prefixes, and they're better than ever before. For those who aren't familiar with them from Retro 9.x, a prefix is a single character that is placed before a name. The notfound handler will check to see if a word exists without the prefix; if so, it passes the xt of the word on the stack to a prefix handler word.

Ok, so in Retro we have five basic prefixes: @ ! + - & these correspond to fetch, store, increment by, decrement by, and address of.

@ and ! correspond to @ and ! (pretty obvious); + and - corespond to +! and -!; and & corresponds to either ' or ['].

Their usage is intended to help cleanup code flow a bit. We can do things like:

&words execute ( execute 'words' )

@here ( return the value at 'here' )

@foo +bar ( increment 'bar' by the value in 'foo' )

If you want to define more prefixes, it's easy to do. For instance, say we want a prefix that returns the dictionary header of a word. notfound will give us the xt, so we can lay down a stub:

: __~ ( a-d ) ; immediate prefix

Note the two _ characters. This is a common attribute of prefix names: the must be three characters long, and start with two underscores. The word prefix moves our word to the prefixes vocabulary. This isn't necessary, but does prove useful by making it easy to disable prefixes globally.

Ok, so filling in the details:

: __~ ( a-d ) xt->d ; immediate prefix

At first glance this looks ok. Testing it reveals a problem:

: __~ ( a-d ) xt->d ; immediate prefix
: foo ~words ;
foo .s
reset
foo .s

To fix this we need to compile as a literal into a definition. Still pretty easy as far as things go because .data will do this for us:

: __~ ( a-d ) xt->d .data ; immediate prefix
: foo ~words ;
foo .s
reset
foo .s

Now we have a working prefix. Take a look at at the implementation of the default prefixes to get ideas on implementing more complex ones. Careful use can enable significant cleanups to messy code, but in some cases it will be better to use discrete words to aid overall readability. Like all advanced features, prefixes should be learned, then used where they will help.

Fossil and Captchas

2010-05-01T20:51:00.001-04:00

I've begun moving to Fossil for my version control needs.

Rx Core: http://rx-core.org/dev/rx.fossil

Retro Forth: http://rx-core.org/dev/retroforth.fossil

The pastebin was clogged with spam posts. I deleted over 2k of them, and added a captcha to hopefully reduce the number of new ones.

Simplifying Things

2010-04-24T22:23:00.002-04:00

Now that the VM code has been condensed and simplified, I'm turning my focus to the image. No new features, but I want a simple build process, with things bound a bit more tightly.

The work in progress can be found at http://drop.io/retroforth

So far I have a working VM, single file for building a new base image, and makefile to automate the process. I've moved files, sockets, and canvas words into this, so the core stuff now includes all the basics needed. I'll probably move 'words' here as well.

Overall it seems to be working out nicely, so I'll keep going.

Shrinking the VM

2010-04-21T22:48:00.002-04:00

I'm working on a simplified C implementation of Ngaro; no major changes, but I'm hoping for something a bit easier to use as a base for future experiments. (And hopefully a bit easier to adapt as a skeleton for new ports, etc).

The new code is one file (retro.c), and is 841 lines long. It still has all the major stuff (console i/o, file i/o, sockets, etc), but drops the statistics tracking, endian conversion, and runtime disassembly tracing. It works nicely on my OS X box; I'll be testing on other platforms in the near future.

Expect to see this checked into the repository within the next day or so.

Online Demonstration

2010-04-17T22:18:00.001-04:00

It's live and all links point to it now.

The code for rebuilding the retroImage.js is now in the repository.

Preparing to deploy updated online demonstration

2010-04-17T10:41:00.002-04:00

One of the more fun parts of Retro is the online demo. Thanks to the Ngaro JS implementation, it's really easy to test Retro without needing to install it.

Unfortunately, the existing demo is now quite old. It's running a 10.2/10.3 hybrid image, which is far out of date. As part of the work done on 10.5, the various forthlets have been updated to run on the latest image, and I'm now doing final tests of a new image + current Ngaro at http://rx-core.org/jsvm/

Soon this will be rolled out to all mirrors and replace the existing demos. Also, it'll be integrated into the repo, so deploying updates should be easier in the future.

Decompiling code in 9.2.10

2010-04-17T10:17:00.002-04:00

Disclaimer: The following code sucks. Badly. Please don't consider this typical of what I write; it's a quick hack I threw together in the last hour or so using parts of earlier code written many years ago.

This is the start of the decompiler for Retro 9.2.10. It's based in part on a decompiler from 8.x, and has a few bits from earlier releases.

: h. ( n- ) base @ swap hex . base ! ;

Displays numbers in hexadecimal format, saving and restoring the base.

: lit 1 ;

' lit 1+ @ ' lit 1+ + cell+ constant dolit

This one provides a constant pointing to the internal routine that pushes numbers to the stack. It's definitely not portable, but works. A bit of explanation is in order. Retro 9.x and earlier were coded in assembly for x86 processors. The compiled code is (generally) a series of call, jump, and return instructions. But the calls/jumps are relative, so we can't just fetch the target, we need to adjust it to the actual base address. So:

Compiled code is basically:

call dolit

dd 1

ret

Call is one byte on x86, so: ' lit 1+ will get us to the target (4 bytes). We @ this, then add the address of the call target (' lit 1+ +), and finally adjust by the size of the call target (cell+). Kind of messy, but it works.

variable xt

variable d

variable flag

: scan ( - )

last repeat @ 0; dup :xt @ xt @ =if d ! -1 flag ! ;then again

;

: xt->d ( a-df )

xt ! 0 flag ! 0 d ! scan d @ flag @ ;

To resolve names, we have a word to get the dictionary entry and a success/failure flag from a base address. The use of variables here is due to laziness.

variable addr

This holds the current address being decompiled.

: shift ( n- ) addr +! ;

Advance the addr variable by some arbitrary number of bytes. Each handler will invoke this to cover the byte range being decompiled.

: .name ( - )

1 shift here addr @ @ addr @ + cell+

dup dolit =if ." <>" drop cr ;then

dup xt->d

if :name unpack type cr ;then drop h. cr ;

An ugly way to display either a word name or the address being called/jumped to.

: tab 9 emit ;

: comment tab ." ( " later ." )" cr ;

Output formatting. Ideally tab would be more intelligent and try to keep columns lined up properly, but that's not crucial. The comment word uses later to wrap output in parenthesis.

: call, ." call " .name 4 shift ;

: jmp, ." jmp " .name 4 shift ;

: ret, ." ret" cr 1 shift ;

: lodsd, ." lodsd" tab comment ." drop" 1 shift ;

: inc, ." inc eax" comment ." 1+" 1 shift ;

: dec, ." dec eax" comment ." 1-" 1 shift ;

: stc, ." stc" comment ." false" 1 shift ;

: clc, ." clc" comment ." true" 1 shift ;

: push, ." push eax" 1 shift comment ." >r";

: mov, ." mov [esi],eax" 2 shift cr ;

: pop, ." pop eax" 1 shift comment ." r>" ;

: or, ." or eax, eax" 2 shift comment ." or" ;

: swap, 1+ ." xchg eax, [esi]" comment ." swap" 2 shift ;

: nip, 2 + ." add esi, 4" comment ." nip" 3 shift ;

: .name 1 shift here addr @ @ addr @ + cell+ h. ;

: jz, ." jz " .name ." <>if" 1 shift cr ;

: jng, ." jng " .name ."

: jnz, ." jnz " .name ." =if" 1 shift cr ;

: jnl, ." jnl " .name ." >if" 1 shift cr ;

: cmp, ." cmp eax, [esi]" 1+ comment ." (if)" 2 shift ;

: nop, ." nop" comment ." then" 1 shift ;

: add, ." add eax,[esi]" comment 2 shift ;

: sub, ." sub [esi],eax" comment 2 shift ;

: mul, ." mul dword [esi]" comment 2 shift ;

: show_it dup c@ >r dup 1+ r type r> dup 1+ shift + ;

: string, ." string" tab comment show_it ;

A bunch of words to handle various sequences used. This was all ripped (with minor changes) from 8.x.

: lookup ( n-n )

addr @ h. tab

dup $e8 =if call, ;then

dup $e9 =if jmp, ;then

dup $c3 =if ret, ;then

dup $ad =if lodsd, ;then

dup $40 =if inc, ;then

dup $48 =if dec, ;then

dup $f9 =if stc, ;then

dup $f8 =if clc, ;then

dup $50 =if push, ;then

dup $89 =if mov, ;then

dup $58 =if pop, ;then

dup $09 =if or, ;then

dup $87 =if swap, ;then

dup $83 =if nip, ;then

dup $74 =if jz, ;then

dup $7e =if jng, ;then

dup $75 =if jnz, ;then

dup $7d =if jnl, ;then

dup $3b =if cmp, ;then

dup $90 =if nop, ;then

dup $03 =if add, ;then

dup $29 =if sub, ;then

dup $f7 =if mul, ;then

dup $eb =if string, ;then

." Unknown: " dup h. 1 shift cr ;

I really hate this part. This is a huge if/then block to call the handlers for each starting byte. It'd be better to use a jump table (which was done in 8.x), but this was quicker to implement for testing purposes.

: see ( n"- ) ' addr ! for addr @ c@ lookup drop next ;

A wrapper word to invoke the decompiler. Pass it the number of instructions to decompile, and follow it by the name of a word:

: big 1 2 + . ;

5 foo big

Ok, so that's that. It's at least partially functional, but still has lots of work needed.

A bit of life in 9.x

2010-04-12T22:54:00.002-04:00

A user on the comp.lang.forth is working on a fork of the 9.x codebase with a focus on moving even more out of the initial assembly kernel. While I'm not supporting continued use of 9.x; I am writing a crude decompiler/disassembler for use with the fork.

More on Prefixes

2010-04-11T09:24:00.002-04:00

In the last post, I briefly touched on the overall implementation of prefixes. I'll try to cover them in a bit more detail now.

Retro's listener will call notfound when a word name is not found, and when the input token can not be converted to a number in the current base. Prior to 10.6, notfound just displays some sort of error message. In 10.6, it has been rewritten and extended to do additional checks related to prefixes.

The code:

label: ___ " ___" $,

t: get ( $-$ ) dup, @, ___ # 2 # +, !, 1+, ;

t: try ( - )

TIB # get find

if d->xt @, ___ # find

if dup, d->xt @, swap, d->class @, with-class

pop, pop, 2drop ; then

drop,

then drop, ;

t: filter ( - ) TIB # getLength 2 # >if try then ;

t: notfound ( - ) filter char: ? # emit cr ;

Ok, first thing to note is that the syntax looks a bit unusual. This is written using the metacompiler, which introduces some new rules:

t: instead of :
# following numbers/constants
, following opcode names

Getting this out of the way, we can proceed to break it down.

label: ___ " ___" $,

This line creates a constant pointing to a string containing three underscores. This string is modified later, with the last underscore replaced by a prefix character.

t: get ( $-$ ) dup, @, ___ # 2 # +, !, 1+, ;

The get receives a string from the stack; moves the first character into the last position of ___ and then returns the rest of the provided string.

t: try ( - )

TIB # get find

if d->xt @, ___ # find

if dup, d->xt @, swap, d->class @, with-class

pop, pop, 2drop ; then

drop,

then drop, ;

This is the largest part of the prefix handling. It passes the token in tib to get and then searches for the name (sans the prefix character) in the dictionary.

If found, the xt of the word is returned, and a search is done for __X, where X is the prefix character. If this is found, the xt and class of the prefix handler are pushed to the stack, and with-class is called to execute everything. Also, if found, control is passed directly back to the listener.

If not found, the stack is cleaned up and control passes back to the notfound handler.

t: filter ( - ) TIB # getLength 2 # >if try then ;

To ensure that the prefixes are not searched for with single character tokens we have a simple filter word. A subtle note here: >if and in Retro are inclusive, so this is actually checking for 2 or greater length, not length greater than 2.

t: notfound ( - ) filter char: ? # emit cr ;

And finally the notfound word itself. It calls filter, which then calls try, and so on. If the token + prefix are not found, a ? is displayed and control is returned to the listener.

Ok, now for the prefixes that are provided by default:

t: __& ( a-n ) .data ;

Usage: &name
Action: Return the address (xt) of the word.
Similar To: ['] name

t: __+ ( a- ) .data ' +! # .word ;

Usage: +name

Action: Add a value from the stack to the value stored at name's xt field.

Similar To: name +!

t: __- ( a- ) .data ' -! # .word ;

Usage: -name

Action: Subtract a value from the value stored at name's xt field.

Similar To: name -!

t: __@ ( a-n ) .data ' @ # .word ;

Usage: @name

Action: Return the value stored at the address (xt) field of the word.

Similar To: name @

t: __! ( a-n ) .data ' ! # .word ;

Usage: !name

Action: Store a value to the address field of a name

Similar To: name !

If you want to create your own prefixes, you can do this as well. For a simple example:

: __~ ( a- ) .data ` :see ; immediate prefix

This will create a new ~ prefix that shows decompiled code for the word. Note the prefix at the end of the definition. This is optional, but moves the new prefix handler to the prefixes vocabulary, allowing it to be grouped cleanly with the other prefixes.

Prefixes

2010-04-10T21:10:00.002-04:00

I introduced prefixes in the 9.x releases. They were a basic shortcut technique: add a single character before a name to alter the behavior. The original code used a non-editable table for determining which characters were valid and the new behaviors the prefixes would introduce. When adding prefixes to the current Retro, I took a different approach.

Prefixes are handled by words, with a consistent naming format: __X where X is the prefix character. I also added a prefixes vocabulary, allowing the prefixes to be toggled on and off simply.

The notfound error handler was extended to search for prefix words if a name isn't found in the dictionary. If the prefix is found, and the rest of the name is found, the word is passed to the prefix handler.

The end result is a clean, extensible way of handling, adding, and replacing prefixes. Plus, they can easily be disabled or enabled as desired.

The State of the Forth - April 2010

2010-04-10T16:16:00.002-04:00

Though I haven't updated this blog in almost a year, I am still actively working on Retro.

The stable release is now 10.5. Since the last update here, Retro has seen a lot of major changes:

profiling
files
sockets
fully cross-platform canvas (javascript, sdl)
nestable vocabularies
time device
reorganized library
new examples

Platform support currently stands at:

Windows (via Mono and .NET)
Cellphones (J2ME/MIDP)
Linux
BSD
BeOS
MacOS X
AIX
iPhoneOS (jailbroke devices)
Browsers (Opera, FireFox, Safari, Chrome)

There is also an Emacs Lisp implementation that can run 10.4 and older images; I hope to have it updated for 10.6 and beyond in the near future.

10.6 (in the new git repository) has added support for word prefixes, and some cleanups to the core.

10.1, and some new stuff

2009-06-04T14:35:00.002-04:00

10.1 is out. Nothing major, but it has some cleanups and bugfixes. It's worth switching to IMO.

Beyond that, 10.2 is being worked on. The development branches (now with live snapshots) has a .NET implementation of the VM, reduced memory requirements, new tools to shrink a saved image, and the start of a debugging tool (already capable of decompiling most compiled code). In the optional modules, I've started work on extending the compiler to generate smaller, faster code.

does>

2009-04-28T13:11:00.002-04:00

A few days ago, someone in the IRC channel asked about implementing does>
The code below provides an implementation, though it does require my extensions
package to be loaded first.

{{
  : reclass last @ d->class ! ;
  : .does compiler @ -1 =if swap literal, compile pop drop ;; then drop ;
---reveal---
  macro: does>
    1 , here 0 ,         ( compile address of code after does> [will be patched] )
    ['] reclass compile  ( compile a call to reclass, which assigns the code     )
                         ( after does> as the class handler for the word.        )
    ['] ;; execute       ( compile an exit, so no code following does> will be   )
                         ( executed when the create/does> sequence is run.       )
    here swap !          ( Patch the address from the 1 , here 0 , line to the   )
                         ( actual start of the code for the does> action.        )
    here literal,        ( Compile the address following does> as a literal so   )
                         ( it can be used by the .does class.                    )
    ['] .does compile    ( And finally, compile a call to the .does class, which )
                         ( will take care of compile-time and interpret-time bits)
  ;
}}

Back on IRC

2009-02-18T13:58:00.002-05:00

After a long break from IRC due to connectivity problems, I will now be back on a regular basis.

Even better, I now have an irc-via-email client, which means that I am now able to monitor the #retro channel continually throughout the day, so if you have questions, I'll be able to respond far more quickly.

Misc. Updates and an introduction to Motu

2009-01-25T19:17:00.004-05:00

10.0 (Stable) has been out for a while now, and I'm nearing completion on 10.0.1 (documentation updates to cover missing words, a fixed version of ' and movement of a few state variables to specific places at the start of the image).

In addition, I've been working on reviving an app I first created back in the Retro 7 days. The current incarnation of this is Motu, a notepad with a Retro system built in. I used to use it to keep notes and bits of code I was working on together.

Motu is implemented as a client-side web app, storing data in a cookie, and allowing trial and debugging of code. I've tested it in Safari, FireFox, and Chrome. (It will *not* work in IE). Chrome users need to start Chrome with the --enable-file-cookies flag for this to work.

Motu uses a 10.0.1 (development) image, and needs access to the state variables I mentioned earlier (it uses them to allow resetting the image to a default state without a restart).

You can get a copy of Motu from the git repository:

git clone git://github.com/crcx/motu

GIT repository open (at last)

2008-12-24T23:55:00.002-05:00

I've still had problems with BerliOS, so I setup a repository at github.com

To checkout a copy:

git clone git://github.com/crcx/retro10

This is up to date with my latest source, so anyone with patches pending should check them against this and then send them in.

For inclusion in the 10.0-stable release, I'll be needing the code/patches by Dec. 31st. (The release should be finished and made public by January 5, 2009)

Nearly there

2008-12-23T18:58:00.002-05:00

I'm almost done cleaning up the source tree for the 10.0-stable release.

It'll include: C implementations (mine and Mat's) for console and framebuffer, javascript vm, Toka, Retro source, and docs.

The library will be made into a second download.

Binaries will be provided for various systems. If you have requests please let me know so I can obtain the relevant systems for building.

Getting closer to a stable release

2008-11-30T15:15:00.003-05:00

I'm finally planning to do a stable release (hopefully in late December or early January) for the 10th generation of Retro.

It's almost ready. The core language is solid, and will see few if any changes. The library still needs works, but it's getting closer. The documentation is a bit light, but has been well received by those who have read it.

A big step forward: as of today a separate image file is no longer needed to get started with Retro. The VM will still try to load a retroImage (or another file, if specified on the command line), but if one can't be found, it'll initialize the stable image from an internal function. If you use the internal image and save your session, a new separate retroImage will be created.

For those wanting to try it out, grab a copy from http://rx-core.org/retro-10.tar.gz and run 'make vm'.

Another update: I'm working on setting up a code repository at Berlios, hopefully this will be ready later this week. This repo will be open to all contributors, so keep an eye out for the announcement when it's ready.

Some notes on my block editor

2008-11-07T14:32:00.005-05:00

In my previous post I mentioned my block editor. This is a small editor that I've used for several years now.

My image contains 1024 blocks of 512 characters each. The blocks are formatted as 8 lines, with 64 character per line. The editor allows the user to enter text on each line, or at a particular place in the line. You can delete lines, or the entire block. It is possible to evaluate code contained in a block. Editing is done through a set of short words.

Though primitive, it's served me well through the years. I see it as a decent starting point for a better editor (in fact, in older Retro systems, this editor was extended to have a VI-style modal editing model and a syntax-highlighted viewer). I'll post more details on it in the near future.

Things I'm working on...

2008-10-30T15:42:00.002-04:00

I've been having trouble with my internet connection, so here's an overview of what I'm working on:

Core

Exposed ok (to allow control of the prompt)

Expanded the forth94 (ANS FORTH) layer. Now more compliant with the spec :)

Additions

extras/bootstrap.retro
extras/retro.nanorc
library/debug.retro
library/does.retro
library/date.retro
library/math.retro
library/value.retro

Removals

core: removed unused bits left over from old compile-time module support
cross compiler: removed unused bits left over from old compile-time module support
library/square-root.retro [merged with library/math.retro]

Documentation

Tutorial

The bootstrap.reto is my personal startup file, which includes the block editor I've used for the last several years.

Release 2008.10.22

2008-10-22T13:52:00.002-04:00

I've finally finished work on a new build. This has a few changes and should help me make it easier to extend Retro in the future.

Three VMs included

Console [curses backend]
Framebuffer [SDL backend]
JavaScript

Major Bugs Fixed:

Saving of images now works reliably on Windows
Scrolling works properly in the console build

Compile-time modules replaced by a new library of modules that can be loaded from the command line
The core now reserves a smaller piece of memory for its own use (10k rather than 32k), leaving a little more room for applications
Small improvements to the 1994 ANS FORTH layer
Documentation has received some minor updates to formatting and a little explanation of how to load modules at the command line.

Documentation is included in PDF and ePub formats. Additional formats can be found on feedbooks.com

This release also has an updated binary package with console binaries for Linux (x86-32, x86-64, and Alpha), MacOS X (Intel), and Windows.

Release 2008.09.21

2008-09-20T20:43:00.008-04:00

Major changes in this release

Support for loading code from a normal text file
This only works for console implementation
Use ./ngaro --with filename
Improved screen clearing
Trying to display a negative character code clears the display.
This allows a cleaner, more portable way of clearing the display without relying on particular escape sequences being emulated.
Console source trees merged
Cleanups to I/O handling in the console implementation
Curses-backend used for console implementation by default

The most significant of these is the addition of --with to the console implementation.

The binary package will be updated within the next few days.

Binary Package

2008-09-01T20:30:00.003-04:00

I just uploaded a set of binaries for Retro. This contains the standard image, the current documentation, and prebuilt copies of the Ngaro VM for many systems.

Console

Linux (x86, x86-64, and Alpha)
Mac OS X (Intel)

Framebuffer

Windows (x86, tested on XP SP3)

The Linux binaries are statically linked, so should work on pretty much any distro for the listed architectures. The OS X build is dynamically linked and built on OS X 10.5. The Windows build includes the SDL.dll and should work on most Windows systems without problems.

I couldn't produce static cross-platform binaries with the framebuffer as some of the Linux systems don't have the SDL libraries and the one that does lacks support for building static SDL applications. For OS X, the framebuffer support is not working yet.

I'd like to expand the number of platforms I can provide binaries for sometime soon. (Perhaps I'll try to get my old box with the BSD's on it up and running again.). For now this should make trying Retro even easier.

Release 2008.08.31

2008-08-31T10:27:00.002-04:00

Well, I've finally pushed out a new release.

This is primarily a cleanup of the core source tree. Some highlights:

Build scripts replaced by Makefiles
Documentation is now included
The default image name is now retroImage
VM's attempt to load retroImage if no image name is specified
Win32 implementation now separate from the standard framebuffer source
Assembler is now separate from the Retro source
Modules renamed and organized better

In addition to the above changes, I've revamped the site a little. It now makes trying Retro easier through use of the javascript implementation of Ngaro.

Also new is an image builder. Rather than provide a static image, this allows you to select the modules you want in your custom image.

Release 2008.07.27

2008-07-27T13:20:00.002-04:00

I've pushed the latest release (2008.07.27) online. This has a few nice improvments, including inlining of primitive words, the latest JavaScript implementation of Ngaro, and lots of internal cleanups.

Due to changes on repo.or.cz, I've been unable to push changes to the GIT repository for a while now. I'm looking for a solution, and hope to be able to resume putting changes into a VCS soon.

The Windows installer has not been updated yet due to hardware problems on the single Windows box I have. I'm hoping to be able to have an updated installer later this week.

The Dashboard widget and Retro Online are now running the latest code with improved reliability and improved interfaces. Users of the Dashboard widget should definitely update as this fixes some resource usage issues and offers significantly greater performance.

Anyway, download the latest snapshot or image and enjoy.

Retro Widget for Apple's Dashboard

2008-07-21T17:04:00.001-04:00

I've adapted the 3rd generation of the Javascript implementation of Ngaro to run as a Dashboard widget. It's running a standard image, and works quite nicely.

If this proves useful, I'll look into supporting other widget platforms in the future.

Get the Retro Dashboard Widget

Misc. Updates

2008-07-19T22:32:00.002-04:00

Due to some ongoing health issues, I haven't been as active as I'd like. Hopefully these will be resolved soon and I'll be able to do some serious work on supporting the extended Ngaro VM, expanding the modules, and further improving the documentation.

For now, a few minor updates:

I've begun updating http://ngaro.jottit.com using the material from the Retro docs as a base. It can now be edited by anyone, so all those working on Ngaro implementations can contribute.
jsvm3: An updated version of the Ngaro in Javascript, this is primarily based on an updated I/O model that allows an even closer approximation of the console builds. This is new, and needs testing.

Improving Performance

2008-07-07T17:01:00.002-04:00

While looking over Mat's extensions to Ngaro I had a thought about inlining the handful of words which correspond to Ngaro instructions. A few tests showed enough promise that I've added a new .inline class to Retro.

The primitives will only be inlined if they are not revectored. This ensures that they will behave as closely to normal Forth words as possible. Devectoring a .inline word will restore the original behaviour. This is not intended for user code, and is only useful if you are adding new opcodes to your Ngaro VM.

Anyway, a look at the benchmarks:

Switched VM

Before:
- Recursive FIB (39)... 208.0144 seconds
- Countdown Loop (1,000,000)... 12.228482 seconds
- Nest/Unnest (256 million pairs)... 42.299306 seconds

Now:
- Recursive FIB (39)... 85.860101 seconds
- Countdown Loop (1,000,000)... 12.38329 seconds
- Nest/Unnest (256 million pairs)... 30.796164 seconds

Threaded VM

Before:
- Recursive FIB (39)... 130.537818 seconds
- Countdown Loop (1,000,000)... 8.325312 seconds
- Nest/Unnest (256 million pairs)... 27.092447 seconds

Now:
- Recursive FIB (39)... 51.989871 seconds
- Countdown Loop (1,000,000)... 8.292265 seconds
- Nest/Unnest (256 million pairs)... 17.577663 seconds

I'll update the documentation to cover the new class in the near future. If you don't want to wait for a new release, you can get this code from the GIT repository. See http://repo.or.cz/w/retro.git for details.

Site Changes

2008-07-06T12:09:00.002-04:00

I've begun reworking the layout of the Retro website. This should be completed within the next few days.

Some highlights of the new design so far:

Cleaner organization
Downloads on the right, will remain easier to spot
Automatically updated links to recent discussions on the mailing list and recent commits to the main GIT repository. (Updated hourly)

Coming soon:

New documentation page, which will allow downloading the documentation in various formats (text, word doc, pdf, html) as well as viewing it online.
The News page will have links to the blogs of those involved in Retro's development, as well as displaying the most recent posts (updated hourly) from each blog.

Possibilities:

Web access to the #retro channel on irc.freenode.net

Release 2008.06.29

2008-06-29T19:00:00.003-04:00

This is a minor update with the updated word class handling (so you can try code from my previous posts) and a few minor fixes and improvements to the source. I've also updated the online interpreter to use both the latest Ngaro.js and image.

Switched to GIT

2008-06-25T21:45:00.002-04:00

I've switched to using GIT for the Retro source repository. Relevant information follows:

To obtain a copy:

git clone git://repo.or.cz/retro.git

The repository can be browsed at:

http://repo.or.cz/w/retro.git

Adding New Word Classes

2008-06-19T23:12:00.003-04:00

In my previous post I discussed the concept of word classes. In this post I'll show how to define and add a word class to Retro. This assumes that you know Forth, and have some experience with Retro. Consult the documentation if you have questions on the words used.

For this example, we'll create a class for strings with the following behaviour:

If interpreting, display the string
If compiling, compile the code needed to display the string

Retro has a convention of using a . as the first character of a class name. We'll call our new class .string

On entry to a class handler, the address (xt) of the word or data element is on the stack. The compiler state (often important to class handlers) is stored in a variable named compiler. With this in mind, we'll define our class:

: .string ( xt -- )
compiler @ 0 =if type ;; then
1 , , ['] type compile ;

The compile-time portion is a bit trickier than the interpret time since it has to lay down the proper code in the target definition. In this case a LIT instruction (opcode 1) is laid down, followed by the xt of the string. This is followed by code to lay down a call to type.

We now need a creator word to attach this class to a value. For this example we'll define displayString: to take a form like the one shown below.

string-addr displayString: name

New dictionary entries are made using create, so we'll use that and change the class to our new .string handler.

: displayString: ( string-addr -- )
create ['] .string last @ d->class !
keepString last @ d->xt ! ;

This uses create to make a new word, then sets the class to .string and the xt of the word to the string. It also makes the string permanent using keepString. last is a variable pointing to the most recently created dictionary entry. The words d->class and d->xt are dictionary field accessors and are used to provide portable access to fields in the dictionary.

We can now test our new class:

" hello, world!" displayString: hello
hello
: foo hello cr ; foo

Introduction to Word Classes

2008-06-19T19:15:00.005-04:00

There are many ways to code a Forth interpreter. When I inherited Retro, there were two separate dictionaries, one for normal words and data, and one for compiler macros. There were two internal token processing loops, one for interpret time and one for compile time. This lead to some confusion as the compiler macro dictionary was only visible when compiling words, and preceeded the normal dictionary. When new functionality (inlining, etc) was added, more dictionaries and internal loops had to be coded.

Some Forths solve this by having flags in the dictionary headers and having the token processor use these to select the proper functionality. This still requires changes to the kernel to handle new flags and add the behaviour in the proper places.

Helmar Wodtke introduced me to a simpler approach which he calls word classes. In this approach each dictionary entry gets an additional value, a pointer to a class handler. Class handlers are just words that take an execution token (in Retro this is just the address of a word) and do something with it.

A default Retro build has three classes:

.word
At compile time, compile a call to the word. At interpret time, execute the word.
.macro
Execute the word. The word may compile code or add data into the heap based on any form of system state.
.data
Return the address or value associated with the requested token.

These cover the core functionality required for a small Forth implementation.

In the next post we'll look at how to define new classes and use them to extend Retro's functionality.

Release 2008.06.14

2008-06-14T19:14:00.003-04:00

I've finally pushed a new release. This is current as of 7:15 pm (EST) and includes the stable releases of Ngaro (C, JavaScript) as well as current source and build scripts.

I also updated the binary release for Windows. It's now an MSI installer package.

Some of the more signficant enhancments:

Ngaro
Gained a new I/O port for queries of emulated devices
C implementations now have more flexible command line processing
Updated threaded C implementation
Retro
Minor bugfixes
All non-data words are now vectors
Now detects hardware settings using the new I/O port in Ngaro
Word classes exposed
Updated benchmarks

The documentation has been updated to reflect the changes.

Exposing Word Classes

2008-06-12T22:24:00.002-04:00

I've been using word classes in Retro 10's implementation since the beginning. This allows a simple, consistent means of controlling the behavior of words.

Until today it was not possible to define a custom word class. Now you can, but it's not automatic. You'll have to redefine with-class, but take care to handle the existing classes (.word, .macro, and .data).

I'll try to document this sometime soon, but take a look in the Retro source to see how it's done if you wish to experiment with it. The code supporting this is in the SVN repository.

Ngaro Updates

2008-06-12T19:05:00.003-04:00

With recent work on Ngaro it's become possible to run Retro on many smaller systems (including phones and pdas). This has lead to a need for supporting smaller memory configurations and screen sizes.

To help reduce the work involved involved in adapting Retro to these devices, I've set I/O port 5 as a hardware query device. This can be queried by writing specific values, waiting, then reading back the result.

Since some implementations are likely to support optional hardware devices in the future, they can use this to provide Retro with information about these devices.

In the SVN repository the C implementations of Ngaro have been adapted to support this. I'm working on the JavaScript implementations and should have them ready tomorrow.

The initial capabilities that can be detected are:

-1 Get Amount of Memory
-2 Get Framebuffer Address (0 if not supported)
-3 Get Framebuffer Width (0 if not supported)
-4 Get Framebuffer Height (0 if not supported)

I've updated the Retro source to detect the framebuffer settings on startup and use them rather than relying on hardcoded values. Retro also detects the amount of memory being provided, but the memory map still needs to be configured in the source.

Improving Ngaro in JavaScript

2008-06-06T10:37:00.002-04:00

Ngaro in JavaScript (jsvm) has been working nicely for quite a while now. Today I did a little more work to help improve the interface. It's now using the innerhtml to remove the textarea for the output. I've tweaked the ngaro.js a bit to handle newlines properly with this. The tracing and error log is not supported yet, but I expect to work that out in the near future.

Feel free to take a look.

Retro meets J2ME

2008-06-06T09:27:00.002-04:00

Martin Polak has ported the Ngaro VM to mobile devices with J2ME support. This is another great step forward as it makes Retro even more accessible to developers.

The sources are in the subversion repository and I'm working on making it buildable using Retro's standard build scripts.

Martin has provided a source and binary package at http://www.nigol.cz/retro/Retroforth.zip.

To try building it you'll need to be using MacOS X and Ant. Of course, you'll also need the relevant Java libraries and development tools.

Benchmarking Retro

2008-05-25T20:57:00.003-04:00

Retro now has a small, but working, set of benchmarks. This is in the svn repo and can be found in bin/bench. To use the benchmarks, you'll need a console build of Ngaro, a Retro image, a Unix-like host system, and Perl.

  cd bin/bench
./make-image
./run-bench

The timings for each test will be displayed when the test completes. There are currently three tests:

Recursive FIB
Countdown Loop
Call/Return (nesting) Pairs

I expect to see this grow into a more complete set of benchmarks in the future. For now, this should suffice for basic performance testing.

Initial results:

Host: ZenWalk 5 (Linux)
GCC: 4.1.2
CPU: Pentium 4 2.8GHz (1 core)
RAM: 1.5gb

Switched VM:
Recursive FIB (39)... 225.540318 seconds
Countdown Loop (1,000,000)... 13.616979 seconds
Nest/Unnest (256 million pairs)... 36.356847 seconds

Threaded VM:
Recursive FIB (39)... 212.555673 seconds
Countdown Loop (1,000,000)... 13.995988 seconds
Nest/Unnest (256 million pairs)... 33.026269 seconds

Updated Documentation

2008-05-25T19:50:00.002-04:00

I have finished the initial updates to Retro's documentation. This is an ongoing process, in which I have been merging all of my notes and separate pieces of documentation into a single, cohesive reference.

Many sections have been expanded, and some new areas were added. I think it's a definite step forward and invite everyone to Check it out.

Loading text files into an image

2008-05-24T11:18:00.003-04:00

While Retro has no direct access to the host system, it is possible to load text files into an image and then save the results, or use it interactively. The process assumes that you are running on some form of Unix. It's pretty simple:

./build
cd bin
cat filename | ./ngaro forth.image

Or, for interactive mode:


cat filename - | ./ngaro forth.image

There is a limiting factor. The standard Retro image only recognizes space as valid whitespace, so any use of tabs or newlines will cause problems. This can be fixed by starting your code with the following line. This should be one single line. Be sure to include a space at the end of the line:

: fix dup 10 =if drop 32 then dup 13 =if drop 32 then
dup 9 =if drop 32 then ; ' fix is (remap-keys)

If you want, this can be disabled later by doing:

devector remap-keys

If you aren't going to drop to an interactive prompt, end with bye. Be sure to save your state if you want any changes to be present when you next load your Retro image.

Threaded Execution in the Ngaro VM

2008-05-23T18:49:00.003-04:00

Thanks to Matthias Schirm there's a new implementation of Ngaro. It's written in C, and supports the text console devices. His implementation uses threaded execution, which is faster. He is working on further performance improvements. (This will likely become the default console build later this summer.) At present it does not work with the SDL-backed I/O; I will try to figure out why later on.

I've not done any significant benchmarking, but he did provide some numbers:

  : test dup for dup . CR 1 – next ; 1000000 test bye

ngaro (old):

  real  [s] 35,47
user [s] 8,27
sys  [s] 1,07

ngaro (new):

 real: [s]  35,13
user [s]  7,46
sys  [s]  1,34

  : fib dup 1 >if dup 1 RECURSE swap 2 RECURSE + EXIT then drop 1 ; 39 fib bye

ngaro (old):

 real  [s]  131,23
user [s]  131,21
sys  [s]  0,03

ngaro (new):

 real  [s]  77,95
user [s]  77,67
sys  [s]  0,04

This is a clear improvement and should continue to get better. I've begun to work on a small benchmark package; once done it'll be easier to test performance tweaks.

Retro on Windows

2008-05-11T19:04:00.003-04:00

Retro has been supported on Windows for a few weeks now. My access to Windows has been very limited, but I finally have a working XP install on my main development box.

I'll be revising the Windows Build Instructions later today, to help clarify a few points and point out the exact files I'm using. In addition, I've built a new installable package for Retro 10. This is a standard MSI package, so should install on XP and Vista machines without problems. It includes all necessary dependencies. I expect to replace the binary package on retroforth.org with this later this week.

Release 2008.05.10

2008-05-10T13:34:00.005-04:00

This is a pretty small release. I've been very busy (and a bit sick), so haven't had time to do much.

The video refresh port is now set properly after outputting a character
Ngaro JS now supports the video refresh port
Ngaro JS now stores the video output apart from the form output textarea
Strings are now rendered after all characters are drawn to the video buffer rather than character by character
update (enable/disable screen redraws) was added
redraw (redraw the screen if update is on) is now in the dictionary
Graphics module now does video updates after drawing shapes
Removed the FFI from Toka.

As shown, most of my time has been spent in enabling proper use of the video refresh support provided by Ngaro. This allows a noticeable performance increase in bits that are text heavy (try words) by delaying renders until the output is finished. It's also helpful with graphics, allowing filled shapes to be rendered more quickly.

I also removed the FFI functionality from Toka for this release. The FFI words are not used by the cross-compiler, and serves only to slow down the build of Toka and increase the number of dependencies. (This only affects the copy of Toka included in the Retro package; the full Toka still has FFI).

Retro on Mobile Devices

2008-05-04T10:32:00.006-04:00

I have scaled down the UI elements in Ngaro JS and am now able to run Retro on some mobile devices.

I have confirmation that it works properly on (platform, browser):

Windows Mobile 5 (Minimo)
Apple iPhone (Mobile Safari)

You can check out Retro Mobile online, or download the files to copy to your device for offline use.

Links:

Retro Mobile (http://retroforth.org/mobile)
Download Retro Mobile (zip, http://retroforth.org/mobile/mobile.zip)
Download Retro Mobile (tar.gz, http://retroforth.org/mobile/mobile.tar.gz)
Minimo (http://www.mozilla.org/projects/minimo/)

Release 2008.05.03

2008-05-03T17:23:00.002-04:00

This release primarily focuses on the build system, but does bring some minor updates to Retro, the modules, and Ngaro.

The build system:

Refactored, now has separate functions for each step of the build system
The clean script has been merged in
Accepts various command line options allowing more control over the build
Basic help (pass --help) to see
Several bugs were fixed in fix-image
Memory usage of fix-image was reduced
image2js.toka is now commented

For the modules:

COUNT and R@ were added to forth94 and forth83.

Retro:

for / next loops have been added

And as finally, Ngaro:

Added a Clear Display button.

Nothing too impressive, but the updated build script will be easier for me to maintain and extend in the future.

Loops

2008-05-03T10:17:00.004-04:00

I've implemented simple for/next loops now. The code below can be entered into a running Retro instance, then you can save the image to get them. (They will also be in the release due later today).

macro: for here 5 , ;
macro: next 6 , 27 , 25 , 8 , , ;

For those who are curious, this involves two compiler macros, which inline raw machine code into the words using the macros. As an example:

: foo 10 for 98 emit next ;

This will compile to:

foo:
lit 10

foo.1: ( 'for' )
push

lit 98
call emit

pop ( 'next' )
1-
0;
jump foo.1

;

There is a limitation that should be mentioned: when the loop counter reaches zero, the word will exit. Factor loops out if this is a problem for you.

SVN Repo (and Trac)

2008-04-28T19:35:00.004-04:00

Retro now has a Subversion repository at http://svn2.assembla.com/svn/retro

The repository can be browsed via Trac. Please be aware that this is primarily intended for my use, but could be useful to anyone wanting to stay current with my work.

Current Work

2008-04-28T12:59:00.005-04:00

A few minor updates:

The image builder now builds an image.js for use with the Ngaro JS
Ngaro JS now has a Clear Display button which may be of some use

I've rewritten the build script for Retro. It's now got a number of internal functions, which have allowed some minor cleanups and the addition of a few new features, including basic help (pass --help to the script to see a list of all flags provided now). This will be included in the 2008.05.03 release.

Image Builder

2008-04-27T09:33:00.002-04:00

As mentioned in my post on the 2008.04.26 release, I am working on an online image builder.

I have finished a quick prototype. It will allow you to select from the majority of the modules that are included in the latest release and build an image including them. This does not save images for long, so download quickly. (The download will be replaced by the next image that is built.)

The image builder will also provide a copy of the image map and the build log. It does not build the image for Ngaro JS yet.

Give it a try.

Release 2008.04.26

2008-04-24T22:21:00.006-04:00

From this point on, I'm going to try to do a release at the end of any week that sees improvements to the Retro codebase.

This week brings a couple of minor changes and improvements:

Ngaro JS now included in the package
A tool (in Toka) to build an image.js for Ngaro JS
(This does not work properly on 64-bit systems, I'll try to fix in the next week or two)
A new (remap-keys) hook has been added to allow alteration of the values returned by key before they can be processed by other words
New whitespace module, which uses (remap-keys) to convert TAB, CR, and LF to SPACE
retro.config renamed to extensions
Minor cleanups to start organizing the modules

My short term plans are to finish organizing the module sources, and to put up an online tool for building a custom forth.image without requiring anything other than a web browser.

Possible change to 'key'

2008-04-24T22:07:00.002-04:00

As mentioned by Snoopy42 on #retro, the Retro 10 system has been ignoring any whitespace separators other than space. Since this can be a problem, I've modified key to convert other whitespace forms (return, tab) into spaces. I'm not sure I like this, so stay tuned.

Ngaro JS performance improvements

2008-04-24T18:52:00.005-04:00

The Ngaro in JavaScript (jsvm) is now fast enough to actually be useable. It's still slower than the Java and C implementations, but I'm now comfortable with the performance.

Now to apply some CSS to improve the look and feel a bit....

For those who are curious about what I did, all it took was:

Put in a loop to call vm_process() more than once per 10ms interval

I also experimented a bit with loop unrolling, but didn't notice enough of a payoff for the code duplication it brought.

New Implementations of Ngaro

2008-04-21T20:17:00.005-04:00

I've been planning to write implementations of Ngaro in other languages for a while now. Over the last few days, I have (with input from erider in the #keow channel on irc.freenode.net) written two new Ngaro implementations.

JavaScript

The first implementation was done in JavaScript. It presents a form-based interface that allows for a crude approximation of the tty console I/O model. Though brutally slow and memory hungry, it does work and seems quite reliable.

Java

While far from my favorite language, I recognize that Java is fairly widespread, so yesterday I wrote a quick and dirty implementation in Java. This is running as a Java applet, and still has some bugs in the I/O device emulation, but does work. It does have an advantage over the JavaScript: this one is very fast.

Future

I'll be cleaning up and working on merging the source for these into the main Retro codebase. Beyond this, I hope to eventually get the JavaScript implementation to run at an acceptable speed. The Java version is fast enough, and will hopefully have its bugs ironed out soon.

Eventually I'd like to have a couple of variations on the Java implementation. It would be nice to run Retro on a Java-capable cell phone or pda, and having an offline version would be useful as well. Given enough time, it's possible that the full graphical console I/O devices could be emulated.

Looking out even further, I am currently doing some research on .NET; an implementation of Ngaro for .NET could also be a big help in improving platform support.

References

RetroNews, 2.0

2008-04-21T19:59:00.000-04:00

A few years ago I kept a blog of news and musings related to RetroForth. I stopped updating it and it faded away after the RetroForth 9.x series and my long pause from development.

Since I'm now actively working on Retro again, I've decided to restart blogging as a means of sharing what I'm working on.