TclWise, a Tcl book with free chapters online

TCLWISE

An introduction to the Tcl programming language

Sponsored Project: The Jim interpreter
A small footprint implementation of Tcl

Send a comment to the author

9. EXTENDING TCL IN TCL

Tcl is a programmable programming language. Almost every functionality of the language is exported to the user through commands: conditionals, creation of procedures, control structures, mathematical expressions, all are implemented as commands, that take string as arguments. Because the user can create new commands, or even replace core commands with user defined ones, the language can be extended and modified in a radical way. You can write commands implementing your own control structures, a new way to deal with mathematical expressions, functionalities that are closely realted to the program you are writing, or an object oriented programming extension. So the expert Tcl programmer will write a program in a new way: in the first stage he will specialize Tcl to make the language better for the task, then he will write the program using this new specialized language.

We already saw how to create Tcl procedures, but in order to create more powerful procedures, more powerful commands are required, this commands are eval, uplevel, upvar, and may others: some of this commands will be explained in this chapter, with real examples on how to program Tcl in Tcl in order to increase the power of the language.

9.1 Programs executing programs: the eval command

In the first chapter of this book, we already shown that Tcl has a very dynamic nature. For instance you can put a command name in a variable and call it like this:


% set a llength
llength
% $a {1 2 3}
3

You may even use interpolation to create the command name (or any of its arguments) at runtime:


% [string range "Xllen" 1 end]gth {1 2 3}
3

Still there is something of very desiderable that we can't do without the use of a command called eval, that's to evaluate a string as a Tcl program. That's exactly what the eval command does: it runs the Tcl script you pass as argument, and uses the return value of the last command executed in that script as return value. That's an example just to figure how it works:


% eval "set foo 10"
10
% set myscript {puts $foo}
puts $foo
% eval $myscript
10

In the first line we called eval with the string "set foo 10" as argument. Eval will just evaluate it, and return the value returned by the script. Then we set the variable myscript to a string representing a valid Tcl command and execute it using eval. This is a very important operation to do, immagine for example that you want to write a new Tcl control structure called repeat. The goal is to avoid a for or while loop when you want to just repeat a given Tcl script a number of times, using instead something like "repeat 10 {lappend mylist foo}", to create a list with 10 elements of value "foo". The second argument of the repeat command is a script, and it is not possible at all to execute this script in a loop without to use eval. This is a possible implementation of the repeat command:


proc repeat {n script} {
    while {[incr n -1] >= 0} {
        eval $script
    }
}

Cut & paste this in the tclsh to check if it's working well:


% repeat 4 {puts Hello}
Hello
Hello
Hello
Hello

Great, we just extended Tcl in Tcl. Actually we'll see in a moment that this implementation of repeat is not very correct, but for know our goal is to understand very well how eval works. An important information about eval is that it can take multiple arguments, they will be concatenated using the concat command, and the result of the concatenation will be evaluated. So actually to write:


eval $foo $bar

is the same to call


eval [concat $foo $bar]

The concat command does a simple operation: it removes every space at the right and at the left of every argument passed, and then concatenate all the resulting strings using a single space as separator between every string, returning the result obtained to the caller. It's possibly simpler to show the behaviour using some example:


% concat {a  } b
a b
% concat a b c
a b c
% concat {1 2 3} {a b c}
1 2 3 a b c
%

While concat performs just an operation on strings, you should notice that if all its arguments are valid lists, the result is a single list that is the concatenation of all the original lists. So concat can be used to concatenate lists. Because eval concatenates the arguments like concat, we have to study the concat behaviour very carefully when the arguments are pieces of scripts so that can we can apply what we understand about concat in order to use eval very well.

Let's start with an example, but first we need to recall the last version of the + procedure we wrote talking about procedures with variable number of arguments.


proc + {x args} {
    foreach e $args {
        set x [expr $x+$e]
    }
    return $x
}

Now suppose you have a list of integers, and you want to sum all this integers using the + procedure. + is designed to take every integer to sum as a separated argument, so we can't write something like "+ $mylist", but it's possible to get the work done using eval:


% set mylist [list 1 2 3 4 5]
1 2 3 4 5
% eval + $mylist
15

It works, and the behaviour is explained recalling that eval concatenates it's arguments like concat. We can call concat instead of eval, using exactly the same arguments to check what's the resulting script that eval will finally evaluate after the concatenation step:


% set mylist [list 1 2 3 4 5]
1 2 3 4 5
% concat + $mylist
+ 1 2 3 4 5

The resulting script is "+ 1 2 3 4 5", and it is perfect because the + procedure will be called with every list element as a different argument. This behaviour is useful, but we want also a way to avoid it, that's, what about if I want to use eval with an argument that should be treated as an single` argument even if it contains spaces? Just passing the arguments to eval will not work:


% set a "puts"
puts
% set b "Hello World"
Hello World
% eval $a $b
can not find channel named "Hello"
while evaluating {eval $a $b}

An error is returned, because [concat $a $b] will produce "puts Hello World", so puts will be called with three arguments, instead of two like in "puts {Hello World}". In order to fix the problem, we have to quote the arguments using the list command. First of all we can try to see the difference in the produced string calling concat directly:


% set a "puts"
puts
% set b "Hello World"
Hello World
% concat $a $b
puts Hello World
% concat [list $a $b]
puts {Hello World}

As you can see, in the last line we used list in order to quote the command: list will take care to use braces around arguments containing spaces or other characters interpreted by Tcl in a special way. Using or not using the list command to quote arguments makes we able to select what argument of eval should be expanded like in the case of the + procedure used to sum a list of numbers, and what should be not. For example in the code:


eval $a [list $b $c $d] $e

if $a and $e contain spaces, the content will be interpreted as more then one argument, while $b $c and $d will always be interpreted as a single argument because of the quoting done by the list command.

Don't worry if all this is not very clear for now, it's a complex topic that you will master after some pratice. For now let's introduce a new command that will allow us to write a new, better implementation of repeat.

9.2 Breaking the rules with uplevel

In the previous section we wrote this implementation of the repeat command.


proc repeat {n script} {
    while {[incr n -1] >= 0} {
        eval $script
    }
}

It works in some case, like if we use it to print four times "Hello World" on the screen like this:


repeat 4 {puts "Hello World"}

But if you look at the implementation of repeat, there is something of strange. The script argument, containing the body of the loop, is executed in the context of the repeat procedure, not in the context of the caller. In the case of a script like {puts "Hello World"} there are no problems because this script can be executed in any context without troubles, it does not refer to local variables, it's just a command with a constant argument. Instead this will create troubles:


% set a 10
10
% repeat 2 {puts $a}
can't read "a": no such variable
while evaluating {repeat 2 {puts $a}}

What happens is that the script "puts $a" is executed in the context of the repeat procedure, where the $a variable does not exists. What we need to fix this procedure, is a command very similar to eval but able to run a script in the context of the caller. This command is called uplevel. The uplevel command is exactly like eval, but it takes as first argument the level where the script will be executed. This is the command signature:

uplevel ?level? arg ?arg ...?

For instance, if level is 1, the code is executed one level back, in the context of the caller procedure, if level is 2 the code is executed in the context of the caller of the caller, and so on. Look at the following example carefully:


proc a {} {
    set myvar "Tcl is a programmable programming language"
    b
}


proc b {} {
    c
}


proc c {} {
   uplevel 2 {puts $myvar}
}

The output of this program is "Tcl is a programmable programming language", but it's important to realize why. The program creates three procedures, a, b and c, and starts calling the procedure a. The a procedure will create a local variable myvar with the "Tcl is a ..." string as content, and will call b, that will call c, that will execute the script "puts $myvar" in the context of the caller of the caller (that's, the a procedure). Because in the context of a the myvar variable exists and is set to a given value, the program will print it, and exit.

Now what you need to change in the repeat procedure to make it working is to replace eval with uplevel 1, and it will be sane.


proc repeat {n script} {
    while {[incr n -1] >= 0} {
        uplevel 1 $script
    }
}

This version will work as expected: you can't distinguish it from a core command, you just have a new control structure to use when needed. This new procedure of course will work when the last implementation used to fail:


% set a 10
10
% repeat 2 {puts $a}
10
10

The only difference between repeat and other Tcl commands performing loops is that it does not work with break and continue. Of course it's possible to fix this problem too, and we will see how to do it in the chapter about the error handling features of Tcl.

9.3 Passing variable names to procedures

Now that we have a tool like uplevel we can figure how it's possible to implement procedures able to get variable names as arguments, and to modify or use the content of this variables from the caller's context. This is the case of commands like incr, append, lappend, foreach. In some way this is the Tcl way to do pass by reference, but instead of a reference what is passed is the name of a variable of the caller.

Let's try to implement a procedure that given a variable name converts the string stored in that variable in uppercase, setting the uppercase version of the string as the new content of the variable.


proc toupper varname {
    set oldval [uplevel 1 [list set $varname]]
    set newval [string toupper $oldval]
    uplevel 1 [list set $varname $newval]
    return {}
}

The usage is simple:


% set myvar "tcl"
tcl
% toupper myvar
% puts $myvar
TCL

Note that the implementation of toupper returns a null string, in order to stress on the work the function does as side effect, and not on its return value. You should be able to understand how toupper works, but we will analyze it step by step.

Starting from the first line of the procedure:


set oldval [uplevel 1 [list set $varname]]

This command set the oldval variable to the value of the script "set $varname" executed in the context of the caller. Do you remember that the set command with only one argument returns the value of the specified variable? But this variable lives in the context of the caller so we need to use uplevel, and quote the script with list because the value of $varname may contain spaces.

The next line of the procedure is just basic Tcl:


set newval [string toupper $oldval]

We use the string command to set an uppercase version of $oldval as value of the newval variable. At this point, we are ready to modify the variable with name $varname in the caller's context to store the new uppercased value of the string:


uplevel 1 [list set $varname $newval]

This is very similar to the first line of the procedure, but here set is called (again, on the caller's context thanks to uplevel) with three arguments, the last being the new value to assign. Again list is required in order to make sure that even if $varname or $newval contain spaces, they will be handled as a single argument.

Finally the last line of the procedure returns the empty string.

This all works well... but there is a problem: it is a bit too much complex to be comfortable, so Tcl has a command called upvar to make this simpler. The following is the equivalent program using upvar instead of uplevel:


proc toupper varname {
    upvar 1 $varname var
    set var [string toupper $var]
    return {}
}

This looks simpler, but how it works? upvar is able to bind a variable name living in the current procedure, with one living in a different level. This is the procedure signature:

upvar ?level? otherVar myVar ?otherVar myVar ...?

The level argument works exactly like the level argument of uplevel, so a level value of 1 means to bind with a variable of living in the context of the caller. In the above example, the code:


upvar 1 $varname var

means: bind the local variable name var, with the caller's variable name $varname. Every time the toupper procedure will use the variable var the effect will be to access the caller's variable named $varname (the actual value of varname of course depends on the value of the argument passed to the toupper procedure).

It's possible to specify more than one pair of variable names to upvar like in:


upvar 1 $listname list $somename foo

in order to bind more variables with a single upvar command.

9.4 Mapping scripts to lists

Now that we are able to extend the language, we can write a procedure that will prove to be very useful in a number of circumstances. This procedure called map is used to map a script to a Tcl list. Every element of a list is used as argument of a Tcl script, and the result is used to generate a new list (composed of the resulting elements). For example mapping a script that returns the square of its input to the list "1 2 3 4" we will obtain the list "1 4 9 25", and so on. The following is a first implementation of map, but we will write a better one in the chapter about Functional programming with Tcl.


proc map {varname mylist body} {
    upvar 1 $varname var
    set res {}
    foreach var $mylist {
        lappend res [uplevel 1 $body]
    }
    return $res
}

This map procedure is very similar to the foreach command, but at every iteration, the return value of the script is appended to a list that is returned by map at the end of the loop. For example in order to compute the squares of the first 5 natural numbers you can write:


map x {1 2 3 4 5} {expr $x*$x}

The return value of this command is the list {1 4 9 16 25}. Another example is to use map to turn a list of strings in the list of their lengths. This time tclsh is used for an interactive session:


% set l {I will be translated into a list of length} 
I will be translated into a list of length
% map x $l {string length $x}
1 4 2 10 4 1 4 2 6

It makes a lot of task simpler! Actually what map does is to encapsulate in a procedure a piece of code that you otherwise will type many times, that's something like this:


set result {}
foreach e $list {
    lappend result [string length $e]
}

This explains why experienced programmers think that programming languages with great extension capabilites are better: the programmer can look for patterns in the code (symptoms of repetitive actions that the programmer is doing, and that the language may do for him), and write a procedure to make a specific task occurring frequently much simpler.

9.5 The rename command

In the first chapters we mentioned that using proc it is possible to create procedures with names of already existing commands (both core commands or used defined ones). Not only it's possible to overwrite an existing command name, it is also possible to rename existing commands using the rename command. The usage is very simple:

rename oldName newName

rename will change the name of the command oldName, into the new name newName, with one exception: if the newName argument is an empty string, the oldName command will be removed from Tcl at all. This feature may be used in order to make the language more secure in critical contexts (removing all the commands that are a potential security problem), but we will see this better talking about Safe Interpreters in the next chapters.

The rename command enters in the picture traced by this chapter because the ability to rename a command leads to the ability to wrapper existing commands. For example you may like to wrapper the puts command in order to act as the eval command when called with multiple arguments, concatenating all the arguments using a space as separator and printing the result on the screen, but at the same time, we need to take the original puts implementation under some other name, because our new puts will need to call it to perform the output. We can do all this using rename:


rename puts _puts
proc puts args {
	_puts [join $args]
}

After the above code is executed, puts will work with multiple arguments:


% puts multiple arguments passed to puts
multiple arguments passed to puts

Wrappering procedures may be interesting, for example you may like to extend the proc functionalities in order to define the description of the procedure as additional argument, and then write a command to get the documentation from the command name, or you may wrapper set in order to print debugging information about modified variables in a log file in order to trace some bug, and so on.

9.6 Expanding lists into arguments in Tcl 8.5

Talking about eval we shown how it is possible to use this command in order to perform expansion of the elements of a list as single command arguments. In the example we used this feature and the + procedure to compute the sum of a list of integers:


% set mylist [list 1 2 3 4 5]
1 2 3 4 5
% eval + $mylist
15

Starting from Tcl 8.5 there is a better way (both performance-wise and from the point of view of the user) in order to perform argument expansion. It's not a command, but new syntax. Basically prepending an argument with the string {expand} has the result that the Tcl interpreter will expand every element of the original argument interpreted as a Tcl list, into a single argument. For instance the scripts:


{expand}{puts Hello}

and


set mylist [list puts Hello]
{expand}$mylist

are both equivalent to just


puts Hello

We can rewrite the sum of integers element of a list using {expand} without to use eval:


+ {expand}$mylist

Every element of $mylist will be expanded as an argument of the + command.

The introduction of {expand} in the Tcl syntax was controversial, many people didn't liked the syntax, or the idea to make the Tcl syntax more complex (fortunately the idea of simplicity is very important among people of the Tcl comunity). Strictly speaking it's possible to write every kind of program without to use {expand} at all, but actually there are many situations where argument expansion is needed, and eval is not the best tool for argument expansion (the main goal of eval is different, it is to evaluate Tcl scripts). I'm confident that after all the decision of the Tcl Core Team to introduce the {expand} syntax into the language was a good one: now that Tcl 8.5 is near to be released the Tcl comunity is starting to appreciate how comfortable and fast argument expansion can be.