Yacas under the hood

---

The Yacas architecture

Generally, all core functions have plain names and almost all are not "bodied" or infix operators. The file src/yacasapi.cpp in the source tree lists declarations of all kernel functions callable from Yacas; consult it for reference. For many of the core functions, the script library already provides convenient aliases. For instance, the addition operator "+" is defined in the script scripts/standard while the actual addition of numbers is performed through the built-in function MathAdd.

---

Startup, scripts and .def files

---

Object types

The type of an object is returned by the built-in function Type, for example:

In> Type(a); Out> ""; In> Type(F(x)); Out> "F"; In> Type(x+y); Out> "+"; In> Type({1,2,3}); Out> "List";

Internally, atoms are stored as strings and compounds as lists. (The Yacas lexical analyzer is case-sensitive, so List and list are different atoms.) The functions String() and Atom() convert between atoms and strings. A Yacas list {1,2,3} is internally a list (List 1 2 3) which is the same as a function call List(1,2,3) and for this reason the "type" of a list is the string "List". During evaluation, atoms can be interpreted as numbers, or as variables that may be bound to some value, while compounds are interpreted as function calls.

Note that atoms that result from an Atom() call may be invalid and never evaluate to anything. For example, Atom(3X) is an atom with string representation "3X" but with no other properties.

Currently, no other lowest-level objects are provided by the core engine, besides numbers, atomes, strings, and lists. There is, however, a possibility to link externally compiled code that will provide additional types of objects. Those will be available in Yacas as "generic objects."

---

Yacas evaluation scheme

Internally, all expressions are either atoms or lists (perhaps nested). Use FullForm() to see the internal form of an expression. A Yacas list expression written as {a, b} is represented internally as (List a b), equivalently to a function call List(a,b).

Evaluation of an atom goes as follows: if the atom is bound locally as a variable, the object it is bound to is returned, otherwise, if it is bound as a global variable then that is returned. Otherwise, the atom is returned unevaluated.

Internal lists of atoms are generally interpreted in the following way: the first atom of the list is some command, and the atoms following in the list are considered the arguments. The engine first tries to find out if it is a built-in command (core function). In that case, the function is executed. Otherwise, it could be a user-defined function (with a "rule database"), and in that case the rules from the database are applied to it. If none of the rules are applicable, or if no rules are defined for it, the object is returned unevaluated.

Application of a rule to an expression transforms it into a different expression to which other rules may be applicable. Transformation by matching rules continues until no more rules are applicable, or until a "terminating" rule is encountered. A "terminating" rule is one that returns Hold() or UnList() of some expression. Calling these functions gives an unevaluated expression because it terminates the process of evaluation itself.

The main properties of this scheme are the following. When objects are assigned to variables, they generally are evaluated (except if you are using the Hold() function) because assignment var := value is really a function call to Set(var, value) and this function evaluates its second argument (but not its first argument). When referencing that variable again, the object which is its value will not be re-evaluated. Also, the default behaviour of the engine is to return the original expression if it could not be evaluated. This is a desired behaviour if evaluation is used for simplifying expressions.

One major design flaw in Yacas (one that other functional languages like LISP also have) is that when some expression is re-evaluated in another environment, the local variables contained in the expression to be evaluated might have a different meaning. In this case it might be useful to use the functions LocalSymbols and TemplateFunction. Calling

LocalSymbols(a,b) a*b;

results in "a" and "b" in the multiplication being substituted with unique symbols that can not clash with other variables that may be used elsewhere. Use TemplateFunction instead of Function to define a function whose parameters should be treated as unique symbols.

---

Rules

A function is identified by its name as returned by Type and the number of arguments, or "arity". The same name can be used with different arities to define different functions: f(x) is said to 'have arity 1' and f(x,y) has arity 2. Each of these functions may possess its own set of specific rules, which we shall call a "rule database" of a function.

Each function should be first declared with the built-in command RuleBase as follows:

RuleBase("FunctionName",{argument list});

So, a new (and empty) rule database for f(x,y) could be created by typing RuleBase("f",{x,y}). The names for the arguments "x" and "y" here are arbitrary, but they will be globally stored and must be later used in descriptions of particular rules for the function f. After the new rulebase declaration, the evaluation engine of Yacas will begin to really recognize f as a function, even though no function body or equivalently no rules have been defined for it yet.

The shorthand operator := for creating user functions that we illustrated in the tutorial is actually defined in the scripts and it makes the requisite call to the RuleBase() function. After a RuleBase() call you can specify parsing properties for the function; for example, you could make it an infix or bodied operator.

Now we can add some rules to the rule database for a function. A rule simply states that if a specific function object with a specific arity is encountered in an expression and if a certain predicate is true, then Yacas should replace this function with some other expression. To tell Yacas about a new rule you can use the built-in Rule command. This command is what does the real work for the somewhat more aesthetically pleasing ... # ... <-- ... construct we have seen in the tutorial. You do not have to call RuleBase() explicitly if you use that construct.

Here is the general syntax for a Rule() call:

Rule("foo", arity, precedence, pred) body;

This specifies that for function foo with given arity (foo(a,b) has arity 2), there is a rule that if pred is true, then body should be evaluated, and the original expression replaced by the result. Predicate and body can use the symbolic names of arguments that were declared in the RuleBase call.

All rules for a given function can be erased with a call to Retract(funcname, arity). This is useful, for instance, when too many rules have been entered in the interactive mode. This call undefines the function and also invalidates the RuleBase declaration.

You can specify that function arguments are not evaluated before they are bound to the parameter: HoldArg("foo",a) would then declare that the a arguments in both foo(a) and foo(a,b) should not be evaluated before bound to a. Here the argument name a should be the same as that used in the RuleBase() call when declaring these functions. Inhibiting evaluation of certain arguments is useful for procedures performing actions based partly on a variable in the expression, like integration, differentiation, looping, etc., and will be typically used for functions that are algorithmic and procedural by nature.

Rule-based programming normally makes heavy use of recursion and it is important to control the order in which replacement rules are to be applied. For this purpose, each rule is given a precedence. Precedences go from low to high, so all rules with precedence 0 will be tried before any rule with precedence 1.

You can assign several rules to one and the same function, as long as some of the predicates differ. If none of the predicates are true, the function is returned with its arguments evaluated.

This scheme is slightly slower for ordinary functions that just have one rule (with the predicate True), but it is a desired behaviour for symbolic manipulation. You can gradually build up your own functions, incrementally testing their properties.

---

Examples of using rules

In> RuleBase("f",{n}); Out> True; In> Rule("f", 1, 10, n=0) 1; Out> True; In> Rule("f", 1, 20, IsInteger(n) \ And n>0) n*f(n-1); Out> True;

This definition is entirely equivalent to the one in the tutorial. f(4) should now return 24, while f(a) should return just f(a) if a is not bound to any value.

The Rule commands in this example specified two rules for function f with arity 1: one rule with precedence 10 and predicate n=0, and another with precedence 20 and the predicate that returns True only if n is a positive integer. Rules with lowest precedence get evaluated first, so the rule with precedence 10 will be tried before the rule with precedence 20. Note that the predicates and the body use the name "n" declared by the RuleBase() call.

After declaring RuleBase() for a function, you could tell the parser to treat this function as a postfix operator:

In> Postfix("f"); Out> True; In> 4 f; Out> 24;

There is already a function Function defined in the standard scripts that allows you to construct simple functions. An example would be

Function ("FirstOf", {list}) list[ 1 ] ;

which simply returns the first element of a list. This could also have been written as

Function("FirstOf", {list}) [ list[1] ; ];

As mentioned before, the brackets [ ] are also used to combine multiple operations to be performed one after the other. The result of the last performed action is returned.

Finally, the function FirstOf could also have been defined by typing

FirstOf(list):=list[1] ;

---

Structured programming and control flow

Function("ForEach",{foreachitem, foreachlist,foreachbody}) [ Local(foreachi,foreachlen); foreachlen:=Length(foreachlist); foreachi:=0; While (foreachi < foreachlen) [ foreachi++; MacroLocal(foreachitem); MacroSet(foreachitem, foreachlist[foreachi]); Eval(foreachbody); ]; ]; Bodied("ForEach"); UnFence("ForEach",3); HoldArg("ForEach",foreachitem); HoldArg("ForEach",foreachbody);

Functions like this should probably be defined in a separate file. You can load such a file with the command Load("file"). This is an example of a macro-like function. Let's first look at the last few lines. There is a Bodied(...) call, which states that the syntax for the function ForEach() is ForEach(item,{list}) body; -- that is, the last argument to the command ForEach should be outside its brackets. UnFence(...) states that this function can use the local variables of the calling function. This is necessary, since the body to be evaluated for each item will probably use some local variables from that surrounding.

Finally, HoldArg("function",argument) specifies that the argument "argument" should not be evaluated before being bound to that variable. This holds for foreachitem and foreachbody, since foreachitem specifies a variable to be set to that value, and foreachbody is the expression that should be evaluated after that variable is set.

Inside the body of the function definition there are calls to Local(...). Local() declares some local variable that will only be visible within a block [ ... ]. The command MacroLocal() works almost the same. The difference is that it evaluates its arguments before performing the action on it. This is needed in this case, because the variable foreachitem is bound to a variable to be used as the loop iterator, and it is the variable it is bound to that we want to make local, not foreachitem itself. MacroSet() works similarly: it does the same as Set() except that it also first evaluates the first argument, thus setting the variable requested by the user of this function. The Macro... functions in the built-in functions generally perform the same action as their non-macro versions, apart from evaluating an argument it would otherwise not evaluate.

To see the function in action, you could type:

ForEach(i,{1,2,3}) [Write(i); NewLine();];

This should print 1, 2 and 3, each on a new line.

Note: the variable names "foreach..." have been chosen so they won't get confused with normal variables you use. This is a major design flaw in this language. Suppose there was a local variable foreachitem, defined in the calling function, and used in foreachbody. These two would collide, and the interpreter would use only the last defined version. In general, when writing a function that calls Eval(), it is a good idea to use variable names that can not collide with user's variables. This is generally the single largest cause of bugs when writing programs in Yacas. This issue should be addressed in the future.

---

Additional syntactic sugar

• Lists, like for example {a,b}. This then is parsed into the internal notation (List a b) , but will be printed again as {a,b};
• Statement blocks like [ statement1 ; statement2;];. This is parsed into a Lisp object (Prog (statement1 ) (statement2 )), and printed out again in the proper form.
• Object argument accessors in the form of expr[ index ]. These are mapped internally to Nth(expr,index). The value of index=0 returns the operator of the object, index=1 the first argument, etc. So, if expr is foo(bar), then expr[0] returns foo, and expr[1] returns bar. Since lists of the form {...} are essentially the same as List(...), the same accessors can be used on lists.

• Function blocks such as

  While (i < 10) [ Write(i); i:=i+1; ];

  The expression directly following the While(...) block is added as a last argument to the While(...) call. So While(a)b; is parsed to the internal form (While a b).

This scheme allows coding the algorithms in an almost C-like syntax.

Strings are generally represented with quotes around them, like "this is a string". Backslash \ in a string will unconditionally add the next character to the string, so a quote can be added with \" (a backslash-quote sequence).

---

Using "Macro rules" (e.g. NFunction)

This is accomplished using functions MacroRuleBase, MacroRule, MacroRulePattern. These functions evaluate their arguments (including the rule name, predicate and body) and define the rule that results from this evaluation.

Normal, "non-Macro" calls such as Rule() will not evaluate their arguments and this is a desired feature. For example, suppose we defined a new predicate like this,

RuleBase("IsIntegerOrString, {x}); Rule("IsIntegerOrString", 1, 1, True) IsInteger(x) And IsString(x);

If the Rule() call were to evaluate its arguments, then the "body" argument, IsInteger(x) And IsString(x), would be evaluated to False since x is an atom, so we would have defined the predicate to be always False, which is not at all what we meant to do. For this reason, the Rule calls do not evaluate their arguments.

Consider however the following situation. Suppose we have a function f(arglist) where arglist is its list of arguments, and suppose we want to define a function Nf(arglist) with the same arguments which will evaluate f(arglist) and return only when all arguments from arglist are numbers, and return unevaluated Nf(arglist) otherwise. This can of course be done by a usual rule such as

Rule("Nf", 3, 0, IsNumericList({x,y,z})) <-- "f" @ {x,y,z};

Here IsNumericList is a predicate that checks whether all elements of a given list are numbers. (We deliberately used a Rule call instead of an easier-to-read <-- operator to make it easier to compare with what follows.)

However, this will have to be done for every function f separately. We would like to define a procedure that will define Nf, given any function f. We would like to use it like this:

NFunction("Nf", "f", {x,y,z});

After this function call we expect to be able to use the function Nf.

Here is how we could naively try to implement NFunction (and fail):

NFunction(new'name, old'name, arg'list) := [ MacroRuleBase(new'name, arg'list); MacroRule(new'name, Length(arg'list), 0, IsNumericList(arg'list) ) new'name @ arg'list; ];

Now, this just does not do anything remotely right. MacroRule evaluates its arguments. So IsNumericList(arg'list) will evaluate to False and the new rule will be defined with a predicate that is always False, i.e. it will be never applied.

The right way to figure this out is to realize that the MacroRule call evaluates all its arguments and passes the results to a Rule call. So we need to see exactly what Rule() call we need to produce and then we need to prepare the arguments of MacroRule so that they evaluate to the right values. The Rule() call we need is something like this:

Rule("actual new name", , 0, IsNumericList({actual arg list}) ) "actual new name" @ {actual arg list};

Note that we need to produce expressions such as "new name" @ arg'list and not results of evaluation of these expressions. We can produce these expressions by using UnList(), e.g.

UnList({Atom("@"), "Sin", {x}})

produces

"Sin" @ {x};

but not Sin(x), and

UnList({IsNumericList, {1,2,x}})

produces the expression

IsNumericList({1,2,x});

which is not further evaluated.

Here is a second version of NFunction() that works:

NFunction(new'name, old'name, arg'list) := [ MacroRuleBase(new'name, arg'list); MacroRule(new'name, Length(arg'list), 0, UnList({IsNumericList, arg'list}) ) UnList({Atom("@"), old'name, arg'list}); ];

Note that we used Atom("@") rather than just the bare atom @ because @ is a prefix operator and prefix operator names as bare atoms do not parse (they would be confused with applications of a prefix operator to what follows).

Finally, there is a more concise (but less general) way of defining NFunction() for functions with known number of arguments, using the backquoting mechanism. The backquote operation will first substitute variables in an expression, without evaluating anything else, and then will evaluate the resulting expression a second time. The code for functions of just one variable may look like this:

N1Function(new'name, old'name) := `( @new'name(x_IsNumber) <-- @old'name(x) );

This executes a little slower than the above version, because the backquote needs to traverse the expression twice, but makes for much more readable code.

---

Scope of variable bindings

When entering a block, a new stack frame is pushed for the local variables; it means that the code inside a block doesn't see the local variables of the caller either! You can tell the interpreter that a function should see local variables of the calling environment; to do this, declare

UnFence(funcname, arity)

on that function.