NeoWebScript™ Theory of Operations

Apache Module Info

Programming information on the Apache API, from the Apache User's Guide.

General Theory

Overview from C

The primary module, mod_neoscript.c, creates a Tcl (master) interpreter and loads in various Tcl packages like Extended Tcl, NeoTcl, interfaces to Postgres and GD if available. The master interpreter then sources in init.tcl, which performs the complex tasks of creating and setting up the safe (slave) interpreters that make the NeoWebScript™ environment.

Another module, mod_log_neo.c creates a special TransferLog containing more information than the normal CLF files, and formatted in a Tcl list-based structure. The command estimate_hits_per_hour relies on this log format to work.

The module mod_auth_tcl.c implements user authentication through the Tcl interpreter.

The module mod_neo_userdir.c is a starting attempt to implement user directory access policy through the Tcl interpreter.

Overview from Tcl

init.tcl initializes the master interpreter; loads in additional commands from other source files; creates safe slave interpreters as needed; and establishes various interfaces between the master and safe slaves, and between the module and the interpreters. Source files are also loaded into the slave as needed. Autoloading is even supported in the neoscript-tcl directory, but be sure to create the proper index file.

Processing a Webpage

When a webpage is requested, if it matches the criteria for a server side include and a neoscript directive is found in the page, the C handle_neowebscript routine is called from the C code in the module that sends parsed content.

We make use the request_config data in the request record structure. This allows each module to set a pointer and find it later. We follow the request_rec structure from ours back to the main one (if we aren't the main one) to find a non-NULL pointer for our module in the request_config structure. If there isn't one to be found, it is the first invocation of NeoWebScript™ for this page.

If it's the first use of NeoWebScript™ for this page, or if this page is being included in another page but is not owned by the same owner as the other page, we create a safe interpreter, set it into the request_config structure for our module, register for the cleanup routine to destroy the interpreter, and configure and propagate the variables to the safe interpreter.

Tcl_request_rec is a global pointer to the current request_rec being processed. We save the previous value on the stack while we're executing send_parsed_content, and put it back afterwards. That way Tcl procedures don't have to all pass around a magic cookie to find the request rec, yet multiple interpreters can exist and code in existant interpreters can be invoked for subordinate includes when the criteria for doing that are met.

If it's the first use of NeoWebScript™ for this page, then if a safe interpreter previously existed, it is destroyed. A new safe interpreter is created, and various environment variables, server variables, etc, are exported to it by propagate_vars_to_nws.

handle_neowebscript builds up the command to be executed. It sets up to call handle_neowebscript_request (which is written in Tcl). The first argument is the name of the safe interpreter. The second is the key of the key-value pair read from the server-side-include. Last comes the value of the key-value pair.

It is most important that neither the key nor the value are ever evaluated within the trusted interpreter, or the user will have defeated the protection mechanisms provided by the safe interpreter.

Since we build up the command to be executed as a list, the key and value will be quoted to make a valid list element, so they won't be evaluated by the trusted interpreter when handle_neowebscript_request is called.

We then evaluate the constructed command. If there is an error, we send the error result to the webpage being constructed.

Finally we restore the Tcl_request_rec global request rec pointer with whatever was in there before. It is intialized to NULL at the start, and should automatically be restored to that when the top level page of NeoWebScript™ code completes.

handle_neowebscript_request

This command is responsible for actually processing embedded NeoWebScript™ requests from within webpages. It is defined in init.tcl.

It switches on the tag, which can currently be return, code, eval, var or expr. Anything else is treated as an error.

Valid tags are handled using handle_server_side_return, handle_server_side_eval, handle_server_side_variable, and handle_server_side_expr. These routines are called with the name of the safe interpreter and the code (or variable, or expression) to evaluate. The code is evaluated within the safe interpreter. Errors are caught and traced back.

These routines were defined as procs by init.tcl, and they use the interp eval command to do the right thing to the untrusted interpreter to cause code to be executed, results to be returned, etc.

devel.tcl

devel.tcl is used by people who are modifying NeoWebScript™ by changing code that runs in the trusted code base. This should only be done by people who really know what they're doing, because errors here can compromise the security of your webserver.

If debugging is set to 1 in init.tcl, then every time a page begins NeoWebScript™ execution, devel.tcl is sourced in, and the devel_setup procedure is executed. This makes it easy to work on new functions, without having to restart the webserver on every change.

If debugging is set to 0 in init.tcl, then devel.tcl is loaded it at server startup time only, although devel_setup is still run each time an interpreter is set up to do something to a page.

Unknown Packages and Procs in Safe Interpreters

As of version 2.0, there are better ways to support package require and unknown procedures within the safe interpreter. Previously, only unknown procedures could be resolved, and they were resolved by providing access to the source command (to a restricted list of directories) among others. In this way, an essentially standard unknown and auto_loading procedure could take place within the safe interpreter. There has been no previous support for package require.

Unknown Procedures

Previous versions of NeoWebScript™ borrowed from Michael McClennan's Itcl2.1 implementation of unknown, the mechanism for resolving calls to unknown procs in the safe interpreter. However, along with the integration of Tcl 8.0.3, it now uses the new Safe Base mechanisms specifically designed for creating and manipulating safe interpreters. It automatically manages the slaves' auto_path and tcl_library; and contains predefined aliases for source, load, file, and exit commands allowing for their restricted use.

However, NeoWebScript™ performs some internal caching of certain procs into the master interpreter. The Tcl proc SAFE_autoload is given first crack to resolve a command for the safe interpreter. It checks two global arrays, the safe_proc_cache and the safe_alias_cache. These arrays contain the proc or alias definition to be evaled into the safe interpreter. If it does not find a valid definition, it then passes autoloading to the Safe Base in the safe interpreter.

Resolving Package Requests

To package require in the safe interpreter, during the setup of the slave, a package ifneeded command can be evaled into the slave giving a specific script to execute when the package is requested.