quisp, a simple dynamic web content system

quisp (QUIck Server Pages) is a compact self-contained environment for generating server-side html content on-the-fly with things like database interaction using a bundled sorta-SQL flatfile database (shsql), invoking shell commands and capturing results, html form-building, as well as a collection of functions for datatype checking, arithmetic, working with strings, lists, dates, times, and misc.

The basic approach, similarly to systems like PHP, is to use your text editor to create files that contain html intermingled with quisp #directives that do the dynamic things. Your web server invokes quisp as an ordinary CGI, then quisp interprets the files you create, to produce the final displayed HTML.

Quisp can serve for demos, rapid prototyping, or many types of small-to-medium sized production applications. The package also includes quisputil(1), a command line utility for filtering and formatting data.

Quick links:   Syntax     Shell     SQL     Forms     Functions     App hints     Config


First example
Suppose you type the following into your text editor and save as 'test1'.
#cgivar xval
<h2>My first QUISP page</h2>
#if @xval = ""
  parameter xval was not supplied
#else
  you passed xval as @xval
#endif
<br><br>
<form action="http://myserver.com/cgi-bin/quisp" >
#formtarget test1
Enter a value: <input name=xval value="@xval" size=6>
<input type=submit value="Go">
</form>
This won't actually work until things are set up (see below). But, you can get the jist of what it will do... it's ordinary HTML overlaid with a few quisp #directives and @variable references.


To set up a quisp project
This procedure is similar to that given on the shsql page. If you do the procedure here, shsql will be taken care of automatically.
• decide on a project file system location and a projname
• mkdir projname; cd projname
• mkdir ./pages ./textchunks ./logs ./tmp ./data ./indexes ./locks
• create a config file; it can be named ./config
• set environment variable SHSQL_CONFIG to full pathname of config file
• if you need to build quisp and shsql, do that
• copy 'quisp' executable to your cgi-bin and check its runtime privilege level
• in cgi-bin create a symlink to your config file named 'quisp.cnf' (you can use any name in your cgi-bin as long as the names match).


First test and getting going
• Create a test file, name it test1 and save it in your ./pages directory... then with your web browser go to eg.   http://myserver.com/cgi-bin/q?rtn=test1   ... and you should see your test page.
• Create more pages in your ./pages directory. Use html with the quisp directive constructs shown below. For html links between pages in your app use eg. <a href="@CGIPROG?rtn=test1">click here</a>
• Hints for developing your application




Quisp preprocessing syntax rules
• lines beginning with // are comments
• things like #include or #if are quisp directives
• tokens beginning with @ are assumed to be variables and evaluated
• @variables are delimited by space and punct chars except for period (.)
• any line that's not a directive is written to stdout (with @variables evaluated)
• things like $arith() and $addmember() are functions
• if/else, loops, and includes can be freely nested



Directives – flow of control, vars, I/O

#set varname = value
    Set quisp variable varname to value. Examples:
    #set lastname = "Jones"
    #set fullname = @firstname " " @lastname
    #set x = 47.5
    #set total = $arith( @x+15 )
(a function)

#setifnotgiven varname = value
    Same as #set but executes only when varname has no value.
    Use it to set defaults for user vars and parameters.

#cgivar varnames
Get one or more CGI user variable varnames and load into quisp variables of same name. Each varname can have an optional datatype constraint code appended. Example: #cgivar id(i*) mode(t) order(t) desc(s). Codes include: i=integer n=numeric t=token s=safestring [More info]. An asterisk following the code means that the user var must be present.

#cookie varnames
Same as #cgivar above but gets an HTTP cookie instead of a CGI user variable. To set a cookie print an HTTP set-cookie statement directly as the first line in your page, eg. Set-Cookie: MY_COOKIE=hello; path=/;   Quisp will finagle the headers appropriately.

#chkvar varnames
Same as #cgivar above but performs datatype checking on a variable that's already in use (it does everything except loading CGI user variables).

[More about getting user vars and cookies]



#include filename
    Include quisp codechunk from filename. The filename is
    relative to ./pages dir or it can be a full pathname beginning with "/".
    The included code executes "inline". Nested #includes are ok.

#return
    Stop processing an included file and return to the includer.


#if cond   #elseif cond   #else   #endif     Example construct:
#if @mode = 1
    Hello, mode is 1<br>
#elseif @mode = 2
    Hi, mode is 2<br>
#else
    Well, mode is something else<br>
#endif


The cond can have vars, constants, and functions. More examples:
@lastname = "Jones"
@balance < 0.0 and @status = "A"
$arith( @x+@y ) > 100
$strlen( @uin ) < 3
$datevalid( @uin ) = 1
@color in "red,green,blue"
@tag like "J*"                 @tag !like "J*"
@status = ""                   @status != ""
@name inlike "a*,b*,c*"       @name !inlike "a*,b*"
@zscore inrange -0.1,0.1       @zscore outrange -2.5,2.5
@authors hasword "smith"
@colorlist hasmember "blue"
@latitude inrange 38.1,38.5 or @latitude = ""

Supported operators: =   !=   >   >=   <   <=   like   !like   in   !in   inlike   !inlike   inrange   outrange   hasmember   hasword
Wild card chars: *   ?       Case: = and != are case-sensitive, "like" is not.
Several conditional terms can be connected using && or ||. Parentheses are not supported and there's no control over precedence, so do not mix && and || in the same statement. There's no logical negation operator.


#for varname in commalist   #endloop     Example construct:
    #set colorlist = red,blue,orange
    #for color in @colorlist
        Color is @color<br>
    #endloop


#while cond   #endloop
    While-loop, using same conditional constructs as used with #if

#loop   #endloop
    Endless loop, must be terminated with a #break statement

#break   #continue
    Loop control.


#call $function( arglist )
    Call a quisp function without retaining any return value.


#write pathname [append]     #endwrite
When quisp encounters #write it begins redirecting quisp output to the given file instead of stdout, until #endwrite is reached. Just about anything (directives, nesting, etc.) can be used within #write/#endwrite. Here's an example that uses the $uniquename() function to generate a tmpfile name, then appends to the file. Note that @PROJDIR is also used.
#set tmpfile = $uniquename()
#write @PROJDIR/tmp/@tmpfile append
  #for color in red,blue,green
    The color is @color ...
  #endloop
#endwrite


#cat pathname1 .. pathnameN
Copy the contents of the given file(s) directly to stdout. Files can be text or binary files.


#nonhtml
Tells quisp to refrain from supplying a Content-Type text/html header. You must then output your own Content-type header first thing. If #nonhtml is used it must be at the top of your page file.

#+
    Same as no directive at all, but allows tighter control over
    the amount of leading space, occasionally useful.

#exit
    Stop processing... terminate page result.



Directives – running shell commands and capturing results

#shell   #endshell
Run the given shell command(s). After var evaluation the quisp #shell directive attempts to rub out any shell-dangerous characters found, then the command is executed. Running shell commands in a CGI context involves risks. For instance guard against hacks when building pathnames eg. #set fname = $change( "..", "", @fname )

Example 1: Print the result directly.
#shell
    cat /etc/motd
#endshell


Example 2: Build a command based on context, run it, use the #processrows option to capture results, and use $shellrow() function to get each rows and assign result fields to quisp vars id, x, and y. Note that quisp if/else, loops, set, and include can be used inside an #shell/#endshell... but most other quisp directives cannot.
#shell #processrows
    cat /home/scg/mydata |
    #if @mode = num
        sort -nr
    #else
        sort
    #endif
#endshell
#set total = 0
#while $shellrow( id, x, y ) = 0
    #set total = $arith( @total+@x )
#endloop




Directives – SQL database interactions

#sql   [#endsql]
Run the given shsql command and capture any results. SQL commands can be given in a one-line construct or multiline construct (see examples).

[Click here for details on submitting queries and building SQL].

Example 1: Issue an SQL command and use the #load option to copy the values of "name" and "desc" into quisp vars of same name.
#sql #load select name, desc from dict where id = "@reqid"


Example 2: Issue an SQL command and process result rows one by one.
#sql select name, desc from dict order by name
#while $sqlrow() = 0
    #+ Name = @name .... description = @desc <br>
#endloop


Example 3: Issue an SQL delete command
#sql delete from places where name = "Trelawney"

Example 4: Using the multiline construct, build an sql command based on context, run it, and if there are any result rows process then one at a time.
#sql
    select name, val from totals
    #if @ord = alpha
        order by name
    #elseif @ord = num
        order by val (num desc)
    #endif
#endsql
#if $sqlrowcount() > 0
    #while $sqlrow() = 0
        Name = @name .... description = @desc <br>
    #endloop
#endif

Note that quisp if/else, loops, set, and include can be used inside an #sql/#endsql... but most other quisp directives cannot.



Directives – building html forms, and capturing user input
Quisp includes a number of features for building html forms on the fly, as well as capturing user form input. [Click here for full info].





Functions
Quisp includes a fairly well-rounded collection of functions for datatype checking, arithmetic/math, and working with strings, lists, dates, times, and misc. Function names begin with a dollar-sign ($). Functions usually return values and are used with #set, #if/#elseif, and #call as seen in examples above.

[Click here for complete function reference]   (info applies to quisp, just ignore the "plotting" functions). A few commonly used functions:

$arith( expr [, fmt] ) ... evaluate simple arithmetic expr having numerics and operators + - / * for example: #set total = $arith( @x+@y )
expr should have no embedded spaces or parentheses and evaluation is strictly left-to-right. The "fmt" arg is an optional printf-style format, for example #set pct = $arith( @total*100, "%.2f" )

$change( old, new, string ) ... in string, change all occurrances of old to new. For example: #set fname = $change( "..", "", @fname )

$strlen( str ) .... return the number of chars in str
For example: #if $strlen( @title ) < 15

$getenv( varname ) ... return current value of environment variable varname
For example:   #set ip = $getenv( REMOTE_ADDR )

$uniquename() ... return a unique 9-char string, useful for tmpfile names
For example: #set tmpname = $uniquename()

$fileexists( dir, pathname ) ... returns 1 if a certain file exists, relative to dir, 0 otherwise. dir may be "/" or "scriptdir" or "tmpdir" or "datadir". Example:
#set pathname = @PROJDIR "/tmp/" @handle
#if $fileexists( /, @pathname ) = 1
    ....
#endif


Additional info on:   dates   times   (info applies to quisp)



Hints for designing applications
The quisp executable handles all your app's CGI calls. Your quisp executable's url can be conveniently referred to using the special var @CGIPROG.
Build html links between pages in your app like this (for ./pages/test1): <a href="@CGIPROG?rtn=test1">click here</a> ...and similarly for forms use <form action="@CGIPROG">. For images and other files that are served directly don't use @CGIPROG, just specify the url directly. If you have a lot of pages you can organize them into subdirectories. Page names are relative to ./pages so you could have eg. <a href="@CGIPROG?rtn=people/form">...

It is often convenient to have all your pages #include the same codefile for page header and another one for page footer. For instance, header codefile can set variables generally needed by the app, include your CSS, and generate html for a standardized page header, so that all pages in the app have a uniform look. If your app requires user passwords, session cookies, etc. this housekeeping work can be done in the header file. You can also log user hits there (see below).

Depending on your environment, your quisp executable may need to be granted sufficient runtime privilege so that your app can update files. One way to see is to create a quisp test page that simply writes to a file like /tmp/test001 ... visit the test page with your browser then do -ls -l /tmp/test001. If this file ownership is what you want then you need do nothing. Otherwise you'll need to make your quisp executable setuid to the UID you want. Do this at your own risk; escalating runtime privilege of a CGI program can increase your risk if you get hacked. Consider having two quisp executables (@CGIPROG and @WRITEPROG), one unpriviledged and the other a setuid copy for the pages that need to update files.

Cleaning up of temp files... if your app generates temp files that accumulate in, say, /home/scg/db1/tmp the following linux find(1) command can be invoked nightly in your crontab to clean them up:
/usr/bin/find /home/scg/db1/tmp -mtime +1 -exec rm -f {} \;



Security / safety issues
Always remember that user variables, cookies, and http vars like REMOTE_HOST can be easily manipulated and can have really anything in them. Use the #cgivar datatypes as strictly as possible, and when building file pathnames guard against hacks eg. #set fname = $change( "..", "", @fname ). Avoid allowing user variable content to be present in #shell commands; if it's unavoidable use caution and confirm that #shell dangerous character suppression is working as you expect. Quisp has some additional safety features such as rejection of invalid page names, and quisp has been used without incident for a number of years in fairly low-profile settings, but there are no guarantees. Keep an eye on usage and web_error logs and watch for suspicious activity. Once your application goes live on the internet anyone in the world can probably find it and some out there may try to hack it.

Use these tools at your own risk and understand exactly what you're doing at all times.



Logs
Quisp errors will be logged in the file ./logs/web_errorlog. Any shsql errors will be logged in ./logs/dberrorlog You may wish to create log files in advance and chmod them to 777 so ordinary unprivileged httpd can write to them. Any other logging that you wish to do must be coded explicitly. The following is a code chunk that logs each hit with date, time, user IP address, and http query_string:
#call $setdatefmt( "yyyy/mm/dd" )
#set CURRENT_DATE = $todaysdate()
#set CURRENT_TIME = $time()
#set QUERY_STRING = $getenv( "QUERY_STRING" )
#set IP = $getenv( "REMOTE_ADDR" )
#cgivar rtn
#write @PROJDIR/logs/usagelog append
#+ @CURRENT_DATE @CURRENT_TIME page=@rtn q=@QUERY_STRING ip=@IP
#endwrite




Config file for quisp (and shsql)
This important file is read first-thing by every quisp process to get its bearings. Create using a text editor when you set up a quisp project and then set your environment var SHSQL_CONFIG to its pathname. Note that quisp and shsql share one config file; parameters for both packages are discussed below. Suppose the projname is "db1":
projectdir: /home/scg/db1
tmpdir: /home/scg/db1/tmp
shsql_bin: /home/scg/db1/bin
putenv: LC_ALL=C
putenv: SHSQL_CONFIG=/home/scg/db1/config
varvalue: CGIPROG=http://myserver.com/cgi-bin/q
varvalue: PROJDIR=/home/scg/db1
quisp_charset: UTF-8
dateformat: yyyy/mm/dd
Mandatory settings:
• projectdir .... full pathname of project dir (where ./pages ./data etc. located)
• tmpdir .... full pathname of a temp file dir (usually ./tmp)
• shsql_bin ..... full pathname of shsql bin
• putenv .... set an environment variable, to set up environment for spawned child processes. LC_ALL is important; it controls sort(1) behavior which must be consistent with internal qsort() to avoid anomolies.

These settings are optional:
• varvalue .... set a quisp variable to a value. In this case we're setting variable CGIPROG to the url of our quisp executable for convenience in building intrernal hyperlinks within our app.
• quisp_pgvar .... user var for conveying the quisp page name (default="rtn")
• quisp_fallbackpg .... quisp page to show if no page requested (default="")
• quisp_charset .... for example quisp_charset: utf-8 ... if this is specified the charset is stated for each page eg: Content-type: text/html; charset=utf-8
• quisp_cpulimit .... sets the maximum number of cpu seconds allowed for page processing before abort (default=20)
• quisp_okcookies .... if specified only the listed cookies are accessed. A commalist of cookie names or wildcard patterns matching cookie names. Example: quisp_okcookies: CLF_* ... causes the application to retrieve only those cookies with names beginning with CLF_.
• quisp_oktags .... defines a list of html tags that are allowed in user textarea input; all others are rubbed out. The default is a b i br p li ul sup sub small font h2 h3 h4 tt pre
• shellmetachars .... list of dangerous chars to disallow in shell commands. Default is \"'`$\\;|", (double-quote, single-quote, back-quote, dollar-sign, backslash, semi-colon, and pipe symbol).
• dateformat .... set a default date format
• A few other optional shsql-related settings are described here.




Command line utility – quisputil
While quisp's main use is to be invoked as a CGI and generate html, there's also a command line utility version available called quisputil for other things such as filtering and formating data. Just like quisp, you create a template file with the directives and variables as described above. Then you invoke quisputil command and result appears on stdout. Or, you can pipe a stream of data to quisputil -dp, and it will evaluate the page template repeatedly, once for each line of incoming data.

Usage: cat mydata | quisputil [options] templatefile [varvalue-pairs] > out

Options:
-dp .... data processing mode ... a shsql-compliant stream will be read on stdin. Data fields will be available in your template as @1, @2, etc.
-h .... used with -dp, indicates that the data stream begins with a field name header. Data fields can then be referenced by name in your template.
-H .... same as -h but field name header is echoed to stdout. Typical use is with a template that massages then outputs another result data stream.
-c script .... the quisp template content is given directly on the command line, as one shell arg. and no templatefile arg should be given. Be sure to escape out shell metacharacters, eg. \$
-ver ... print the quisp version number and exit.

Varvalue-pairs... allow quisp variables to be initialized on the command line, eg. quisputil -dp report total=0

To set quisputil's shell exit code use a directive such as: #exit 2   All directives mentioned on this page are usable in quisputil except those that apply only in a web context such as #cookie or anything to do with generating html forms. You must have a config file and your working environment must have SHSQL_CONFIG set to the full pathname of the config file.