© 2008 Michael Thompson Software Documentation [ Main Index ]

lMaker: a text-file driven website generator

lMaker is a php class designed for web masters and programmers who want a simple way to generate complex, dynamic web sites from easily maintainable text files. It is designed to help automate some of the most repetitive features of website design thus leaving the web site maintainer free to focus on the unique elements of the content. Using lMaker to generate web pages on the fly from text files typically reduces the amount of data in files the web master has to maintain by at least a factor of 4 .

lMaker was originally conceived as a way to handle the repetitive elements of HTML markup for the long lists of links often found within portals, site indexes or site maps. Providing a consistent appearance and persistent navigation area in web pages can result in a lot of repetition within the HTML source. With lMaker, only the unique information is required for each link; the url, the text or image, and optionally a description. All the repetitive elements that may accompany each link, such as markup for HTML tables, lists or any other decorative features associated with each link need only be defined once.

lMaker makes it easy to implement the popular “tab” style controls by generating multiple views using only one php script together with one or more text files.

The ability to generate an entire website such that each page has a uniform appearance and persistent navigation area without resorting to the use of frames is another key feature. Website usability studies have shown that frames “break the fundamental user model of the web page,” interfere with bookmarking^[1] and can make the website problematic for search engines^[2].

There are many other methods that may be used to achieve the same results as the ones shown in the following examples but lMaker's systematic approach can greatly reduce the amount of staffing required to maintain many complex websites.

Fundamental Concepts

lMaker generates web pages by processing plain tab-delimited text files. Each text file should consist of lines in the pattern of keyword [tab] value. The complete documentation for each of the defined keywords is available at the project website ^[3]

lMaker processes text files from top to bottom and the order in which the case sensitive keywords are encountered is important. When some keywords are processed the value associated with it is held for later use. Other keywords are designed to generate output as soon as they are encountered. It depends on the way the keyword is designed to function.

In the lMaker processing model, a valid input file is expected to contain one or more views. Within a view we may specify one or more output modes and typically a list of links. When links are encountered they are output as HTML markup in a format dictated by the current mode.

The expected keyword sequence is: view then mode then link.

The keyword view is used to identify which sets of lines within a file will be processed when the given view is selected. For example, the line view[tab]view1 specifies that the lines following it are to be processed only when value of the view argument passed the lMaker processMap function equals “view1”. The lines after view[tab]view2 would be normally be associated only with the selection “view2” and so on. A view continues until the next view statement is encountered or the file ends. Listing 1 illustrates this concept.

Listing 1. lMaker usage example illustrating the concept of separate views. Here [tab] is used to represent the tab character (\x09) <?php require_once("lMaker.php"); $it = new lMaker(); # processMap($file,$view) $it->processMap("site.lis", “view1”); ?> # site.lis view[tab]view1 # anything that follows belongs to view1 ... ... view[tab]view2 # anything that follows belongs to view2 ... ... view[tab]viewN # anything that follows belongs to viewN ... ...

The keyword mode determines what HTML markup we want associated with each link. For example, mode[tab]table indicates we want each link to occupy an entire row in an HTML table, whereas mode[tab]cell would indicate we want each link to be within a cell of an HTML table. Other possible values for the mode are: list, olist and plain.

lMaker is designed to handle any HTML markup needed (if any) to 'open' and 'close' the output mode. For example, if the value of the mode is table then the first link is preceded at least by the HTML tag <table> and the last is followed by the </table> tag. The same concept applies to the HTML lists; given mode[tab]olist each link that follows is wrapped by the tags <li> and </li> and whole list of links is wrapped by the <ol>, </ol> tags. The keyword edom can be used to close the mode immediately instead of waiting for the default behavior which is to close the mode when another mode statement is encountered or when the view or file ends. Listing 2 illustrates the basic usage of the mode and link keywords.

Listing 2. a more detailed usage example # site.lis view main mode cell modePrefix <table width=”100%”><tr> modeSuffix </tr></table> tdAttrib valign=top # this will be prepended to each link that follows linkPrefix /index.php?view= # here comes a list of links, in this case, selectable views title First View link view1 title Second View link view2 # ... and so on title Nth View link viewN # The output from lMaker given the above with the view argument # having a value of “main” would then be as follows: <table width=”100%”><tr> <td valign=top><a href=”./index.php?view=view1”>First View</a></td> <td valign=top><a href=”./index.php?view=view2”>Second View</a></td> <td valign=top><a href=”./index.php?view=viewN”>Nth View</a></td> </tr></table>

Note also in Listing 2 the modePrefix, modeSuffix statements. Just about all of the definables elements can be assigned prefixes and suffixes. For example, in the case of a view, a typical viewPrefix might be <div class=”some_class”> with a corresponding viewSuffix of </div>.

The link keyword generates output as soon as it is encountered. This means that any extra information associated with the link such as the title, img, desc (description), and any other optional attributes need to have been defined previously or it will not be displayed as expected.

Value Assignments

lMaker provides a variety of methods to assign values to the keywords. Most often the value we use is the text remaining after the tab character. Here are examples showing each method:

-- simple assignment:

keyword[tab]string of text to the end of the current line

-- heredocs method^[4]:

keyword[tab]<<<END
the value may be set using the heredocs method,
which is a useful method for handling multiple lines of text,
potentially troublesome characters like
the [tab] which is the value of our delimiter
or other atypical or unusually lengthy data.
END

-- from a file or URL:

keyword[tab] <<! ./inc/man1p2.inc.html
keyword[tab] <<! http://mydomain.com?output_contents.php

-- from a pipe:

keyword[tab] <<|cat files[0-9].txt|sed s/”\n”/”<br>”/1
keyword[tab] <<|php -q '<?php rand(0,99); ?>'

The pipe method creates a lot of possibilities. Imagine the problem of setting the value of a link based on the current date. Here is one way to handle it with lMaker:

title Today in History
linkPrefix ./history .php?ymd=
link <<| date '+%Y%m%d'
# the result always includes the current date like this
# <a href=”./history .php?ymd=20050913”>Today in History</a>

Consider a simple design for displaying the most recent image from a web cam:

linkPrefix /to/images/
# ls -1t where 1= file path only, t=sort from newest to oldest
img <<| ls -1t ./camera0/*.png | head -1

# first set a line break as the value of desc
desc <br clear=all>
# then append desc with the current system date and time
# the dot “.” preceding next desc statement means “append”
.desc <<| date
link /to/camera/
# the final results will be similar to this...
# <a href=”/to/camera”><img src=”/to/images/latest.png”></a>
# <br clear=all>Tue Sep 13 12:35:57 GMT-0500 2005

A key lMaker feature is the variety of ways that keyword values can be assigned. Listing 3 shows how each is implemented.

Listing 3. Four ways to get the value of a variable; 'heredocs', from a file, a pipe, or most typically, from the current line and assigning that value to a variable. # getting the value for a variable $value = (count($ar)>1 ? $ar[1] : null); if( $value!=null ) { if( substr($value,0,3)=="<<<" ) # <<< signals heredocs   { $end = trim(str_replace("<<<","",$value)) . "\n";     $value = "";     while( !feof($fh) )     { $buff = fgets($fh);      if( $buff==$end ) break;      $value .= (strlen($value)?"\n":"") . rtrim($buff);     }   }   elseif( substr($value,0,3)=="<<!" ) # <<! read from file   { $rf = trim(substr($value,3));     if( ($bMust || $bInView) && strlen($rf)>0 )     { $value = @implode("",@file($rf));     }   }   elseif( substr($value,0,3)=="<<|" ) # read from a pipe   { $rf = trim(substr($value,3));     if( ($bMust || $bInView) && strlen($rf)>0 )     { $value = "";       if( ($ph=@popen($rf,"r")) )       {         while( !feof($ph) )         { $value .= fgets($ph);         }         fclose($ph);       }     }   }   else $value = $ar[1]; # remainder of current line } # assigning the value to the variable name ${$ar[0]}=$value; $this->arV[$ar[0]]=$value; if( $value==null ) { unset(${$ar[0]}); unset($this->arV[$ar[0]]); continue; }

Conditional Assignments

lMaker also supports conditional assignments. Normally the lines of a file are only processed if we are within the correct view but we can override this behavior. In this example we make use of the literal keyword which outputs its value immediately:

:literal <html><head><title>
:?view==1 literal[tab]First View
:?view==2 literal[tab]Second View
:?view==N literal[tab]Nth View
:literal </title></head><body>
:!view==1 title[tab]&lt;&lt;Previous
:!view==1 link[tab]./index.php?go=-1
:literal ::You are here::
:!view==N literal[tab]Next&gt;&gt;
:!view==N link[tab]./index.php?go=+1

Lines preceded by a colon ':' are processed in all views. Keywords preceded by a question mark '?' are processed only if the associated condition is true. Keywords preceded by an exclamation point are processed only if the associated condition is not true.

Given the example above, if the selected view is '2', our output will be as follows:

<html><head><title>
Second View
</title></head><body>
<a href=”./index.php?go=-1”>&lt;&lt;Previous</a>
::You are here::
<a href=”./index.php?go=+1”>Next&gt;&gt;</a>

If instead the selected view is '1', our output would be as follows:

<html><head><title>
First View
</title></head><body>
::You are here::
<a href=”./index.php?go=+1”>Next&gt;&gt;</a>

In this example index.php is given the responsibility for incrementing and decrementing the view number, perhaps like this:

<?php
...
if( isset($go) ) $ view = $view + $go;
$it->processMap(“site.lis”,$view);
...
?>

It is important to note in the previous example where we used the built-in conditional statements that these have a very specific, limited capability. The left side of the comparison must be a globally accessible variable and the right side must be a constant. The comparison accepts the usual conditional operators: ==, >=, >, <, <=, !=

To handle more complex conditions there is a provision for a user-defined callback function. Here is how the input file would differ:

:fx myCallbackFunction
:literal <html><head><title>

# if callback function returns true...
:? literal[tab]First View
:? literal[tab]Second View
:? literal[tab]Nth View
: </title></head><body>

# if callback function returns false...
:! literal[tab]<< Previous ::
: literal[tab]You are here
:! literal[tab]:: Next >>

The user-defined callback function should return true or false and must accept the following five arguments: the keyword, the keyword's current value, the file name, the view, and an array containing the keyword-value pairs defined prior to the current line. If the callback function is undefined lMaker behaves as if the callback function had returned false.

We are not restricted to the defined keywords. Sometimes it is useful to set a value for user-defined variables for the benefit of the callback function. The variable names chosen need only conform to the usual variable naming conventions but it is wise to plan for future versions of lMaker which may include additional keywords. It is a good practice is to use safe variable naming conventions such as prefacing user-defined variable names with something like “user_ “ or “my_”

The value of the view keyword usually consists of the name of a single view but lMaker also accepts a list of view names separated by vertical bar '|'. This makes it possible to have lines processed in more than one view. Here is a brief example of the concept which is particularly useful for implementing expandable outlines.:

# the following lines will be processed when the
# view is “all” or when the view is “section1”
view all|section1
...
view all|section2
...
view all|sectionN
...

More About the Keywords

All of the reserved keywords can be divided into three categories. Some keywords – the “verbs” – generate output as soon as they are encountered: view, mode, edom, literal, link

Other keywords – the “adverbs” – affect the output only if a corresponding verb is encountered:

viewPrefix, viewSuffix in conjunction with view
modePrefix, modeHeading. modeSuffix in conjunction with mode or edom
title,img,desc, and the following attributes, if they are set, augment link

itemPrefix, itemSuffix, itemPointer, titlePrefix, titleSuffix, alt
descPrefix, descSuffix, linkPrefix, linkSuffix

The remaining keywords generate no output at all directly but are instead used to control what happens during the processing of the input file: fx, push, pull, clear
The keywords push and pull are primarily used to stack and unstack keyword values but they can also be used with the mode keyword to create nested lists ( i.e. <ol>, <ul>)

Consider the case shown in Listing 4, where most of the links in a list are to be given one sort of format but one expection in the list is to be handled differently.

Listing 4. An illustration of the use of keywords push and pull # site.lis view 1 mode plain itemSuffix <br> descPrefix &nbsp;-&nbsp; linkPrefix ./rss.php?rss= title Daily link today.rss push itemSuffix push linkPrefix # at this point itemSuffix and linkPrefix are null # but their previous values are saved on the stack # as the next link does not need the usual prefix title {Hour} desc news of the day<br> link http://myhourlynews.com/ pull itemSuffix pull linkPrefix # at this point itemSuffix and linkPrefix # have been returned to their previous values title Weekly desc news of the week link week.rss title Monthly desc news of the month link month.rss # The resulting output from processing view '1' will be like this: <a href=”./rss.php?rss=today.rss”>Daily</a> <a href=”http://myhourlynews.com/”>{Hour}</a> - news of the day<br> <a href=”./rss.php?rss=weekly.rss”>Weekly</a> - news of the week<br> <a href=”./rss.php?rss=monthly.rss”>Monthly</a> - news of the month<br> # For further clarity, here is a view of the result # without showing the markup Daily {Hourly} – news of the day Weekly – news of the week Monthly – news of the month

Implementing Tabbed Views

Now that the basics have been covered, consider Listing 5. It is a simplistic example of how lMaker can be used to handle the tabbed views of a website. There are just a few key points to consider; displaying all the selectable tabs, highlighting the selected tab and restricting the content to the selected tab.

Listing 5. Tabbed view <?php # index.php require_once("lMaker.php"); $it = new lMaker(); $it->processMap("site.lis",$view); ?> # site.lis # the selectable tabs must appear in all views :modePrefix <table width=”100%”><tr> :mode cell :modeSuffix </tr></table> :linkPrefix ./index.php?view= # only the selected tab will have the yellow background :?view==1 tdAttrib bgcolor=”#ffff00” :?view!=1 tdAttrib bgcolor=”#999999” :title Tab1 :link 1 :?view==2 tdAttrib bgcolor=”#ffff00” :?view!=2 tdAttrib bgcolor=”#999999” :title Tab2 :link 2 :?view==N tdAttrib bgcolor=”#ffff00” :?view!=N tdAttrib bgcolor=”#999999” :title TabN :link N # close the current mode, i.e. </tr></table> :edom # this will be processed only if the first tab is selected view 1 literal <<! ./inc/tab1.inc.html # this will be processed only if the second tab is selected view 2 literal <<! ./inc/tab2.inc.html # this will be processed only if the Nth tab is selected view N literal <<! ./inc/tabN.inc.html

Website Automation

Because the input files for lMaker are plain text files, it is simple to quickly generate an entire website or certain portions of a website from a variety of sources. Consider the case of populating a company intranet with a collection of Microsoft Word^TM documents. A simple shell script as shown in Listing 6 could do the job in moments. Further, since all that is required to regenerate input files is a simple shell script, it is easy enough to automate site updates by scheduling them with cron^[5].

Listing 6. A simple shell script to quickly create an lMaker input file. In this example docReader represents a command line utility such as antiword[6] #!/bin/sh : cat <<ECAT view allfiles mode table tdAttrib valign=top linkPrefix /cgi-bin/docReader?f= ECAT for FILE in `find . -type f -name “*.doc” -print` do # the name of the file will serve as title of the link     NAME=`basename $FILE` # strip off the extension     TITLE=`echo $NAME | sed s/”\.doc”//1`     echo “title $TITLE” # first 10 lines of the file will serve as the description     echo “desc <<<END”     docReader $FILE | head     echo “END”     echo “link $FILE” done # when the resulting text file is processed by lMaker # the output from lMaker will be similar to this... <table> <tr><td valign=top> <a href=”/cgi-bin/docReader?f=Doc1”>Doc1</a></td> <td>first 10 lines of Doc1...</td></tr> <tr><td valign=top> <a href=”/cgi-bin/docReader?f=Doc2”>Doc2</a></td> <td>first 10 lines of Doc2...</td></tr> <tr><td valign=top> <a href=”/cgi-bin/docReader?f=DocN”>DocN</a></td> <td>first 10 lines of DocN...</td></tr> </table>

Listing 7. A shell script to create lMaker input files for a list of files indexed alphabetically #!/bin/sh : cat <<ECAT # since we want this portion to appear in all views # we preface each keyword with a colon “:” :modePrefix <center><table><tr> :modeSuffix </tr></table></center> :mode cell :linkPrefix /index.php?view= :title A through F :link [a-f] :title G through K :link [g-k] :title L through R :link [l-r] :title S through Z :link [s-z] :mode table :tdAttrib valign=top :linkPrefix /cgi-bin/docReader?f= ECAT # alphabetical indexing for VIEW in [a-f] [g-k] [l-r] [s-z] do     echo “view $VIEW”     for FILE in `find . -type f -name “$VIEW*.doc” -print | sort`     do         NAME=`basename $FILE`         TITLE=`echo $NAME | sed s/”\.doc”//1`         echo “title $TITLE”         echo “desc <<<END”         docReader $FILE | head         echo “END”         echo “link $FILE”     done done

What's Next?

lMaker's capabilities will continue to grow. The work in progress includes; modes specifically for XHTML and XML, more flexibility in the area of variable assignments, and more advanced conditional assignments.

References

[1] Top 10 Mistakes in Website Design: http://www.useit.com/alertbox/9605a.html
[2] Frames and Search Engines: http://www.searchengines.com/frames.html
[3] lMaker : Syntax
[4] Heredocs: http://www.onlamp.com/lpt/a/3367
[5] cron http://www.uwsg.iu.edu/usail/automation/cron.html
[6] antiword: http://www.winfield.demon.nl/

Author: Michael Thompson
The author is a software engineer currently working as the IT Director and chief software architect for a multinational manufacturer.