# Documentation

# Usage

- Searching
  - Suggest Search
    - Integrating search suggest into your own page design
- Stopwords
  - Manual creation
  - Automatic creation
    - Add N new
    - Add top N
- Logging
  - Log view
    - Log filter
  - Log statistics
- Edit and delete an index

# Searching

To run a search, open your search page or the page we prepared for you called 'tsepsearch.php' in your browser and input the words to be searched. The search words are not case sensitive.

TSEP supports boolean search if you have a MySQL version equal or higher than 4. Below are some the boolean search features. Important: Your tables must be MyISAM tables if the boolean search should work. (they should be MyISAM when we created them)

#### Search for the words

apple banana
  ...find page that contain at least one of these words.
+apple +juice
  ...both words
+apple macintosh
  ...word "apple", but rank it higher if it also contain "macintosh"
+apple -macintosh
  ...word "apple" but not "macintosh"
+apple +(>pie <strudel)
  ..."apple" and "pie", or "apple" and "strudel" (in any order), but rank "apple pie" higher than "apple strudel".
apple*
  ..."apple", "apples", "applesauce", and "applet"

This will be familiar to you if you know MySQL Full text search. There is also help available directly on the search page, next to the search button.

Here are some general boolean search operators:

- + The word is mandatory in all rows returned.
- - The word cannot appear in any row returned.
- < The word that follows has a lower relevance than other words, although rows containing it will still match
- > The word that follows has a higher relevance than other words.
- ( ) Used to group words into subexpressions.
- ~ The word following contributes negatively to the relevance of the row (which is different to the '-' operator, which specifically excludes the word, or the '<' operator, which still causes the word to contribute positively to the relevance of the row.
- * The wildcard, indicating zero or more characters. It can only appear at the end of a word.
- " Anything enclosed in the double quotes is taken as a whole (so you can match phrases, for example).

The minimum length of a search term is 4, see MySQL restrictions below for details and a work around! (User defined) stopwords are not marked in the results and not used in the database query.

## Suggest Search (not yet implemented!)

In version 0942 we have introduced a feature called suggest search. This grabs the user input as he types in his search term(s) and suggests words which have been searched before by users. The user can use a suggested search term for his search. He also sees how many results these search terms produced and how many people used the search terms.

How it works - technical but important for you to understand the differences in the configuration options you can choose from:

Table log is not changed. It stores the idlog, logstring, etc. A single logstring (e.g. "Olaf Nöhring") will be stored as many times as it's used.
Table loghits stores idlog, number of times a query is executed (nr_hits) and the number of pages returned (returned_pages). The field "idlog" contains the id of first record of a logstring. So if the log table contains:

IDLOG | LOGSTRING
1 | Olaf
2 | Toon
3 | Olaf

Then table loghits contains:

IDLOG | NR_HITS | RETURNED_PAGES
1 | 2 | x
2 | 1 | x

In log.php the "nr_hits" field is updated for each query.
In search.php the "returned_pages" is updated for each query. There are 4 different ways of calculating "returned_pages":

1. The actual number of pages returned for the query that was executed just a couple of milliseconds ago is stored as is.
2. The actual number of pages returned just now is averaged with the already stored number:
   "returned_pages" = round( ( ("returned_pages" * ("nr_hits"-1) ) + "actual number of pages returned") / "nr_hits" )
3. The minimum number of returned pages is stored:
   "returned_pages" = min( "returned_pages", "actual number of pages returned" )
4. The maximum number of returned pages is stored:
   "returned_pages" = max( "returned_pages", "actual number of pages returned" )

So in conclusion:

- each query is stored in log.
- each unique logstring is stored in loghits.
- "returned_pages" in loghits is updated with or without regard to previous queries depending on the admins settings.

### Integrating search suggest into your own page design

The easiest way to incorporate the TSEP search suggest feature into your website is to use the tsepsearch.php script. By changing a few lines in the CSS file you can make the colors, etc. fit the design of your website.

If this won't do you can tweak the search.php script to exactly display as you want it to. It's a bit technical but don't let that scare you off.

The search suggest is displayed with one or two lines of PHP code, depending on what you want. First you have to display the search form by calling the class method SearchSuggest::SearchForm(). It needs a whole bunch of parameters that you need to forget about. For you the only important one is right at the very end: $includeSuggestTable
The SearchSuggest::SearchForm() method creates the searchform and the JavaScript code needed for search suggest. The $includeSuggestTable parameter can be set to TRUE or FALSE.

If you set it to TRUE the method creates the HTML code (a set of DIV's and SPAN's that look like a table structure) to display the suggestions. If you use TRUE you don't need any more PHP code to get it to work.

If you set it to FALSE this code isn't created and you need to call SearchSuggest::echoSearchTable() somewhere it fits you best to include the display HTML code. You could call this method from inside a table cell or whatever you want.

# Stopwords

Stopwords are words which will not be searched on your pages. This means that when using a stopword as a searchterm, it will not be marked as a searchterm in the results.

Stopwords are not case sensitive. This means that if you enter "Apple" in the stopwords section and the users searches for "apple" this word will be treated as a stopword.

At this time you can easily add stopwords by entering severel words at once into the "add new stopwords" field on the stopwords.php page.

We provide lists of stopwords for several languages. Please take a look into the language folder (/language), and in the folder of the language you desire stopwords for (for english: /language/en). You will find a file called stopword.txt. This is a simple text file which can be opened using any texteditor/-viewer. The file contains some meta information (lines beginning with the ";" character) and a list of stopwords which looks much like this: a,a's,able,about,above As you see, the stopwords are seperated by coma. You can simply copy and paste all the stopwords into the "add new stopwords"-field and click the submit button. TSEP will automatically add all the stopwords to your stopword list!

Please note that there are MySQL restrictions on stopwords as well!

You can add, update and delete your own stopwords.

### Manual

Simply enter a stopword in the form field and press the "add" button. You can also enter more than once stopword at once, seperating them by comma.

Example: this,is,a,stopword,list

### Automatic

Stopwords can be automatically generated based on the occurrences of a typical word included in a website. All stopwords that are added are displayed above the stopword list after a search is conducted.

We have implemented two, slightly different working methods.

#### Add N new

The administrator has the ability to enter an integer value of stopwords. The value entered will be the number of stopwords searched for. The most frequently found words will then be added to the stopwords database. By default stopwords that are already in the database are overwritten.

#### Add top N

In the case that the administrator only wants new stopwords, he or she may check the option "Retrieve new stopwords only".

# Logging

This was introduced in version 0.911. The administrator can define in the setup file whether and what search activity should be logged. All log entries are accompanied by a timestamp. The admin can decide to log the following: IP address, search term and clicks on the results.

The administrator may want to analyse what users are searching for on his site and make navigation to those points easier.

The administrator can also log the IP address of the person searching. Be aware that people might not like the idea of you "spying" on them. But we thought this might be a useful feature - maybe especially for Intranets. In those, if someone is totally lost the administrator can take him by his hand and help directly.

The administrator may want / need to notify the users if their actions are being loged, especially when logging their IP address.

For sorting the log entries by IP adddress, MySQL v3.23 or higher is required. (MySQL restriction on IP sorting)

Note: When you search from the search field in the 'indexer.php, this search will not be logged!

## Log view

The logged data can be seen in the file /admin/logview.php

You need a pretty wide screen (preferably 1280x1024 or wider) to see all log entries nicely side by side. If the are to long they are not completely displayed to prevent overlapping. You can move (and hold) your mouse over the entries to see the complete entry in the title tag.

### Log filter

In version 0.941 we introduced a filter function for the log. The log display is always filtered! The administrator can change the following parameters:

- logview type
- searchterm
- date/time. You can click into the fields to open a calendar. (formats for date: YYYY-[M]M-[D]D, YYYY/[M]M/[D]D, YYYYMMDD and HH:MM:SS for time)
- IP address (Only format accepted are variations on: 999.999.999.999, limits: 000.000.000.000 through 255.255.255.255)
- resolved IP
- stopwords in search terms

You can move your mouse over some of the fields to see example values.

## Log statistics

In version 0.941 we started statistics for the log. The statistics show for example:

- Totals: Log records, search queries, clicked URLs
- Details: Unique search words / stopwords / IP addresses / domains
- Top N: Search queries, clicked URLs

# Edit and delete an index

Some people asked how they can delete a word from the index or correct a word. In version 0.910 we introduced the possibility to do this right from TSEP.

Open the indexoverview.php page either directly or from any page in the admin folder (indexer.php for example). Look for the page you want to edit. Click in the title column on the link (title of the page). This takes you to the editing. Be aware: This behaviour has changed in 0.938 - if you click on the URL (in the URL column) you will open the page itself!

In version 0942 we introduced new fields in the indexedit:

Additional Information
  This is for additional information the administrator want's to attach to a page. The contents of this can be shown to the searching user in the results - in a future version of TSEP.
Last Indexing date
  This specifies when the page was indexed the last time
Last Edit date
  this specifies when the indexentry of the page was edited the last time

Tipp: You can still call the indexedit.php directly (with no parameters). This will show you more page details at once and you can still add completly new pages to the index! The number of pages you see there is defined in the configuration.php by this value in the Limits group:

How many index entries in the show complete index page in one html document? Be careful not to set a too high number as the page might get very big!