The extent to which these databases reflect the “contents” of the Web in an accurate and timely manner is now under considerable doubt, and in any event, it is apparent that the methods used by the search engines for finding new and modified Web documents are not scaling well. To ameliorate these problems, we have been exploring the use of a “data push” model for notifying Web changes, to replace the current “data pull” model, which uses aglets (aka servlets or peerlets) to distribute the indexing task.

This is a continuation of my earlier posts on How do search engine robots work and Methods for highlighting search queries by Search Engines.

To sum up from these earlier posts

Search engines which use current “data pull” model consist of discrete software components like the spider.

Spider : a robotic browser like program that downloads webpages. It finds new work in two main ways. Firstly it repetitively follows hyperlinks in known documents to find unknown documents. In the initial stages of spidering, seed URLs are given to the spider and the hyperlink structure is used to populate the index. In later stages, when documents are added or modified, the hyperlinks within these documents can be followed in a similar fashion. Secondly, the spider tracks changes and deletions in documents already indexed, by requesting header information and checking document time-stamps.

Now Tracking changes and deletions through HTTP is less efficient, because it is based on a “data-pull” communication model and because each request is in terms of a single document. If there is new work for an indexer to do, the only way to find out through HTTP is to send a request for information on each document indexed. For large web indexes this means up to 50 million requests before all documents have been checked, only a very small fraction of which will result in new work. One way of addressing this inefficiency is to reduce the number of HTTP requests a spider must send to find new work. A notable proposal is for a strong>sitelist.txt standard, where it becomes each web server’s responsibility to provide a single text file listing all files and their time-stamps. However, in such a system the amount of communication is still great regardless of the amount of new work, if any.

Data Pull vs Data Push

Instead of 50 million largely useless requests or one large request per site, a small notification message is sent at an appropriate time, describing in a concise fashion all new work. Novel server push methods have already been proposed using, for example, a request for email notification whenever a stated document changes.

A “data-push” model for finding new indexing work requires a certain amount of computation and state storage on the web server end. This makes it an ideal application for aglet technology, because not only can this remote computation take place, but the techniques used are determined by the indexer, so new technologies or indexing priorities can be reflected in new versions of the aglet software. An aglet would be dispatched by the spider/indexer to a web server, acting as its agent, working on behalf of the spider to access and perhaps predigest local information changes at the server, then Page 3 sending them to the indexer.

The search industry is at a state of play

Due to the very large volume of documents available, large web search services are expensive to run, both in terms of the cost of spidering and the cost of index building and searching. Because of this, only a small number of larger web indexes exist, and these are all commercial services (Google, Ask, Yahoo, Live, AltaVista, Excite, HotBot, Lycos and InfoSeek).

Even these services are forced to compromise for efficiency, only partially covering each site and polling documents infrequently. In fact in some services it is possible for an index to be more than three months out of date with respect to changes in a particular document. If the cost of spidering is reduced, the overall cost of running a search service also decreases and the coverage of search services may increase. For this reason there is much industry enthusiasm for finding new ways of efficiently detecting change in remote documents. The solution suggested here, a data-push model employing aglets, would offer indexers not only very efficient change notification, but could be used in the transmission of the documents themselves, by compressing, pre-indexing or even sending only changed portions of documents.

What is an Aglet?

An aglet is a Java-based mobile software agent. The term software agent has been given many definitions; here we refer to a piece of software that can halt its execution on one host, transfer to another host, and then continue execution from where it left off on the remote host.

The aglet framework was created by a team lead by Danny B. Lange at the IBM Tokyo Research Laboratory in Japan.

Aglets are designed around an event-driven callback programming model that has similarities with the Java Applet programming model. An aglet can experience any of the following events in its life:

Creation: An aglet is instantiated, and its main thread begins executing.

Disposal: An aglet is destroyed, all information is lost.

Cloning: The aglet is replicated, with current state but new identity.

Dispatch: The aglet and state is sent to a remote host.

Retract: A previously dispatched aglet is pulled back from a remote host.

Deactivation: Aglet and its state are transfered to persistent storage.

Activation:Aglet and its state are transfered from persistent storage.

Before any of these events occur, an aglet is notified of the upcoming event through a call to the appropriate callback method. For example, when an aglet is created, the OnCreation() method is invoked. A programmer can override this method with one that initialises the state of the newly formed aglet

How Aglets move between hosts

Aglets are moved from host to host using the JDK’s Object Serialisation feature. The aglet object, all serialisable objects reachable from it, and the aglet’s heap are converted into a byte stream and sent across the network. The receiving host can then reconstruct the aglet and its heap. Java does not allow access to execution stacks of the virtual machine, and so not all state is preserved in the transition. However, state can be effectively restored if the aglet is programmed like a finite state machine. Before dispatch, the current ‘‘state’’ can be recorded in variables on the heap, so then when execution begins at the receiving node, the state variables on the heap can be consulted to determine what to do next.

Aglets and Spiders
Aglets suggest themselves as an important part of the solution to the problem of the scalability of web spidering for a number of reasons:

• support of a “data push” model should decrease the network traffic and wall time required to locate changes in the web
• the possibility exists for performance-related contracts to be agreed between data suppliers and indexers to ensure that document changes are reflected in indexes in a timely, or at least predicatable, manner (opportunities for contractual consideration exist on both sides – the data provider pays in local computational power, and the indexing site guarantees index presence)
• the functionality of the aglet code is under the control of the indexing site, and so can be updated to reflect its requirements, and its core knowledge of the indexing task
• the binding of an aglet into a local site requires the consent of that site, but in this application such consent is likely (unlike, say, aglets produced by individual persons and dispatched for personal business)
• the spidering problem is fundamentally a graph-traversal problem – the web can be seen as a graph with many alternative arcs (paths) between nodes with different cost-related weights.
• The ability of aglets to clone, to have an initial intinerary, and to react to local conditions, permits numerous alternative traversal strategies to be explored.