2.1 Introduction

Knowing what you need is crucial before you can start a search. Therefore first the key criteria to define the quality and efficiency of the search are introduced in Recall and precision.

Then the process of defining and finding relevant documents is described in Defining relevant results for your research and Working through a set of documents

Knowing the key criteria and what results are needed enables the selection of a set of documents in Selecting a set of documents and knowing when the search can be finished in When to stop.

2.2 Recall and precision

Recall and precision are key criteria for the assessment of the search process, because the quality and efficiency of the search process are important. Although recall is difficult to measure, it can still be used to understand the quality of your search.

In practice, 100% recall and 100% precision at the same time is not possible. Performing a search is therefore always a balance between effort and quality.

Recall (how complete it is) is defined as:

ratio of relevant documents found compared with all relevant documents.

$$\text{recall} = \frac{\text{relevant documents found}}{\text{all relevant documents}}$$

Precision is defined as:

ratio of relevant documents found compared with all documents found.

$$\text{precision} = \frac{\text{relevant documents found}}{\text{all documents found}}$$

The collections of documents used for both criteria are shown in figure 1.

A proper quality of a search for patent documents is in many cases important. In for example product development, it is important that your questions about the patent information to be used for your development are completely answered. Missing a relevant document can lead to unnecessary development work or to the inability to sell a product. This can have serious financial consequences.

The recall is thus a good criteria to assess the quality of the search. Normally the achieved recall can only be estimated by redoing the search by another person. It is therefore not a criterium that can be easily used to assess the quality of the search. In the following parts is described how to work with the recall criterium.

The efficiency of a search has an important effect on the costs of a search. The precision criterium can be used for assessing the efficiency. This can easily be measured after the search is finished by comparing the total number of documents selected to the relevant documents. This comparison has on its own almost no value. When comparing different searches or with peers, this comparison can be used for assessing efficiency. Also in the following parts the use of this criterium is described.

In general experience recall and precision work in opposite directions: As recall improves, precision decreases, and as precision increases, recall decreases. During searching you will have to make a choice about this balance for yourself. The big problem with this is that normally you do not know the recall of your search. Experience with searching will help to make better estimations for this balance. This should also become clear in the following parts.

For more information about recall and precision, see for example wikipedia.

2.3 Defining relevant results for your research

Before starting a search you should have a clear picture of what the relevant results should be. The relevant results are determined by the questions to be answered in the research.

For example for a patent examiner the claims of the patent application determine how relevant the searched documents are. This is a relative clear criterion to use for the selection of the documents. If you are doing a research for technical solutions the criterion to select documents is less clear, because often you are searching for concepts or all solutions to a certain problem. What helps in this case is to determine a technical feature or a few features that have to be present in a document to be relevant. This will allow you to quickly select possible relevant documents.

Another important reason to have a clear definition of the relevant results, is that it allows you to determine what not to look for. If it is possible to exclude certain technical fields or applications, this will reduce the amount of documents (and thus increase the precision).

2.4 Working through a set of documents

Once you have defined what the relevant results are, you can start selecting the relevant documents.

It is often difficult to quickly determine if a document is really relevant. To determine this you normally have to read the document. Reading too many documents is not possible in a limited time. Therefore as a first pass you select possible relevant documents from a workable set of documents (see figure 2). You select possible relevant documents by using your criterion for selecting these documents as described in the previous paragraph.

In this first pass it is normally not necessary to read the documents. It is often sufficient to look at the figures or read the abstract (or also a part of the description) to see if a document will pass or fail your criterion. In this way you create a selected group of possible relevant documents in the first pass.

When the first pass is done, it is time to have a more accurate look at the selected group of documents. By reading the documents the really relevant documents can be found (see figure 3). These documents give the answers to the questions of your research.

2.5 Selecting a set of documents

Selecting a workable set containing all relevant documents is the most important part of a search, because this has a huge influence on the quality and amount of work to go through the set (recall and precision). See figure 4 to get an impression of the number of documents.

In the previous paragraph is described how to work through the set to finally select only the really relevant documents. This is “just” a lot of work. Selecting this workable set does not have to take a lot of time, but is the most difficult part of finding information from patents.

In practise it is very difficult to select all relevant documents (to have a 100% recall). Therefore a workable goal is to come as close as possible to a 100% recall with a workable amount of documents (high precision). This part will give an introduction into the most used methods to achieve this goal.

The available methods are determined by the information in the databases and by the functionalities of the current search engines to perform a search. These possibilities will be explained in the following paragraphs in relation to achieving the goal to select a workable set with the relevant documents.

2.6 When to stop?

The question when to stop your search can be answered when the following question can be answered:

Did you select all relevant documents?

This question can only be answered with uncertainty. In practise it is not possible to check that you have selected all relevant documents (recall = 100%). Instead some indications can be used to answer the question.

• If you have enough knowledge of the classification and technology of a certain technical field (similar to a patent examiner) then you are able to select one or a limited number a classification codes, whereby you have a high certainty that most relevant documents will be selected. Selecting a set with these classification codes and working through this set is then sufficient for the search. The search can then be stopped after this. This method is therefore the most simple.

• The set you have selected should return a significant number of relevant or closely related documents. If this is not the case, then you have probably selected a set without relevant documents (recall is low).

• When looking at cited and citing documents of the really relevant documents does not retrieve new relevant documents or technical fields, then this is an indication that most relevant documents have been found.

If you come to the conclusion that your search is not yet complete enough, then a new selection has to be made. The information from the previous results can be used for this. For example new classification codes found on relevant documents can be used.