Search Engines. Gertjan van Noord. August 31, 2017

Transcription

1 Search Engines Gertjan van Noord August 31, 2017

2 About the course Information about the course is available from:

3 Search Engines / Information Retrieval Individuals, administrations, organizations have lots of digital information how to organize and store it? how to retrieve documents? how to retrieve info inside them? An IR system is a tool to facilitate retrieval of such information

4 Search Engines / Information Retrieval Here is what the book says: Information retrieval is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).

5 Search Engines / Information Retrieval Important aspects of this definition: finding material large collections unstructured, usually text information need

6 Finding material documents part of documents tweets facts (e.g., what is the Dutch name for peregrine falcon )...

7 Large collections Large world-wide-web or parts of it Twitter specific collections (legal, medical,... ) your own computer information within a company (manuals Boeing)

8 Unstructured arbitray text versus databases even text is always structured somewhat beyond text: image, sound, video (not in this course)

9 Database search versus Information Retrieval structured versus unstructured search of fields versus full text search exact search versus inexact order of results alfanumerical versus order of results by goodness (... )

10 Information Need Is not the same as the query you type in Google In web search: find specific web-site find information about some topic find concrete answer to a question...

11 Information Need and Evaluation Search engines produce often a lot of results When are you satisfied with the results? How can we evaluate a system?

12 Evaluation: Precision and recall Key statistics for evaluation with a test set (fixed questions, set of documents, evaluations of documents for the queries available). Precision: what fraction of the system results are relevant to the information need? Recall: what fraction of the relevant documents in the collection were returned by the system?

13 Information Need and Evaluation Precision and Recall are widely used evaluation metrics. It depends on your information need if these metrics are really appropriate.

14 Course objectives IR terminology IR models and IR processes Understanding of indexing methods, querying, retrieving, ranking, evaluation Practical experience by implementing basic techniques in Python Learn about program efficiency: what happens in case you apply particular algorithms for very large datasets

15 Chapter 1: Boolean retrieval The first IR systems were Boolean systems. Queries are formulated with the Boolean operators AND, OR and NOT. Examples: Brutus AND Caesar (Brutus OR Caesar) AND NOT Cleopatra Brutus OR (Caesar AND NOT Cleopatra) NOT Brutus

16 Information from documents Each document in the collection needs a unique identifier Each document is tokenized. Tokenization is the process of splitting a text into separate tokens (terms, words). This is harder than you might think. For a boolean system: we need to know which tokens (terms, words) are present in which document

17 Term document incidence matrix For a boolean system: we need to know which tokens (terms, words) are present in which document. Doc1 Doc2 Doc3 Doc4 Antony Brutus Caesar Cleopatra What to do for a query such as: Antony AND Brutus AND NOT Cleopatra

18 Term document incidence matrix For a huge collection, this matrix becomes very big (how big?). What would the row for Cleopatra look like in a real system? More compact representation: only represent the 1 s.

19 How big? Example: Dutch Wikipedia. 256 Million words, 16 Million sentences, almost 5 Million articles. Number of different words. Types. Vocabulary. 3,5 Million (!). The term document incidence matrix would contain 5 Million 3,5 Million bits, i.e., more than 2 terrabytes.

20 Alternative representation: Inverted File For each term (type, word), have an ordered list of document identifiers The ordered list of document identifiers is called postings list Antony Brutus Caesar Cleopatra [Doc1,Doc2] [Doc1,Doc2,Doc3] [Doc1,Doc2,Doc4] [Doc1] What to do for a query such as: Antony AND Brutus AND NOT Cleopatra

21 How big? Dump of the Dutch Wikipedia. 256 Million words, 16 Million sentences, almost 5 Million articles. Number of different words. Types. Vocabulary. 3,5 Million (!). The term document incidence matrix would contain 5 Million 3,5 Million bits, is more than 2 terrabytes. The length of all posting lists together: 256 Million. 3 bytes for each document id. 256 Million * 3 amounts to something like 800 megabytes.

22 Efficient merging algorithm if posting lists are sorted In order to find the common elements in two ordered lists (intersection): l1 points at the next element of list1 l2 points at the next element of list2 if list1[l1] = list2[l2]: add the common element to the result, and move l1 and l2 one step forward otherwise move either l1 or l2 one step forward (whichever is the smallest). stop if either list is exhausted

23 Efficient merging algorithm if posting lists are sorted. Example. Example: list1: [1,6,12,14,20,21]; list2: [2,10,14,16,21]. Initially, l1 points to first element of list1; l2 points to first element of list2. The result is intially empty, of course. l1 list1 list2 l2 result ==> 1 2 <==

24 Efficient merging algorithm if posting lists are sorted. Example (1) l1 list1 list2 l2 result ==> 1 2 <==

25 Efficient merging algorithm if posting lists are sorted. Example (2) l1 list1 list2 l2 result 1 2 <== ==>

26 Efficient merging algorithm if posting lists are sorted. Example (3) l1 list1 list2 l2 result 1 2 <== 6 10 ==>

27 Efficient merging algorithm if posting lists are sorted. Example (4) l1 list1 list2 l2 result <== ==>

28 Efficient merging algorithm if posting lists are sorted. Example (5) l1 list1 list2 l2 result ==> <==

29 Efficient merging algorithm if posting lists are sorted. Example (6) l1 list1 list2 l2 result <== ==>

30 Efficient merging algorithm if posting lists are sorted. Example (7) l1 list1 list2 l2 result <== ==>

31 Efficient merging algorithm if posting lists are sorted. Example (8) l1 list1 list2 l2 result ==> <== 21

32 Efficient merging algorithm if posting lists are sorted. Example (9) l1 list1 list2 l2 result <== ==> 21

33 Efficient merging algorithm if posting lists are sorted To find identical elements in two ordered lists, the number of steps depends on the size of both lists. If the length of the two lists are x and y, then the number of steps roughly is x + y. The number of steps is linear in the length of the input: if the input is ten times as long, then computation takes about ten times as long. If the lists are unordered, the number of steps roughly is x y. The number of steps is quadratic in the length of the input. If the input is ten times as long, computation takes about hundred times as long. This is an important difference in case x and y become large. x y x + y x y

34 Intersection of two ordered lists in Python (1): the naive way def intersect (l1,l2 ): return [ obj for obj in l1 if obj in l2 ] or, equivalently, without the use of list-comprehension: def intersect (l1,l2 ): sol = [] for obj in l1: if obj in l2: sol. append ( obj ) return sol You have to compare each element in l1 with each element in l2 If those lists 11 and l2 have n and m elements respectively, then you have to perform n m comparisons You do not use the information that the lists are ordered

35 Can we do better? Yes of course.

36 Intersection of two ordered lists in Python (2): merge def intersect (l1,l2 ): it1 = iter (l1) it2 = iter (l2) sol = [] n1 = next ( it1 ) n2 = next ( it2 ) while True : try : if n1 == n2: sol. append (n1) n1 = next ( it1 ) n2 = next ( it2 ) else : if n1 < n2: n1 = next ( it1 ) else : n2 = next ( it2 ) except StopIteration : return sol

37 Intersection of two ordered lists in Python (2): merge Go through both lists, from top to bottom, in parallel Number of steps: n+m

38 Intersection of two ordered lists in Python (3): sets Use Python s set datastructure Intersection of two sets is built-in (and very fast) This is efficient as long as you can work with sets throughout; it is not recommended to convert lists to sets for the sole purpose of computing the intersection aset = set (l1) bset = set (l2) intersect_set = aset & bset

39 Experiment de timeit module in Python library l1 and l2 both have N elements, intersection is empty timings in second for N=10000, N=20000, N= lists (naive) lists (merge) lists converted to sets sets