Data Structures in Web Information Retrieval:Fingerprints

By -
Fingerprints

Fingerprints are short strings that represent larger strings and have the following properties:

• If the fingerprints of two strings are different then the strings are guaranteed to be different.

• If the fingerprints of two strings are identical then there is only a small probability that the strings are different.

If two different strings have the same fingerprint, it is called a collision. Thus, fingerprints are a type of hash functions where the hash table is populated only very sparsely in exchange for a very low collision probability.

Fingerprints are very useful. For example, search engines can use them to quickly check whether a URL that is contained on a web page is already stored in the index. We will describe how they are useful for finding near-duplicate documents in the next section. One fingerprinting scheme is due to Rabin [12] and is equivalent to cyclic redundancy checks. We follow here the presentation by Broder [3]. Let s be a binary string and let st = 1s, i.e. st equals s prefixed with a 1. The string st = (s1, s2,... , sm) induces a polynomial S(t) over Z2 as follows:

S(t) = s1tm−1 + s2tm−2 + ··· + sm.

Let P (t) be an irreducible polynomial of degree k over Z2. The fingerprint of s is defined to be the polynomial

f (s) = S(t) mod P (t)

over Z2, which can be simply represented as a string of its coefficients. As shown in [3] the probability that two distinct strings of length m from a set of n strings have the same fingerprint is less than nm2/2k. Thus, given n and m the probability of collision can be reduced to the required value simply by increasing k, i.e. the length of a fingerprint.

Rabin’s fingerprints have the property that given two overlapping strings s = (s1, s2,..., sm) and t = (t1, t2,... , tm) such that ti+1 = si for all 1 ≤ i < m then the fingerprint of t can be computed from the fingerprint of s very efficiently: If

image

Hence f (t) can be computed from f (s) as follows: Assume f (s) is stored in a shift register and Q(t) is stored in another register. Shift left with tm as input and if r1 = 1 perform a bit-wise EX-OR operation with Q(t). 32-bit word computer.

See [3] for a software implementation for a typical Other ways of computing fingerprints are cryptographic checksums like Sha-1 or MD5.

Comments

Post a Comment

Popular posts from this blog

Data Structure Visualization:Introduction and Value of Data Structure Rendering

By -
Introduction Important advances in programming languages have occurred since the early days of assembly languages and op codes, and modern programming languages have significantly simplified the task of writing computer programs. Unfortunately, tools for un d e rstanding programs have not achieved the accompanying levels of improvement. Software developers still face difficulties in understanding, analyzing, debugging, and improving code. As an example of the difficulties still evident, consider a developer who has just implemented a new algorithm and is now ready to examine the program’s behavior. After issuing a command to begin program execution, the developer examines output data and/or inter- active behavior, attempting to ascertain if the program functioned correctly, and if not, the reasons for the program’s failure. This task can be laborious and usually requires the use of a debugger or perhaps even the addition of explicit checking code, usually through output statements. Furt
Read more

Collision Detection:Penetration Depth Computation

By -
Image
Penetration Depth Computation In this section, we give a brief overview of penetration depth (PD) computation algorithms between convex polytopes and general polyhedral models. The PD of two inter-penetrating objects A and B is defined as the minimum translation distance that one object undergoes to make the interiors of A and B disjoint. It can be also defined in terms of the TCSO. When two objects are overlapping, the origin of the Minkowski sum of A and − B is contained inside the TCSO. The penetration depth corresponds to the minimum distance from the origin to the surface of TCSO [18]. PD computation is often used in motion planning [37], contact resolution for dynamic simulation [38, 39] and force computation in haptic rendering [40]. For example, computation of dynamic response in penalty-based methods often needs to perform PD queries for imposing the non-penetration constraint for rigid body simulation. In addition, many applications, such as motion planning and dynamic simula
Read more

Concurrent Data Structures:Linked Lists

By -
Linked Lists Consider implementations of concurrent search structures supporting insert, delete, and search operations. If these operations deal only with a key value, then the resulting data structure is a se t ; if a data value is associated with each key, we have a dic tionary [24]. These are closely related data structures, and a concurrent set implementation can often be adapted to implement a dictionary. In the next three sections, we concentrate on implementing sets using different structures: linked lists, hash tables, and trees. Suppose we use a linked list to implement a set. Apart from globally locking the linked list to prevent concurrent manipulation, the most popular approach to concurrent lock- based linked lists is ha nd-over-hand locking (sometimes called l o c k coupling ) [14, 89]. In this approach, each node has an associated lock. A thread traversing the linked list releases a node’s lock only after acquiring the lock of the next node in the list, thus preventin
Read more