Differential privacy

Differential privacy (DP) is a system for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset. The idea behind differential privacy is that if the effect of making an arbitrary single substitution in the database is small enough, the query result cannot be used to infer much about any single individual, and therefore provides privacy.
Another way to describe differential privacy is as a constraint on the algorithms used to publish aggregate information about a statistical database which limits the disclosure of private information of records whose information is in the database. For example, differentially private algorithms are used by some government agencies to publish demographic information or other statistical aggregates while ensuring confidentiality of survey responses, and by companies to collect information about user behavior while controlling what is visible even to internal analysts.
Roughly, an algorithm is differentially private if an observer seeing its output cannot tell if a particular individual's information was used in the computation. Differential privacy is often discussed in the context of identifying individuals whose information may be in a database. Although it does not directly refer to identification and reidentification attacks, differentially private algorithms probably resist such attacks.
Differential privacy was developed by cryptographers and thus is often associated with cryptography, and draws much of its language from cryptography.
History.
Official statistics organizations are charged with collecting information from individuals or establishments, and publishing aggregate data to serve the public interest. For example, the 1790 United States Census collected information about individuals living in the United States and published tabulations based on sex, age, race, and condition of servitude. Statistical organizations have long collected information under a promise of confidentiality that the information provided will be used for statistical purposes, but that the publications will not produce information that can be traced back to a specific individual or establishment. To accomplish this goal, statistical organizations have long suppressed information in their publications. For example, in a table presenting the sales of each business in a town grouped by business category, a cell that has information from only one company might be suppressed, in order to maintain the confidentiality of that company's specific sales.
The adoption of electronic information processing systems by statistical agencies in the 1950s and 1960s dramatically increased the number of tables that a statistical organization could produce and, in so doing, significantly increased the potential for an improper disclosure of confidential information. For example, if a business that had its sales numbers suppressed also had those numbers appear in the total sales of a region, then it might be possible to determine the suppressed value by subtracting the other sales from that total. But there might also be combinations of additions and subtractions that might cause the private information to be revealed. The number of combinations that needed to be checked increases exponentially with the number of publications, and it is potentially unbounded if data users are able to make queries of the statistical database using an interactive query system.
In 1977, Tore Dalenius formalized the mathematics of cell suppression.
In 1979, Dorothy Denning, Peter J. Denning and Mayer D. Schwartz formalized the concept of a Tracker, an adversary that could learn the confidential contents of a statistical database by creating a series of targeted queries and remembering the results. This and future research showed that privacy properties in a database could only be preserved by considering each new query in light of (possibly all) previous queries. This line of work is sometimes called "query privacy," with the final result being that tracking the impact of a query on the privacy of individuals in the database was NP-hard.
In 2003, Kobbi Nissim and Irit Dinur demonstrated that it is impossible to publish arbitrary queries on a private statistical database without revealing some amount of private information, and that the entire information content of the database can be revealed by publishing the results of a surprisingly small number of random queries—far fewer than was implied by previous work. The general phenomenon is known as the Fundamental Law of Information Recovery, and its key insight, namely that in the most general case, privacy cannot be protected without injecting some amount of noise, led to development of differential privacy.
In 2006, Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam D. Smith published an article formalizing the amount of noise that needed to be added and proposing a generalized mechanism for doing so. Their work was a co-recipient of the 2016 TCC Test-of-Time Award and the 2017 Gödel Prize.
Since then, subsequent research has shown that there are many ways to produce very accurate statistics from the database while still ensuring high levels of privacy.
ε-differential privacy.
The 2006 Dwork, McSherry, Nissim and Smith article introduced the concept of ε-differential privacy, a mathematical definition for the privacy loss associated with any data release drawn from a statistical database. (Here, the term "statistical database" means a set of data that are collected under the pledge of confidentiality for the purpose of producing statistics that, by their production, do not compromise the privacy of those individuals who provided the data.)
The intuition for the 2006 definition of ε-differential privacy is that a person's privacy cannot be compromised by a statistical release if their data are not in the database. Therefore, with differential privacy, the goal is to give each individual roughly the same privacy that would result from having their data removed. That is, the statistical functions run on the database should not overly depend on the data of any one individual.
Of course, how much any individual contributes to the result of a database query depends in part on how many people's data are involved in the query. If the database contains data from a single person, that person's data contributes 100%. If the database contains data from a hundred people, each person's data contributes just 1%. The key insight of differential privacy is that as the query is made on the data of fewer and fewer people, more noise needs to be added to the query result to produce the same amount of privacy. Hence the name of the 2006 paper, "Calibrating noise to sensitivity in private data analysis."
The 2006 paper presents both a mathematical definition of differential privacy and a mechanism based on the addition of Laplace noise (i.e. noise coming from the Laplace distribution) that satisfies the definition.
Definition of ε-differential privacy.
Let ε be a positive real number and formula_1 be a randomized algorithm that takes a dataset as input (representing the actions of the trusted party holding the data).
Let formula_2 denote the image of formula_1. The algorithm formula_1 is said to provide formula_5-differential privacy if, for all datasets formula_6 and formula_7 that differ on a single element (i.e., the data of one person), and all subsets formula_8 of formula_2:
formula_10where the probability is taken over the randomness used by the algorithm.
Differential privacy offers strong and robust guarantees that facilitate modular design and analysis of differentially private mechanisms due to its composability, robustness to post-processing, and graceful degradation in the presence of correlated data.
Composability.
(Self-)composability refers to the fact that the joint distribution of the outputs of (possibly adaptively chosen) differentially private mechanisms satisfies differential privacy.
Sequential composition. If we query an ε-differential privacy mechanism formula_11 times, and the randomization of the mechanism is independent for each query, then the result would be formula_12-differentially private. In the more general case, if there are formula_13 independent mechanisms: formula_14, whose privacy guarantees are formula_15 differential privacy, respectively, then any function formula_16 of them: formula_17 is formula_18-differentially private.
Parallel composition. If the previous mechanisms are computed on "disjoint" subsets of the private database then the function formula_16 would be formula_20-differentially private instead.
Robustness to post-processing.
For any deterministic or randomized function formula_21 defined over the image of the mechanism formula_1, if formula_1 satisfies ε-differential privacy, so does formula_24.
Together, composability and robustness to post-processing permit modular construction and analysis of differentially private mechanisms and motivate the concept of the "privacy loss budget". If all elements that access sensitive data of a complex mechanisms are separately differentially private, so will be their combination, followed by arbitrary post-processing.
Group privacy.
In general, ε-differential privacy is designed to protect the privacy between neighboring databases which differ only in one row. This means that no adversary with arbitrary auxiliary information can know if one particular participant submitted his information. However this is also extendable. We may want to protect databases differing in formula_25 rows, which amounts to an adversary with arbitrary auxiliary information knowing if formula_25 particular participants submitted their information. This can be achieved because if formula_25 items change, the probability dilation is bounded by formula_28 instead of formula_29, i.e., for D1 and D2 differing on formula_25 items:
formula_31
Thus setting ε instead to formula_32 achieves the desired result (protection of formula_25 items). In other words, instead of having each item ε-differentially private protected, now every group of formula_25 items is ε-differentially private protected (and each item is formula_35-differentially private protected).
ε-differentially private mechanisms.
Since differential privacy is a probabilistic concept, any differentially private mechanism is necessarily randomized. Some of these, like the Laplace mechanism, described below, rely on adding controlled noise to the function that we want to compute. Others, like the exponential mechanism and posterior sampling sample from a problem-dependent family of distributions instead.
Sensitivity.
Let formula_36 be a positive integer, formula_37 be a collection of datasets, and formula_38 be a function. The "sensitivity" of a function, denoted formula_39, is defined by
formula_40
where the maximum is over all pairs of datasets formula_6 and formula_7 in formula_37 differing in at most one element and formula_44 denotes the formula_45 norm.
In the example of the medical database below, if we consider formula_46 to be the function formula_47, then the sensitivity of the function is one, since changing any one of the entries in the database causes the output of the function to change by either zero or one.
There are techniques (which are described below) using which we can create a differentially private algorithm for functions with low sensitivity.
The Laplace mechanism.
The Laplace mechanism adds Laplace noise (i.e. noise from the Laplace distribution, which can be expressed by probability density function formula_48, which has mean zero and standard deviation formula_49). Now in our case we define the output function of formula_50 as a real valued function (called as the transcript output by formula_50) as formula_52 where formula_53 and formula_54 is the original real valued query/function we planned to execute on the database. Now clearly formula_55 can be considered to be a continuous random variable, where
which is at most formula_57. We can consider formula_58 to be the privacy factor formula_59. Thus formula_60 follows a differentially private mechanism (as can be seen from the definition above). If we try to use this concept in our diabetes example then it follows from the above derived fact that in order to have formula_50 as the formula_59-differential private algorithm we need to have formula_63. Though we have used Laplace noise here, other forms of noise, such as the Gaussian Noise, can be employed, but they may require a slight relaxation of the definition of differential privacy.
According to this definition, differential privacy is a condition on the release mechanism (i.e., the trusted party releasing information "about" the dataset) and not on the dataset itself. Intuitively, this means that for any two datasets that are similar, a given differentially private algorithm will behave approximately the same on both datasets. The definition gives a strong guarantee that presence or absence of an individual will not affect the final output of the algorithm significantly.
For example, assume we have a database of medical records formula_6 where each record is a pair (Name, X), where formula_65 is a Boolean denoting whether a person has diabetes or not. For example:
Now suppose a malicious user (often termed an "adversary") wants to find whether Chandler has diabetes or not. Suppose he also knows in which row of the database Chandler resides. Now suppose the adversary is only allowed to use a particular form of query formula_47 that returns the partial sum of the first formula_67 rows of column formula_65 in the database. In order to find Chandler's diabetes status the adversary executes formula_69 and formula_70, then computes their difference. In this example, formula_71 and formula_72, so their difference is 1. This indicates that the "Has Diabetes" field in Chandler's row must be 1. This example highlights how individual information can be compromised even without explicitly querying for the information of a specific individual.
Continuing this example, if we construct formula_7 by replacing (Chandler, 1) with (Chandler, 0) then this malicious adversary will be able to distinguish formula_7 from formula_6 by computing formula_76 for each dataset. If the adversary were required to receive the values formula_47 via an formula_5-differentially private algorithm, for a sufficiently small formula_5, then he or she would be unable to distinguish between the two datasets.
Randomized response.
A simple example, especially developed in the social sciences, is to ask a person to answer the question "Do you own the "attribute A"?", according to the following procedure:
(The seemingly redundant extra toss in the first case is needed in situations where just the "act" of tossing a coin may be observed by others, even if the actual result stays hidden.) The confidentiality then arises from the refutability of the individual responses.
But, overall, these data with many responses are significant, since positive responses are given to a quarter by people who do not have the "attribute A" and three-quarters by people who actually possess it.
Thus, if "p" is the true proportion of people with "A", then we expect to obtain (1/4)(1-"p") + (3/4)"p" = (1/4) + "p"/2 positive responses. Hence it is possible to estimate "p".
In particular, if the "attribute A" is synonymous with illegal behavior, then answering "Yes" is not incriminating, insofar as the person has a probability of a "Yes" response, whatever it may be.
Although this example, inspired by randomized response, might be applicable to microdata (i.e., releasing datasets with each individual response), by definition differential privacy excludes microdata releases and is only applicable to queries (i.e., aggregating individual responses into one result) as this would violate the requirements, more specifically the plausible deniability that a subject participated or not.
Stable transformations.
A transformation formula_80 is formula_25-stable if the Hamming distance between formula_82 and formula_83 is at most formula_25-times the Hamming distance between formula_85 and formula_86 for any two databases formula_87. Theorem 2 in asserts that if there is a mechanism formula_88 that is formula_5-differentially private, then the composite mechanism formula_90 is formula_91-differentially private.
This could be generalized to group privacy, as the group size could be thought of as the Hamming distance formula_92 between
formula_85 and formula_86 (where formula_85 contains the group and formula_86 doesn't). In this case formula_90 is formula_98-differentially private.
Other notions of differential privacy.
Since differential privacy is considered to be too strong or weak for some applications, many versions of it have been proposed. The most widespread relaxation is (ε, δ)-differential privacy, which weakens the definition by allowing an additional small δ density of probability on which the upper bound ε does not hold.
Adoption of differential privacy in real-world applications.
To date there are over 12 real-world deployments of differential privacy, the most important being: 
Public purpose considerations.
There are several public purpose considerations regarding differential privacy that are important to consider, especially for policymakers and policy-focused audiences interested in the social opportunities and risks of the technology: