Consumer Marketers, Looking to Test New Data Categories? Try These

We are all trying to create and sustain customers, using data to discover new patterns, new audiences, and new prospects — and that requires a lot of testing, and innovative data sets to explore (responsibly). Let’s make it experiential, as well as experimental.

We in the data marketing business love to test — at least, we should. And what we should test for is new data categories.

Expanding the marketing universe — and stretching the marketing budget — depends on higher efficiency in our lists, offers, and creative. We should be eager to test new proofs of concepts and new categories of data sources as they enter the market … if only to know whether or not they produce incrementally or otherwise.

I’m still surprised when I hear some of my data-vendor friends say that a good number of their clients pass on testing — and just go all-in on new lists and data sources. It seems like testing is still too much work for some, or they feel the only way to test is with an entire data source. Guess these client-side folks have money to burn, or are operating very much on-the-fly.

In some ways, digital marketers have it all over offline marketers in their ability to test, cycle, test again, and so on — often, many times over by the time a direct mail or direct-response print or broadcast test cycle has run its course. Yet, in this speed, have we sacrificed some quality in our prospecting strategies?

Online audience algorithms can produce some highly categorized niche segments, based on site visits and app usage — much of it de-identified, from a personal perspective. But how do these segments really stack up against a transaction database, or response lists, or even compiled lists, based on personally identifiable information? Thankfully, we can test for this, or even overlay data! (I am not advocating re-identification here, nor should you. Oh California, please don’t force us to identify non-PII. It’s soooo anti-privacy.)

Recently, the Direct Marketing Club of New York (DMCNY) held a very interesting breakfast program titled “Beyond Demographics: The Data You Need to Max Out Marketing Performance.”

Some Fresh Categories for New Reach and Affinity Discovery

Consider some of these data sources for testing:

  • Values Data — Test cohorts based on “shared values,” rather than simply choosing audiences based on demographics or psychographics. David Allison, principal, David Allison Inc., and author of “We Are All the Same Age Now,” pointed to his firm’s internal research that shows that popularly defined age groups rarely (or barely) match on what they agree upon, or value, as a generation. For example, Baby Boomers agree with each other about 13% of the time; Gen X, about 11% of the time; and Millennials, 15% of the time. Thus, targeting based on demographics alone can be extremely wasteful if the marketer is assuming some sort of shared attribute among them, other than age.However, when targeting based on shared “values” — Adventurers, Savers, and Techsters, and the like — all of a sudden affinities jump sky-high. In these cases, 89%, 76%, and 81%, respectively. These “valuegraphics” are based on “big data” segments — rather than small data (response lists, for example). Still, when compared to demographics targeting alone, shared-value targeting offers an eight-time lift!  Well, that’s worth testing.
  • Attitudinal Data — Another perspective on “beyond demographics” came from Mark Himmelsbach, co-founder, Episode Four, a creator of “brand hits,” such as this one for Charles Schwab. We often have stereotypical views of many demographic and other audience categories — and too many algorithms, he said. But analyze the data for unusual patterns, and suddenly you can find “who knew?” commonalities among certain audience segments that would wow any of us.Who knew that ultra-high net worth individuals are electronic dance music enthusiasts? Who knew that African-American married women are high on the e-sports genre? Or that young Hispanic/Latino adventurers are really into escape rooms? These discoveries give brands new advertising, product placement, and sponsorship opportunities, for example, which might otherwise go untapped. I’m still trying to get my head around these reported affinities, based no doubt by my own preconceptions.
  • Location Data — According to the World Economic Forum, 90% of the world will soon have or already has a supercomputer in their pocket — a smartphone. We’re actually closing in on four connected devices per person, reports Jeff White, founder and CEO, Gravy Analytics. With smartphones alone, as constant companions, we have a huge opportunity to leverage responsibly use of location data. Location can provide huge “affinity” targeting opportunities.A casual wine user might search and buy online his or her wine. But a wine aficionado visits a winery (Location X), or attends a wine tasting (Location Y), and now you have a true affinity opportunity. Granted, location data has a level of sensitivity that carries, more often than not, an opt-in requirement — but the marketing lift can be a significant reward for the advertiser who strategically applies such insights from it. Makes me want to tag every latitude and longitude for some hobby or interest!
  • Experiential Data — Live Nation may own concert venues, Ticketmaster, online game communities and music/culture festivals — but across these many first-party experiences, the company can provide deep analytics that help monetize its various audiences through enriched second-party relationships, said Anubhav Mehrotra, VP, Live Nation. Hilton, American Express, and Uber are just some of the brands Live Nation has teamed up with to enrich brand users with engaging experiences, such as backstage tours and “meet the artists.”

We are all trying to create and sustain customers, using data to discover new patterns, new audiences, and new prospects — and that requires a lot of testing, and innovative data sets to explore (responsibly). Let’s make it experiential, as well as experimental: I sure hope to meet some ultra-high-net-worth individuals at the next Electronic Dance Festival I attend. Or not.

It’s Decision Time for Data Privacy (or Will Be Soon)

Chet Dalzell’s recent thoughtful piece on “Our Digital Selves” came along at the same time I (and probably a gazillion others) were pondering the increasingly pressing question of data privacy in the digital age.


It’s a much bigger question than what data can be used to target potential customers for the latest widget or widget club or to stop you in your tracks at the supermarket in front of the pet food shelves to tell you that Fido, your beloved Fido, seen in the picture on your cell phone, absolutely must have the new, nutritious and tasty Dogbit,s or he may bite your fingers off if you try to give him anything else.

The data question goes to the heart of how we see ourselves in the digital world. And how we see ourselves is in no way clear — even to ourselves.

“Bottom line: If Facebook’s users in the United States are similar to most Americans (and studies suggest they are), large majorities don’t want personalized ads — and when they learn how companies find out information about them, even greater percentages don’t want them.”

That’s what Joseph Turow, a professor of communications and Chris Jay Hoofnagle, an adjunct professor of law, say in The New York Times using various research to support their thesis. The problem is what people tell researchers is not always what they do. Facebook’s quarterly earnings statement showed these enlightening KPIs.

  • Monthly active users (MAUs) — MAUs were 2.32 billion as of Dec. 31, 2018, an increase of 9%, year-over-year.
  • We estimate that around 2.7 billion people now use Facebook, Instagram, WhatsApp or Messenger (our “Family” of services) each month, and more than 2 billion people use at least one of our Family of services every day, on average.

It has been said over and over again that everything has its price. Assuming that this is largely true, how much value or benefit should the consumer expect in return for how much and which data? As I wrote in a comment to Chet’s article, this is sure to be the data-use question we’ll all be turning in our minds as the algorithms get smarter and the temptations greater.

Imagine that you could put a value on each element of your personal, demographic, psychographic and behavioral data, and anyone wanting to use that data would have to pay your price, whether or not you ended up making a purchase or taking a desired action? Imagine further that a data user wanted to use $20 worth of your data to try to sell you a product you wanted, priced at $100? It would be an easy transaction, if the seller were willing to offer you a 20% or even a greater discount for the specific permission to use the data. You would have the product, the seller would have the sale and everyone would be happy.

However fanciful that scenario, it is not nearly as crazy as it sounds. In fact, in one form or another, that is exactly what is happening in the real marketplace; although without your specific permission. As a marketer, I have to spend money to acquire your data and, by making an attractive offer (say a 20% discount), I am offering to compensate you for your data, which allows me to talk to you.

Of course, I have over-simplified the argument. As stated earlier: How much value or benefit should the consumer expect in return for how much and which data?

I think we would all agree that this determination is much too complicated, so we let the “invisible hand of the market” do its magic. Which reduces the decision to a very simple one: Do we perceive that we get enough value from having our data out there in the marketplace to be manipulated however the marketers wish to and simply lie back and enjoy all the offers and benefits? Or should we bite the bullet, give our cell phones to a needy child, do without Waze and get lost again and again, be prepared to stand in the endless line at the bank, throw the “delete everything” switch and effectively remove ourselves from the digital economy? It is getting near decision time for all of us.

I remember many years ago in London, as “one of those Americans,” being lectured over lunch by a very traditional British publisher about the horrors of books being sold by mail order and direct mail and assuring me that the British wouldn’t have anything to do with book clubs or the like. Just when the bill had been paid and we were preparing to depart, she reached into her handbag and pulled out an all singing and all dancing mailing piece from the Readers Digest, offering a very handsome discount on their superb motorist bible, the “Book of the Road.”

She was going to order it right away.


Marketing Success Sans ‘Every Breath They Take, Every Move They Make’

Last month, I talked about how to measure success when there are many conflicting goals and available metrics flying around (refer to “Marketing Success Metrics: Response or Dollars?”). This time, let’s start thinking about how to act on data and intelligence that we’ve gathered. And that means we get to touch different kinds of advanced analytics.


But before we get into boring analytics talk, citing words like “predictive analytics” and “segmentation,” let’s talk about what kind of data are required to make predictions better and more accurate. After all, no data, no analytics.

I often get questions like what the “best” kind of data are. And my answer is, to the inquirer’s disappointment, “it depends.” It really depends on what you are trying to predict, or ultimately, do. If you would like to have an accurate forecast of futures sales, such an effort calls for a past sales history (but not necessarily on an individual or transactional level); past and current marcom spending by channel; web and other channel traffic data; and environmental data, such as economic indicators, just to start off.

Conversely, if you’d like to predict an individual’s product affinity, preferred offer types or likelihood to respond to certain promotion types, such predictive modeling requires data about the past behavior of the target. And that word “behavior” may evoke different responses, even among seasoned marketers. Yes, we are all reflections of our past behavior, but what does that mean? Every breath you take, every move you make?

Thanks to the Big Data hype a few years back, many now believe that we should just collect anything and everything about everybody. Surely, cost for data collection, storage and maintenance has decreased quite a bit over the years, but that doesn’t mean that we should just hoard data mindlessly. Because you may be deferring inevitable data hygiene, standardization, categorization and consolidation to future users — or machines — who must sort out unorganized and unrefined data and provide applicable insights.

So, going back to that question of what makes up data about human behavior, let’s define what that means in a categorical fashion. With proliferation of digital data collection and analytics, the definition of behavioral data has expanded considerably.

In short, what people casually refer to as “behavioral data” may include this to measure success:

  • Online Behavior: Web data regarding click, view and other shopping behavior.
  • Purchase: Transactional data, made of who, what, when, how much and through what channel.
  • Response: Response history, in relation to specific promotions, covering open, click-through, opt-out, view, shopping basket, conversion/transaction. Offline response may be as simple as product purchase.
  • Channel: Channel usage data, not necessarily limited to shopping behavior.
  • Payment: Payment and related delinquent history — essential for credit purchases and continuity and subscription businesses.
  • Communication: Call, chat or other communication log data, positive or negative in nature.
  • Movement: Physical proximity or movement data, in store or store area, for example.
  • Survey: Responses to various surveys.
  • Opt-in/Opt-out: Sign-up specific 2-way communications and channel specific opt-out requests.
  • Social Media: Product review, social media posting and product/service-related sentiment data.

I am sure some will think of more categories. But before we create an exhaustive list of data types, let’s pause and think about what we are trying to do here.

First off, all of these data traceable to a person are being collected for one major reason (at least for marketers): To sell more things to them. If the goal is to predict the who, what, when and why of buying behavior, do we really need all of this?

The ‘Who’ of Buying Behavior

In the prediction business, predicting “who” (as in “who will buy this product?”) is the simplest kind of action. We’d need some PII (personally identifiable information) that can link to buying behaviors of the target. After all, the whole modeling technique was invented to rank target individuals and set up contact priority — in that order. Like sending expensive catalogs only to high-score individuals, in terms of “likely to respond,” or sales teams contacting high “likely to convert” targets as priorities in B2B businesses.

The ‘What’ of Buying Behavior

The next difficulty level lies with the prediction of “what” (as in “what is that target individual going to buy next?”). This type of prediction is generally a hit-or-miss, so even mighty Amazon displays multiple product offers at the end of a successful transaction, by saying “Customers who purchased this item are also interested in these products.” Such a gentle push, based on collaborative filtering, requires massive purchase history by many buyers to be effective. But, provided with ample amounts of data, it is not terribly difficult, and the risk of being wrong is relatively low. Pinpointing the very next product for 1:1 messaging can be challenging, but product basket analysis can easily lead to popular combinations of products, at the minimum.




  1. How do I find out the names of individuals who visit my website?
    There are two ways to de-anonymize the website visit. First, add a registration invitation to your site. This could be an email sign-up, or a piece of gated content, like a white paper or research report, in exchange for providing important data elements like name, title, company name, address, phone and email.
    Second, use the IP address to identify the company from which the visitor arrived. This can be done by hand, using Google Analytics, or more easily by using any number of services that enable IP address look-up. Marketing automation systems are increasingly baking this option into their tools.

    But the IP address method will still not get you the name of the visitor. You can infer the visitor’s interests and, possibly, role by looking at the time spent on various pages. And you can drop a cookie and retarget the visitor with text or banner ads later.

  2. Job titles are increasingly inconsistent-and proliferating. Categories like marketing manager and financial analyst don’t seem to work anymore.
    Several companies offer job title standardization services, called something like title mapping, title translation or title beautification. A resource like that is a good first step.

    Then, consider sending an outbound email, perhaps with a follow-up phone call, positioned as a “contact verification” message. Invite the target to indicate his or her functional job title, from a list.

    After that, you will be left with a relatively smaller list of remaining titles. At that point, you need to decide on a default for the rest of them. For example, anything that sounds like IT will go in an IT functional bucket. And, depending on how often you query your customers, you can always gather answers to this question over time.

    Then, you are faced with the remaining issue, which is far more difficult, namely the crazy new titles that some people are using these days. We’ve seen bizarre titles like Chief Instigating Officer and Marketing Diva. With these, you have two options.

    • Force aberrant titles into your standards, by hand, using your best guess. Use a default code for anything you can’t really figure out.
    • Leave them as they are, and link them to a table of standardized job functions. But maintain the self-reported wacky title, too, so you can still address the person the way he or she wants to be addressed.

    You might also consider using forced drop-down menus for job function and job title, at the point of key entry.

  3. How should I handle job changes? When an employee leaves and goes to another company, does his or her history with my company go along?
    We are going to assume—a big assumption—that you actually know the person has gone to a new company. It’s more likely that you will not know. This is why it’s a good idea to do periodic de-duplications by functional title to get a sense of new names that have popped up at the companies in your database.

    When you know that there is a job change and you have the new information, you must move the contact to the new company in your database. It’s a good idea to send along behavioral data like communications preferences. You might also add a LinkedIn profile URL to the record. If you believe the prior behavioral data is important, then take it as a duplicate, and put it in a separate field, not attributing it to the new company record.

    The purchase history belongs with the original company, and should stay there. Indicate in the company record that the individual has left.
    As a general rule, in marketing databases, never overwrite. Keep everything data stamped.

  4. We want our sales people to be selling, and keep administrative tasks to a minimum. But these people are also the closest resources to our customers. How can we motivate them to capture important data about the customers and prospects they are interacting with?
    Boil down the mission to just one or two key data points that reps are asked to collect and report. Job title, buying role and email address are among the most likely to change, and perhaps the most important to keep current. Train and reward the reps on consistent reporting on the selected elements.
  5. In an effort to improve web-form response rates, we are asking for only name and email address. What’s the best way to create a company record in this situation?
    We recommend that you consider hiring a service that will fill in the company record on the spot, as a start. Or send the file out to a third party compiler to append the records you need.

    Another way is to parse the email address. Take the letters after the @ and before the .com. For example, if the email is formatted as, the meaningful letters are hp. Search for other emails with these letters in this position in your file, and build a business rule that every email with these letters shall be assigned that company name. If you have a standard record on your file, import it.

    If the email address is a generic one, like or, it’s more difficult. Email the prospect and ask for more data. You could also consider preventing email addresses other than those from company domains from being accepted on the web form. But keep in mind that there is some evidence that individuals filling out web forms with personal email addresses tend to be more responsive over time.

  6. We need to get our international customer data under control. Where should we start?
    First, add country name as a required field in your web forms and other response vehicles, so that future data collection will be set. Use a dropdown menu to improve capture of a standardized country name. Prevent the record from moving forward until the country is specified.

    Then, look at what parts of the world you do business in. Estimate how many countries, and how many customer records in each country, so you can see how big an issue this is.

    Then, figure out which records in the database are non-U.S. This will take some effort. Many databases don’t have a non-domestic indicator. There is no easy way around it.

    Country names are increasingly important as laws change. Consider Canada’s onerous new email law, which requires proven opt in before emailing. You can’t assume that those email addresses ending with .ca are the only Canadian emails on your file. One suggestion is to update your web forms with a message like “If you are in Canada, opt in here.”

You can find more thorny data issues and solutions in our new white paper, available for free download. Please submit any other issues you may be facing, using the comments section here, and we’ll be happy to suggest some solutions.

Not All Databases Are Created Equal










1. Depth



2. Width



3. Accuracy









4. Recency




5. Consistency




6. Connectivity




7. Delivery Mechanisms


8. User-Friendliness



9. Cost





Business First



Beyond RFM Data




1. Descriptive Data
2. Behavioral Data (mostly Transaction Data)
3. Attitudinal Data.



RFM Data Is Just a Good Start



Creating a Buyer-Centric Portrait




Remember R, F, M, P and C




· Number of Transactions
· Total Dollar Amount
· Number of Days (or Weeks) since the Last Transaction
· Number of Days (or Weeks) since the First Transaction



· Average Dollar Amount per Customer
· Average Dollar Amount per Transaction
· Average Dollar Amount per Year
· Lifetime Highest Amount per Item
· Lifetime Lowest Amount per Transaction
· Average Number of Days Between Transactions
· Etc., etc…



The Time Factor


Analytical Sandbox


Big Data Must Get Smaller

Like many folks who worked in the data business for a long time, I don’t even like the words “Big Data.” Yeah, data is big now, I get it. But so what? Faster and bigger have been the theme in the computing business since the first calculator was invented. In fact, I don’t appreciate the common definition of Big Data that is often expressed in the three Vs: volume, velocity and variety. So, if any kind of data are big and fast, it’s all good? I don’t think so. If you have lots of “dumb” data all over the place, how does that help you? Well, as much as all the clutter that’s been piled on in your basement since 1971. It may yield some profit on an online auction site one day. Who knows? Maybe some collector will pay good money for some obscure Coltrane or Moody Blues albums that you never even touched since your last turntable (Ooh, what is that?) died on you. Those oversized album jackets were really cool though, weren’t they?


Seriously, the word “Big” only emphasizes the size element, and that is a sure way to miss the essence of the data business. And many folks are missing even that little point by calling all decision-making activities that involve even small-sized data “Big Data.” It is entirely possible that this data stuff seems all new to someone, but the data-based decision-making process has been with us for a very long time. If you use that “B” word to differentiate old-fashioned data analytics of yesteryear and ridiculously large datasets of the present day, yes, that is a proper usage of it. But we all know most people do not mean it that way. One side benefit of this bloated and hyped up buzzword is data professionals like myself do not have to explain what we do for living for 20 minutes anymore by simply uttering the word “Big Data,” though that is a lot like a grandmother declaring all her grandchildren work on computers for living. Better yet, that magic “B” word sometimes opens doors to new business opportunities (or at least a chance to grab a microphone in non-data-related meetings and conferences) that data geeks of the past never dreamed of.

So, I guess it is not all that bad. But lest we forget, all hypes lead to overinvestments, and all overinvestments leads to disappointments, and all disappointments lead to purging of related personnel and vendors that bear that hyped-up dirty word in their titles or division names. If this Big Data stuff does not yield significant profit (or reduction in cost), I am certain that those investment bubbles will burst soon enough. Yes, some data folks may be lucky enough to milk it for another two or three years, but brace for impact if all those collected data do not lead to some serious dollar signs. I know how the storage and processing cost decreased significantly in recent years, but they ain’t totally free, and related man-hours aren’t exactly cheap, either. Also, if this whole data business is a new concept to an organization, any money spent on the promise of Big Data easily becomes a liability for the reluctant bunch.

This is why I open up my speeches and lectures with this question: “Have you made any money with this Big Data stuff yet?” Surely, you didn’t spend all that money to provide faster toys and nicer playgrounds to IT folks? Maybe the head of IT had some fun with it, but let’s ask that question to CFOs, not CTOs, CIOs or CDOs. I know some colleagues (i.e., fellow data geeks) who are already thinking about a new name for this—”decision-making activities, based on data and analytics”—because many of us will be still doing that “data stuff” even after Big Data cease to be cool after the judgment day. Yeah, that Gangnam Style dance was fun for a while, but who still jumps around like a horse?

Now, if you ask me (though nobody did yet), I’d say the Big Data should have been “Smart Data,” “Intelligent Data” or something to that extent. Because data must provide insights. Answers to questions. Guidance to decision-makers. To data professionals, piles of data—especially the ones that are fragmented, unstructured and unformatted, no matter what kind of fancy names the operating system and underlying database technology may bear—it is just a good start. For non-data-professionals, unrefined data—whether they are big or small—would remain distant and obscure. Offering mounds of raw data to end-users is like providing a painting kit when someone wants a picture on the wall. Bragging about the size of the data with impressive sounding new measurements that end with “bytes” is like counting grains of rice in California in front of a hungry man.

Big Data must get smaller. People want yes/no answers to their specific questions. If such clarity is not possible, probability figures to such questions should be provided; as in, “There’s an 80 percent chance of thunderstorms on the day of the company golf outing,” “An above-average chance to close a deal with a certain prospect” or “Potential value of a customer who is repeatedly complaining about something on the phone.” It is about easy-to-understand answers to business questions, not a quintillion bytes of data stored in some obscure cloud somewhere. As I stated at the end of my last column, the Big Data movement should be about (1) Getting rid of the noise, and (2) Providing simple answers to decision-makers. And getting to such answers is indeed the process of making data smaller and smaller.

In my past columns, I talked about the benefits of statistical models in the age of Big Data, as they are the best way to compact big and complex information in forms of simple answers (refer to “Why Model?”). Models built to predict (or point out) who is more likely to be into outdoor sports, to be a risk-averse investor, to go on a cruise vacation, to be a member of discount club, to buy children’s products, to be a bigtime donor or to be a NASCAR fan, are all providing specific answers to specific questions, while each model score is a result of serious reduction of information, often compressing thousands of variables into one answer. That simplification process in itself provides incredible value to decision-makers, as most wouldn’t know where to cut out unnecessary information to answer specific questions. Using mathematical techniques, we can cut down the noise with conviction.

In model development, “Variable Reduction” is the first major step after the target variable is determined (refer to “The Art of Targeting“). It is often the most rigorous and laborious exercise in the whole model development process, where the characteristics of models are often determined as each statistician has his or her unique approach to it. Now, I am not about to initiate a debate about the best statistical method for variable reduction (I haven’t met two statisticians who completely agree with each other in terms of methodologies), but I happened to know that many effective statistical analysts separate variables in terms of data types and treat them differently. In other words, not all data variables are created equal. So, what are the major types of data that database designers and decision-makers (i.e., non-mathematical types) should be aware of?

In the business of predictive analytics for marketing, the following three types of data make up three dimensions of a target individual’s portrait:

  1. Descriptive Data
  2. Transaction Data / Behavioral Data
  3. Attitudinal Data

In other words, if we get to know all three aspects of a person, it will be much easier to predict what the person is about and/or what the person will do. Why do we need these three dimensions? If an individual has a high income and is living in a highly valued home (demographic element, which is descriptive); and if he is an avid golfer (behavioral element often derived from his purchase history), can we just assume that he is politically conservative (attitudinal element)? Well, not really, and not all the time. Sometimes we have to stop and ask what the person’s attitude and outlook on life is all about. Now, because it is not practical to ask everyone in the country about every subject, we often build models to predict the attitudinal aspect with available data. If you got a phone call from a political party that “assumes” your political stance, that incident was probably not random or accidental. Like I emphasized many times, analytics is about making the best of what is available, as there is no such thing as a complete dataset, even in this age of ubiquitous data. Nonetheless, these three dimensions of the data spectrum occupy a unique and distinct place in the business of predictive analytics.

So, in the interest of obtaining, maintaining and utilizing all possible types of data—or, conversely, reducing the size of data with conviction by knowing what to ignore, let us dig a little deeper:

Descriptive Data
Generally, demographic data—such as people’s income, age, number of children, housing size, dwelling type, occupation, etc.—fall under this category. For B-to-B applications, “Firmographic” data—such as number of employees, sales volume, year started, industry type, etc.—would be considered as descriptive data. It is about what the targets “look like” and, generally, they are frozen in the present time. Many prominent data compilers (or data brokers, as the U.S. government calls them) collect, compile and refine the data and make hundreds of variables available to users in various industry sectors. They also fill in the blanks using predictive modeling techniques. In other words, the compilers may not know the income range of every household, but using statistical techniques and other available data—such as age, home ownership, housing value, and many other variables—they provide their best estimates in case of missing values. People often have some allergic reaction to such data compilation practices siting privacy concerns, but these types of data are not about looking up one person at a time, but about analyzing and targeting groups (or segments) of individuals and households. In terms of predictive power, they are quite effective and results are very consistent. The best part is that most of the variables are available for every household in the country, whether they are actual or inferred.

Other types of descriptive data include geo-demographic data, and the Census Data by the U.S. Census Bureau falls under this category. These datasets are organized by geographic denominations such as Census Block Group, Census Tract, Country or ZIP Code Tabulation Area (ZCTA, much like postal ZIP codes, but not exactly the same). Although they are not available on an individual or a household level, the Census data are very useful in predictive modeling, as every target record can be enhanced with it, even when name and address are not available, and data themselves are very stable. The downside is that while the datasets are free through Census Bureau, the raw datasets contain more than 40,000 variables. Plus, due to the budget cut and changes in survey methods during the past decade, the sample size (yes, they sample) decreased significantly, rendering some variables useless at lower geographic denominations, such as Census Block Group. There are professional data companies that narrowed down the list of variables to manageable sizes (300 to 400 variables) and filled in the missing values. Because they are geo-level data, variables are in the forms of percentages, averages or median values of elements, such as gender, race, age, language, occupation, education level, real estate value, etc. (as in, percent male, percent Asian, percent white-collar professionals, average income, median school years, median rent, etc.).

There are many instances where marketers cannot pinpoint the identity of a person due to privacy issues or challenges in data collection, and the Census Data play a role of effective substitute for individual- or household-level demographic data. In predictive analytics, duller variables that are available nearly all the time are often more valuable than precise information with limited availability.

Transaction Data/Behavioral Data
While descriptive data are about what the targets look like, behavioral data are about what they actually did. Often, behavioral data are in forms of transactions. So many just call it transaction data. What marketers commonly refer to as RFM (Recency, Frequency and Monetary) data fall under this category. In terms of predicting power, they are truly at the top of the food chain. Yes, we can build models to guess who potential golfers are with demographic data, such as age, gender, income, occupation, housing value and other neighborhood-level information, but if you get to “know” that someone is a buyer of a box of golf balls every six weeks or so, why guess? Further, models built with transaction data can even predict the nature of future purchases, in terms of monetary value and frequency intervals. Unfortunately, many who have access to RFM data are using them only in rudimentary filtering, as in “select everyone who spends more than $200 in a gift category during the past 12 months,” or something like that. But we can do so much more with rich transaction data in every stage of the marketing life cycle for prospecting, cultivating, retaining and winning back.

Other types of behavioral data include non-transaction data, such as click data, page views, abandoned shopping baskets or movement data. This type of behavioral data is getting a lot of attention as it is truly “big.” The data have been out of reach for many decision-makers before the emergence of new technology to capture and store them. In terms of predictability, nevertheless, they are not as powerful as real transaction data. These non-transaction data may provide directional guidance, as they are what some data geeks call “a-camera-on-everyone’s-shoulder” type of data. But we all know that there is a clear dividing line between people’s intentions and their commitments. And it can be very costly to follow every breath you take, every move you make, and every step you take. Due to their distinct characteristics, transaction data and non-transaction data must be managed separately. And if used together in models, they should be clearly labeled, so the analysts will never treat them the same way by accident. You really don’t want to mix intentions and commitments.

The trouble with the behavioral data are, (1) they are difficult to compile and manage, (2) they get big; sometimes really big, (3) they are generally confined within divisions or companies, and (4) they are not easy to analyze. In fact, most of the examples that I used in this series are about the transaction data. Now, No. 3 here could be really troublesome, as it equates to availability (or lack thereof). Yes, you may know everything that happened with your customers, but do you know where else they are shopping? Fortunately, there are co-op companies that can answer that question, as they are compilers of transaction data across multiple merchants and sources. And combined data can be exponentially more powerful than data in silos. Now, because transaction data are not always available for every person in databases, analysts often combine behavioral data and descriptive data in their models. Transaction data usually become the dominant predictors in such cases, while descriptive data play the supporting roles filling in the gaps and smoothing out the predictive curves.

As I stated repeatedly, predictive analytics in marketing is all about finding out (1) whom to engage, and (2) if you decided to engage someone, what to offer to that person. Using carefully collected transaction data for most of their customers, there are supermarket chains that achieved 100 percent customization rates for their coupon books. That means no two coupon books are exactly the same, which is a quite impressive accomplishment. And that is all transaction data in action, and it is a great example of “Big Data” (or rather, “Smart Data”).

Attitudinal Data
In the past, attitudinal data came from surveys, primary researches and focus groups. Now, basically all social media channels function as gigantic focus groups. Through virtual places, such as Facebook, Twitter or other social media networks, people are freely volunteering what they think and feel about certain products and services, and many marketers are learning how to “listen” to them. Sentiment analysis falls under that category of analytics, and many automatically think of this type of analytics when they hear “Big Data.”

The trouble with social data is:

  1. We often do not know who’s behind the statements in question, and
  2. They are in silos, and it is not easy to combine such data with transaction or demographic data, due to lack of identity of their sources.

Yes, we can see that a certain political candidate is trending high after an impressive speech, but how would we connect that piece of information to whom will actually donate money for the candidate’s causes? If we can find out “where” the target is via an IP address and related ZIP codes, we may be able to connect the voter to geo-demographic data, such as the Census. But, generally, personally identifiable information (PII) is only accessible by the data compilers, if they even bothered to collect them.

Therefore, most such studies are on a macro level, citing trends and directions, and types of analysts in that field are quite different from the micro-level analysts who deal with behavioral data and descriptive data. Now, the former provide important insights regarding the “why” part of the equation, which is often the hardest thing to predict; while the latter provide answers to “who, what, where and when.” (“Who” is the easiest to answer, and “when” is the hardest.) That “why” part may dictate a product development part of the decision-making process at the conceptual stage (as in, “Why would customers care for a new type of dishwasher?”), while “who, what, where and when” are more about selling the developed products (as in “Let’s sell those dishwashers in the most effective ways.”). So, it can be argued that these different types of data call for different types of analytics for different cycles in the decision-making processes.

Obviously, there are more types of data out there. But for marketing applications dealing with humans, these three types of data complete the buyers’ portraits. Now, depending on what marketers are trying to do with the data, they can prioritize where to invest first and what to ignore (for now). If they are early in the marketing cycle trying to develop a new product for the future, they need to understand why people want something and behave in certain ways. If signing up as many new customers as possible is the immediate goal, finding out who and where the ideal prospects are becomes the most imminent task. If maximizing the customer value is the ongoing objective, then you’d better start analyzing transaction data more seriously. If preventing attrition is the goal, then you will have to line up the transaction data in time series format for further analysis.

The business goals must dictate the analytics, and the analytics call for specific types of data to meet the goals, and the supporting datasets should be in “analytics-ready” formats. Not the other way around, where businesses are dictated by the limitations of analytics, and analytics are hampered by inadequate data clutters. That type of business-oriented hierarchy should be the main theme of effective data management, and with clear goals and proper data strategy, you will know where to invest first and what data to ignore as a decision-maker, not necessarily as a mathematical analyst. And that is the first step toward making the Big Data smaller. Don’t be impressed by the size of the data, as they often blur the big picture and not all data are created equal.