2.3: Ancient Mesopotamia
-
- Last updated
- Save as PDF
Learning Objectives
By the end of this section, you will be able to:
- Identify characteristics of civilization in Ancient Mesopotamia
- Discuss the political history of Mesopotamia from the early Sumerian city-states to the rise of Old Babylon
- Describe the economy, society, and religion of Ancient Mesopotamia
In the fourth millennium BCE, the world’s first great cities arose in southern Mesopotamia , or the land between the Tigris and Euphrates Rivers, then called Sumer . The ancient Sumerians were an inventive people responsible for a host of technological advances, most notably a sophisticated writing system. Even after the Sumerian language ceased to be spoken early in the second millennium BCE, Sumerian literary works survived throughout the whole of Mesopotamia and were often collected by later cities and stored in the first libraries.
The Rise and Eclipse of Sumer
The term Mesopotamia , or “the land between the rivers” in Greek, likely originated with the Greek historian Herodotus in the fifth century BCE and has become the common name for the place between the Tigris and Euphrates Rivers in what is now Iraq. The rivers flow north to south, from the Taurus Mountains of eastern Turkey to the Persian Gulf, depositing fertile soil along their banks. Melting snow and rain from the mountains carry this topsoil to the river valleys below. In antiquity, the river flow was erratic, and flooding was frequent but unpredictable. The need to control it and manage the life-giving water led to the building of cooperative irrigation projects.
Agricultural practices reached Mesopotamia by around 8000 BCE, if not earlier. However, for about two millennia afterward, populations remained quite small, typically living in small villages of between one hundred and two hundred people. Beginning around 5500 BCE, some had begun to establish settlements in southern Mesopotamia, a wetter and more forbidding environment. It was here that the Sumerian civilization emerged (Figure 3.8). By around 4500 BCE, some of the once-small farming villages had become growing urban centers, some with thousands of residents. During the course of the fourth millennium BCE (3000s BCE), urbanization exploded in the region. By the end of the millennium, there were at least 124 villages with about one hundred residents each, twenty towns with as many as two thousand residents, another twenty small urban centers of about five thousand residents, and one large city, Uruk , with a population that may have been as high as fifty thousand. This growth helped make Sumer the earliest civilization to develop in Mesopotamia.
The fourth millennium BCE in Sumer was also a period of technological innovation. One important invention made after 4000 BCE was the process for manufacturing bronze, an alloy of tin and copper, which marked the beginning of the Bronze Age in Mesopotamia. In this period, bronze replaced stone as the premier material for tools and weapons and remained so for nearly three thousand years. The ancient Sumerians also developed the plow, the wheel, and irrigation techniques that used small channels and canals with dikes for diverting river water into fields. All these developments allowed for population growth and the continued rise of cities by expanding agricultural production and the distribution of agricultural goods. In the area of science, the Sumerians developed a sophisticated mathematical system based on the numbers sixty, ten, and one.
One of the greatest inventions of this period was writing. The Sumerians developed cuneiform , a script characterized by wedge-shaped symbols that evolved into a phonetic script, that is, one based on sounds, in which each symbol stood for a syllable (Figure 3.9). They wrote their laws, religious tracts, and property transactions on clay tablets, which became very durable once baked, just like the clay bricks the Sumerians used to construct their buildings. The clay tablets held records of commercial exchanges, including contracts and receipts as well as taxes and payrolls. Cuneiform also allowed rulers to record their laws and priests to preserve their rituals and sacred stories. In these ways, it helped facilitate both economic growth and the formation of states.
Dueling Voices
The Invention of Writing in Sumer
Writing developed independently in several parts of the world, but the earliest known evidence of its birth has been found in Sumer, where cuneiform script emerged as a genuine writing system by around 3000 BCE, if not earlier. But questions remain about how and why ancient peoples began reproducing their spoken language in symbolic form.
Archaeologist Denise Schmandt-Besserat argued in the 1990s that small clay representations of numbers and objects, often called “tokens,” date from thousands of years before the development of cuneiform writing and were its precursor. These tokens, she believed, were part of an accounting system, and each type represented a different good: livestock, grains, and oils. Some were found within hollow baseball-sized clay balls now called “bullae,” which were marked with pictures of the tokens inside. Schmandt-Besserat believed the pictures portray the type of transaction in which the goods represented by the tokens were exchanged, and thus they were a crucial step toward writing. Over time, she suggested, the marked bullae gave way to flat clay tablets recording the transactions, and the first truly written records emerged (Figure 3.10).
Schmandt-Besserat’s linear interpretation is still one of the best-known explanations for the emergence of writing. But it is hardly the only one. One scholar who offers a different idea is the French Assyriologist Jean-Jacques Glassner. Glassner believes that rather than being an extension of accounting techniques, early writing was a purposeful attempt to render the Sumerian language in script. He equates the development of writing, which gives meaning to a symbol, to the process by which Mesopotamian priests interpreted omens for divining the future. Writing allowed people to place language, a creation of the gods, under human control. Glassner’s argument is complex and relies on ancient works of literature and various theoretical approaches, including that of postmodernist philosopher Jacques Derrida.
Many disagree with Glassner’s conclusions, and modern scholars concede that tokens likely played an important role, but probably not in the linear way Schmandt-Besserat proposed. Uncertainty about the origin of writing in Sumer still abounds, and the scholarly debate continues.
- Why do you think Schmandt-Besserat’s argument was once so appealing?
- If you lived in a society with no writing, what might prompt you to develop a way to represent your language in symbolic form?
Cuneiform was a very complex writing system, and literacy remained the monopoly of an elite group of highly trained writing specialists, the scribes. But the script was also highly flexible and could be used to symbolize a great number of sounds, allowing subsequent Mesopotamian cultures such as the Akkadians, Babylonians, and many more to adapt it to their own languages. Since historians deciphered cuneiform in the nineteenth century, they have read the thousands of clay tablets that survived over the centuries and learned much about the history, society, economy, and beliefs of the ancient Sumerians and other peoples of Mesopotamia.
The Sumerians were polytheists , people who revered many gods. Each Sumerian city had its own patron god, however, one with whom the city felt a special connection and whom it honored above the others. For example, the patron god of Uruk was Inanna, the goddess of fertility; the city of Nippur revered the weather god Enlil; and Ur claimed the moon god Sin. Each city possessed an immense temple complex for its special deity, which included a site where the deity was worshipped and religious rituals were performed. This site, the ziggurat , was a stepped tower built of mud-brick with a flat top (Figure 3.11). At its summit stood a roofed structure that housed the sacred idol or image of the temple’s deity. The temple complex also included the homes of the priests, workshops for artisans who made goods for the temple, and storage facilities to meet the needs of the temple workers.
Sumerians were clearly eager to please their gods by placing them at the center of their society. These gods could be fickle, faithless, and easily stirred to anger. If displeased with the people, they might bring famine or conquest. Making sure the gods were praised and honored was thus a way of ensuring prosperity. Praising them, however, implied different things for different social tiers in Sumer . For common people, it meant living a virtuous life and giving to the poor. For priests and priestesses, it consisted of performing the various rituals at the temple complexes. And for rulers honoring the gods, it meant ensuring that the temples were properly funded, maintained, and regularly beautified and enlarged if possible.
By the Early Dynastic Period (c. 2650 BCE–2400 BCE), powerful dynasties of kings called lugals had established themselves as rulers of the cities. In each city, the lugals rose to power primarily as warlords, since the Sumerian cities often waged war against each other for control of farmland and access to water as well as other natural resources. Lugals legitimized their authority through the control of the religious institutions of the city. For example, at Ur, the daughter of the reigning lugal always served as the high priestess of the moon god Sin, the chief deity at Ur.
The lugals at Ur during this period, the so-called First Dynasty of Ur, were especially wealthy, as reflected in the magnificent beehive-shaped tombs in which they were buried. In these tombs, precious goods such as jewelry and musical instruments were stored, along with the bodies of servants who were killed and placed in the tomb to accompany the rulers to the Land of the Dead. One of the more spectacular tombs belonged to a woman of Ur called Pu-Abi, who was buried wearing an elaborate headdress and might have been a queen (Figure 3.12). The most famous lugal in all Sumer in this early period was Gilgamesh of Uruk, whose legendary exploits were recounted later in fantastical form in the Epic of Gilgamesh .
Link to Learning
The Epic of Gilgamesh is one of the world’s earliest examples of epic literature. To understand this ancient tale, first written down in the form we know today around 2100 BCE, read the overview of the Epic of Gilgamesh provided by the Metropolitan Museum of Art, which has a notable collection of ancient Mesopotamian artifacts.
The Rise of the World’s First Empire
Around 2300 BCE, the era of the independent Sumerian city-state , a political entity consisting of a city and surrounding territory that it controls, came to an end. Sumer and indeed all of Mesopotamia was conquered by Sargon of Akkad , who created the first-known empire, in this case, a number of regional powers under the control of one person. The word “Akkad” in his name was a reference to the Akkadians, a group that settled in central Mesopotamia, north of Sumer, around the ancient city of Kish. Over time, the Akkadians adopted Sumerian culture and adapted cuneiform to their own language, a language of the Semitic family that includes the Arabic and Hebrew spoken today. They also identified their own gods with the gods of the Sumerians and adopted Sumerian myths. For example, the Akkadians identified the fertility goddess Inanna with their own goddess Ishtar.
Sargon conquered not only Sumer but also what is today northern Iraq, Syria, and southwestern Iran. While the precise details of his origin and rise to power are not known, scholars believe the story Sargon told about himself, at least, has likely been accurately preserved in the Legend of Sargon , written two centuries after his death as a purported autobiography. It is a familiar story of a scrappy young hero born in humble circumstances and rising on his own merits to become a great leader. The Legend relates how, when Sargon was a baby, his unwed mother put him in a basket and cast it on the Euphrates River. A farmer found and raised him, and Ishtar loved Sargon and elevated him from a commoner to a great king and conqueror.
This interesting tale would have certainly been a powerful piece of propaganda justifying Sargon’s rule and endearing him to the common people, and some of it may even be true. But from what historians can tell, Sargon’s rise to power likely occurred during a period of turmoil as his kingdom of Kish, of which he had likely seized control, came under attack by another king named Lugalzagesi. Sargon’s eventual defeat of Lugalzagesi and conquest of all of Sumer proved to be the beginning of a larger conquest of Mesopotamia. The Akkadian Empire that Sargon created lasted for about a century and a half, officially coming to an end in the year 2193 BCE (Figure 3.13).
One of the rivals of the Akkadian Empire was the city-state of Ebla, located in northwestern Syria. At some point, its people had adapted Sumerian cuneiform to their own language, which, like Akkadian, belonged to the Semitic family of languages, and archaeologists have discovered thousands of cuneiform tablets at the site. These tablets reveal that Ebla especially worshipped the storm god Adad, who was honored with the title “Ba‘al” or lord. More than one thousand years later in the Iron Age, people in this region still worshipped Baal, who was the main rival of Yahweh for the affections of the ancient Israelites.
Other rivals of the Akkadians were the Elamites , who inhabited the region to the immediate southeast of Mesopotamia in southwest Iran and whose city of Susa arose around 4000 BCE. The art and architecture of the Elamites suggest a strong Sumerian influence. They developed their own writing system around 3000 BCE, even though they adapted Sumerian cuneiform to their language later in the third millennium BCE. The Elamites also worshipped their own distinct deities, such as Insushinak, the Lord of the Dead. Both Elam and Ebla eventually suffered defeat at the hands of the Akkadians.
In the year 2193 BCE, however, the Akkadian Empire collapsed. The precise reason is not entirely clear. However, some ancient accounts point to the incursions of the nomadic Guti tribes, whose original homes were located in the Zagros Mountains of northwestern Iran, northwest of Mesopotamia. These Guti were originally pastoralists , who lived off their herds of livestock and moved from place to place to find pasture for their animals. While the Guti tribes certainly did move into the Akkadian Empire toward its end, modern scholarship suggests that the empire was likely experiencing internal decline and famine before this. The Guti appear to have exploited this weakness rather than triggering it. Regardless, for around a century, the Guti ruled over Sumer and adopted its culture as their own. Around 2120 BCE, however, the Sumerians came together under the leadership of the cities of Uruk and Ur and expelled the Guti from their homeland.
Later Empires in Mesopotamia
While Sargon’s empire lasted only a few generations, his conquests dramatically transformed politics in Mesopotamia. The era of independent city-states waned, and over the next few centuries, a string of powerful Mesopotamian rulers were able to build their own empires, often using the administrative techniques developed by Sargon as a model. For example, beginning about 2112 BCE, all Sumer was again united under the Third Dynasty of Ur as the Guti were driven out. The rulers of this dynasty held the title of lugal of all Sumer and Akkad, and they were also honored as gods. They built temples in the Sumerian city of Nippur, which was sacred to the storm god Enlil, the ruler of the gods in the Sumerian pantheon. The most famous lugal of this dynasty was Ur-Nammu (c. 2150 BCE), renowned for his works of poetry as well as for the law code he published.
At its height, the Third Dynasty extended its control over both southern and northern Mesopotamia. But by the end of the third millennium, change was on the horizon. Foreign invaders from the north, east, and west put tremendous pressure on the empire, and its rulers increased their military preparedness and even constructed a 170-mile fortification wall to keep them out. While these strategies were somewhat effective, they appear to have only postponed the inevitable as Amorites, Elamites, and other groups eventually poured in and raided cities across the land. By about 2004 BCE, Sumer had crumbled, and even Ur was violently sacked by the invaders.
Link to Learning
The sack of Ur by the Elamites and others was the inspiration for a lament or song of mourning that became a classic of Sumerian literature. Read The Lament for Urim and pay attention to the way the writer attributes the destruction to the caprice of the gods; the actual invaders are merely tools. For descriptions of the destruction itself, focus on lines 161–229.
In the centuries after 2004 BCE, the migration of Amorites into Mesopotamia resulted in the gradual disappearance of Sumerian as a spoken language. People in the region came to speak Amorite, which belonged to the family of Semitic languages. Nonetheless, scribes continued to preserve and write works in Sumerian and Akkadian cuneiform. Sumerian and Akkadian became the languages of religious rituals, hymns, and prayers, as well as classic literary works such as the Epic of Gilgamesh . Consequently, the literary output of these earlier cultures was preserved and transmitted to the new settlers. When nomadic Amorite tribes settled in Mesopotamia, they eventually established new cities such as Mari, Asshur, and Babylon, and they adopted much of the culture they encountered. The ancient Sumerian cities of Larsa and Isin of this era also preserved these cultural traditions, even as they came under the rule of Amorite kings.
Hammurabi , the energetic ruler of Babylon during the first half of the eighteenth century BCE, defeated the kings of the rival cities of Mari and Larsa and created an empire that encompassed nearly all of Mesopotamia. To unify this new empire, Hammurabi initiated the construction of irrigation projects, built new temples at Nippur, and published his legal edicts throughout his realm. Hammurabi had these edicts inscribed on stone pillars erected in different places in the empire to inform his subjects about proper behavior and the laws of the land. Being especially clear, the Code of Hammurabi far outlived the king who created it. It also provides us with a fascinating window into how Mesopotamian society functioned at this time.
In Their Own Words
The Law in Old Babylon
Remarkable for its clarity, the Code of Hammurabi may have introduced concepts like the presumption of innocence and the use of evidence. It informed legal systems in Mesopotamia for many centuries after Hammurabi’s death (Figure 3.14).
The Code of Hammurabi promoted the principle that punishment should fit the crime, but penalties often depended on social class:
199. If [a man] put out the eye of a man’s slave, or break the bone of a man’s slave, he shall pay one-half of its value.
202. If any one strike the body of a man higher in rank than he, he shall receive sixty blows with an ox-whip in public.
Many edicts concern marriage, adultery, children, and marriage property.
129. If a man’s wife be surprised with another man, both shall be tied and thrown into the water, but the husband may pardon his wife and the king his slaves.
150. If a man give his wife a field, garden, and house and a deed therefor, if then after the death of her husband the sons raise no claim, then the mother may bequeath all to one of her sons whom she prefers, and need leave nothing to his brothers.
A good number of the code’s edicts concern the settling of commercial disputes:
9. If anyone lose an article, and find it in the possession of another [who says] “A merchant sold it to me, I paid for it before witnesses,” . . . The judge shall examine their testimony—both of the witnesses before whom the price was paid, and of the witnesses who identify the lost article on oath. The merchant is then proved to be a thief and shall be put to death. The owner of the lost article receives his property, and he who bought it receives the money he paid from the estate of the merchant.
48. If anyone owe a debt for a loan, and a storm prostrates the grain, or the harvest fail, or the grain does not grow for lack of water; in that year he need not give his creditor any grain, he washes his debt-tablet in water and pays no rent for this year.
—"Hammurabi’s Code of Laws,” c. 1780 BCE, translated by L.W. King
- What do these edicts suggest about the different social tiers in Babylonian society? How were they organized?
- Was marriage similar to or different from marriage today?
- Do the edicts for resolving economic disputes seem fair to you? Why or why not?
While Hammurabi’s empire lasted a century and a half, much of the territory he conquered began falling away from Babylon’s control shortly after he died. The empire continued to dwindle in size until 1595 BCE, when an army of Hittites from central Anatolia in the north (modern Turkey) sacked the city of Babylon. Shortly thereafter, Kassites from the Zagros Mountains of northwestern Iran conquered Babylon and southern Mesopotamia and settled there, unlike the Hittites who had returned to their Anatolian home. The Kassites established a dynasty that ruled over Babylon for nearly five hundred years, to the very end of the Bronze Age . Like the Guti and the Amorites before them, over time, the Kassite rulers adopted the culture of their Mesopotamian subjects.
Society and Religion in Ancient Mesopotamia
Thanks to the preservation of cuneiform clay tablets and the discovery and translation of law codes and works of literature, historians have at their disposal a wealth of information about Mesopotamian society. The study of these documents and the archaeological excavations carried out in Mesopotamia have allowed them to reconstruct the empire’s economy.
We know now that temples and royal palaces were not merely princely residences and places for religious rituals; they also functioned as economic redistribution centers. For example, agricultural goods were collected from farmers as taxes by civic and religious officials, who then stored them to provide payments to the artisans and merchants they employed. Palaces and temples thus needed to possess massive storage facilities. Scribes kept records in cuneiform of all the goods collected and distributed by these institutions. City gates served as areas where farmers, artisans, and merchants could congregate and exchange goods. Precious metals such as gold often served as a medium of exchange, but these goods had to be weighed and measured during commercial exchanges, since coinage and money as we understand it today did not emerge until the Iron Age, a millennium later.
Society in southern Mesopotamia was highly urban. About 70 to 80 percent of the population lived in cities, but not all were employed as artisans, merchants, or other traditional urban roles. Rather, agriculture and animal husbandry accounted for a majority of a city’s economic production. Much of the land was controlled by the temples, kings, or other powerful landowners and was worked by semi-free peasants who were tied to the land. The rest of the land included numerous small plots worked by the free peasants who made up about half the population. A much smaller portion was made up of enslaved people, typically prisoners of war or persons who had committed crimes or gone into debt. A man could sell his own children into slavery to cover a debt.
Much of the hard labor performed in the fields was done by men and boys, while the wives, mothers, and daughters of merchants and artisans were sometimes fully engaged in running family businesses. Cuneiform tablets tell us that women oversaw the business affairs of their families, especially when husbands were merchants who often traveled far from home. For example, cuneiform tablets from circa 1900 BCE show that merchants from Ashur in northern Mesopotamia conducted trade with central Anatolia and wrote letters to their female family members back home. Women were also engaged in the production of textiles like wool and linen. They not only produced these textiles in workshops with their own hands, but some appear to have held managerial positions within the textile industry.
Free peasant farmers, artisans, and merchants were all commoners. This put them in a higher social position than the semi-free peasants and slaves but lower than the elite nobility, who made up a very small percentage of the population and whose ranks included priests, official scribes, and military leaders. This aristocratic elite often received land in payment for their services to the kings and collected rents in kind from their peasant tenants. Social distinctions were also reflected in the law. For example, aspects of Hammurabi’s law code called for punishments for causing physical harm to another to be equal to the harm inflicted. This principle is best summarized in the line “an eye for an eye and a tooth for a tooth.” However, the principle applied only to victims and perpetrators of the same social class.
Men and women were not equal under the Code of Hammurabi . A man was free to have multiple wives and divorce a wife at will, whereas a woman could divorce her husband only if she could prove he had been unkind to her without reason. However, a woman from a family of means could protect her position in a marriage if her family put up a dowry, which could be land or goods. Upon marriage, the husband obtained the dowry, but if he divorced or was unkind to his wife, he had to return it to her and her family.
Cuneiform tablets have also allowed historians to read stories about the gods and heroes of Mesopotamian cultures. Mesopotamians revered many different gods associated with forces of nature. These were anthropomorphic deities who not only had divine powers but also frequently acted on very human impulses like anger, fear, annoyance, and lust. Examples include Utu, the god of the sun (Figure 3.15); Inanna (known to the Akkadians as Ishtar), the goddess of fertility; and Enlil (whose equivalent in other Mesopotamian cultures was Marduk), the god of wind and rain. The ancient Mesopotamians held that the gods were visible in the sky as heavenly bodies like stars, the moon, the sun, and the planets. This belief led them to pay close attention to these bodies, and over time, they developed a sophisticated understanding of their movement. This knowledge allowed them to predict astronomical events like eclipses and informed their development of a twelve-month calendar.
People in Mesopotamia believed human beings were created to serve the gods (Figure 3.16). They were expected to supply the gods with food through the sacrifice of sheep and cattle in religious rituals, and to honor them with temples, religious songs or hymns, and expensive gifts. People sought divine support from their gods. But they also feared that their worship might be insufficient and anger the deity. When that happened, the gods could bring death and devastation through floods and pestilence. Stories of gods wreaking great destruction, sometimes for petty reasons, are common in Mesopotamian myths. For example, in one Sumerian myth, the storm god Enlil nearly destroyed the entire human race with a flood when the noise made by humans annoyed him and kept him from sleep.
The ancient Mesopotamians’ belief that the gods were fickle, destructive, and easily stirred to anger is one reason many historians believe they had a generally pessimistic worldview. From the literature they left behind, we can see that while they hoped for the best, they were often resigned to accept the worst. Given the environment in which Mesopotamian civilization emerged, this pessimism is somewhat understandable. River flooding was common and could often be unpredictable and destructive. Wars between city-states and the destruction that comes with conflict were also common. Life was difficult in this unforgiving world, and the profiles of the various gods of the Mesopotamians reflect this harsh reality.
Evidence of Mesopotamians’ pessimism is also present in their view of the afterlife. In their religion, after death all people spent eternity in a shadowy underworld sometimes called “the land of no return.” Descriptions of this place differ somewhat in the details, but the common understanding was that it was a gloomy and frightening place where the dead were consumed by sorrow, eating dust and clay and longing pitifully and futilely to return to the land of the living.


13: Database Development Process
A core aspect of software engineering is the subdivision of the development process into a series of phases, or steps, each of which focuses on one aspect of the development. The collection of these steps is sometimes referred to as the software development life cycle (SDLC) . The software product moves through this life cycle (sometimes repeatedly as it is refined or redeveloped) until it is finally retired from use. Ideally, each phase in the life cycle can be checked for correctness before moving on to the next phase.
Software Development Life Cycle – Waterfall
Let us start with an overview of the waterfall model such as you will find in most software engineering textbooks. This waterfall figure, seen in Figure 13.1, illustrates a general waterfall model that could apply to any computer system development. It shows the process as a strict sequence of steps where the output of one step is the input to the next and all of one step has to be completed before moving onto the next.
Figure 13.1. Waterfall model.We can use the waterfall process as a means of identifying the tasks that are required, together with the input and output for each activity. What is important is the scope of the activities, which can be summarized as follows:
- Establishing requirements involves consultation with, and agreement among, stakeholders about what they want from a system, expressed as a statement of requirements.
- Analysis starts by considering the statement of requirements and finishes by producing a system specification. The specification is a formal representation of what a system should do, expressed in terms that are independent of how it may be realized.
- Design begins with a system specification, produces design documents and provides a detailed description of how a system should be constructed.
- Implementation is the construction of a computer system according to a given design document and taking into account the environment in which the system will be operating (e.g., specific hardware or software available for the development). Implementation may be staged, usually with an initial system that can be validated and tested before a final system is released for use.
- Testing compares the implemented system against the design documents and requirements specification and produces an acceptance report or, more usually, a list of errors and bugs that require a review of the analysis, design and implementation processes to correct (testing is usually the task that leads to the waterfall model iterating through the life cycle).
- Maintenance involves dealing with changes in the requirements or the implementation environment, bug fixing or porting of the system to new environments (e.g., migrating a system from a standalone PC to a UNIX workstation or a networked environment). Since maintenance involves the analysis of the changes required, design of a solution, implementation and testing of that solution over the lifetime of a maintained software system, the waterfall life cycle will be repeatedly revisited.
Database Life Cycle
We can use the waterfall cycle as the basis for a model of database development that incorporates three assumptions:
- We can separate the development of a database – that is, specification and creation of a schema to define data in a database – from the user processes that make use of the database.
- We can use the three-schema architecture as a basis for distinguishing the activities associated with a schema.
- We can represent the constraints to enforce the semantics of the data once within a database, rather than within every user process that uses the data.
Using these assumptions and Figure 13.2, we can see that this diagram represents a model of the activities and their outputs for database development. It is applicable to any class of DBMS, not just a relational approach.
Database application development is the process of obtaining real-world requirements, analyzing requirements, designing the data and functions of the system, and then implementing the operations in the system.
Requirements Gathering
The first step is requirements gathering . During this step, the database designers have to interview the customers (database users) to understand the proposed system and obtain and document the data and functional requirements. The result of this step is a document that includes the detailed requirements provided by the users.
Establishing requirements involves consultation with, and agreement among, all the users as to what persistent data they want to store along with an agreement as to the meaning and interpretation of the data elements. The data administrator plays a key role in this process as they overview the business, legal and ethical issues within the organization that impact on the data requirements.
The data requirements document is used to confirm the understanding of requirements with users. To make sure that it is easily understood, it should not be overly formal or highly encoded. The document should give a concise summary of all users’ requirements – not just a collection of individuals’ requirements – as the intention is to develop a single shared database.
The requirements should not describe how the data is to be processed, but rather what the data items are, what attributes they have, what constraints apply and the relationships that hold between the data items.
Analysis
Data analysis begins with the statement of data requirements and then produces a conceptual data model. The aim of analysis is to obtain a detailed description of the data that will suit user requirements so that both high and low level properties of data and their use are dealt with. These include properties such as the possible range of values that can be permitted for attributes (e.g., in the school database example, the student course code, course title and credit points).
The conceptual data model provides a shared, formal representation of what is being communicated between clients and developers during database development – it is focused on the data in a database, irrespective of the eventual use of that data in user processes or implementation of the data in specific computer environments. Therefore, a conceptual data model is concerned with the meaning and structure of data, but not with the details affecting how they are implemented.
The conceptual data model then is a formal representation of what data a database should contain and the constraints the data must satisfy. This should be expressed in terms that are independent of how the model may be implemented. As a result, analysis focuses on the questions, “What is required?” not “How is it achieved?”
Logical Design
Database design starts with a conceptual data model and produces a specification of a logical schema; this will determine the specific type of database system (network, relational, object-oriented) that is required. The relational representation is still independent of any specific DBMS; it is another conceptual data model.
We can use a relational representation of the conceptual data model as input to the logical design process. The output of this stage is a detailed relational specification, the logical schema, of all the tables and constraints needed to satisfy the description of the data in the conceptual data model. It is during this design activity that choices are made as to which tables are most appropriate for representing the data in a database. These choices must take into account various design criteria including, for example, flexibility for change, control of duplication and how best to represent the constraints. It is the tables defined by the logical schema that determine what data are stored and how they may be manipulated in the database.
Database designers familiar with relational databases and SQL might be tempted to go directly to implementation after they have produced a conceptual data model. However, such a direct transformation of the relational representation to SQL tables does not necessarily result in a database that has all the desirable properties: completeness, integrity, flexibility, efficiency and usability. A good conceptual data model is an essential first step towards a database with these properties, but that does not mean that the direct transformation to SQL tables automatically produces a good database. This first step will accurately represent the tables and constraints needed to satisfy the conceptual data model description, and so will satisfy the completeness and integrity requirements, but it may be inflexible or offer poor usability. The first design is then flexed to improve the quality of the database design. Flexing is a term that is intended to capture the simultaneous ideas of bending something for a different purpose and weakening aspects of it as it is bent.
Figure 13.3 summarizes the iterative (repeated) steps involved in database design, based on the overview given. Its main purpose is to distinguish the general issue of what tables should be used from the detailed definition of the constituent parts of each table – these tables are considered one at a time, although they are not independent of each other. Each iteration that involves a revision of the tables would lead to a new design; collectively they are usually referred to as second-cut designs , even if the process iterates for more than a single loop.
Figure 13.3. A summary of the iterative steps involved in database design.First, for a given conceptual data model, it is not necessary that all the user requirements it represents be satisfied by a single database. There can be various reasons for the development of more than one database, such as the need for independent operation in different locations or departmental control over “their” data. However, if the collection of databases contains duplicated data and users need to access data in more than one database, then there are possible reasons that one database can satisfy multiple requirements, or issues related to data replication and distribution need to be examined.
Second, one of the assumptions about database development is that we can separate the development of a database from the development of user processes that make use of it. This is based on the expectation that, once a database has been implemented, all data required by currently identified user processes have been defined and can be accessed; but we also require flexibility to allow us to meet future requirements changes. In developing a database for some applications, it may be possible to predict the common requests that will be presented to the database and so we can optimize our design for the most common requests.
Third, at a detailed level, many aspects of database design and implementation depend on the particular DBMS being used. If the choice of DBMS is fixed or made prior to the design task, that choice can be used to determine design criteria rather than waiting until implementation. That is, it is possible to incorporate design decisions for a specific DBMS rather than produce a generic design and then tailor it to the DBMS during implementation.
It is not uncommon to find that a single design cannot simultaneously satisfy all the properties of a good database. So it is important that the designer has prioritized these properties (usually using information from the requirements specification); for example, to decide if integrity is more important than efficiency and whether usability is more important than flexibility in a given development.
At the end of our design stage, the logical schema will be specified by SQL data definition language (DDL) statements, which describe the database that needs to be implemented to meet the user requirements.
Implementation
Implementation involves the construction of a database according to the specification of a logical schema. This will include the specification of an appropriate storage schema, security enforcement, external schema and so on. Implementation is heavily influenced by the choice of available DBMSs, database tools and operating environment. There are additional tasks beyond simply creating a database schema and implementing the constraints – data must be entered into the tables, issues relating to the users and user processes need to be addressed, and the management activities associated with wider aspects of corporate data management need to be supported. In keeping with the DBMS approach, we want as many of these concerns as possible to be addressed within the DBMS. We look at some of these concerns briefly now.
In practice, implementation of the logical schema in a given DBMS requires a very detailed knowledge of the specific features and facilities that the DBMS has to offer. In an ideal world, and in keeping with good software engineering practice, the first stage of implementation would involve matching the design requirements with the best available implementing tools and then using those tools for the implementation. In database terms, this might involve choosing vendor products with DBMS and SQL variants most suited to the database we need to implement. However, we don’t live in an ideal world and more often than not, hardware choice and decisions regarding the DBMS will have been made well in advance of consideration of the database design. Consequently, implementation can involve additional flexing of the design to overcome any software or hardware limitations.
Realizing the Design
After the logical design has been created, we need our database to be created according to the definitions we have produced. For an implementation with a relational DBMS, this will probably involve the use of SQL to create tables and constraints that satisfy the logical schema description and the choice of appropriate storage schema (if the DBMS permits that level of control).
One way to achieve this is to write the appropriate SQL DDL statements into a file that can be executed by a DBMS so that there is an independent record, a text file, of the SQL statements defining the database. Another method is to work interactively using a database tool like SQL Server Management Studio or Microsoft Access. Whatever mechanism is used to implement the logical schema, the result is that a database, with tables and constraints, is defined but will contain no data for the user processes.
Populating the Database
After a database has been created, there are two ways of populating the tables – either from existing data or through the use of the user applications developed for the database.
For some tables, there may be existing data from another database or data files. For example, in establishing a database for a hospital, you would expect that there are already some records of all the staff that have to be included in the database. Data might also be brought in from an outside agency (address lists are frequently brought in from external companies) or produced during a large data entry task (converting hard-copy manual records into computer files can be done by a data entry agency). In such situations, the simplest approach to populate the database is to use the import and export facilities found in the DBMS.
Facilities to import and export data in various standard formats are usually available (these functions are also known in some systems as loading and unloading data). Importing enables a file of data to be copied directly into a table. When data are held in a file format that is not appropriate for using the import function, then it is necessary to prepare an application program that reads in the old data, transforms them as necessary and then inserts them into the database using SQL code specifically produced for that purpose. The transfer of large quantities of existing data into a database is referred to as a bulk load . Bulk loading of data may involve very large quantities of data being loaded, one table at a time so you may find that there are DBMS facilities to postpone constraint checking until the end of the bulk loading.
Guidelines for Developing an ER Diagram
Note: These are general guidelines that will assist in developing a strong basis for the actual database design (the logical model).
- Document all entities discovered during the information-gathering stage.
- Document all attributes that belong to each entity. Select candidate and primary keys. Ensure that all non-key attributes for each entity are full-functionally dependent on the primary key.
- Develop an initial ER diagram and review it with appropriate personnel. (Remember that this is an iterative process.)
- Create new entities (tables) for multivalued attributes and repeating groups. Incorporate these new entities (tables) in the ER diagram. Review with appropriate personnel.
- Verify ER modeling by normalizing tables.
Key Terms
analysis : starts by considering the statement of requirements and finishes by producing a system specification
bulk load : the transfer of large quantities of existing data into a database
data requirements document :used to confirm the understanding of requirements with the user
design : begins with a system specification, produces design documents and provides a detailed description of how a system should be constructed
establishing requirements : involves consultation with, and agreement among, stakeholders as to what they want from a system; expressed as a statement of requirements
flexing : a term that captures the simultaneous ideas of bending something for a different purpose and weakening aspects of it as it is bent
implementation : the construction of a computer system according to a given design document
maintenance : involves dealing with changes in the requirements or the implementation environment, bug fixing or porting of the system to new environments
requirements gathering : a process during which the database designer interviews the database user to understand the proposed system and obtain and document the data and functional requirements
second-cut designs :the collection of iterations that each involves a revision of the tables that lead to a new design
software development life cycle (SDLC) : the series of steps involved in the database development process
testing : compares the implemented system against the design documents and requirements specification and produces an acceptance report
waterfall model : shows the database development process as a strict sequence of steps where the output of one step is the input to the next
waterfall process :a means of identifying the tasks required for database development, together with the input and output for each activity (see waterfall model)
Exercises
- Describe the waterfall model. List the steps.
- What does the acronym SDLC mean, and what does an SDLC portray?
- What needs to be modified in the waterfall model to accommodate database design?
- Provide the iterative steps involved in database design.
Attribution
This chapter of Database Design (including all images, except as otherwise noted) is a derivative copy of The Database Development Life Cycle by the Open University licensed under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
The following material was written by Adrienne Watt:
- Key Terms
- Exercises


1.2: Chemistry in Context
-
- Last updated
- Save as PDF
Learning Objectives
By the end of this section, you will be able to:
- Outline the historical development of chemistry
- Provide examples of the importance of chemistry in everyday life
- Describe the scientific method
- Differentiate among hypotheses, theories, and laws
- Provide examples illustrating macroscopic, microscopic, and symbolic domains
Throughout human history, people have tried to convert matter into more useful forms. Our Stone Age ancestors chipped pieces of flint into useful tools and carved wood into statues and toys. These endeavors involved changing the shape of a substance without changing the substance itself. But as our knowledge increased, humans began to change the composition of the substances as well—clay was converted into pottery, hides were cured to make garments, copper ores were transformed into copper tools and weapons, and grain was made into bread.
Humans began to practice chemistry when they learned to control fire and use it to cook, make pottery, and smelt metals. Subsequently, they began to separate and use specific components of matter. A variety of drugs such as aloe, myrrh, and opium were isolated from plants. Dyes, such as indigo and Tyrian purple, were extracted from plant and animal matter. Metals were combined to form alloys—for example, copper and tin were mixed together to make bronze—and more elaborate smelting techniques produced iron. Alkalis were extracted from ashes, and soaps were prepared by combining these alkalis with fats. Alcohol was produced by fermentation and purified by distillation.
Attempts to understand the behavior of matter extend back for more than 2500 years. As early as the sixth century BC, Greek philosophers discussed a system in which water was the basis of all things. You may have heard of the Greek postulate that matter consists of four elements: earth, air, fire, and water. Subsequently, an amalgamation of chemical technologies and philosophical speculations was spread from Egypt, China, and the eastern Mediterranean by alchemists, who endeavored to transform “base metals” such as lead into “noble metals” like gold, and to create elixirs to cure disease and extend life (Figure \(\PageIndex{1}\)).
From alchemy came the historical progressions that led to modern chemistry: the isolation of drugs from natural sources, such as plants and animals. But while many of the substances extracted or processed from those natural sources were critical in the treatment of diseases, many were scarce. For example, progesterone, which is critical to women's health, became available as a medicine in 1935, but its animal sources produced extremely small quantities, limiting its availability and increasing its expense. Likewise, in the 1940s, cortisone came into use to treat arthritis and other disorders and injuries, but it took a 36-step process to synthesize. Chemist Percy Lavon Julian turned to a more plentiful source: soybeans. Previously, Julian had developed a lab to isolate soy protein, which was used in firefighting among other applications. He focused on using the soy sterols—substances mostly used in plant membranes—and was able to quickly produce progesterone and later testosterone and other hormones. He later developed a process to do the same for cortisone, and laid the groundwork for modern drug design. Since soybeans and similar plant sources were extremely plentiful, the drugs soon became widely available, saving many lives.
Chemistry: The Central Science
Chemistry is sometimes referred to as “the central science” due to its interconnectedness with a vast array of other STEM disciplines (STEM stands for areas of study in the science, technology, engineering, and math fields). Chemistry and the language of chemists play vital roles in biology, medicine, materials science, forensics, environmental science, and many other fields (Figure \(\PageIndex{2}\)). The basic principles of physics are essential for understanding many aspects of chemistry, and there is extensive overlap between many subdisciplines within the two fields, such as chemical physics and nuclear chemistry. Mathematics, computer science, and information theory provide important tools that help us calculate, interpret, describe, and generally make sense of the chemical world. Biology and chemistry converge in biochemistry, which is crucial to understanding the many complex factors and processes that keep living organisms (such as us) alive. Chemical engineering, materials science, and nanotechnology combine chemical principles and empirical findings to produce useful substances, ranging from gasoline to fabrics to electronics. Agriculture, food science, veterinary science, and brewing and wine making help provide sustenance in the form of food and drink to the world’s population. Medicine, pharmacology, biotechnology, and botany identify and produce substances that help keep us healthy. Environmental science, geology, oceanography, and atmospheric science incorporate many chemical ideas to help us better understand and protect our physical world. Chemical ideas are used to help understand the universe in astronomy and cosmology.
What are some changes in matter that are essential to daily life? Digesting and assimilating food, synthesizing polymers that are used to make clothing, containers, cookware, and credit cards, and refining crude oil into gasoline and other products are just a few examples. As you proceed through this course, you will discover many different examples of changes in the composition and structure of matter, how to classify these changes and how they occurred, their causes, the changes in energy that accompany them, and the principles and laws involved. As you learn about these things, you will be learning chemistry , the study of the composition, properties, and interactions of matter. The practice of chemistry is not limited to chemistry books or laboratories: It happens whenever someone is involved in changes in matter or in conditions that may lead to such changes.
The Scientific Method
Chemistry is a science based on observation and experimentation. Doing chemistry involves attempting to answer questions and explain observations in terms of the laws and theories of chemistry, using procedures that are accepted by the scientific community. There is no single route to answering a question or explaining an observation, but there is an aspect common to every approach: Each uses knowledge based on experiments that can be reproduced to verify the results. Some routes involve a hypothesis , a tentative explanation of observations that acts as a guide for gathering and checking information. A hypothesis is tested by experimentation, calculation, and/or comparison with the experiments of others and then refined as needed.
Some hypotheses are attempts to explain the behavior that is summarized in laws. The laws of science summarize a vast number of experimental observations, and describe or predict some facet of the natural world. If such a hypothesis turns out to be capable of explaining a large body of experimental data, it can reach the status of a theory. Scientific theories are well-substantiated, comprehensive, testable explanations of particular aspects of nature. Theories are accepted because they provide satisfactory explanations, but they can be modified if new data become available. The path of discovery that leads from question and observation to law or hypothesis to theory, combined with experimental verification of the hypothesis and any necessary modification of the theory, is called the scientific method (Figure \(\PageIndex{3}\)).
The Domains of Chemistry
Chemists study and describe the behavior of matter and energy in three different domains: macroscopic, microscopic, and symbolic. These domains provide different ways of considering and describing chemical behavior.
Macro is a Greek word that means “large.” The macroscopic domain is familiar to us: It is the realm of everyday things that are large enough to be sensed directly by human sight or touch.  The macroscopic domain includes everyday and laboratory chemistry, where we observe and measure physical and chemical properties such as density, solubility, and flammability.
Micro comes from Greek and means “small.” The microscopic domain of chemistry is often visited in the imagination. Some aspects of the microscopic domain are visible through standard optical microscopes, for example, many biological cells. More sophisticated instruments are capable of imaging even smaller entities such as molecules and atoms (see Figure \(\PageIndex{4}\): ( b )).
However, most of the subjects in the microscopic domain of chemistry are too small to be seen even with the most advanced microscopes and may only be pictured in the mind. Other components of the microscopic domain include ions and electrons, protons and neutrons, and chemical bonds, each of which is far too small to see.
The symbolic domain contains the specialized language used to represent components of the macroscopic and microscopic domains. Chemical symbols (such as those used in the periodic table), chemical formulas, and chemical equations are part of the symbolic domain, as are graphs, drawings, and calculations. These symbols play an important role in chemistry because they help interpret the behavior of the macroscopic domain in terms of the components of the microscopic domain. One of the challenges for students learning chemistry is recognizing that the same symbols can represent different things in the macroscopic and microscopic domains, and one of the features that makes chemistry fascinating is the use of a domain that must be imagined to explain behavior in a domain that can be observed.
A helpful way to understand the three domains is via the essential and ubiquitous substance of water. That water is a liquid at moderate temperatures, will freeze to form a solid at lower temperatures, and boil to form a gas at higher temperatures (Figure \(\PageIndex{4}\)) are macroscopic observations. But some properties of water fall into the microscopic domain—what cannot be observed with the naked eye. The description of water as comprising two hydrogen atoms and one oxygen atom, and the explanation of freezing and boiling in terms of attractions between these molecules, is within the microscopic arena. The formula H 2 O, which can describe water at either the macroscopic or microscopic levels, is an example of the symbolic domain. The abbreviations ( g ) for gas, ( s ) for solid, and ( l ) for liquid are also symbolic.


6.2: Measures of Variability
-
- Last updated
- Save as PDF
The statistics that we’ve discussed so far all relate to
central tendency
. That is, they all talk about which values are “in the middle” or “popular” in the data. However, central tendency is not the only type of summary statistic that we want to calculate. The second thing that we really want is a measure of the
variability
of the data. That is, how “spread out” are the data? How “far” away from the mean or median do the observed values tend to be? For now, let’s assume that the data are interval or ratio scale, so we’ll continue to use the
afl.margins
data. We’ll use this data to discuss several different measures of spread, each with different strengths and weaknesses.
Range
The
range
of a variable is very simple: it’s the biggest value minus the smallest value. For the AFL winning margins data, the maximum value is 116, and the minimum value is 0. We can calculate these values in R using the
max()
and
min()
functions:
max( afl.margins )
## [1] 116
min( afl.margins )
## [1] 0
where I’ve omitted the output because it’s not interesting. The other possibility is to use the
range()
function; which outputs both the minimum value and the maximum value in a vector, like this:
range( afl.margins )
## [1] 0 116
Although the range is the simplest way to quantify the notion of “variability”, it’s one of the worst. Recall from our discussion of the mean that we want our summary measure to be robust. If the data set has one or two extremely bad values in it, we’d like our statistics not to be unduly influenced by these cases. If we look once again at our toy example of a data set containing very extreme outliers…
−100,2,3,4,5,6,7,8,9,10
… it is clear that the range is not robust, since this has a range of 110, but if the outlier were removed we would have a range of only 8.
Interquartile range
The
interquartile range
(IQR) is like the range, but instead of calculating the difference between the biggest and smallest value, it calculates the difference between the 25th quantile and the 75th quantile. Probably you already know what a
quantile
is (they’re more commonly called percentiles), but if not: the 10th percentile of a data set is the smallest number x such that 10% of the data is less than x. In fact, we’ve already come across the idea: the median of a data set is its 50th quantile / percentile! R actually provides you with a way of calculating quantiles, using the (surprise, surprise)
quantile()
function. Let’s use it to calculate the median AFL winning margin:
quantile( x = afl.margins, probs = .5)
## 50%
## 30.5
And not surprisingly, this agrees with the answer that we saw earlier with the
median()
function. Now, we can actually input lots of quantiles at once, by specifying a vector for the
probs
argument. So lets do that, and get the 25th and 75th percentile:
quantile( x = afl.margins, probs = c(.25,.75) )
## 25% 75%
## 12.75 50.50
And, by noting that 50.5−12.75=37.75, we can see that the interquartile range for the 2010 AFL winning margins data is 37.75. Of course, that seems like too much work to do all that typing, so R has a built in function called
IQR()
that we can use:
IQR( x = afl.margins )
## [1] 37.75
While it’s obvious how to interpret the range, it’s a little less obvious how to interpret the IQR. The simplest way to think about it is like this: the interquartile range is the range spanned by the “middle half” of the data. That is, one quarter of the data falls below the 25th percentile, one quarter of the data is above the 75th percentile, leaving the “middle half” of the data lying in between the two. And the IQR is the range covered by that middle half.
Mean absolute deviation
The two measures we’ve looked at so far, the range and the interquartile range, both rely on the idea that we can measure the spread of the data by looking at the quantiles of the data. However, this isn’t the only way to think about the problem. A different approach is to select a meaningful reference point (usually the mean or the median) and then report the “typical” deviations from that reference point. What do we mean by “typical” deviation? Usually, the mean or median value of these deviations! In practice, this leads to two different measures, the “mean absolute deviation (from the mean)” and the “median absolute deviation (from the median)”. From what I’ve read, the measure based on the median seems to be used in statistics, and does seem to be the better of the two, but to be honest I don’t think I’ve seen it used much in psychology. The measure based on the mean does occasionally show up in psychology though. In this section I’ll talk about the first one, and I’ll come back to talk about the second one later.
Since the previous paragraph might sound a little abstract, let’s go through the
mean absolute deviation
from the mean a little more slowly. One useful thing about this measure is that the name actually tells you exactly how to calculate it. Let’s think about our AFL winning margins data, and once again we’ll start by pretending that there’s only 5 games in total, with winning margins of 56, 31, 56, 8 and 32. Since our calculations rely on an examination of the deviation from some reference point (in this case the mean), the first thing we need to calculate is the mean, \(\bar{X}\). For these five observations, our mean is \(\bar{X}\)=36.6. The next step is to convert each of our observations X
i
into a deviation score. We do this by calculating the difference between the observation Xi and the mean \(\bar{X}\). That is, the deviation score is defined to be X
i
−\(\bar{X}\). For the first observation in our sample, this is equal to 56−36.6=19.4. Okay, that’s simple enough. The next step in the process is to convert these deviations to absolute deviations. As we discussed earlier when talking about the
abs()
function in R (Section 3.5), we do this by converting any negative values to positive ones. Mathematically, we would denote the absolute value of −3 as |−3|, and so we say that |−3|=3. We use the absolute value function here because we don’t really care whether the value is higher than the mean or lower than the mean, we’re just interested in how
close
it is to the mean. To help make this process as obvious as possible, the table below shows these calculations for all five observations:
| the observation | its symbol | the observed value |
|---|---|---|
| winning margin, game 2 | X2 | 31 points |
| winning margin, game 5 | X5 | 32 points |
| winning margin, game 1 | X1 | 56 points |
| winning margin, game 3 | X3 | 56 points |
| winning margin, game 4 | X4 | 8 points |
Now that we have calculated the absolute deviation score for every observation in the data set, all that we have to do to calculate the mean of these scores. Let’s do that:
\[
\dfrac{19.4+5.6+19.4+28.6+4.6}{5}=15.52
\nonumber\]
And we’re done. The mean absolute deviation for these five scores is 15.52.
However, while our calculations for this little example are at an end, we do have a couple of things left to talk about. Firstly, we should really try to write down a proper mathematical formula. But in order do to this I need some mathematical notation to refer to the mean absolute deviation. Irritatingly, “mean absolute deviation” and “median absolute deviation” have the same acronym (MAD), which leads to a certain amount of ambiguity, and since R tends to use MAD to refer to the median absolute deviation, I’d better come up with something different for the mean absolute deviation. Sigh. What I’ll do is use AAD instead, short for average absolute deviation. Now that we have some unambiguous notation, here’s the formula that describes what we just calculated:
\[
(X)=\dfrac{1}{N} \sum_{i=1}^{N}\left|X_{i}-\bar{X}\right|
\nonumber\]
The last thing we need to talk about is how to calculate AAD in R. One possibility would be to do everything using low level commands, laboriously following the same steps that I used when describing the calculations above. However, that’s pretty tedious. You’d end up with a series of commands that might look like this:
X <- c(56, 31,56,8,32) # enter the data
X.bar <- mean( X ) # step 1. the mean of the data
AD <- abs( X - X.bar ) # step 2. the absolute deviations from the mean
AAD <- mean( AD ) # step 3. the mean absolute deviations
print( AAD ) # print the results
## [1] 15.52
Each of those commands is pretty simple, but there’s just too many of them. And because I find that to be too much typing, the
lsr
package has a very simple function called
aad()
that does the calculations for you. If we apply the
aad()
function to our data, we get this:
library(lsr)
aad( X )
## [1] 15.52
No suprises there.
Variance
Although the mean absolute deviation measure has its uses, it’s not the best measure of variability to use. From a purely mathematical perspective, there are some solid reasons to prefer squared deviations rather than absolute deviations. If we do that, we obtain a measure is called the variance , which has a lot of really nice statistical properties that I’m going to ignore, 71 (X)$ and Var(Y) respectively. Now imagine I want to define a new variable Z that is the sum of the two, Z=X+Y. As it turns out, the variance of Z is equal to Var(X)+Var(Y). This is a very useful property, but it’s not true of the other measures that I talk about in this section.] and one massive psychological flaw that I’m going to make a big deal out of in a moment. The variance of a data set X is sometimes written as Var(X), but it’s more commonly denoted s 2 (the reason for this will become clearer shortly). The formula that we use to calculate the variance of a set of observations is as follows:
\[
\begin{aligned}
&\operatorname{Var}(X)=\dfrac{1}{N} \sum_{i=1}^{N}\left(X_{i}-\bar{X}\right)^{2}\\
&\operatorname{Var}(X)=\dfrac{\sum_{i=1}^{N}\left(X_{i}-\bar{X}\right)^{2}}{N}
\end{aligned}
\nonumber\]
As you can see, it’s basically the same formula that we used to calculate the mean absolute deviation, except that instead of using “absolute deviations” we use “squared deviations”. It is for this reason that the variance is sometimes referred to as the “mean square deviation”.
Now that we’ve got the basic idea, let’s have a look at a concrete example. Once again, let’s use the first five AFL games as our data. If we follow the same approach that we took last time, we end up with the following table:
Table 5.1: Basic arithmetic operations in R. These five operators are used very frequently throughout the text, so it’s important to be familiar with them at the outset.
| Notation [English] | i [which game] | X i [value] | X i −\(\bar{X}\) [deviation from mean] | (Xi−\(\bar{X}\)) 2 [absolute deviation] |
|---|---|---|---|---|
| 5 | 32 | -4.6 | 21.16 | |
| 2 | 31 | -5.6 | 31.36 | |
| 1 | 56 | 19.4 | 376.36 | |
| 3 | 56 | 19.4 | 376.36 | |
| 4 | 8 | -28.6 | 817.96 |
That last column contains all of our squared deviations, so all we have to do is average them. If we do that by typing all the numbers into R by hand…
( 376.36 + 31.36 + 376.36 + 817.96 + 21.16 ) / 5
## [1] 324.64
… we end up with a variance of 324.64. Exciting, isn’t it? For the moment, let’s ignore the burning question that you’re all probably thinking (i.e., what the heck does a variance of 324.64 actually mean?) and instead talk a bit more about how to do the calculations in R, because this will reveal something very weird.
As always, we want to avoid having to type in a whole lot of numbers ourselves. And as it happens, we have the vector
X
lying around, which we created in the previous section. With this in mind, we can calculate the variance of
X
by using the following command,
mean( (X - mean(X) )^2)
## [1] 324.64
and as usual we get the same answer as the one that we got when we did everything by hand. However, I
still
think that this is too much typing. Fortunately, R has a built in function called
var()
which does calculate variances. So we could also do this…
var(X)
## [1] 405.8
and you get the same… no, wait… you get a completely different answer. That’s just weird. Is R broken? Is this a typo? Is Dan an idiot?
As it happens, the answer is no.
72
It’s not a typo, and R is not making a mistake. To get a feel for what’s happening, let’s stop using the tiny data set containing only 5 data points, and switch to the full set of 176 games that we’ve got stored in our
afl.margins
vector. First, let’s calculate the variance by using the formula that I described above:
mean( (afl.margins - mean(afl.margins) )^2)
## [1] 675.9718
Now let’s use the
var()
function:
var( afl.margins )
## [1] 679.8345
Hm. These two numbers are very similar this time. That seems like too much of a coincidence to be a mistake. And of course it isn’t a mistake. In fact, it’s very simple to explain what R is doing here, but slightly trickier to explain why R is doing it. So let’s start with the “what”. What R is doing is evaluating a slightly different formula to the one I showed you above. Instead of averaging the squared deviations, which requires you to divide by the number of data points N, R has chosen to divide by N−1. In other words, the formula that R is using is this one
\[
\dfrac{1}{N-1} \sum_{i=1}^{N}\left(X_{i}-\bar{X}\right)^{2}
\nonumber\]
It’s easy enough to verify that this is what’s happening, as the following command illustrates:
sum( (X-mean(X))^2 ) / 4
## [1] 405.8
This is the same answer that R gave us originally when we calculated
var(X)
originally. So that’s the
what
. The real question is
why
R is dividing by N−1 and not by N. After all, the variance is supposed to be the
mean
squared deviation, right? So shouldn’t we be dividing by N, the actual number of observations in the sample? Well, yes, we should. However, as we’ll discuss in Chapter 10, there’s a subtle distinction between “describing a sample” and “making guesses about the population from which the sample came”. Up to this point, it’s been a distinction without a difference. Regardless of whether you’re describing a sample or drawing inferences about the population, the mean is calculated exactly the same way. Not so for the variance, or the standard deviation, or for many other measures besides. What I outlined to you initially (i.e., take the actual average, and thus divide by N) assumes that you literally intend to calculate the variance of the sample. Most of the time, however, you’re not terribly interested in the sample
in and of itself
. Rather, the sample exists to tell you something about the world. If so, you’re actually starting to move away from calculating a “sample statistic”, and towards the idea of estimating a “population parameter”. However, I’m getting ahead of myself. For now, let’s just take it on faith that R knows what it’s doing, and we’ll revisit the question later on when we talk about estimation in Chapter 10.
Okay, one last thing. This section so far has read a bit like a mystery novel. I’ve shown you how to calculate the variance, described the weird “N−1” thing that R does and hinted at the reason why it’s there, but I haven’t mentioned the single most important thing… how do you interpret the variance? Descriptive statistics are supposed to describe things, after all, and right now the variance is really just a gibberish number. Unfortunately, the reason why I haven’t given you the human-friendly interpretation of the variance is that there really isn’t one. This is the most serious problem with the variance. Although it has some elegant mathematical properties that suggest that it really is a fundamental quantity for expressing variation, it’s completely useless if you want to communicate with an actual human… variances are completely uninterpretable in terms of the original variable! All the numbers have been squared, and they don’t mean anything anymore. This is a huge issue. For instance, according to the table I presented earlier, the margin in game 1 was “376.36 points-squared higher than the average margin”.  I’ve watched a lot of footy games, and never has anyone referred to “points squared”. It’s not a real unit of measurement, and since the variance is expressed in terms of this gibberish unit, it is totally meaningless to a human.
Standard deviation
Okay, suppose that you like the idea of using the variance because of those nice mathematical properties that I haven’t talked about, but – since you’re a human and not a robot – you’d like to have a measure that is expressed in the same units as the data itself (i.e., points, not points-squared). What should you do? The solution to the problem is obvious: take the square root of the variance, known as the standard deviation , also called the “root mean squared deviation”, or RMSD. This solves out problem fairly neatly: while nobody has a clue what “a variance of 324.68 points-squared” really means, it’s much easier to understand “a standard deviation of 18.01 points”, since it’s expressed in the original units. It is traditional to refer to the standard deviation of a sample of data as s, though “sd” and “std dev.” are also used at times. Because the standard deviation is equal to the square root of the variance, you probably won’t be surprised to see that the formula is:
\[
s=\sqrt{\dfrac{1}{N} \sum_{i=1}^{N}\left(X_{i}-\bar{X}\right)^{2}}
\nonumber\]
and the R function that we use to calculate it is
sd()
. However, as you might have guessed from our discussion of the variance, what R actually calculates is slightly different to the formula given above. Just like the we saw with the variance, what R calculates is a version that divides by N−1 rather than N. For reasons that will make sense when we return to this topic in Chapter@refch:estimation I’ll refer to this new quantity as \(\hat{\sigma}\) (read as: “sigma hat”), and the formula for this is
\[
\hat{\sigma}=\sqrt{\dfrac{1}{N-1} \sum_{i=1}^{N}\left(X_{i}-\bar{X}\right)^{2}}
\nonumber\]
With that in mind, calculating standard deviations in R is simple:
sd( afl.margins )
## [1] 26.07364
Interpreting standard deviations is slightly more complex. Because the standard deviation is derived from the variance, and the variance is a quantity that has little to no meaning that makes sense to us humans, the standard deviation doesn’t have a simple interpretation. As a consequence, most of us just rely on a simple rule of thumb: in general, you should expect 68% of the data to fall within 1 standard deviation of the mean, 95% of the data to fall within 2 standard deviation of the mean, and 99.7% of the data to fall within 3 standard deviations of the mean. This rule tends to work pretty well most of the time, but it’s not exact: it’s actually calculated based on an assumption that the histogram is symmetric and “bell shaped.” 73 As you can tell from looking at the AFL winning margins histogram in Figure 5.1, this isn’t exactly true of our data! Even so, the rule is approximately correct. As it turns out, 65.3% of the AFL margins data fall within one standard deviation of the mean. This is shown visually in Figure 5.3.
Median absolute deviation
The last measure of variability that I want to talk about is the median absolute deviation (MAD). The basic idea behind MAD is very simple, and is pretty much identical to the idea behind the mean absolute deviation (Section 5.2.3). The difference is that you use the median everywhere. If we were to frame this idea as a pair of R commands, they would look like this:
# mean absolute deviation from the mean:
mean( abs(afl.margins - mean(afl.margins)) )
## [1] 21.10124
# *median* absolute deviation from the *median*:
median( abs(afl.margins - median(afl.margins)) )
## [1] 19.5
This has a straightforward interpretation: every observation in the data set lies some distance away from the typical value (the median). So the MAD is an attempt to describe a typical deviation from a typical value in the data set. It wouldn’t be unreasonable to interpret the MAD value of 19.5 for our AFL data by saying something like this:
The median winning margin in 2010 was 30.5, indicating that a typical game involved a winning margin of about 30 points. However, there was a fair amount of variation from game to game: the MAD value was 19.5, indicating that a typical winning margin would differ from this median value by about 19-20 points.
As you’d expect, R has a built in function for calculating MAD, and you will be shocked no doubt to hear that it’s called
mad()
. However, it’s a little bit more complicated than the functions that we’ve been using previously. If you want to use it to calculate MAD in the exact same way that I have described it above, the command that you need to use specifies two arguments: the data set itself
x
, and a
constant
that I’ll explain in a moment. For our purposes, the constant is 1, so our command becomes
mad( x = afl.margins, constant = 1 )
## [1] 19.5
Apart from the weirdness of having to type that
constant = 1
part, this is pretty straightforward.
Okay, so what exactly is this
constant = 1
argument? I won’t go into all the details here, but here’s the gist. Although the “raw” MAD value that I’ve described above is completely interpretable on its own terms, that’s not actually how it’s used in a lot of real world contexts. Instead, what happens a lot is that the researcher
actually
wants to calculate the standard deviation. However, in the same way that the mean is very sensitive to extreme values, the standard deviation is vulnerable to the exact same issue. So, in much the same way that people sometimes use the median as a “robust” way of calculating “something that is like the mean”, it’s not uncommon to use MAD as a method for calculating “something that is like the standard deviation”. Unfortunately, the
raw
MAD value doesn’t do this. Our raw MAD value is 19.5, and our standard deviation was 26.07. However, what some clever person has shown is that, under certain assumptions
74
, you can multiply the raw MAD value by 1.4826 and obtain a number that is directly comparable to the standard deviation. As a consequence, the default value of
constant
is 1.4826, and so when you use the
mad()
command without manually setting a value, here’s what you get:
mad( afl.margins )
## [1] 28.9107
I should point out, though, that if you want to use this “corrected” MAD value as a robust version of the standard deviation, you really are relying on the assumption that the data are (or at least, are “supposed to be” in some sense) symmetric and basically shaped like a bell curve. That’s really
not
true for our
afl.margins
data, so in this case I wouldn’t try to use the MAD value this way.
Which measure to use?
We’ve discussed quite a few measures of spread (range, IQR, MAD, variance and standard deviation), and hinted at their strengths and weaknesses. Here’s a quick summary:
- Range . Gives you the full spread of the data. It’s very vulnerable to outliers, and as a consequence it isn’t often used unless you have good reasons to care about the extremes in the data.
- Interquartile range . Tells you where the “middle half” of the data sits. It’s pretty robust, and complements the median nicely. This is used a lot.
- Mean absolute deviation . Tells you how far “on average” the observations are from the mean. It’s very interpretable, but has a few minor issues (not discussed here) that make it less attractive to statisticians than the standard deviation. Used sometimes, but not often.
- Variance . Tells you the average squared deviation from the mean. It’s mathematically elegant, and is probably the “right” way to describe variation around the mean, but it’s completely uninterpretable because it doesn’t use the same units as the data. Almost never used except as a mathematical tool; but it’s buried “under the hood” of a very large number of statistical tools.
- Standard deviation . This is the square root of the variance. It’s fairly elegant mathematically, and it’s expressed in the same units as the data so it can be interpreted pretty well. In situations where the mean is the measure of central tendency, this is the default. This is by far the most popular measure of variation.
- Median absolute deviation . The typical (i.e., median) deviation from the median value. In the raw form it’s simple and interpretable; in the corrected form it’s a robust way to estimate the standard deviation, for some kinds of data sets. Not used very often, but it does get reported sometimes.
In short, the IQR and the standard deviation are easily the two most common measures used to report the variability of the data; but there are situations in which the others are used. I’ve described all of them in this book because there’s a fair chance you’ll run into most of these somewhere.


3.3.5: The Mayan Numeral System
-
- Last updated
- Save as PDF
Background
As you might imagine, the development of a base system is an important step in making the counting process more efficient. Our own base-ten system probably arose from the fact that we have 10 fingers (including thumbs) on two hands. This is a natural development. However, other civilizations have had a variety of bases other than ten. For example, the Natives of Queensland used a base-two system, counting as follows: “one, two, two and one, two two’s, much.” Some Modern South American Tribes have a base-five system counting in this way: “one, two, three, four, hand, hand and one, hand and two,” and so on. The Babylonians used a base-sixty (sexigesimal) system. In this chapter, we wrap up with a specific example of a civilization that actually used a base system other than 10.
The Mayan civilization is generally dated from 1500 B.C.E to 1700 C.E. The Yucatan Peninsula (see map[i]) in Mexico was the scene for the development of one of the most advanced civilizations of the ancient world. The Mayans had a sophisticated ritual system that was overseen by a priestly class. This class of priests developed a philosophy with time as divine and eternal.[ii] The calendar, and calculations related to it, were thus very important to the ritual life of the priestly class, and hence the Mayan people. In fact, much of what we know about this culture comes from their calendar records and astronomy data. Another important source of information on the Mayans is the writings of Father Diego de Landa, who went to Mexico as a missionary in 1549.
There were two numeral systems developed by the Mayans - one for the common people and one for the priests. Not only did these two systems use different symbols, they also used different base systems. For the priests, the number system was governed by ritual. The days of the year were thought to be gods, so the formal symbols for the days were decorated heads,[iii] like the sample to the left[iv] Since the basic calendar was based on 360 days, the priestly numeral system used a mixed base system employing multiples of 20 and 360. This makes for a confusing system, the details of which we will skip.
The Mayan Number System
Instead, we will focus on the numeration system of the “common” people, which used a more consistent base system. As we stated earlier, the Mayans used a base-20 system, called the “vigesimal” system. Like our system, it is positional, meaning that the position of a numeric symbol indicates its place value. In the following table you can see the place value in its vertical format.[v]
\(\begin{array}{|c|c|l|}
\hline \text { Powers } & \text { Base-Ten Value } & \text { Place Name } \\
\hline 20^{7} & 12,800,000,000 & \text { Hablat } \\
\hline 20^{6} & 64,000,000 & \text { Alau } \\
\hline 20^{5} & 3,200,000 & \text { Kinchil } \\
\hline 20^{4} & 160,000 & \text { Cabal } \\
\hline 20^{3} & 8,000 & \text { Pic } \\
\hline 20^{2} & 400 & \text {Bak} \\
\hline 20^{1} & 20 & \text { Kal } \\
\hline 20^{0} & 1 & \text { Hun } \\
\hline
\end{array}\)
In order to write numbers down, there were only three symbols needed in this system. A horizontal bar represented the quantity 5, a dot represented the quantity 1, and a special symbol (thought to be a shell) represented zero. The Mayan system may have been the first to make use of zero as a placeholder/number. The first 20 numbers are shown in the table to the right.[vi]
Unlike our system, where the ones place starts on the right and then moves to the left, the Mayan systems places the ones on the bottom of a vertical orientation and moves up as the place value increases.
When numbers are written in vertical form, there should never be more than four dots in a single place. When writing Mayan numbers, every group of five dots becomes one bar. Also, there should never be more than three bars in a single place…four bars would be converted to one dot in the next place up. It’s the same as 10 getting converted to a 1 in the next place up when we carry during addition.
Example 12
What is the value of this number, which is shown in vertical form?
Solution
Starting from the bottom, we have the ones place. There are two bars and three dots in this place. Since each bar is worth 5, we have 13 ones when we count the three dots in the ones place. Looking to the place value above it (the twenties places), we see there are three dots so we have three twenties.
Hence we can write this number in base-ten as:
\[\begin{align*}
\left(3 \times 20^{1}\right)+\left(13 \times 20^{0}\right) &=(3 \times 20)+(13 \times 1) \\
&=60+13 \\
&=73
\end{align*} \nonumber \]
Example 13
What is the value of the following Mayan number?
Solution
This number has 11 in the ones place, zero in the 20’s place, and 18 in the 20 2 =400’s place. Hence, the value of this number in base-ten is:
\[18 \times 400+0 \times 20+11 \times 1=7211 \nonumber \]
Example 14
Convert the base 10 number \(3575_{10}\) to Mayan numerals.
Solution
This problem is done in two stages. First we need to convert to a base 20 number. We will do so using the method provided in the last section of the text. The second step is to convert that number to Mayan symbols.
The highest power of 20 that will divide into \(3575\) is \(20^{2}=400,\) so we start by dividing that and then proceed from there:
\[ \begin{align*} 3575 \div 400 &=8.9375 \\[4pt] 0.9375 \times 20 &=18.75 \\[4pt] (0.75 \times 20 &=15.0 \end{align*} \nonumber \]
This means that \(3575_{10}=8,18,15_{20}\)
The second step is to convert this to Mayan notation. This number indicates that we have 15 in the ones position. That’s three bars at the bottom of the number. We also have 18 in the 20’s place, so that’s three bars and three dots in the second position. Finally, we have 8 in the 400’s place, so that’s one bar and three dots on the top. We get the following
Note that in the previous example a new notation was used when we wrote \(8,18,15_{20}\). The commas between the three numbers 8, 18, and 15 are now separating place values for us so that we can keep them separate from each other. This use of the comma is slightly different than how they’re used in the decimal system. When we write a number in base 10, such as 7,567,323, the commas are used primarily as an aide to read the number easily but they do not separate single place values from each other. We will need this notation whenever the base we use is larger than 10.
Writing numbers with bases bigger than 10
When the base of a number is larger than 10, separate each “digit” with a comma to make the separation of digits clear.
For example, in base \(20,\) to write the number corresponding to \(17 \times 20^{2}+6 \times 20^{1}+\) \(13 \times 20^{0},\) we'd write \(17,6,13_{20}\).
Try it Now 11
Convert the base 10 number \(10553_{10}\) to Mayan numerals.
- Answer
-
\(10553_{10}=1,6,7,13_{20}\)
Try it Now 12
Convert the base 10 number 5617 to Mayan numerals.
- Answer
-
\(5617_{10}=14,0,17_{20}\). Note that there is a zero in the 20’s place, so you’ll need to use the appropriate zero symbol in between the ones and 400’s places.
Adding Mayan Numbers
When adding Mayan numbers together, we’ll adopt a scheme that the Mayans probably did not use but which will make life a little easier for us.
Example 15
Add, in Mayan, the numbers 37 and 29: [vii]
Solution
First draw a box around each of the vertical places. This will help keep the place values from being mixed up.
Next, put all of the symbols from both numbers into a single set of places (boxes), and to the right of this new number draw a set of empty boxes where you will place the final sum:
You are now ready to start carrying. Begin with the place that has the lowest value, just as you do with Arabic numbers. Start at the bottom place, where each dot is worth 1. There are six dots, but a maximum of four are allowed in any one place; once you get to five dots, you must convert to a bar. Since five dots make one bar, we draw a bar through five of the dots, leaving us with one dot which is under the four-dot limit. Put this dot into the bottom place of the empty set of boxes you just drew:
Now look at the bars in the bottom place. There are five, and the maximum number the place can hold is three. Four bars are equal to one dot in the next highest place .
Whenever we have four bars in a single place we will automatically convert that to a dot in the next place up. We draw a circle around four of the bars and an arrow up to the dots' section of the higher place. At the end of that arrow, draw a new dot. That dot represents 20 just the same as the other dots in that place. Not counting the circled bars in the bottom place, there is one bar left. One bar is under the three-bar limit; put it under the dot in the set of empty places to the right.
Now there are only three dots in the next highest place, so draw them in the corresponding empty box.
We can see here that we have 3 twenties (60), and 6 ones, for a total of 66. We check and note that \(37 + 29 = 66\), so we have done this addition correctly. Is it easier to just do it in base-ten? Probably, but that’s only because it’s more familiar to you. Your task here is to try to learn a new base system and how addition can be done in slightly different ways than what you have seen in the past. Note, however, that the concept of carrying is still used, just as it is in our own addition algorithm.
Try it Now 13
Try adding 174 and 78 in Mayan by first converting to Mayan numbers and then working entirely within that system. Do not add in base-ten (decimal) until the very end when you check your work.
- Answer
-
A sample solution is shown.
Conclusion
In this chapter, we have briefly sketched the development of numbers and our counting system, with the emphasis on the “brief” part. There are numerous sources of information and research that fill many volumes of books on this topic. Unfortunately, we cannot begin to come close to covering all of the information that is out there.
We have only scratched the surface of the wealth of research and information that exists on the development of numbers and counting throughout human history. What is important to note is that the system that we use every day is a product of thousands of years of progress and development. It represents contributions by many civilizations and cultures. It does not come down to us from the sky, a gift from the gods. It is not the creation of a textbook publisher. It is indeed as human as we are, as is the rest of mathematics. Behind every symbol, formula and rule there is a human face to be found, or at least sought.
Furthermore, we hope that you now have a basic appreciation for just how interesting and diverse number systems can get. Also, we’re pretty sure that you have also begun to recognize that we take our own number system for granted so much that when we try to adapt to other systems or bases, we find ourselves truly having to concentrate and think about what is going on.
[i] www.gorp.com/gorp/location/latamer/map_maya.htm
[ii] Bidwell, James; Mayan Arithmetic in Mathematics Teacher , Issue 74 (Nov., 1967), p. 762-68.
[iii] www.ukans.edu/~lctls/Mayan/numbers.html
[iv] www.ukans.edu/~lctls/Mayan/numbers.html
[v] Bidwell
[vi] www.vpds.wsu.edu/fair_95/gym/UM001.html
[vii] forum.swarthmore.edu/k12/mayan.math/mayan2.html