Chapter 9: Role of Data and Information
The canteen in a school in Jaipur sells idli, poha and lemon rice. Every afternoon, the cook writes on a slip of paper how many plates of each item were sold that day, and drops the slip into a box. By the end of the month the box is full of slips. Look at any one slip on its own and it tells you almost nothing — a few item names and a few counts, with no order and no meaning.
Then a teacher empties the box and arranges all the slips into a single table, one row for each day of the week. Suddenly a pattern shows up. Some days are busy, some are quiet, and one item sells far better than the others. The cook now knows what to prepare more of, and what to stop making.
Nothing new was added. The same numbers were simply organised until they began to mean something. That small jump — from a pile of numbers to something you can act on — is the idea this whole chapter is built around. See Fig. 9.0-1.
Diagram
Fig. 9.0-1 – on the left, a heap of loose paper slips with scattered item names and counts, labelled “raw data”; a curved arrow in the middle labelled “organise, add context”; on the right, the same values shown as a neat day-wise table with a small bar chart beside it, labelled “information”
This chapter will answer each of these five questions as you read through it.
9.1 What Is Data?
Think about an ordinary school morning. The class teacher marks who is present and who is absent. The sports teacher notes down how long each student took to run one round of the ground. The lab assistant writes the room temperature in a register before the practical begins. None of these people are trying to prove anything yet. They are simply recording what happened.
Everything they wrote down is data — raw facts and figures that have been collected but not yet made sense of. Data is what you have before anyone has studied it, arranged it or drawn a conclusion from it.
Data comes in many forms
It is easy to assume that data means numbers. It usually does not stop there. A voice note is data. A photograph of a classroom noticeboard is data. The names in the attendance register are data. A reading from a thermometer is data. All of them are recordings of something real, kept in a form that can be stored and studied later. Fig. 9.1-1 shows a few everyday sources feeding into the same idea.
Diagram
Fig. 9.1-1 – four small labelled sources on the left — a thermometer, an attendance register page, a mobile phone camera, and a microphone — with arrows all pointing into one common box in the centre labelled “DATA (raw facts, not yet organised)”
Data also arrives in two ways. Sometimes a person records it by hand, the way a shopkeeper writes each sale in a notebook. Sometimes a machine records it on its own, the way a step counter in a watch keeps counting without being asked. The second kind can pile up very quickly, because a machine never gets tired of recording.
Raw data has no meaning yet
Here is the important part. Suppose you find a slip of paper with only 36.6 written on it. Is that a temperature in degrees Celsius? A distance in metres? The price of something? You cannot tell. The number is perfectly real, but on its own it carries no meaning, because nothing tells you what it measures, when it was recorded, or why.
That is true of raw data in general. It has no context, it may be in no particular order, and it does not answer a question by itself. Something has to be done to it first. That “something” is the subject of the next section.
One honest warning before we move on. Data is only as good as the care taken while collecting it. A reading noted down in the wrong column, a name spelt two different ways on two days, or a sensor left switched off for an hour all become part of the data and look exactly like everything else. Nobody can spot such mistakes later just by staring at the numbers, so careful collection matters far more than it seems at the time.
9.2 From Data to Information
Go back to the canteen slips from the start of this chapter. Every slip was genuine data. Yet as long as they lay in a box, nobody could answer the one question the cook actually cared about: what should be prepared more of tomorrow?
The slips only became useful once someone sorted them, put them in day order and looked at them together. What came out of that is called information — data that has been processed and organised so that it means something and can be used to decide, explain or plan.
Notice that nothing was invented in between. The counts on the slips were the same before and after. Only their arrangement changed, and that was enough to turn a pile of numbers into an answer.
9.2.1 How Data and Information Differ
Students often treat the two words as the same thing, because both look like “stuff written down”. They are not the same. Data is the input; information is what comes out after that input has been worked on. Table 9.1 sets the difference out clearly.
The last row is worth reading twice. The same counts appear in both columns. What changes is that the second one tells you which day and what was counted.
9.2.2 The Steps That Turn Data into Information
Turning data into information is not one single action. It is a sequence, and each step in it does one small job. Fig. 9.2-1 shows the whole path in order.
Diagram
Fig. 9.2-1 – a left-to-right flow of eight labelled boxes joined by arrows: Data collection → Data processing → Data storage → Data analysis → Contextualisation → Organisation → Presentation → Interpretation; the first box drawn beside a small heap of loose slips and the last box beside a person looking at a chart
Walking through the steps once
Suppose the canteen slips for one week are emptied out and the counts of idli plates read: 24, 65, 42, 38, 55, 40. In that form the numbers are useless. Nobody knows which day is which, or even whether they are in day order.
Now add the context and the order. That is steps 5 and 6 of the method above, and the result is Table 9.2.
The table shows the same six counts, but each one is now attached to a day, and the column heading says what was counted. Friday stands out as the busiest day and Saturday as the quietest — and Saturday being a half day explains why. That explanation is step 8: interpretation.
A single summary figure often makes a table easier to talk about, so let us work one out.
The average is \(\text{average} = \dfrac{\text{total of all values}}{\text{number of values}}\), where the total is the sum of the daily counts and the number of values is the number of days counted. One number now stands in for the whole week, which makes it easy to compare this week against the next.
9.3 Types of Data
A canteen register, a song saved on a phone, a photograph of the school gate and a message to a friend are all data, as Section 9.1 showed. But a computer cannot treat them in the same way. Counting plates of idli and playing back a song need very different handling.
So data is sorted into types, and the type tells you how the data will be stored and what can sensibly be done with it. The four types you need to know are audio, visual, numeric and text data.
Before we look at each one, there is a small puzzle to settle. A computer stores everything as 0s and 1s. Sound, however, does not arrive in steps — it is a smooth, continuous wave, and so is the light falling on a camera. Data that varies smoothly like this cannot be written down as 0s and 1s directly. It first has to be measured at regular moments, and each measurement written as a number. That conversion happens inside the microphone or camera circuit, before the data ever reaches a file.
9.3.1 Audio Data
Audio data is sound stored in a form a computer can hold — a recorded song, a voice note, or the beep a machine makes.
To store a sound, the recording device measures the strength of the sound wave over and over again, thousands of times every second. Each measurement is called a sample, and each sample is stored as a number. Play the numbers back in the same order, at the same speed, and the ear hears the original sound again. See Fig. 9.3-1.
Diagram
Fig. 9.3-1 – a smooth sound wave drawn across the page, with evenly spaced vertical dotted lines meeting it; a small dot marked at each meeting point and a short column of numbers written below, labelled “each sample stored as a number”
How many samples are taken every second is called the sampling rate, and it is measured in hertz (Hz), which simply means “per second”. Music on an audio CD, for example, uses the long-standing standard of 44,100 samples per second. The number of bits used to write down each single sample is called the bit depth; more bits allow a sample to be recorded more finely.
9.3.2 Visual Data
Visual data is data you look at — photographs, drawings, screenshots and video.
A digital image is built out of tiny coloured squares. Each square is a pixel, the smallest unit of an image, and it holds the colour at that one spot. Millions of pixels placed side by side make a picture your eye reads as smooth. Fig. 9.3-2 shows what happens when you zoom far into one.
Diagram
Fig. 9.3-2 – a small photograph of a school building on the left, with a square region magnified on the right to reveal a grid of individual coloured squares, one square outlined and labelled “one pixel”
The resolution of an image is how many pixels it contains, usually written as width × height. A higher resolution image holds more detail, but it also takes up more storage space.
An image file also carries metadata — extra details stored alongside the picture itself, such as its size in pixels, the date it was taken and the camera settings used. Metadata is data about the data.
Video is visual data with one addition. A video is a run of still images shown quickly one after another, and how many are shown each second is the frame rate, measured in frames per second (fps). Films are commonly shot at 24 fps and much everyday video at 30 fps. Because a video holds so many images, it needs far more storage and far more processing power than a single photograph.
9.3.3 Numeric Data
Numeric data is data made of numbers that stand for quantities — marks, temperatures, counts of plates sold, the price of a notebook, the time taken to run a round of the ground.
This is the easiest type to work with, because arithmetic applies to it directly. Numeric values can be added, averaged, sorted and compared, exactly as was done with the canteen counts in Section 9.2.2. Numeric data comes in two common shapes: whole numbers such as a count of 42 plates, and decimal numbers such as a temperature of 36.6 degrees Celsius.
9.3.4 Text Data
Text data is data made of letters, words and sentences — a student’s name in a register, a message, a page of a textbook, the answer someone types into a form.
A computer cannot store a letter as a letter. Every character is stored as a number using an agreed code, so that a file written on one machine reads correctly on another. The older code is ASCII (American Standard Code for Information Interchange), which covers English letters, digits and common punctuation marks. The wider code used today is Unicode, which also covers scripts such as Devanagari, Tamil and Bengali, so text in Hindi or Marathi can be stored just as easily as text in English.
Plain text, with no styling at all, is usually kept in files such as .txt or .csv. Text that carries formatting — headings, bold, page layout — is kept in files such as .docx or .pdf.
9.4 Why Data Is Pivotal to Artificial Intelligence
Some problems can be solved by writing down a rule. If a school says a student passes when marks are 33 or more, a programmer can write that rule once and the program will be right every time. No data is needed beyond the rule itself and the marks of the student in front of it.
Now try to write a rule for reading handwriting. Everyone’s letter “a” is a little different — some are round, some are squeezed, some lean over. There is no sentence you can write that captures every possible “a” and rejects everything else. Nobody has ever managed it.
This is where artificial intelligence (AI) is used: for problems where the rule cannot be written out, but plenty of examples of the right answer exist. Instead of being given the rule, the program is given thousands of examples and works out for itself which patterns separate one answer from another.
How examples become a model
A very young child is not taught a definition of a cow. The child is shown cows — in fields, in pictures, in books — until the shape becomes familiar. An AI system is built in a similar way, though what happens inside is arithmetic, not understanding.
The examples are fed in, along with the correct answer for each one. The program adjusts its internal numbers a little at a time, so that its output moves closer to the correct answer across all the examples. This adjusting stage is called training, and what comes out of it is called a model.
Fig. 9.4-1 shows the whole path, from examples to a model that can handle something new.
Diagram
Fig. 9.4-1 – on the left, a stack of labelled examples (small handwritten letters each tagged with the correct letter) feeding into a box marked “training”; an arrow from training to a box marked “model”; below, a single new unseen handwritten letter entering the model box and an arrow leaving it marked “predicted letter”
Nothing in this picture works without the stack on the left. That is the point of this section: for an ordinary rule-based program, data is only what you feed in at the end; for an AI system, data is what the program is built out of in the first place.
What data does inside an AI system
Data is not used at one moment and then forgotten. It has several distinct jobs.
Training the model. The examples are what the model learns its patterns from. Without them there is nothing to train on and no model at all.
Finding useful features. Not every part of the data matters. Studying the data shows which parts actually separate one answer from another — in handwriting, the shape of the stroke matters and the colour of the ink usually does not.
Recognising patterns. Once trained, the model matches a new input against the patterns found in the data and produces the answer those patterns point to.
Improving the model. Fresh data collected later shows where the model is getting things wrong, and it can be trained again on the wider set so that it handles those cases better.
Personalising. A system that suggests videos or shop items uses data about what a user has chosen before to decide what to place in front of them next.
Why so much of it is needed
A model can only recognise the kinds of examples it was trained on. Train it on neat handwriting alone and it will do badly on hurried handwriting, because it never saw any. So it is not just the amount of data that matters, but the variety — the training examples must cover the range of cases the model will actually meet.
One honest limitation is worth stating plainly. A model does not check whether its answer makes sense; it computes the output its trained values point to, and it will do that just as confidently when the input is unlike anything in its training data. The safeguard is not in the model but in the data behind it, and in a person checking the results.
9.5 Identifying, Acquiring and Exploring Data
Section 9.4 showed that an AI system is built out of its data. That raises an obvious practical question: where does that data come from, and how do you know it is the right data?
The work happens in three stages, and they always run in this order. First you decide which data the problem needs. Then you go and get it. Then you look at what you actually got, before using it for anything. Fig. 9.5-1 shows the three stages together.
Diagram
Fig. 9.5-1 – three boxes joined left to right by arrows, each with a short question underneath: “Identify — which data would answer the question?”, “Acquire — where will it come from?”, “Explore — what does the data we collected actually look like?”
The collected data, once it is gathered together in one place, is called a dataset. Everything that follows in an AI project rests on this dataset, which is why these three stages come before any model is built.
9.5.1 Identifying the Data a Problem Needs
Start from the question, never from the data. Write the question down in one plain sentence, then ask what facts would be needed to answer it.
Suppose a school wants to reduce the food wasted in its canteen. The question becomes: how many plates of each item should be prepared tomorrow? Now the useful data almost lists itself — how many plates of each item were sold on each past day, which day of the week it was, and whether that day was a holiday, an exam day or a sports day.
Just as useful is knowing what to leave out. The names of the students who bought lunch, or their roll numbers, would not help answer this question at all. Data that does not help should not be collected, however easy it is to collect.
9.5.2 Acquiring the Data
Once you know what you need, you have to obtain it. There are four common sources, and most projects use more than one.
Records that already exist. Attendance registers, library issue records, canteen sales books and exam mark sheets are all data a school already holds. This is the cheapest source, because the collecting has already been done.
Surveys and forms. When the data does not exist yet, you ask for it — a printed form or an online form filled in by the people concerned. This gives exactly the fields you asked for, but only as honestly and carefully as people fill them in.
Sensors and devices. A temperature sensor, a light sensor or a step counter records data by itself, at regular intervals, without anyone writing anything down. This suits any quantity that changes through the day.
Public datasets. Governments and research bodies publish datasets for anyone to use. India’s open data portal at
data.gov.in, for instance, carries datasets on rainfall, transport, health and much else. These are large and already cleaned up, but they were collected for someone else’s purpose, so they may not contain exactly the fields you want.
9.5.3 Exploring the Data
Data almost never arrives clean. Before it is used for anything, you look through it — this stage is called exploring the data, and its whole purpose is to find problems while they are still cheap to fix.
Table 9.3 holds a short dataset from a temperature sensor placed in a school computer lab, which recorded one reading every hour.
Four things are worth noticing, and each of them comes from a question you should ask of any dataset.
How much data is there? Eight time slots, but only seven readings. Knowing the size tells you whether there is enough to work with at all.
Is anything missing? The 12:00 reading is blank. A blank is not a zero — nothing was recorded, perhaps because the sensor lost power for a moment. You either leave that row out or note clearly that it is missing.
What is the normal range? The valid readings sit between 24.5 °C and 27.2 °C, which is a sensible range for a classroom.
Does anything look impossible? The 14:00 reading of 205.0 °C does. A room does not reach that temperature, and the readings on either side of it are close to 27 °C. This is almost certainly a fault — a loose wire, or the sensor being touched — and not a real measurement.
That last one matters more than it looks, as the following calculation shows.
One bad value out of seven has pushed the average to roughly twice the true figure. Exploring the data is what catches it.
9.6 Data as a Precursor to the AI Project Cycle
Nobody builds a house by starting with the roof. There is an order — the ground is cleared, the foundation is laid, the walls go up, and only then does the roof make sense.
Building an AI system works the same way. The fixed sequence of stages it goes through, from first understanding the problem to checking whether the finished system actually works, is called the AI project cycle. Its stages are Problem Scoping, Data Acquisition, Data Exploration, Modelling and Evaluation.
You will study each of these stages closely later. What matters here is one thing: data comes before all of them and runs through all of them. That is what “a precursor to the AI project cycle” means — the data work is not one stage among five, it is the ground the whole cycle stands on. Fig. 9.6-1 shows this.
Diagram
Fig. 9.6-1 – five boxes arranged in a ring, labelled clockwise Problem Scoping, Data Acquisition, Data Exploration, Modelling, Evaluation, with an arrow from each to the next and a longer return arrow from Evaluation back to Data Acquisition; the centre of the ring holds a single box labelled “DATA” with short lines joining it to all five stages
What data does at each stage
Problem Scoping. This is where the problem is stated clearly. Knowing what data exists shapes what can sensibly be attempted — a question no available data could ever answer is not yet a workable project.
Data Acquisition. The data identified during scoping is collected, from the kinds of sources listed in Section 9.5.2. What can be obtained here fixes the limits of everything that follows.
Data Exploration. The collected data is examined for gaps, mistakes and odd values, and arranged so that its patterns can be seen. This is the checking work of Section 9.5.3, done before any model exists.
Modelling. The prepared data is used to train the model. The kind of data available also decides which method is suitable, since a method that expects numbers cannot be trained on photographs.
Evaluation. The model is tested on data it has not seen during training, to check whether it works on new cases rather than only on the examples it was shown.
The cycle turns
Notice the return arrow in Fig. 9.6-1. Evaluation often shows that the model handles some cases badly, and the usual reason is that those cases were thin or absent in the data. The response is to go back and collect more, then work forward through the stages again.
So an AI project is rarely finished in one pass. Each turn of the cycle usually begins the same way — with more or better data.
9.7 Summary
- Data is raw, unprocessed facts and figures that have been collected but not yet made sense of. On its own it has no context and answers no question.
- Data is recorded either by people, such as in a register, or automatically by machines and sensors.
- Information is data that has been processed, organised and given context, so that it carries a clear meaning and can be used to decide, explain or plan.
- The same values can be data at one moment and information at the next. What changes is the arrangement and the context added, not the values themselves.
- Data becomes information through a sequence of steps: data collection, data processing, data storage, data analysis, contextualisation, organisation, presentation and interpretation.
- The four types of data are audio, visual, numeric and text.
- Audio data is sound stored as a list of samples. The sampling rate is how many samples are taken each second, measured in hertz.
- Visual data covers images and video. A pixel is the smallest unit of an image, resolution is how many pixels an image contains, and metadata is the extra detail stored alongside the image. For video, the frame rate is how many frames are shown each second.
- Numeric data is numbers standing for quantities, which can be added, averaged and compared. Numbers used only as labels, such as roll numbers, should not be treated as quantities.
- Text data is letters and words, stored using a character code such as ASCII or the wider Unicode.
- An AI system is used for problems where the rule cannot be written down but examples of the right answer exist. It is trained on those examples, and the result of training is a model.
- Data is what an AI system is built out of. It trains the model, shows which features matter, supports pattern recognition, allows the model to be improved later, and lets a system personalise what it offers a user.
- The variety of the training data matters as much as the amount. A model handles only the kinds of cases it was trained on, and data that leaves a group out produces a model that is unfair to that group.
- Before any model is built, three stages of data work come first: identifying which data the question needs, acquiring it, and exploring what was actually collected.
- Data can be acquired from existing records, surveys and forms, sensors and devices, or public datasets. Only the fields the question needs should be collected.
- Exploring a dataset means checking its size, looking for missing values, checking the normal range, and spotting impossible values. A single faulty value can badly distort a result such as an average.
- The AI project cycle runs through Problem Scoping, Data Acquisition, Data Exploration, Modelling and Evaluation. Data comes before all of these stages and runs through every one of them.
- Evaluation uses data the model has not seen during training, and its result often sends the project back to collect more data, turning the cycle again.
9.8 Key Terms
- AI project cycle: the fixed sequence of stages an artificial intelligence project passes through — Problem Scoping, Data Acquisition, Data Exploration, Modelling and Evaluation.
- Artificial intelligence (AI): the building of systems that work out an answer from examples, used for problems where the rule cannot be written down directly.
- Audio data: sound stored in a form a computer can hold, as a long list of numbered samples.
- Bias: one-sidedness in a dataset, such as a group of people being left out, which passes into the model’s outputs and makes them unfair.
- Data: raw, unprocessed facts and figures that have been collected and can be stored and analysed later.
- Dataset: the collected data for a project, gathered together in one place.
- Frame rate: how many still frames of a video are shown each second, measured in frames per second (fps).
- Information: data that has been processed, organised and given context, so that it carries a clear meaning and can be used to make decisions.
- Metadata: extra details stored alongside a file describing the file itself, such as an image’s size in pixels, the date it was taken and the camera settings used.
- Model: the result of training a program on data — a set of internal values that turns a new input into a predicted output, judged on data it has not seen before.
- Numeric data: data made of numbers that stand for quantities, such as marks, counts or temperatures.
- Pixel: the smallest unit of a digital image, holding the colour at one spot.
- Resolution: how many pixels an image contains, usually written as width × height.
- Sampling rate: how many samples of a sound are taken each second while recording it, measured in hertz (Hz).
- Text data: data made of letters, words and sentences, stored using a character code such as ASCII (American Standard Code for Information Interchange) or Unicode.
- Visual data: data that is looked at, such as photographs, drawings, screenshots and video.
9.9 Objective Questions
(The smallest unit of a digital image is called:)
- a sample
- a pixel
- a frame
- a byte
(The number of samples taken from a sound wave each second is called the __________, and it is measured in hertz.)
(State whether true or false: data and information mean the same thing, because both are made up of the same values.)
(A school register holds a column of roll numbers. Which one of these is a sensible thing to do with that column?)
- Find the average roll number
- Add all the roll numbers together
- Count how many roll numbers there are
- Find the total of the roll numbers
(Match each term in Column A with its correct meaning in Column B.)
Column A Column B a. Resolution i. Details stored about a file, such as the date it was taken b. Frame rate ii. How many pixels an image contains c. Metadata iii. How many samples of a sound are taken each second d. Sampling rate iv. How many still frames of a video are shown each second (Data that is made of letters, words and sentences, and is stored using a character code such as ASCII or Unicode, is called __________ data.)
(A sensor in a school hall recorded these four temperature readings: 22.8 °C, 23.4 °C, 180.0 °C and 24.1 °C. Name the reading that could not be a real measurement of a hall, and say in one line how you can tell.)
9.10 Concept Questions
(Define data and define information in one sentence each, and then state one clear difference between them.)
(Give one example each of audio data, visual data, numeric data and text data taken from an ordinary school day.)
(A computer can hold only 0s and 1s, but sound reaches a microphone as a smooth wave. Explain what the recording device actually stores, and what is lost in doing so.)
(A shopkeeper says, “I have four years of bills in that cupboard, so I already have information about my shop.” Explain why the shopkeeper is not correct, and describe what would have to be done to those bills first.)
(The number of books issued from a school library on five days was 18, 25, 30, 22 and 25. Work out the average number of books issued per day, showing your working. Then work out what the average would become if the last value had been wrongly typed as 250 instead of 25, and say what this shows about checking data before using it.)
(Explain why the variety of the training data matters as much as the amount of it. Support your answer with one example of what can go wrong when a group of people is missing from the data.)
9.11 Case-Based Questions
(A school library wants to work out how many copies of its most popular book it should keep on the shelf. A student team is asked to prepare the data. The team writes down how many copies of that book were issued on each day of one week, and produces the table below.)
Day Copies issued Monday 12 Tuesday 15 Wednesday (no entry) Thursday 14 Friday 1400 - The team also wants to record the name and class of every student who borrowed the book. Explain whether this data is relevant to the question the library asked, and give a reason.
- Two values in the table need attention before any calculation is done. Identify both, say what is wrong with each, and state how each should be handled.
- Name the stage of the AI project cycle in which the work described in part (b) belongs.
(A shop in a small town wants a system that suggests which items to restock each week. A team collects two things: the shop’s sales records for the past year, and photographs taken of customers standing at the billing counter.)
- Of the two sources collected, state which one is relevant to the question being asked, and explain why the other one is not.
- State one problem with collecting photographs of customers in this way, and describe what the team should do instead.
- The team suggests skipping straight to building the model, since the sales records are already available in the billing computer. Explain why this is a poor idea.