Chapter 16: Data Acquisition
Every afternoon, the school canteen in Neha’s school throws away food. Some days the rajma runs out by half past one and thirty students go back hungry. Other days two full trays are left over and go into the bin. The canteen in-charge, Mr. Solanki, cooks the same amount every day because he has no better way of guessing.
Neha’s class decides to help. They want to build a small system that can predict how many plates of each item will be needed tomorrow. Their teacher listens, then asks one simple question: “What information do you have about the last three months?”
The class goes quiet. Nobody has written anything down. They remember that Fridays are busier, and that fewer students eat during exams, but memories are not something a computer can work with.
So the project does not begin with clever code. It begins with a notebook, a date column, and a decision about what is worth recording.
Diagram
Fig. 16.0-1 – a school canteen counter with a queue of students on one side; on the other side a student writing in a register with columns marked “Date”, “Day”, “Item”, “Plates cooked”, “Plates sold”; a dotted arrow leads from the register to a laptop screen showing a simple table
This chapter answers all five of these questions, one step at a time.
16.1 Why Every AI Project Begins with Data
Think about how you decide what to wear before leaving for school. You look at the sky, you remember that it rained yesterday, and you check whether your raincoat is dry. You are using facts you have collected. Without those facts, the decision would only be a guess.
A computer system is in exactly the same position, only stricter. It has no memory of yesterday unless somebody records yesterday for it. This is why an artificial intelligence project never starts with a program. It starts with facts, written down in a form a computer can read.
Those recorded facts are called data. Data is raw, unprocessed material — numbers, words, pictures, sounds or sensor readings — that has been collected, observed or measured. On its own, a single line of data usually tells you very little.
From data to information
Look at the canteen’s records for one week in Table 16.1. Each row is a plain record of what happened on that day. Nothing in the table draws a conclusion.
Now read the table again and think about it. Friday sold every single plate, which means some students probably went without. Saturday left 46 plates unsold, because 120 − 74 = 46. That understanding — busiest on Friday, quietest on Saturday — is information. Information is what you get after data has been organised and interpreted so that it means something.
Data is the raw material; information is what you build from it. The canteen already had the events. It simply never recorded them, so no information could be drawn.
Why the machine needs the data
A model is the part of an artificial intelligence (AI) system that has been prepared using data so that it can produce an output for a new input. A model is not given the rule “Fridays are busy” by a programmer. Instead it is shown many past days, and it works out mathematically that the number for Friday is usually higher than the rest.
If there are no past days to show, there is nothing for the model to work out. It cannot invent a pattern out of nothing. This is the plain reason data comes first.
Naming the stage
The whole business of deciding what to record, finding it, and bringing it together is called data acquisition. It is the second stage of the AI project framework, coming after the problem has been scoped and before the data is explored.
The collection of records that comes out of this stage is called a dataset. Table 16.1 is a very small dataset — six rows and three columns. A real project would need months of such rows, not one week.
16.2 Types of Data
Before you collect anything, it helps to know what kind of thing you are collecting. A shopkeeper stores rice in a sack, milk in a can and change in a drawer, because each behaves differently. Data is the same. How a piece of data is shaped decides how easily a computer can work with it.
Data is sorted in two useful ways. The first way looks at how neatly it is arranged. The second looks at what the values themselves are.
16.2.1 Structured and Unstructured Data
Open the school’s attendance register. Every page has the same columns, every row means the same thing, and every entry sits in its proper place. Anything arranged like this — in fixed rows and columns, with each column holding one known kind of value — is called structured data.
Spreadsheets and databases hold structured data. So does the canteen register from Table 16.1. A computer can pick out one column, add it up, or sort it, without being told anything extra about the file.
Now think of the photographs on a school trip, the voice notes on a phone, or the messages in a class group. There is plenty of information in them, but nothing is arranged in columns. This is unstructured data.
Unstructured data is not useless — most AI systems that see or listen are built on it. It simply needs far more work before a model can use it. Someone must label the photographs, or convert the speech into text, before any pattern can be counted. See Fig. 16.2-1 for the difference at a glance.
Diagram
Fig. 16.2-1 – left side labelled “Structured”, showing a neat table with column headings Date, Day, Plates sold; right side labelled “Unstructured”, showing a loose pile of a photograph, a sound wave, a paragraph of handwritten text and a video frame
16.2.2 Numbers, Categories, Text and Time-Series Data
The second way of sorting data looks at the values themselves. Four kinds turn up again and again.
Numerical data is data made of measurable quantities — a value you can count or measure, and sensibly add or average. The number of plates sold, a student’s height in centimetres, the temperature of a room, and the price of a book are all numerical.
Categorical data is data made of labels that place a record into a group. The day of the week, the name of a dish, a district in Rajasthan, and whether a bus is running or cancelled are all categorical. You can count how many records fall into each group, but averaging the labels themselves would mean nothing.
Text data is data made of ordinary written language, such as a complaint written in the suggestion box or a one-line review of a canteen dish. It carries a lot of meaning, but a model cannot use a sentence directly. The sentence has to be turned into numbers first.
Time-series data is data in which the same quantity is recorded again and again over time, so the order of the records matters. Daily rainfall, hourly temperature and the canteen’s weekly plate count are all time-series data. Shuffle the rows of a marks list and you lose nothing; shuffle the rows of a time series and the pattern is destroyed.
16.3 Data Features
A dataset is a table, and a table has columns. Not every column plays the same part. Some columns describe the situation, and one column usually holds the thing you are trying to work out. Sorting out which is which is the next job after collecting data.
16.3.1 What a Feature Is
Suppose the class has now recorded three months of canteen data. A few rows look like Table 16.3.
Read one row across. The day, the item, whether it rained and whether it was exam week all describe the conditions of that day. The last column records what actually happened. The describing columns are the inputs the model will be given; the last column is what it must produce.
Each measurable column that is used as an input is called a data feature, or simply a feature.
The column the model has to produce is the target. In the canteen dataset the target is the number of plates sold. When a row already carries the correct target value, as every row in Table 16.3 does, that value is called the label of the row.
Features are not always taken straight from the raw columns. Often a more useful column is built out of the ones you have. From a date you can work out the day of the week; from a start time and an end time you can work out how long something took. Creating, selecting and reshaping features in this way is called feature engineering, and it sits between raw data and modelling, as Fig. 16.3-1 shows.
Diagram
Fig. 16.3-1 – three boxes joined left to right by arrows, labelled “Raw data”, “Features” and “Modelling”; a fourth box labelled “Feature engineering” sits below and feeds an arrow up into the “Features” box
16.3.2 Choosing Features That Actually Help
More columns do not make a better model. Every extra column gives the model one more thing to look at, and a column that has no real connection to the target only adds noise. The model may then latch on to an accidental coincidence in the recorded rows and repeat it on new days, where it does not hold.
Three simple checks help you decide.
Does it relate to the answer?
Ask whether you can explain, in one plain sentence, why the column should change the target. “Fewer students stay for lunch during exam week” is such a sentence. “The plate colour changes appetite” is not.
Will you know it in time?
A feature must be available before the prediction is needed. The canteen has to cook in the morning, so it can use tomorrow’s day of the week and tomorrow’s menu, both known in advance. It cannot use tomorrow’s actual queue length, because that is only known once lunch has already started.
Is it saying the same thing twice?
If you record both “Date” and “Day of the week”, the second is worked out from the first. Two columns that carry the same information add size without adding knowledge, and one of them can go.
16.4 Data Sources
You now know what to record. The next question is where it will come from. Some of it the canteen team will have to write down themselves. Some of it already exists — the attendance is in the school office, and the exam dates are on the notice board.
The place a piece of data comes from is called its data source.
A source is not the same as the data. The attendance figures are data; the school’s attendance software is the source. Knowing the source matters, because it tells you who recorded the values, how often they are updated, and whether you are allowed to use them.
16.4.1 Common Kinds of Data Sources
Data sources come in many forms. Table 16.4 lists the ones you are most likely to meet, with an example of each.
A word about two of these. A flat file is a plain file in which each line holds one record, with the fields separated by a marker such as a comma — a .csv file is the commonest example, and any spreadsheet can save one. An API, short for application programming interface, is a doorway a website opens on purpose so that programs can ask it for data directly, instead of a person reading the page.
Pulling data off a web page by program, when no such doorway is offered, is called web scraping. It works, but it is fragile: the moment the page layout changes, the program breaks.
Fig. 16.4-1 shows how several sources can feed one dataset.
Diagram
Fig. 16.4-1 – four small boxes on the left labelled “Canteen register”, “Attendance software”, “School calendar” and “Weather service”, each with an arrow pointing into a single larger box on the right labelled “Project dataset”
16.4.2 Collecting Your Own Data
Sources split neatly into two groups depending on who did the collecting.
Data you gather yourself, for your own problem, is called primary data. The canteen register is primary data: the class decided the columns, and they filled them in.
Data collected by somebody else, which you then obtain and use, is called secondary data. The attendance figures from the school office are secondary data, and so is a dataset downloaded from a government portal such as data.gov.in.
Primary data is usually the better fit, because you chose the columns. Its cost is time. Three months of canteen records take three months to collect, and there is no way to hurry that.
There are three ordinary ways to collect primary data. You can observe and record, as the canteen team does. You can ask, using a short survey. Or you can measure, using a sensor that logs values on its own.
Each has a weakness worth knowing. Observation depends on someone remembering to fill the register every single day, and a week of gaps ruins a time series. Surveys depend on people answering honestly and in the same way. Sensors depend on the device being placed correctly and not going faulty — a temperature sensor sitting in direct sunlight will report the sun, not the room.
16.5 Acquiring Relevant Data from Reliable Sources
If a stranger at a bus stop tells you the fare has gone up, you might believe it or you might not. If the same notice is pasted on the conductor’s window, you believe it at once. The fact is identical. What changed is your trust in where it came from.
Data works the same way, and two separate questions have to be asked of every dataset you obtain.
- Is it relevant — does it actually describe the problem you are solving?
- Is it reliable — was it recorded carefully, by someone who can be checked?
A dataset can easily pass one test and fail the other. Rainfall figures for Jaipur are reliable if they come from the meteorological department, but they are not relevant to a question about library fines. A neighbour’s guesses about the canteen queue are highly relevant, and completely unreliable.
16.5.1 What Makes a Source Reliable
There is no single stamp that marks a source as trustworthy. Instead there are signs you can look for, and they take only a few minutes to check.
- Who published it. A government department, a university, a hospital or a well-known company puts its name at risk if the figures are wrong. An anonymous upload risks nothing.
- Whether it is documented. A good dataset comes with a note saying what each column means, what units are used, and how the values were collected. Without that note you are only guessing.
- When it was last updated. Bus timings from six years ago are not wrong exactly, but they no longer describe today.
- Whether it is complete and consistent. Missing values, repeated rows and impossible entries all point to careless recording.
- Whether it can be cross-checked. If a second, independent source gives roughly the same figures, both become more believable.
- Whether you are allowed to use it. A dataset published for public use will say so. If nothing says so, assume you may not republish it.
Reading a dataset with your eyes open
Table 16.6 shows the first few rows of an imaginary dataset downloaded from a website with no documentation. Read it before you read the sentence after it.
Four problems are visible without any calculation. The 3 September row appears twice, so any total will be too high. The 4 September value is missing. The 5 September value is negative, and a negative number of plates cannot happen. The 6 September value of 6100 is larger than the whole school. None of this proves the source is dishonest, but it does prove that nobody checked the file, and a source that does not check its own file will not have checked the rest of it either.
16.5.2 Why Reliable and Relevant Data Matters
Careful sourcing is not fussiness. It changes the outcome of the project in seven distinct ways.
- Accuracy of results. Predictions can only be as good as the records they were built from. Poor data leads to wrong conclusions that look perfectly confident on screen.
- Model performance. Relevant, well-recorded features let a model pick out a real pattern. Irrelevant or noisy columns bury the pattern instead.
- Fairness. If a dataset over-represents one group of people and barely includes another, the system built on it will work well for the first group and badly for the second. This tilt in the data is called bias.
- Reproducibility. Work that others can repeat is work others can trust. If you can name the source and the date you obtained it, another class can download the same file and check your result.
- Compliance and ethics. Data about people is governed by law. In India, the Digital Personal Data Protection Act, 2023 sets rules for handling personal data; other countries have their own, such as the GDPR in Europe. Data taken from a trustworthy source is far more likely to have been collected lawfully in the first place.
- Less cleaning work. Every duplicate and blank in Table 16.6 has to be found and dealt with by hand before modelling can begin. Clean sources save that effort.
- Long-term supply. A source that publishes every month lets your project keep running next year. A one-off file uploaded by an unknown user does not.
16.6 Sources of Dataset in AI – Kaggle Platform
Collecting three months of canteen records is honest work, but it is slow. For many problems somebody has already collected the data and published it. What you then need is a place where such collections are gathered, described and kept in one searchable heap — a data hub.
The best known of these is Kaggle, an online platform for data science and machine learning, owned by Google. It is free to join, and anyone can browse and download from it.
16.6.1 What You Can Find on Kaggle
Five parts of the platform are worth knowing.
- Datasets. A very large collection of published datasets covering subjects such as agriculture, health, transport, sport and cinema. Each one has its own page carrying a description, the file itself, and a note on how it may be used.
- Notebooks. A notebook is a page in which code, its output and written explanation sit together in one document, and Kaggle notebooks run inside the browser, so nothing has to be installed on the school computer. In older material these are called kernels — the same thing under an earlier name.
- Courses. Short, free lessons on topics such as Python, handling data and the basics of machine learning, each with exercises you complete in a notebook.
- Competitions. An organisation publishes a problem and a dataset, and anyone may submit predictions, which are scored automatically against data the competitor never sees. Some competitions offer prizes; many are set up purely for practice.
- Community. Discussion forums where people ask questions, share what worked, and point out problems in a dataset. Reading the discussion attached to a dataset often tells you more about its faults than the description does.
Diagram
Fig. 16.6-1 – screenshot of a dataset page on Kaggle, with the dataset title, the short description, the list of files, the licence line and the column summary each clearly visible
16.6.2 Using a Downloaded Dataset Sensibly
Everything in Section 16.5 applies here without exception. A file downloaded in two clicks still has to earn your trust.
Step 8 is easy to skip and matters more than it looks. Two datasets on a hub may have similar names and quite different contents, and a dataset can be edited by its uploader after you download it. Recording the name, uploader and date is what lets somebody else repeat your work.
16.7 Training and Testing Data
Suppose your teacher gives you thirty practice questions with the answers, and then sets a test made of those same thirty questions. You would score well. But nobody — including you — would learn anything about whether you actually understand the chapter.
A model can be fooled in exactly the same way, and for exactly the same reason. So the dataset a project collects is never used all at once. It is deliberately divided.
16.7.1 Splitting a Dataset
The part of the dataset used to build the model is called training data. Each row of it carries the features and, alongside them, the correct answer for that row. That correct answer is called the label, and in Table 16.3 it was the “Plates sold” column.
During training, the model is shown these pairs again and again and its internal numbers are adjusted until its outputs come close to the labels. That is all “training” means.
The part kept aside is called testing data.
The testing rows also carry their labels. The difference is that the model is not shown them while it is being built. Afterwards, the model is given only the features of those rows, and its outputs are compared with the labels that were held back. See Fig. 16.7-1.
Diagram
Fig. 16.7-1 – one long box labelled “Full dataset” splitting into two boxes, a larger one labelled “Training data – used to build the model” and a smaller one labelled “Testing data – kept aside, unseen”; an arrow from the training box into a box labelled “Model”, and a dashed arrow from the testing box into the same Model box marked “only after training”
How big should each part be? There is no fixed law. Typically about 70 to 80 per cent of the rows are used for training and the rest for testing, because the model needs plenty of examples to work from, while a modest number is enough to judge it by.
16.7.2 Why Testing Data Must Stay Unseen
A model has two ways of doing well on the rows it was trained on. It can find a real pattern — plate counts rise with attendance and fall in exam weeks. Or it can simply store the rows and repeat their answers back.
From the training rows alone, these two look identical. Both give right answers. The difference shows up only on a day the model has never met, and that is precisely what the held-back rows provide. A model that found the pattern still does well; a model that only stored the answers has nothing to fall back on.
This is why the split has to be decided before training and honoured afterwards. If even a few testing rows leak into training, the test stops measuring what it was meant to measure, and the result looks better than the truth.
One caution about time
Splitting rows at random is fine for a marks list, where the order means nothing. It is the wrong thing to do for a time series. If the canteen’s 90 days are shuffled and split at random, the model ends up trained on some days in October and tested on days in September — that is, tested on the past after being shown the future.
For time-series data, split by date instead: train on the earlier days and test on the later ones. That matches what the system will actually be asked to do, which is to predict a day that has not happened yet.
16.8 System Maps
The canteen register describes days. It says nothing about the canteen itself — that the cook orders vegetables from a supplier two days in advance, that the timetable decides when the rush arrives, that leftovers change what is cooked tomorrow.
A table is good at holding records. It is poor at showing how parts of a situation pull on one another. For that you need a picture, and the picture used at this stage of an AI project is a system map.
Drawing one is worth the twenty minutes it takes, because it forces you to name every part before you decide what to record. Parts you cannot name, you will not measure.
16.8.1 Reading a System Map
Fig. 16.8-1 is a system map of the school canteen. Look at it before reading on.
Diagram
Fig. 16.8-1 – boxes labelled “Students”, “Class timetable”, “Canteen counter”, “Cook”, “Vegetable supplier”, “Leftover food” and “Daily register”; arrows: Class timetable → Students, Students → Canteen counter labelled “demand”, Cook → Canteen counter labelled “plates cooked”, Vegetable supplier → Cook labelled “ingredients, ordered 2 days ahead”, Canteen counter → Leftover food, Canteen counter → Daily register labelled “plates sold”, Leftover food → Cook drawn as a curved dashed arrow labelled “feedback”
Three things make up any system map.
- Parts. Each box is one component of the system — a person, a group, a place or a store of something. Keep them concrete. “Students” is a part; “school spirit” is not.
- Connections. Each arrow shows that one part affects another, and the arrow points in the direction of the effect. The label on the arrow says what actually travels along it: food, money, information, or a request.
- Loops. Sometimes you can follow the arrows and come back to where you started. Such a closed path is called a feedback loop, because the result of a process comes back and changes the process itself.
The dashed arrow in Fig. 16.8-1 is the canteen’s feedback loop. Large leftovers today make the cook prepare less tomorrow, which makes leftovers smaller, which lets the cook prepare more again. The loop steadies the system — but only slowly, and only if someone actually notices the leftovers.
Some system maps go one step further and mark each arrow with a sign: a plus where more of one part means more of the other, and a minus where more of one means less. On this map, “Students → Canteen counter” would carry a plus, and “Leftover food → Cook” a minus.
Once drawn, the map earns its keep in three ways. It shows where a delay builds up, which is called a bottleneck — here, the two-day ordering gap. It shows where a single failure would stop everything, such as the counter. And it shows which arrows carry information you could actually record, which is exactly what data acquisition needs to know.
16.8.2 Drawing a System Map for a School Project
There is no special software needed. A pencil and one sheet of paper are enough.
Step 2 is where most maps go wrong. If a part is too vague to point at, it cannot be measured, and an arrow leading to it cannot be checked. Replace it with something you could photograph.
16.9 Summary
- Data is raw, unprocessed material — numbers, text, images, sounds or sensor readings — that has been collected, observed or measured. Information is what you get after data has been organised and interpreted.
- An AI project begins with data because a model finds patterns only by comparing many recorded examples. With no records, there is nothing to compare.
- Data acquisition is the stage of an AI project in which relevant data is identified, collected and gathered. The collection of records it produces is called a dataset.
- Structured data is arranged in fixed rows and columns and can be used almost as it is. Unstructured data — photographs, audio, video, free text — has no such arrangement and needs labelling or conversion first.
- By value, data is numerical, categorical, text or time-series. In time-series data the order of the records matters.
- Data features are the measurable attributes of a dataset that are given to a model as inputs. The column the model has to produce is the target.
- A feature is worth keeping only if you can explain why it could change the target, and only if its value is available at the moment the prediction is needed.
- Feature engineering means building a more useful column out of the columns you already have, and dropping the ones that add nothing.
- A data source is the origin or location from which data is obtained — a file, a database, a sensor, a web service, a website or manual entry.
- Primary data is collected by you for your own problem. Secondary data was collected by somebody else and then obtained by you.
- Every dataset must pass two separate tests: relevance to the problem, and reliability of the source.
- A source looks reliable when the publisher is named, the columns are documented, the file is current, the values are complete and consistent, the figures can be cross-checked, and the licence permits use.
- Reliable and relevant data matters for seven reasons: accuracy of results, model performance, fairness, reproducibility, compliance with law and ethics, less cleaning work, and a dependable supply of data over time.
- Bias is a tilt in the data towards some group of people and away from another. A system built on biased data works well for the group that is well represented and badly for the group that is not.
- Kaggle is an online platform serving as a community and resource hub for data science and machine learning, offering datasets, notebooks, courses, competitions and discussion forums.
- Anyone may upload to a data hub, so a downloaded dataset must still be checked. Record its name, uploader and download date so that your work can be repeated.
- Training data is the part of the dataset used to build the model, with each record’s features paired with its correct output, called the label.
- Testing data is a separate part, kept unseen during training and used afterwards to check the model. Typically 70 to 80 per cent of the rows are used for training and the rest for testing.
- Testing on unseen records is the only way to tell a model that found a pattern from one that merely stored the answers. For time-series data, split by date rather than at random.
- A system map is a visual representation of the parts of a system, showing how they are connected and how they affect one another. Boxes are parts, arrows are effects, and a path that closes back on itself is a feedback loop.
- A system map reveals delays, points of failure and the connections along which something could actually be recorded, which is what makes it useful before data collection begins.
16.10 Key Terms
bias — A tilt in a dataset towards some group and away from another, which makes the system built on it work well for one and badly for the other.
categorical data — Data made of labels that place a record into a group, such as the day of the week or the name of a dish.
data — Raw, unprocessed material that has been collected, observed or measured, such as numbers, text, images, sounds or sensor readings.
data acquisition — The stage of an artificial intelligence project in which relevant data is identified, collected and gathered from suitable sources.
data features — The measurable attributes or characteristics of a dataset that are given to a model as inputs.
data source — The origin or location from which data is obtained, such as a file, a database, a sensor or a website.
dataset — A collection of records held together, usually as rows and columns.
feature engineering — Building a more useful column out of the columns already collected, and dropping the ones that add nothing.
feedback loop — A closed path in a system, where the result of a process comes back and changes that same process.
flat file — A plain file in which each line holds one record and the fields are separated by a marker such as a comma.
information — What data becomes after it has been organised and interpreted so that it means something.
Kaggle — An online platform that serves as a community and resource hub for data science and machine learning, offering datasets, notebooks, courses, competitions and forums.
label — The correct output recorded alongside the features of a record, used to guide the model during training.
model — The part of an artificial intelligence system that has been prepared using data so that it can produce an output for a new input.
numerical data — Data made of measurable quantities that can sensibly be counted, added or averaged.
primary data — Data you collect yourself, for your own problem.
secondary data — Data collected by somebody else, for their own purpose, which you then obtain and use.
structured data — Data arranged in a fixed form such as rows and columns, where every record has the same fields.
system map — A visual representation of the parts of a system, showing how they are connected and how they affect one another.
target — The column a model is meant to produce, as opposed to the features it is given.
testing data — The part of a dataset kept aside during training and used afterwards to check the model on records it has never seen.
text data — Data made of ordinary written language, which must be converted into numbers before a model can use it.
time-series data — Data in which the same quantity is recorded repeatedly over time, so that the order of the records matters.
training data — The part of a dataset used to build a model, in which each record’s features are paired with its correct output.
unstructured data — Data with no fixed row-and-column arrangement, such as photographs, audio, video or free-flowing text.
web scraping — Pulling data off a web page by program, when the site offers no proper doorway for programs to request it.
16.11 Objective Questions
Which one of the following is an example of unstructured data?
- A spreadsheet of monthly rainfall figures
- A voice recording of a school assembly
- A
.csvfile of library issue records - A database table of student attendance
Fill in the blank: The part of a dataset that is kept aside during training and used afterwards to check the model is called ________ data.
State whether true or false: A dataset with fifty thousand rows is more reliable than one with three hundred rows.
Which one of the following describes feature engineering?
- Collecting more rows for the dataset
- Building a more useful column out of the columns already collected
- Splitting the dataset into training and testing parts
- Drawing the parts of a system as labelled boxes
A register records four columns — Date, Day, Students present and Plates sold — and the aim is to predict how many plates will be sold tomorrow. Name the target column, and name any one feature.
Match each data source in Column A with the example in Column B.
| Column A | Column B |
|---|---|
| a. Sensor | i. Bus timings printed on a transport department page |
| b. Flat file | ii. A canteen register filled in by hand each day |
| c. Manual entry | iii. A thermometer logging room temperature every minute |
| d. Website | iv. A marks.csv file exported from a spreadsheet |
- On a system map, an arrow is traced from one box and, after passing through other boxes, returns to the box it started from. What is such a closed path called?
- A bottleneck
- A feedback loop
- A data source
- A flat file
16.12 Concept Questions
Define data and information, and explain the difference between them using one example of your own.
Differentiate between structured and unstructured data on any three points.
List any four signs that tell you a data source is reliable, and say briefly why each one matters.
A student wants to predict how many umbrellas a shop will sell tomorrow, and suggests including a column called “umbrellas returned unsold in the evening”. Give one reason why this column cannot be used as a feature.
A dataset of 120 records is to be split between training and testing in the ratio 75 : 25. Calculate how many records go into each part, and show your working.
Explain why a model must be tested on records it has never seen during training. What could go wrong if the same records were used for both?
What would happen if a time-series dataset of daily temperature readings were split at random instead of by date? Give a reason for your answer.
16.13 Case-Based Questions
- A school wants to predict how late its bus will reach the gate each morning, so that the first period can be adjusted. A teacher has been recording the details shown below for several weeks.
| Date | Day | Weather | Buses on route | Arrival delay (minutes) |
|---|---|---|---|---|
| 05 Jan | Monday | Clear | 4 | 3 |
| 06 Jan | Tuesday | Fog | 4 | 18 |
| 07 Jan | Wednesday | Fog | 3 | 22 |
| 08 Jan | Thursday | Clear | 4 | 2 |
| 09 Jan | Friday | Clear | 4 | 5 |
- Name the target column, and name any two features.
- State which of the recorded columns hold categorical data.
- The teacher finally has 100 such records and splits them in the ratio 80 : 20. Calculate how many records will be used for testing, and explain why those records must not be shown to the model while it is being built.
- A vegetable shop owner in a small town wants a system that suggests how many kilograms of tomatoes to buy from the wholesale market each morning. Her nephew offers to build it. He finds a dataset of daily vegetable prices uploaded to an online data hub by an unnamed user, with no description of the columns.
- Name one source of primary data and one source of secondary data that this project could use.
- State three checks the nephew should make before using the downloaded dataset, and say what each check protects against.
- Draw a system map of the shop with at least four parts, labelling what travels along each arrow, and mark one feedback loop on it.