Chapter 17: Data Exploration

The school canteen keeps a simple register. Every evening, someone writes down how many plates of each snack were sold that day. By the end of a term, the register holds hundreds of rows of numbers.

One day the canteen has to decide how much to prepare for the coming week. The register has the answer somewhere inside it, but nobody can see it. Reading row after row of numbers tells you very little. Your eyes slide over them.

So you copy a few weeks of the register into a computer and draw a picture of it — one bar for each day of the week, the height of the bar showing how many plates were sold. Now something jumps out at once. Some bars are clearly taller than others. A dip appears in the middle of the week, every week.

Nothing new was added. The same numbers were sitting in the register all along. All you did was change how you looked at them, and the pattern came out of hiding. That change of view is what this chapter is about.

Diagram

Fig. 17.0-1 – a two-panel picture. Left panel: a page from a canteen register showing rows of handwritten numbers, with a puzzled student looking at it. Right panel: the same information drawn as a simple bar chart with one bar per weekday, the pattern of tall and short bars clearly visible. An arrow labelled “same data, different view” points from the left panel to the right panel.

NoteCuriosity Corner

Q1. The numbers were already in the register, so what does drawing a picture of them actually add? Q2. Why do people study a set of data carefully before using it to build anything? Q3. A bar graph and a histogram look almost the same. What really makes them different? Q4. How can a chart show whether two things rise and fall together, such as hours of study and marks scored? Q5. Why must the slices of a pie chart always add up to 360°?

This chapter answers all five of these questions.

17.1 What Is Data Exploration?

Before cooking a meal, you spread the vegetables out on the counter and look at them. How much do you have? Is anything spoiled? Is one of them missing? You are not cooking yet. You are finding out what you are working with, because deciding anything before that would only be guesswork.

Work with information follows the same order. Data means the raw facts and figures we collect — temperatures, marks, prices, sales counts, sensor readings. When those facts are collected together in an organised way, usually as rows and columns, we call the collection a dataset. A register of daily canteen sales is a dataset. So is a file holding the marks of every student in a class.

A dataset that has just been collected is unknown territory. You may know what it was meant to contain, but not what it actually contains. Looking at it carefully, before doing anything else with it, is called data exploration.

NoteDefinition

(Data exploration is the first careful study of a dataset, done to find out what it contains, what patterns it holds, and what is wrong with it, before the data is used for anything further.)

What you are looking for

Data exploration is not aimless staring. There are four things you are trying to find out.

The first is what the data is made of — how many rows there are, what each column means, and what kind of value each column holds. The second is how the values are spread — where most of them sit, and how far they stretch.

The third is whether anything is wrong — values that are missing, repeated, or far outside the normal range. The fourth is whether the columns are connected to each other in some way, such as one value rising whenever another rises.

Q. Is data exploration the same as analysing the data?

A. No. Exploration comes first and only tells you what the data is like. The analysis, and any model built from it, comes afterwards and uses what the exploration revealed.

Where it sits in a project

You have seen that an artificial intelligence project moves through stages. First the problem is scoped, then the data is acquired. Data exploration is the stage that follows, and it comes before any modelling. Fig. 17.1-1 shows this order.

Diagram

Fig. 17.1-1 – a horizontal flow of four labelled boxes joined by arrows: “Problem Scoping”, “Data Acquisition”, “Data Exploration”, “Modelling and Evaluation”. The third box, “Data Exploration”, is highlighted, with a small caption below it reading “you are here”.

The position matters. Anything you fail to notice at this stage travels forward into every stage after it. A missing column, a wrongly recorded value, a column that turns out to be useless — all of these are cheap to fix now and expensive to fix later.

Why we draw pictures of data

A dataset with three hundred rows is unreadable as rows. Your eyes take in one number at a time and forget the previous one. But a picture of those three hundred rows can be taken in at a single glance.

Turning data into charts and graphs so that it can be understood by looking is called data visualisation. It is the main tool of data exploration, and the rest of this chapter is about the charts that do this work.

Q1. The numbers were already in the register, so what does drawing a picture of them actually add?

A. It adds nothing to the data — the numbers are exactly the same. What it changes is how much of the data you can hold in your mind at once. Reading a register, you see one number at a time. Looking at a chart, you see all of them together, so differences in height, a rise over time, or one odd value standing apart become visible immediately.

17.2 Why Data Exploration Is Needed

A tailor measures the cloth before cutting it. Not because measuring is enjoyable, but because a cut cannot be undone. Data exploration is the same kind of step. It costs a little time at the start and saves a great deal of trouble later.

There are several separate reasons for doing it, and they are worth taking one at a time.

Knowing what you actually have

The person who collects data and the person who uses it are often not the same person. Column names can be unclear. A column called temp might hold temperature in degrees Celsius, or it might hold a temporary code number. Exploration settles such questions early, so that nothing is used in the wrong way.

Finding what is wrong

Real data is rarely clean. Four kinds of trouble turn up again and again.

A missing value is a place where a value should be but nothing was recorded. A duplicate is the same record entered twice. An inconsistency is the same thing written in two different ways, such as Jaipur in one row and jaipur in another. And an outlier is a value that lies far outside the range of the rest.

Table 17.1 holds one week of recorded maximum temperatures for a small weather station. All four problems are hiding in it.

Table 17.1 – recorded maximum temperature for each day of one week

Day Maximum temperature (°C) Station name
Monday 38 Sikar
Tuesday 39 Sikar
Wednesday 415 Sikar
Thursday 40 sikar
Friday (blank) Sikar
Saturday 41 Sikar
Saturday 41 Sikar

Wednesday’s reading of 415 °C is an outlier, and an impossible one — most likely 41.5 typed without its decimal point. Friday’s reading is missing. Thursday’s station name is spelt differently from the rest, which a computer would treat as a different station altogether. And Saturday has been entered twice.

Seven rows can be checked by eye. Seven thousand cannot, which is why exploration uses summaries and charts to bring such problems to the surface.

deleting every outlier as soon as you see one What goes wrong: An outlier is treated as a mistake and removed without thinking. But some outliers are real and important — the one flood in a rainfall record, the one machine that failed. The fix: First ask whether the value is possible. A temperature of 415 °C at a weather station is not, so it is an error. A rainfall figure ten times the usual one may be perfectly true, and throwing it away would hide the very thing worth studying.

Deciding how to clean it

Once you know what is wrong, you can decide what to do about it. Fixing these problems — filling in or removing missing values, deleting duplicates, correcting spellings, bringing values to a common scale — is called data cleaning. You cannot plan the cleaning before you have explored, because until then you do not know what needs cleaning.

Q. Why not simply clean the data first and explore afterwards?

A. Cleaning means changing values. Changing values without first knowing what is normal for this dataset means you may “correct” something that was right all along.

Seeing patterns and relationships

Exploration also shows how the columns behave together. Sales may climb every weekend. Marks may rise as attendance rises. Two columns may hold almost the same information, so that keeping both is pointless.

Each column of a dataset that describes something about a record is called a feature. Knowing which features carry useful information, and which carry none, decides what you feed into the next stage.

Checking what you assumed

People usually begin a project with an idea already in their head — that a scheme worked, that one shop sells more than another, that a machine slows down in the afternoon. Exploration is where such an idea meets the data. Sometimes the data supports it. Often it does not, and it is far better to learn that at the start than after weeks of work.

Better models, better decisions

A model is a system built from data that produces an answer for a new case it is given. A model can only be as good as the data behind it. Clean, well-understood data leads to a model that behaves sensibly; data full of errors nobody noticed leads to a model that repeats those errors confidently.

The same holds for decisions made by people. A shopkeeper who has looked at last year’s sales chart is deciding from evidence. One who has not is deciding from memory.

Exploration is also the moment to check who is present in the data and who is not. A dataset of bus travel collected only on weekday mornings barely represents people who travel at night, and any system built from it will serve them badly. Check the balance of your data before building anything on it, and say plainly what your data does not cover.

Q2. Why do people study a set of data carefully before using it to build anything?

A. Because everything built later rests on the data, and mistakes in it are cheap to fix now and expensive to fix afterwards. Exploration shows what the dataset really contains, what is missing or wrong in it, which features are worth using, and whether the ideas you started with are actually supported by the data.

17.3 Ways of Representing Data

The same set of facts can be put in front of a reader in several different forms. A railway timetable, a station announcement board, and a route map on the wall all carry train information, but each suits a different question. Data is no different.

Showing data as a table

The plainest way to present data is a table, with one row for each record and one column for each thing recorded. You have already seen one in Table 17.1.

A table is exact. Every single value is there and can be read off to the last decimal place. That is its strength, and also its weakness — a table gives you the values but not the shape of them. Nobody can look at four hundred rows and say where most of the values sit.

Showing data as a picture

A chart or graph draws the same values as lengths, positions, or areas. You lose a little exactness, because you now read a bar’s height off a scale instead of reading the number itself. In return you see all the values at once.

This chapter studies five such pictures, in this order:

  1. Bar graph
  2. Histogram
  3. Line graph
  4. Scatter plot
  5. Pie chart

There are other forms too. An infographic mixes charts with drawings and short text to tell a story to a general reader. A time-series visualisation shows how something changed over a long stretch of time. Both are built from the same basic charts you are about to study.

The parts of a chart

Every chart in this chapter shares a common skeleton, shown in Fig. 17.3-1. Learning the parts once saves explaining them five times.

Diagram

Fig. 17.3-1 – a plain bar chart skeleton with callout labels pointing to each part: “title” above the chart, “y-axis (values)” running up the left with tick marks and numbers, “x-axis (categories)” running along the bottom with category names, “axis label” beside each axis, one bar labelled “bar”, a small box at the right labelled “legend”, and a bracket beside the y-axis numbers labelled “scale”.

The x-axis is the horizontal line along the bottom. The y-axis is the vertical line up the left side. Between them they fix where every point or bar sits.

The numbers marked along an axis form its scale. The words beside the axis, saying what the numbers mean and in what unit, are the axis labels. A chart without labels is unreadable — a bar of height 40 tells you nothing until you know whether that is 40 rupees, 40 plates, or 40 degrees.

When one chart shows more than one set of values, the different colours or line styles are explained in a small box called the legend.

Q. A chart is drawn with no numbers marked on the y-axis. What can still be said about it?

A. Only which bar is taller and roughly by how much. The actual quantities cannot be read at all, so any statement about amounts would be a guess.

Two kinds of data

Which chart suits your data depends mostly on what kind of data it is.

Categorical data places each record into a group with a name — subject, city, colour, day of the week. The groups have no natural arithmetic. It makes no sense to add Chemistry to Physics.

Numerical data records a measured or counted quantity — height, temperature, marks, number of plates sold. These values can be added, averaged, and put in order along a scale.

Table 17.2 – categorical data vs numerical data

Feature Categorical data Numerical data
What a value is a name or a label a number that measures or counts
Examples subject, city, day of the week marks, temperature, plates sold
Natural order usually none always, from smallest to largest
Arithmetic on values not meaningful meaningful
Usually shown by bar graph, pie chart histogram, line graph, scatter plot

The last row of Table 17.2 is a preview, not a rule to memorise. Each of the sections that follow explains one chart and what it is good at, and section 17.9 puts the whole choice together.

17.4 Bar Graph

Suppose the school library wants to know which subject’s books are borrowed most. The librarian has the counts, subject by subject. The question is not “how much in total” but “which is biggest, and by how far”. That comparison between named groups is exactly the job a bar graph does.

A bar graph shows a quantity for each category as a rectangular bar, with the length of the bar standing for the quantity. Longer bar, larger quantity. Because the eye is very good at comparing lengths side by side, the ranking of the categories can be read in a moment.

Table 17.3 holds the library’s counts for two terms.

Table 17.3 – number of books issued by the school library, by subject, in two terms

Subject Term 1 Term 2
Science 48 52
Mathematics 30 41
English 35 33
Hindi 22 20
Social Science 18 26

Fig. 17.4-1 draws the Term 1 column of this table as a bar graph.

Diagram

Fig. 17.4-1 – a vertical bar graph titled “Books issued in Term 1”. The x-axis is labelled “Subject” and carries five evenly spaced bars named Science, Mathematics, English, Hindi and Social Science. The y-axis is labelled “Number of books issued” with a scale from 0 to 60 marked every 10. Bar heights are Science 48, Mathematics 30, English 35, Hindi 22, Social Science 18. All bars are the same width and the same colour, with equal gaps between them.

Three details in that figure are not decoration. Every bar has the same width, so that only height carries meaning. The bars stand apart with clear gaps, because the categories are separate groups and not a continuous scale. And the y-axis begins at zero, so that a bar twice as tall really does mean twice as many books.

bar graph What it is: A chart that compares a quantity across several named categories. How it works: Each category gets one bar. The bar’s length is drawn in proportion to the category’s value, read against a numbered scale on the other axis. Used for: Comparing how many library books each subject lent out in a term. Watch out: It compares categories only. It cannot show how values are spread inside a category, and it becomes unreadable once there are too many categories to label.

Vertical or horizontal

Bars may run upwards from a horizontal axis, as in Fig. 17.4-1, or sideways from a vertical axis. The two carry exactly the same information.

Horizontal bars are the better choice when the category names are long, because the names then sit comfortably along the left side instead of being squeezed under narrow bars. Vertical bars are more common for short names and for anything read as a height.

Comparing more than one set

Table 17.3 has two terms in it, and often the interesting question is not the size of each subject but the change between the terms. For this, each category gets more than one bar.

In a grouped bar graph, the bars for the different sets are placed side by side within each category, in different colours, with a legend saying which colour is which term. Fig. 17.4-2 shows this.

Diagram

Fig. 17.4-2 – a grouped bar graph titled “Books issued by subject, Term 1 and Term 2”. The x-axis is labelled “Subject” with the same five categories; the y-axis is labelled “Number of books issued”, scale 0 to 60 marked every 10. Each subject has two bars side by side: Science 48 and 52, Mathematics 30 and 41, English 35 and 33, Hindi 22 and 20, Social Science 18 and 26. A legend at the right identifies the first colour as Term 1 and the second as Term 2.

In a stacked bar graph, the bars sit on top of one another instead, so each category has a single tall bar whose total height is the sum of the sets. Stacking is useful when the total matters as much as the parts, but it makes the individual pieces harder to compare, because only the bottom piece starts from the same line.

which subject’s borrowing rose the most between the two terms

Given: The Term 1 and Term 2 counts of Table 17.3.

Find: The subject with the largest rise from Term 1 to Term 2.

Working:

Change = Term 2 count − Term 1 count.

Science: 52 − 48 = 4 books

Mathematics: 41 − 30 = 11 books

English: 33 − 35 = −2 books

Hindi: 20 − 22 = −2 books

Social Science: 26 − 18 = 8 books

The largest of these changes is 11 books.

Mathematics, with a rise of 11 books.

Notice that the grouped bar graph makes this answer visible without any subtraction. The pair of Mathematics bars shows the widest step up of any pair on the chart.

starting the value axis somewhere above zero What goes wrong: To make small differences look clearer, the y-axis is begun at, say, 15 instead of 0. The Science bar then appears several times the size of the Social Science bar, although 48 is not even three times 18. The fix: Begin the value axis at zero for a bar graph. The reader judges bars by length, and a cut axis makes the lengths lie. If the differences are genuinely too small to see, say so in words rather than stretching the picture.

Count how many students in your class come to school by bus, by cycle, on foot, and by any other means. Draw the four counts as a bar graph on squared paper, then draw the same counts again with the value axis starting halfway up. Compare the two pictures.

17.5 Histogram

A test out of 50 has been marked for a class of 30 students. Here are the marks, in the order the answer papers happened to be picked up:

34, 21, 47, 12, 39, 25, 6, 36, 44, 28, 15, 33, 49, 22, 38, 9, 41, 27, 35, 18, 45, 29, 37, 24, 42, 14, 39, 25, 48, 31

Nobody can look at that and say how the class did. There are no categories here to compare, so a bar graph is of no use. The marks are all measurements on one single scale, and the question is a different one: where do most of the marks sit, and how far do they spread out? That question is about the distribution of the data, and the chart that answers it is a histogram.

Grouping the values into bins

The trick is to stop treating each mark separately. Instead, the scale is cut into equal intervals, and you count how many values fall into each interval. Each interval is called a bin, and the count of values in a bin is its frequency.

Taking bins ten marks wide and going through the list once gives Table 17.4.

Table 17.4 – number of students whose marks fall in each ten-mark bin

Marks Number of students
0–9 2
10–19 4
20–29 8
30–39 9
40–49 7

The five frequencies add up to 2 + 4 + 8 + 9 + 7 = 30, which is the number of students. That check is worth doing every time; if the total does not match, a value has been counted twice or missed.

A histogram draws this table as touching bars, one bar per bin, with the height of each bar equal to the frequency of that bin. Fig. 17.5-1 shows it.

Diagram

Fig. 17.5-1 – a histogram titled “Marks in the class test”. The x-axis is labelled “Marks” and is a continuous scale marked at 0, 10, 20, 30, 40 and 50. The y-axis is labelled “Number of students”, scale 0 to 10 marked every 2. Five bars of equal width stand with no gaps between them: 0–9 of height 2, 10–19 of height 4, 20–29 of height 8, 30–39 of height 9, 40–49 of height 7.

Now the class is easy to describe. Most of the marks are bunched in the twenties and thirties, the tail on the low side is thin, and a fair number of students reached the top bin. None of that was visible in the row of thirty numbers.

histogram What it is: A chart showing how a set of numerical values is spread across a scale. How it works: The scale is divided into equal intervals called bins. Each value is counted into the bin it falls in, and each bin is drawn as a bar whose height is that count. The bars touch, because the bins run one into the next along an unbroken scale. Used for: Seeing where most of a class’s marks lie and how widely they spread. Watch out: The picture depends on the bin width chosen. Very wide bins flatten the shape into two or three blocks; very narrow bins leave a ragged row of ones and twos.

Q. Would the same marks drawn with five-mark bins give a different picture?

A. Yes, a more detailed one, with ten shorter bars instead of five taller ones. The data has not changed, so the general shape should stay similar. If the shape changes completely when the bin width changes, treat both pictures with caution.

Why the bars touch

The gaps in a bar graph and the absence of gaps in a histogram are not a matter of style. They say something.

In a bar graph, the categories are separate things and there is nothing between them. There is no point “between Science and Mathematics”, so a gap is drawn. In a histogram, the bins are neighbouring stretches of one continuous scale. The mark 29 and the mark 30 are next to each other in reality, so their bars are drawn next to each other with nothing in between.

This also means the bars of a histogram cannot be reordered. Swapping two bars of a bar graph changes nothing; swapping two bars of a histogram would put the scale itself out of order.

Table 17.5 – bar graph vs histogram

Feature Bar graph Histogram
Kind of data on the x-axis categories with names a numerical scale divided into bins
What a bar’s height means the value for that category how many values fall in that bin
Gaps between bars yes no, bars touch
Can the bars be reordered yes, order is only a choice no, the scale fixes the order
Question it answers which category is largest where do the values sit and how far do they spread

Q3. A bar graph and a histogram look almost the same. What really makes them different?

A. What sits on the x-axis. A bar graph puts separate named categories there, and each bar’s height is that category’s value, so the bars are drawn with gaps and could be arranged in any order. A histogram puts a continuous numerical scale there, cut into equal bins, and each bar’s height is a count of how many values landed in that bin. Its bars touch and their order is fixed by the scale.

17.6 Line Graph

A school pays an electricity bill every month. Taken one at a time, the bills say very little — this month’s is higher than last month’s, and that is all anyone notices. The useful question is different: is the school’s usage climbing steadily, or does it rise and fall with the seasons?

That question is about change over time, and the chart built for it is the line graph. Table 17.6 holds the units of electricity the school used in the first eight months of two years running.

Table 17.6 – units of electricity used by the school each month, this year and last year

Month Units used this year Units used last year
January 820 800
February 850 830
March 980 940
April 1240 1150
May 1310 1400
June 1180 1220
July 900 950
August 870 880

A line graph marks each reading as a point, with time along the x-axis and the measured value up the y-axis, and joins neighbouring points with straight line segments. Fig. 17.6-1 draws this year’s column.

Diagram

Fig. 17.6-1 – a line graph titled “Electricity used by the school this year”. The x-axis is labelled “Month” and carries January to August in order; the y-axis is labelled “Units used”, scale 0 to 1400 marked every 200. Eight points are plotted at 820, 850, 980, 1240, 1310, 1180, 900 and 870 and joined by straight segments, with a small dot at each point.

Reading the slope

The points carry the values, but the meaning is in the slope of the segments between them.

A segment rising to the right means the value grew during that gap. A falling segment means it dropped. A nearly flat segment means it barely changed. And the steepness says how fast: the steepest climb in Fig. 17.6-1 is from March to April, which is where the school’s usage grew most sharply in one month.

The overall direction that a line graph reveals — usage climbing through the hot months and easing off afterwards — is called a trend. Spotting a trend is the main reason for drawing a line graph at all.

line graph What it is: A chart showing how one measured quantity changed over a sequence, usually time. How it works: Each reading is plotted as a point against its position in the sequence, and neighbouring points are joined. The height of the line gives the value; the slope between two points gives the change. Used for: Following a school’s monthly electricity usage across a year. Watch out: The joining line is drawn, not measured. Between April and May the school’s usage did not actually travel along a straight line, and no reading exists at those in-between positions.

Q. If the line between two months is only drawn and not measured, why draw it at all?

A. Because it makes the direction and steepness of the change visible at a glance. It is an honest aid as long as the readings are close enough together that the value cannot have done anything wild in between.

Putting two lines on one chart

The real value of Table 17.6 is that it holds two years. Drawing both as lines on the same axes lets each month be compared directly, as in Fig. 17.6-2.

Diagram

Fig. 17.6-2 – a line graph titled “Electricity used by the school, this year and last year”. Same axes as Fig. 17.6-1. Two lines are plotted: a solid line through 820, 850, 980, 1240, 1310, 1180, 900, 870 and a dashed line through 800, 830, 940, 1150, 1400, 1220, 950, 880. A legend at the right identifies the solid line as “this year” and the dashed line as “last year”.

Both lines follow the same seasonal shape, which is expected. What stands out is May, where the two lines cross: last year’s usage peaked higher than this year’s, while in April this year’s was the higher of the two. A single line could never have shown that.

Two or three lines on one chart work well. Seven or eight turn into a tangle in which no line can be followed, and separate small charts become the clearer choice.

using a line graph for named categories What goes wrong: The five subjects of Table 17.3 are plotted as points and joined into a line. The line then slopes downwards from Science to Mathematics, as though borrowing had fallen on the way from one subject to the other. The fix: Join points with a line only when the x-axis has a real order with meaningful gaps, such as time. For separate named categories, use a bar graph, where nothing is drawn between the bars.

17.7 Scatter Plot

The charts so far have each handled one measured quantity. But many interesting questions involve two at once. Does a shop sell more cold drinks on hotter days? Does a robot’s battery drain faster when it carries a heavier load? Do students who revise more score higher?

Each of these asks whether two numerical quantities move together. To see that, both have to be drawn on the same picture, one on each axis.

A scatter plot puts one quantity on the x-axis and the other on the y-axis, and marks each record as a single point at the position given by its two values. The points are not joined. Table 17.7 records, for ten students, the hours spent revising for a test and the marks scored in it.

Table 17.7 – hours of revision and marks scored, for ten students, marks out of 50

Student Hours of revision Marks out of 50
A 1 18
B 2 22
C 2 26
D 3 25
E 4 31
F 4 35
G 5 34
H 6 40
I 7 43
J 8 46

Fig. 17.7-1 plots these ten pairs.

Diagram

Fig. 17.7-1 – a scatter plot titled “Revision hours and marks”. The x-axis is labelled “Hours of revision”, scale 0 to 9 marked every 1; the y-axis is labelled “Marks out of 50”, scale 0 to 50 marked every 10. Ten separate dots are plotted at (1, 18), (2, 22), (2, 26), (3, 25), (4, 31), (4, 35), (5, 34), (6, 40), (7, 43) and (8, 46). The dots are not joined by any line.

The dots do not fall on a neat line, but they clearly drift upwards from left to right. Students who revised longer tended to score higher — tended to, not always: student D revised for three hours and scored less than student C, who revised for two.

Reading the pattern

The word for a tendency of two quantities to move together is correlation. A scatter plot shows both its direction and its strength.

The direction is the slope of the general drift. When the points drift upwards to the right, as in Fig. 17.7-1, high values of one quantity go with high values of the other, and the correlation is positive. When they drift downwards, high values of one go with low values of the other, and the correlation is negative — the longer a phone’s screen stays on, the lower its battery percentage falls. When the points sit in a shapeless cloud with no drift either way, there is no correlation.

The strength is how tightly the points hug that drift. Points strung close to a single line mean a strong relationship, so that knowing one value tells you a good deal about the other. Points spread widely mean a weak one. Fig. 17.7-2 puts the three cases side by side.

Diagram

Fig. 17.7-2 – three small scatter plots in a row, each with plain unnumbered x and y axes and about fifteen dots. The first, captioned “positive correlation”, has dots rising from bottom left to top right in a narrow band. The second, captioned “negative correlation”, has dots falling from top left to bottom right in a narrow band. The third, captioned “no correlation”, has dots scattered evenly with no direction.

scatter plot What it is: A chart for seeing whether two numerical quantities are related. How it works: Each record contributes one point, placed by its value on the x-axis and its value on the y-axis. The direction of the overall drift of the points gives the direction of the relationship, and how tightly they gather gives its strength. Used for: Checking whether daily temperature and cold-drink sales rise and fall together. Watch out: When many records share the same pair of values, their points land on top of one another and a crowded region looks the same as a single dot.

Q. One point sits far away from all the others on a scatter plot. What does it mean?

A. It is an outlier in the relationship — a record that does not follow the pattern the rest do. It may be a recording error, or it may be a genuine and interesting case. Either way it is worth checking before it is explained away.

reading a cause into a scatter plot What goes wrong: An upward drift is announced as proof that one quantity causes the other. Ice-cream sales and cases of sunburn rise together across the year, but ice cream does not cause sunburn. The fix: A scatter plot shows that two quantities move together, and nothing more. The cause may run either way, or a third quantity — here, hot sunny weather — may be driving both. Establishing a cause needs evidence beyond the chart.

Q4. How can a chart show whether two things rise and fall together, such as hours of study and marks scored?

A. By a scatter plot. Each student becomes one point, placed by their revision hours across and their marks up. If the cloud of points drifts upwards to the right, the two rise together; if it drifts downwards, one rises as the other falls; if it has no direction at all, they are unrelated. How tightly the points gather around that drift tells you how dependable the pattern is — but even a strong pattern only shows that the two move together, not that one causes the other.

17.8 Pie Chart

After a school clean-up drive, the waste collected is sorted and weighed. The question the school wants answered is not how many kilograms of paper were collected, but how much of the total the paper accounted for. Was it a third of everything? Half?

Questions of this shape — how a single whole is divided among its parts — are what a pie chart is for. Table 17.8 holds the sorted waste.

Table 17.8 – mass of each type of waste collected in the school clean-up drive

Type of waste Mass (kg)
Paper 15
Plastic 12
Food waste 9
Metal 6
Other 3

Together these come to 15 + 12 + 9 + 6 + 3 = 45 kg. That total is the whole; every category is a share of it.

A pie chart draws the whole as a circle and gives each category a slice whose angle is in proportion to its share. A category holding a quarter of the total gets a quarter of the circle.

Working out the slices

A full circle turns through 360°. So the angle of a slice is that category’s share of the total, taken out of 360°:

\[\text{angle of a slice} = \frac{\text{value of the category}}{\text{total of all categories}} \times 360°\]

Here the value of the category is that type’s mass in kilograms, and the total is 45 kg.

the slice angles for the clean-up drive

Given: Paper 15 kg, plastic 12 kg, food waste 9 kg, metal 6 kg, other 3 kg; total 45 kg.

Find: The angle of each slice of the pie chart.

Working:

Paper: (15 ÷ 45) × 360° = 0.333… × 360° = 120°

Plastic: (12 ÷ 45) × 360° = 0.266… × 360° = 96°

Food waste: (9 ÷ 45) × 360° = 0.2 × 360° = 72°

Metal: (6 ÷ 45) × 360° = 0.133… × 360° = 48°

Other: (3 ÷ 45) × 360° = 0.066… × 360° = 24°

Check: 120° + 96° + 72° + 48° + 24° = 360°

Paper 120°, plastic 96°, food waste 72°, metal 48°, other 24°.

The check at the end is not optional. If the angles do not add up to 360°, either a category has been left out or the arithmetic has slipped, and the chart would be wrong. Fig. 17.8-1 draws the slices.

Diagram

Fig. 17.8-1 – a pie chart titled “Waste collected in the school clean-up drive”. The circle is divided into five slices in different shades, of 120°, 96°, 72°, 48° and 24°, each marked with its angle inside the slice. A legend at the right names the slices in order: Paper, Plastic, Food waste, Metal, Other.

The same shares can be written as percentages instead of angles, by multiplying by 100 rather than 360. Paper is (15 ÷ 45) × 100 = 33.3%, food waste is exactly 20%, and metal is 13.3%. Percentages rounded to one decimal place may add to a little more or less than 100, which is a rounding effect and not a mistake.

pie chart What it is: A circular chart showing how a single total is divided among its categories. How it works: Each category’s share of the total is turned into a slice of the circle, with the slice angle worked out as that share of 360°. The bigger the share, the wider the slice. Used for: Showing what fraction of collected waste was paper, plastic or food waste. Watch out: The eye judges angles poorly. Two slices of 96° and 100° look identical, so a pie chart cannot support close comparisons.

Q. Can a pie chart show how the school’s waste changed from last year to this year?

A. Not on its own. A pie chart is a snapshot of one total at one moment. Change over time needs a line graph, or two pie charts placed side by side and read together.

drawing a pie chart of things that are not parts of one whole What goes wrong: Students are asked which clubs they belong to, and many name two or three. The counts are drawn as a pie chart, so a chart claiming to show a single whole adds up to far more than everyone in the school. The fix: Use a pie chart only when every record belongs to exactly one category and the categories together make up the total. When records can fall into several categories at once, a bar graph is the honest choice.

When a bar graph is better

Because angles are hard to judge, a pie chart earns its place only when there are few categories and their sizes are clearly different, and when the point being made is about shares of a whole. Five slices work. Fifteen thin slivers, each needing its own colour in the legend, do not — and the same fifteen values as a bar graph would be read off in seconds.

Q5. Why must the slices of a pie chart always add up to 360°?

A. Because the circle stands for the whole quantity, and a full turn around a circle is 360°. Each slice is given the same fraction of the circle as its category holds of the total, worked out as (value ÷ total) × 360°. Since the categories together make up the whole total, their fractions add to 1 and their angles add to 360°. If they do not, a category has been missed or the arithmetic is wrong, and the chart no longer represents a whole.

17.9 Choosing the Right Chart

You now have five charts. The mistake at this point is to pick whichever one is easiest to draw, or whichever looks best, and then hunt for something to say about it. That is working backwards.

The chart is chosen by the question. Decide what you want to find out, check what kind of data you have, and the chart usually chooses itself.

Match the question to the chart

Five questions come up again and again during exploration, and each has a chart built for it.

If you want to compare a quantity across named groups — which subject lends the most books — you are comparing categories, and a bar graph does it.

If you want to know how a single set of numerical values is spread — where most of a class’s marks sit — you want a distribution, and a histogram shows it.

If you want to follow something over time — the school’s electricity usage month by month — you want a trend, and a line graph carries it.

If you want to know whether two numerical quantities move together — revision hours against marks — you want a relationship, and a scatter plot reveals it.

And if you want to show how one total divides among its parts — what fraction of the collected waste was plastic — you want shares of a whole, and a pie chart displays them.

Table 17.9 gathers all five together.

Table 17.9 – which chart suits which question

Question you are asking Data you need Chart to use Main weakness
Which group is largest? one categorical column and one numerical column bar graph says nothing about spread within a group
How are the values spread? one numerical column histogram the picture depends on the bin width chosen
How has it changed over time? readings in time order line graph the joining line is drawn, not measured
Do two quantities move together? two numerical columns scatter plot shows a link, never a cause
How does the whole divide up? categories that together form one total pie chart close slice sizes cannot be told apart

choosing a chart for a dataset Input: The question you want answered, and the columns of the dataset. Output: The chart to draw. Steps:

1. Write the question in one sentence.
2. Note how many columns the question involves, and whether each is categorical or numerical.
3. If the question is about shares of one total, choose a pie chart.
4. Else if the question involves one categorical column and one numerical column, choose a bar graph.
5. Else if the question involves one numerical column on its own, choose a histogram.
6. Else if one of the columns is time and the readings are in order, choose a line graph.
7. Else if the question involves two numerical columns, choose a scatter plot.
8. Draw the chart, label both axes with their quantity and unit, and give it a title.

Why it works: Each chart is built to answer one shape of question, so identifying the shape of the question identifies the chart.

Q. Can two different charts suit the same data?

A. Often, yes. The waste figures of Table 17.8 make a fair pie chart and an equally fair bar graph. The pie is better for showing shares of the whole; the bar is better for comparing two similar categories exactly.

More than one chart, not one

During exploration there is no rule that says one dataset gets one chart. The library data of Table 17.3 supports a plain bar graph of one term’s borrowing and a grouped bar graph comparing both terms, and each answers something the other cannot.

Draw several. Exploration is meant to be a search, and a chart that turns out to show nothing is not wasted work — it has told you that there is nothing there to find.

Whatever you choose, label it

The same three habits apply to every chart in this chapter. Give it a title saying what it shows. Label both axes with the quantity and its unit. Start a value axis at zero unless you have a stated reason not to.

A chart without these is not a shortcut. It is an unreadable picture that invites the reader to guess, and guesses made from a chart are trusted far more than they deserve.

For each of these, decide which chart you would draw and write one line saying why: the number of trees of each species in the school grounds; the daily rainfall in your town through July; whether taller students in your class have longer arm spans.

17.10 Summary

  1. Data is raw facts and figures; a dataset is an organised collection of them, usually in rows and columns.
  2. Data exploration is the first careful study of a dataset, done before the data is used for anything further.
  3. It sits between data acquisition and modelling in a project, so anything missed at this stage travels forward into every stage after it.
  4. Exploration answers four things: what the data contains, how the values are spread, what is wrong with it, and how the columns are related.
  5. Common faults found during exploration are missing values, duplicates, inconsistent spellings, and outliers. An outlier is checked before it is removed, because some outliers are real.
  6. Data cleaning is decided after exploration, not before, since until then you do not know what needs cleaning.
  7. Data visualisation means turning data into charts so that patterns can be seen at a glance. A table gives exact values; a chart gives the shape of the values.
  8. Every chart needs a title, labelled axes with units, a sensible scale, and a legend when it shows more than one set of values.
  9. Categorical data sorts records into named groups; numerical data measures or counts a quantity. The kind of data largely decides the chart.
  10. A bar graph compares a quantity across named categories, one bar per category, with gaps between the bars and the value axis starting at zero. Grouped and stacked bars show more than one set at a time.
  11. A histogram shows how numerical values are spread. The scale is cut into equal bins, each bar’s height is the count of values in that bin, and the bars touch because the bins run one into the next.
  12. A line graph shows change over time. Points are plotted in order and joined, and the slope of each segment gives the direction and speed of the change. The overall direction is called a trend.
  13. A scatter plot shows whether two numerical quantities move together. Each record is one point. An upward drift means positive correlation, a downward drift means negative correlation, and a shapeless cloud means none.
  14. A scatter plot shows only that two quantities move together. It never proves that one causes the other.
  15. A pie chart shows how one total divides among its parts. Each slice angle is (value ÷ total) × 360°, and the angles must add up to 360°.
  16. A pie chart suits few categories of clearly different sizes. It cannot show change over time, and close slice sizes cannot be told apart by eye.
  17. The chart is chosen by the question being asked, not by which chart is easiest to draw. During exploration, drawing several charts of the same data is normal.

17.11 Key Terms

  1. Bar graph: a chart that compares a quantity across named categories, one bar per category, with the bar’s length in proportion to the value.

  2. Bin: one of the equal intervals a numerical scale is cut into for a histogram.

  3. Categorical data: data whose values are names or labels placing each record in a group, such as subject or city.

  4. Correlation: a tendency of two numerical quantities to move together, seen in a scatter plot as an upward or downward drift of the points.

  5. Data: raw facts and figures that have been collected, such as marks, prices, or sensor readings.

  6. Data cleaning: fixing the faults found in a dataset, such as filling in or removing missing values, deleting duplicates, and correcting inconsistent entries.

  7. Data exploration: the first careful study of a dataset, done to find out what it contains, what patterns it holds, and what is wrong with it, before the data is used further.

  8. Data visualisation: turning data into charts and graphs so that it can be understood by looking at it.

  9. Dataset: a collection of data organised for use, usually as rows and columns.

  10. Distribution: the way a set of numerical values is spread across a scale — where most of them sit and how far they stretch.

  11. Feature: a column of a dataset that describes something about each record.

  12. Frequency: the number of values that fall into a particular bin.

  13. Histogram: a chart showing the distribution of numerical values, with touching bars whose heights are the frequencies of the bins.

  14. Legend: the small box on a chart that says what each colour or line style stands for.

  15. Line graph: a chart showing change over time, with readings plotted in order and joined by line segments.

  16. Missing value: a place in a dataset where a value should have been recorded but was not.

  17. Model: a system built from data that produces an answer for a new case it is given.

  18. Numerical data: data whose values are measured or counted quantities that can be ordered and used in arithmetic.

  19. Outlier: a value that lies far outside the range of the rest of the data.

  20. Pie chart: a circular chart showing how one total divides among its parts, each slice angle being that part’s share of 360°.

  21. Scatter plot: a chart that marks each record as one point against two numerical quantities, used to see whether the two are related.

  22. Trend: the overall direction in which a quantity is moving over time, as shown by a line graph.

  23. x-axis: the horizontal reference line of a chart.

  24. y-axis: the vertical reference line of a chart.

17.12 Objective Questions

  1. A teacher wants to see how the marks of one class are spread across the scale — where most of them sit and how far they stretch. Which chart is built for this?
    1. Pie chart
    2. Bar graph
    3. Line graph
    4. Histogram
  2. In a pie chart, the angles of all the slices taken together must add up to:
    1. 100°
    2. 180°
    3. 270°
    4. 360°
  3. Which of these is NOT something data exploration looks for in a dataset?
    1. Missing values
    2. Duplicate records
    3. The same name spelt in two different ways
    4. The colours used in the final chart
  4. Fill in the blanks.
    1. The equal intervals into which a numerical scale is divided for a histogram are called ________.
    2. The small box on a chart that says what each colour or line style stands for is the ________.
    3. A value lying far outside the range of the rest of the data is called an ________.
  5. State whether each statement is true or false.
    1. An upward drift of points on a scatter plot proves that one quantity causes the other.
    2. The bars of a histogram are drawn touching one another.
    3. A pie chart is a good way of showing how a quantity changed over twelve months.
  6. Match each question in the left column with the chart that answers it in the right column.
Question Chart
a. Which house team collected the most waste? i. Line graph
b. How did rainfall change month by month? ii. Scatter plot
c. Do heavier school bags belong to taller students? iii. Pie chart
d. What share of the total waste was plastic? iv. Bar graph
  1. A pie chart of a school’s four house teams has slices of 90°, 120° and 60° for three of the houses. Give the angle of the fourth slice, in one line, with your working.

17.13 Concept Questions

  1. Define data exploration, and state which stage of an artificial intelligence project comes immediately before it and which comes immediately after it.

  2. Explain why the cleaning of a dataset is planned after exploration rather than before it.

  3. Give three differences between a bar graph and a histogram.

  4. A student draws a bar graph whose value axis begins at 20 instead of 0. Give a reason why this is poor practice, and say what a reader of the chart is likely to conclude wrongly.

  5. A shopkeeper has recorded the number of items sold on each of the last thirty days and wants to know whether sales climb towards the weekend. Name the chart you would draw, explain why it suits this question, and state one thing that chart cannot tell the shopkeeper.

  6. A school garden’s harvest for one season was: tomatoes 8 kg, brinjal 6 kg, spinach 4 kg, okra 2 kg. Calculate the slice angle for each vegetable in a pie chart of this harvest, showing your working, and check your four answers.

  7. What would happen to the appearance of a histogram of thirty test marks if the bins were made very narrow — say one mark wide? Explain why this makes the distribution harder to judge.

17.14 Case-Based Questions

  1. A school office records the number of students absent in each month of one half of the session.
Month Students absent
July 18
August 22
September 30
October 26
November 15
December 12
  1. Name the chart the office should draw to study how absence changed across these months, and give one reason for your choice.
  2. Work out the largest fall in absence from one month to the next, naming the two months and giving the size of the fall.
  3. State one thing this chart cannot tell the office about student absence.
  1. A shop near a bus stand records the day’s maximum temperature and the number of cold drinks it sells, for eight days.
Day Maximum temperature (°C) Cold drinks sold
1 28 42
2 30 48
3 31 46
4 33 55
5 34 60
6 36 66
7 37 63
8 39 74
  1. Name the chart the shopkeeper should draw to see whether the two quantities are related, and say what would be plotted on each axis.
  2. Describe the pattern these eight pairs would show on that chart, and state clearly what the pattern does not prove.
  3. A ninth row is later found in the register, reading 20 °C and 250 cold drinks sold. Say what kind of value this is, and what should be checked before deciding to remove it.