Chapter 19: Python: Modules and Packages

The Class 10 students of a school in Nagpur spent a month preparing for their science exhibition. Every morning at nine, Karan noted the temperature outside the laboratory. Every afternoon at three, Nisha noted it again. By the end of the month they had sixty readings filling four pages of a notebook.

Then came the hard part. They had to find the average, find the middle value, and draw a graph on chart paper. Karan added the numbers on a calculator three times and got three different answers. Nisha drew the graph twice, because the first one ran off the edge of the page.

Their teacher asked one question that changed how they worked: does anyone else have to do this same job every day? Weather offices do. Hospitals do. A shopkeeper counting each day’s sales does. Because so many people needed the same work done, programmers wrote that code once and shared it with everyone. In Python you can bring that ready-made code into your own program, as shown in Fig. 19.0-1, and the average, the middle value and the graph take three lines instead of three afternoons.

Diagram

Fig. 19.0-1 – a left-to-right flow in three panels. Panel 1: an open notebook with a column of handwritten temperature readings, labelled “data collected by hand”. Panel 2: a laptop screen showing a short Python program, with the top two lines highlighted and labelled “ready-made code brought in”. Panel 3: a printed average value and a small bar chart, labelled “result in seconds”. Arrows join panel 1 to panel 2 to panel 3.

Curiosity Corner

Q1. What exactly is a module, and how is it different from any other Python file? Q2. If modules already keep our code tidy, why do we also need packages? Q3. What does the line import numpy as np actually do? Q4. Why is a NumPy array better than an ordinary Python list for working with numbers? Q5. How does a program turn a plain table of readings into a graph on the screen?

This chapter will answer all five of these questions.

19.1 Modules

19.1.1 Writing the Same Code Again and Again

Think of a shop that prints bills. The code that adds up the items and works out the total is needed in the billing program, in the daily sales report, and in the stock check at closing time. If the same lines are typed into all three programs, the shopkeeper now has three copies of the same code.

That sounds harmless until something changes. Suppose the tax rate changes and the total has to be worked out differently. Now all three programs must be corrected. Correct two of them and forget the third, and the shop quietly starts printing wrong bills.

There is a simpler way. Write the code once, keep it in its own file, and let every program that needs it fetch it from there. Fix the file, and every program is fixed at once. In Python, such a file is called a module.

A module is nothing more than a Python file with the .py extension. Inside it you can keep functions, variables, and any other Python code. A function, as you have seen earlier, is a named block of code that does one job and can be called whenever that job is needed.

Definition

(A module is a single Python file, saved with a .py extension, that holds code such as functions and variables so it can be used again in other programs.)

Fig. 19.1-1 shows the idea: one module, three programs, one copy of the code.

Diagram

Fig. 19.1-1 – a single box in the centre labelled “marks_tools.py (module)” containing two small labels “total()” and “percentage()”. Three boxes on the right labelled “report card program”, “class average program” and “progress chart program”, each joined to the centre box by an arrow pointing from the module to the program, with the word “import” written on each arrow.

19.1.2 Creating a Module

You do not need any special command to create a module. You open a new file, write the code, and save it with a .py name. That is all.

Suppose a school wants to work with students’ marks. Two jobs come up again and again: adding the marks of all subjects, and turning that total into a percentage. Put both into a file called marks_tools.py.

###### marks_tools.py - small helpers for working with marks {.unnumbered .unlisted}


def total(marks):
    # marks is a list of numbers; return their sum
    return sum(marks)

def percentage(marks, full_marks):
    # return the percentage scored, to one decimal place
    return round(total(marks) / full_marks * 100, 1)

Save this file. Nothing happens when you save it, and nothing is printed. A module is not meant to be run on its own — it is meant to be used by another program.

Curiosity Corner

Q1. What exactly is a module, and how is it different from any other Python file?

A. It is not different at all in the way it is written. Every Python file saved with a .py extension is a module. The word “module” describes how the file is used — as a store of code that other programs bring in, rather than as a program you run directly.

19.1.3 Importing a Module

To use a module in another program, you bring it in with an import statement. The keyword import followed by the module’s name (without the .py part) makes everything inside that file available.

Once imported, you reach a function inside the module by writing the module name, a dot, and the function name. This is called dot notation. Save the program below as a separate file in the same folder as marks_tools.py.

Program

printing a student’s total and percentage Problem: A student has scored marks in five subjects, each out of 100. Print the total and the percentage using the module already written.

import marks_tools

marks = [72, 65, 88, 91, 54]

print("Total:", marks_tools.total(marks))
print("Percentage:", marks_tools.percentage(marks, 500))

Output:

Total: 370
Percentage: 74.0

There are two other useful forms of the import statement. Writing from marks_tools import percentage brings in only that one function, and you can then call it as plain percentage() without the module name in front. Writing import marks_tools as mt gives the module a short nickname, which saves typing when the name is long.

Program

using the two shorter import forms Problem: The same five subjects are now marked out of 50 each. Print the percentage using a direct import, and the total using a nickname for the module.

from marks_tools import percentage
import marks_tools as mt

marks = [40, 48, 35, 44, 39]

print(percentage(marks, 250))
print(mt.total(marks))

Output:

82.4
206

Why the module name is kept in front

Dot notation looks like extra typing, but it earns its place. Two different modules may each contain a function called total(). Because you call them as marks_tools.total() and, say, bill_tools.total(), Python is never in doubt about which one you mean. Each module has its own namespace — its own private list of names — so names in one module cannot clash with names in another.

Common Error

naming your own file after a built-in module What goes wrong: A student saves practice code in a file called math.py. Later, in the same folder, import math fetches that practice file instead of Python’s own maths module, and math.sqrt() fails. The fix: Never name your file after a module Python already provides. Call it my_math.py or math_practice.py instead.

Try Yourself

Add a function highest(marks) to marks_tools.py that returns the largest mark in the list. Import it into a new program and print the highest mark for a list of your own.

19.1.4 Built-in Modules and User-Defined Modules

Modules come from two places. Some you write yourself. Others arrive with Python itself.

A built-in module is one that comes with every Python installation. You do not download it or install it — you simply import it. Together, these modules form Python’s standard library. Some you will meet often are math for mathematical work, random for random numbers, datetime for dates and times, and os for talking to the operating system.

Program

using the built-in math module Problem: Print the square root of 144, the whole-number part of 3.7, and the value of pi.

import math

print(math.sqrt(144))
print(math.floor(3.7))
print(math.pi)

Output:

12.0
3
3.141592653589793

Notice that math.sqrt(144) prints 12.0 and not 12. Square root always gives a decimal number in Python, even when the answer is a whole number.

A user-defined module is one you create yourself, exactly as marks_tools.py was created. Python treats it in the same way as a built-in module; the only difference is where it came from and who is responsible for the code inside it.

Compare Table

Table 19.1 – Built-in modules vs user-defined modules

Feature	Built-in module	User-defined module
Who wrote it	The Python developers	You, or someone on your team
Comes with Python	Yes	No
Installation needed	None	None; the file just has to be findable
Where it is stored	With the Python installation	Usually in the same folder as your program
Examples	`math`, `random`, `datetime`, `os`	`marks_tools`, `bill_tools`
Can you change it	Better not to	Yes, it is your file

19.1.5 Benefits of Modules

Splitting code into modules is a habit, and like most good habits it pays back slowly. Four benefits are worth naming.

Code is reused, not copied. Once marks_tools.py exists, every new program that needs a percentage imports it. There is only ever one copy of that code.

Large programs stay readable. A single file of two thousand lines is hard to move around in. The same code split into eight modules, each with a clear name, is far easier to search.

Names do not collide. Because each module has its own namespace, a variable called count inside one module has nothing to do with a variable called count inside another.

Repairs happen in one place. If the percentage calculation has a mistake, you correct marks_tools.py once. Every program that imports it is corrected the moment it next runs.

There is an honest limit to all this. Splitting code too finely creates its own problem: twenty tiny modules, each with one small function, can be harder to work with than three sensible ones. Also, any code written at the top level of a module — outside every function — runs the moment the module is imported, which can surprise you if you left a stray print() there. Group code by the job it does, and keep the top level of a module clear.

19.2 Packages

19.2.1 What a Package Is

Modules solve the problem of repeated code. They create a smaller problem of their own.

Imagine the school office has now written twenty modules — some for attendance, some for fees, some for the timetable, some for the library. All twenty files sit loose in one folder, mixed with the programs that use them. Finding the right one means reading twenty file names.

You already know the answer from everyday life. Papers that belong together go into one folder, and folders go into one cupboard. Python does exactly the same thing: related modules are kept in a folder, and that folder is called a package.

Definition

(A package is a folder that holds related modules together, along with a special file named __init__.py that tells Python the folder is to be treated as a package.)

The __init__.py file is what turns an ordinary folder into a package. It may be completely empty — its presence alone is the signal. It can also hold set-up code that should run whenever anything from the package is first imported.

Fig. 19.2-1 shows a small package called school_tools.

Diagram

Fig. 19.2-1 – a folder tree drawn as nested boxes. A folder icon labelled “school_tools/” at the top. Indented under it, three file icons labelled “init.py”, “attendance.py” and “fees.py”. A short note beside “init.py” reads “marks the folder as a package”.

Curiosity Corner

Q2. If modules already keep our code tidy, why do we also need packages?

A. A module tidies the code inside one file. Once a project grows to dozens of modules, the files themselves need tidying. A package groups related modules into one folder, so the work of the attendance system stays separate from the work of the fees system, and each group can be found, moved or shared as a whole.

19.2.2 Building a Package Step by Step

Creating a package is a matter of making a folder and putting the right files inside it. Here is the method in words before you try it on a computer.

Algorithm

creating a package Input: The modules you want to group together. Output: A folder Python will treat as a package. Steps:

1. Create a folder in the same place as the program that will use it.
2. Give the folder a short, lower-case name, such as school_tools.
3. Inside the folder, create an empty file named __init__.py.
4. Save each module file inside the same folder.
5. Keep the program that uses the package outside the folder.

Why it works: Python looks for __init__.py when it is asked to import a folder; finding it, Python knows the folder is a package and searches inside it for the module you named.

Now the two modules. The first, attendance.py, works out how many days a student was present.

###### school_tools/attendance.py - attendance calculations {.unnumbered .unlisted}


def days_present(record):
    # record is a list of "P" and "A" marks
    return record.count("P")

def attendance_percent(record):
    return round(days_present(record) / len(record) * 100, 1)

The second, fees.py, works out what is still owed.

###### school_tools/fees.py - fee calculations {.unnumbered .unlisted}


def balance(total_fee, paid):
    return total_fee - paid

Save both inside the school_tools folder, beside the empty __init__.py.

Common Error

writing the special file name with single underscores What goes wrong: The file is saved as _init_.py. Python does not recognise it, the folder stays an ordinary folder, and the import fails. The fix: The name carries two underscores on each side — __init__.py. Check it character by character before saving.

19.2.3 Importing from a Package

Importing from a package works like importing a module, with one extra step: you say which package the module lives in. The program below is saved outside the school_tools folder.

Program

an attendance and fees summary Problem: For a student marked present on 8 of 10 days, print the days present and the attendance percentage. The student’s total fee is ₹24,000 and ₹18,000 has been paid; print the balance.

from school_tools import attendance, fees

record = ["P", "P", "A", "P", "P", "P", "A", "P", "P", "P"]

print(attendance.days_present(record))
print(attendance.attendance_percent(record))
print(fees.balance(24000, 18000))

Output:

8
80.0
6000

You could also write import school_tools.attendance and then call the function in full as school_tools.attendance.days_present(record). Both work. The from form is used more often simply because it keeps the lines shorter.

A package can also hold another package inside it. Such an inner folder is called a sub-package, and it too needs its own __init__.py. Fig. 19.2-2 shows the same package after a reports sub-package has been added.

Diagram

Fig. 19.2-2 – the same folder tree as Fig. 19.2-1, now with a fourth item indented under “school_tools/”: a folder icon labelled “reports/”. Indented one level further under “reports/” are two file icons, “init.py” and “termwise.py”. An arrow points to the import line written alongside: “from school_tools.reports import termwise”.

Each extra level adds one more name and one more dot to the import line. Two levels are common; three are unusual; beyond that, the import lines become long enough to be a nuisance in themselves.

19.2.4 Benefits of Packages

The project has a shape. Anyone opening the folder can see at a glance which part of the code does what, without reading a single line inside the files.

Names stay apart. A package creates its own namespace, just as a module does. Two modules in different packages may each define a function named balance(), and calling them as school_tools.fees.balance() and hostel_tools.mess.balance() keeps them separate.

Whole groups can be reused. A single module can be copied into another project. A package lets you copy the entire attendance system — every module it needs — in one move.

Related code is kept together. When the fee rules change, you know exactly which folder to open. Nothing outside it needs to be touched.

The honest limitation is that a package binds its modules to a folder path. Rename or move the folder and every import line that mentions it stops working, so the structure is worth thinking about once, at the start, rather than rearranging it later.

19.3 Libraries

19.3.1 What a Library Is

So far, all the code has been yours. You wrote the modules, you grouped them into a package, and if something inside them was wrong, you fixed it.

Now think about the work a weather office does with its readings — averages, spreads, graphs. Thousands of people need exactly that work done, and the calculations are the same for all of them. It would be strange for every one of those people to write the same code from scratch, and stranger still for each of them to make their own separate mistakes in it.

So programmers write such code once, test it carefully, and share it. A collection of ready-made code shared in this way is called a library.

Definition

(A library is a collection of ready-written code — usually one or more packages and modules — that other programmers can bring into their own programs instead of writing that code themselves.)

Fig. 19.3-1 shows how the three ideas of this chapter sit inside one another.

Diagram

Fig. 19.3-1 – three nested rounded rectangles, largest on the outside. The outermost is labelled “Library”. Inside it, a smaller one labelled “Package (folder)”. Inside that, a smaller one labelled “Module (.py file)”. Inside the innermost, three small ovals labelled “function”, “function”, “variable”.

Libraries reach you in two ways. Some are already part of Python — the modules of the standard library, such as math and random, need no downloading at all. The rest are written by programmers around the world and are called third-party libraries. They are collected on a website called the Python Package Index (PyPI), and you fetch them from there when you need them.

Four third-party libraries matter for this chapter, and each has one clear job:

NumPy — fast work with large collections of numbers.
Pandas — handling data arranged in rows and columns, like a spreadsheet.
Matplotlib — drawing graphs and charts.
SciPy — scientific and mathematical calculations built on top of NumPy.

19.3.2 Advantages of Using Libraries

Work already done is not done again. Drawing a bar chart from scratch would mean working out where every bar goes, how tall it is, and how to put numbers along the edges. Matplotlib does it in one line.

The code has been tested by many people. A library used by lakhs of programmers has had its mistakes found and corrected. Code you wrote last night has been tested by one person.

Everyone writes it the same way. Because most people use the same libraries for the same jobs, another programmer can read your program and follow it, even if they have never seen your project before.

Hard things become possible. Some calculations are simply too involved to write by hand at this level. Solving a pair of equations, or finding the spread of a thousand readings, is a single line once the right library is in place.

The limitation is worth stating plainly: your program now depends on somebody else’s code. If a library changes the way a function behaves in a later version, a program that worked last year may stop working. For this reason, install only well-known libraries, and prefer the ones that are widely used and actively looked after.

19.3.3 Installing a Library with pip

A third-party library is not on your computer until you fetch it. The tool that fetches it is called pip, the installer that comes with Python. The name is usually read as a recursive acronym: pip installs packages.

pip is not typed into a Python program. It is typed at the command prompt or terminal — the plain text window where you give instructions to the computer itself.

First, check that pip is there.

pip --version

If Python is installed correctly, this prints the version of pip and the folder it is running from. On Windows, if pip alone is not recognised, use py -m pip --version instead; both do the same job.

Then install what you need. A library is installed once per computer, not once per program.

pip install numpy

pip goes to the Python Package Index, downloads the library, and puts it where Python can find it. You can name several libraries at once.

pip install numpy pandas matplotlib scipy

To see whether a library is already installed, ask pip about it.

pip show numpy

Common Error

typing pip commands inside Python What goes wrong: pip install numpy is typed at the >>> prompt inside Python, and Python reports a syntax error. The fix: pip commands belong at the command prompt or terminal, not inside the Python shell. Close the shell, or open a separate terminal window, and type the command there.

Bringing a library into your program

Once a library is installed, you import it exactly as you imported your own modules. Almost every Python programmer uses the same short nicknames for these four libraries, and you should use them too, so that others can read your code.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats

The third line needs a word of explanation. Matplotlib is a large library, and pyplot is one module inside it — the one that draws plots. So the line reaches into the library, picks out that module, and gives it the nickname plt.

Curiosity Corner

Q3. What does the line import numpy as np actually do?

A. It tells Python to find the installed NumPy library, load it, and make everything inside it available under the short name np. After that line, np.mean() means “the mean function that belongs to NumPy”. The nickname saves typing and, because nearly everyone chooses np, it also makes the program easier for others to read.

19.3.4 Module, Package and Library Compared

The three words are often used loosely in conversation, but they mean different things.

Compare Table

Table 19.2 – Module, package and library

Feature	Module	Package	Library
What it is	A single Python file	A folder of related modules	A collection of packages and modules shared for others to use
Size of the job	One small area of work	One part of a project	A whole field of work, such as charts or data handling
Marked by	The `.py` extension	An `__init__.py` file inside the folder	Nothing special; it is a name given to what is shared
How you get it	Written by you, or built into Python	Written by you, or part of a library	Built into Python, or installed with pip
Examples	`math`, `marks_tools`	`school_tools`, `matplotlib.pyplot`	NumPy, Pandas, Matplotlib, SciPy

A short way to remember it: a module is a file, a package is a folder of files, and a library is a body of work someone has shared with the world.

19.4 NumPy

19.4.1 The NumPy Array

Suppose a school records the midday temperature every day for a year. That is 365 numbers. You want to add five degrees to each of them, or find their average, or find how spread out they are.

With an ordinary Python list you would write a loop, take the numbers out one at a time, do the arithmetic, and put the answers back. It works, but for a long list it is slow, and the code is longer than the idea it expresses.

NumPy, short for Numerical Python, exists for exactly this problem. Its central idea is a container called the NumPy array, or ndarray. An array looks like a list, but it is built for numbers and for doing the same operation to all of them at once.

An array differs from a list in one important way: every element must be of the same type. All whole numbers, or all decimal numbers, but not a mixture. That single restriction is what lets NumPy store the values tightly in memory and work through them quickly.

Definition

(A NumPy array is a container that holds many values of the same type, arranged in one or more dimensions, and allows the same operation to be applied to all of them at once.)

Program

making a NumPy array from a week of readings Problem: Store five midday temperature readings in a NumPy array and show what kind of object it is and how many values it holds.

import numpy as np

readings = np.array([31, 33, 30, 34, 32])

print(readings)
print(type(readings))
print(readings.shape)

Output:

[31 33 30 34 32]
<class 'numpy.ndarray'>
(5,)

Two things are worth noticing. The array prints without commas, which is how NumPy shows arrays. And shape reports (5,) — five values along one dimension.

19.4.2 Creating, Indexing and Slicing Arrays

You will not always have the numbers ready to type in. NumPy gives you short ways to build arrays of a chosen size, which is useful when a program has to fill them in later.

Program

building arrays without typing every value Problem: Create a 2-row, 3-column array of zeros, a 3-row, 2-column array of ones, a counting array from 1 to 9, and the same counting array folded into a 3 by 3 grid.

import numpy as np

zeros = np.zeros((2, 3))
ones = np.ones((3, 2))
counting = np.arange(1, 10)
grid = counting.reshape(3, 3)

print(zeros)
print(ones)
print(counting)
print(grid)

Output:

[[0. 0. 0.]
 [0. 0. 0.]]
[[1. 1.]
 [1. 1.]
 [1. 1.]]
[1 2 3 4 5 6 7 8 9]
[[1 2 3]
 [4 5 6]
 [7 8 9]]

The dots after the zeros and ones are not decoration. np.zeros() and np.ones() produce decimal numbers, so 0 is printed as 0. and 1 as 1..

An array with rows and columns, like grid, is a two-dimensional array. This is how NumPy holds a table of data, and it is what makes the library useful for anything laid out in a grid.

Common Error

expecting np.arange to include the last number What goes wrong: A student writes np.arange(1, 10) and expects ten values ending in 10. Only nine appear, ending in 9. The fix: np.arange stops just before the second number. For 1 to 10, write np.arange(1, 11).

Picking values out of an array

Reading a value out of an array works exactly as it does with a list. Counting starts at 0, and a negative position counts back from the end. A slice [a:b] gives you the values from position a up to, but not including, position b. Fig. 19.4-1 shows the positions for a week of rainfall readings.

Diagram

Fig. 19.4-1 – a horizontal row of seven boxes containing the values 12, 0, 45, 30, 8, 0, 22. Below each box, the index numbers 0 to 6 in one row, and the negative indices -7 to -1 in a second row. Above boxes 2, 3 and 4, a curly bracket labelled “rainfall[2:5]”.

Program

reading parts of a rainfall array Problem: From seven daily rainfall readings in millimetres, print the first reading, the last reading, the readings for days three to five, and the first three readings.

import numpy as np

rainfall = np.array([12, 0, 45, 30, 8, 0, 22])

print(rainfall[0])
print(rainfall[-1])
print(rainfall[2:5])
print(rainfall[:3])

Output:

19.4.3 Element-wise Operations on Arrays

Here is where NumPy earns its place. Suppose every student in a class is to be given two grace marks. With a list, you would write a loop:

marks = [68, 74, 55, 90, 81]
new_marks = []
for m in marks:
    new_marks.append(m + 2)

Four lines, and the idea “add two to everything” is buried inside them. With a NumPy array, you write what you mean:

new_marks = marks + 2

When an operation is applied to every value of an array in this way, it is called an element-wise operation. NumPy also matches two arrays position by position: the first value with the first, the second with the second, and so on.

Program

adjusting marks with element-wise arithmetic Problem: Five students have scored marks out of 100. Each student is to receive a different number of grace marks. Print the adjusted marks, then the marks doubled, then the marks reduced by five.

import numpy as np

marks = np.array([68, 74, 55, 90, 81])
grace = np.array([2, 1, 5, 0, 3])

print(marks + grace)
print(marks * 2)
print(marks - 5)

Output:

[70 75 60 90 84]
[136 148 110 180 162]
[63 69 50 85 76]

The two arrays being added must be the same length. If they are not, NumPy cannot decide which value goes with which, and the program stops with an error.

19.4.4 Statistics with NumPy

Once the data is in an array, the usual summary figures are one function call each. The mean, or average, is the familiar one:

\(\text{mean} = \dfrac{\text{sum of all the values}}{\text{number of values}}\)

The median is the middle value once the data is sorted, and with an even number of values it is the average of the middle two. The variance measures how far the values sit from the mean, on average, and the standard deviation is the square root of the variance. A small standard deviation means the values huddle near the mean; a large one means they are spread out. The range is simply the largest value minus the smallest.

Let us work these out by hand first, so that the program’s answers can be checked rather than trusted.

Example 19.1

summary figures for eight quiz scores

Given: Eight students score 2, 4, 4, 4, 5, 5, 7 and 9 marks out of 10.

Find: The mean, median, variance, standard deviation and range.

Working:

Sum of the scores = 2 + 4 + 4 + 4 + 5 + 5 + 7 + 9 = 40 marks.

Mean = 40 ÷ 8 = 5 marks.

Sorted, the scores are 2, 4, 4, 4, 5, 5, 7, 9. The two middle values are the 4th and 5th, that is 4 and 5, so the median = (4 + 5) ÷ 2 = 4.5 marks.

Distances from the mean: −3, −1, −1, −1, 0, 0, 2, 4.

Their squares: 9, 1, 1, 1, 0, 0, 4, 16, which add up to 32.

Variance = 32 ÷ 8 = 4 (marks squared).

Standard deviation = \(\sqrt{4}\) = 2 marks.

Range = 9 − 2 = 7 marks.

Mean 5 marks, median 4.5 marks, variance 4, standard deviation 2 marks, range 7 marks.

Now the same figures from NumPy.

Program

summary figures for a set of quiz scores Problem: Print the mean, median, variance, standard deviation and range of the eight quiz scores.

import numpy as np

scores = np.array([2, 4, 4, 4, 5, 5, 7, 9])

print("Mean:", np.mean(scores))
print("Median:", np.median(scores))
print("Variance:", np.var(scores))
print("Standard deviation:", np.std(scores))
print("Range:", np.ptp(scores))

Output:

Mean: 5.0
Median: 4.5
Variance: 4.0
Standard deviation: 2.0
Range: 7

Every figure matches the hand working. np.ptp is short for “peak to peak”; you could equally write np.max(scores) - np.min(scores) and get the same 7.

One summary figure is missing from that list. The mode — the value that occurs most often — has no direct NumPy function. For that you need SciPy, which you will meet in Section 19.7.

19.4.5 NumPy Arrays and Python Lists Compared

A list and an array both hold many values, so it is fair to ask when each should be used.

Compare Table

Table 19.3 – NumPy array vs Python list

Feature	NumPy array	Python list
Types of value	All values must be of the same type	Numbers, text and other types can be mixed
Arithmetic	Applied to every value at once, as in `arr * 2`	Needs a loop to reach each value
Rows and columns	Built in; an array can have two or more dimensions	Only by putting lists inside lists
Speed on large data	Fast, because values are stored together in memory	Slower, because each value is handled separately
Size	Fixed once created, though it can be reshaped	Items can be added or removed freely
Ready-made functions	Many, for statistics and mathematics	Few; other libraries are needed
Installation	Needs NumPy to be installed	Part of Python itself

Neither is better in every situation. For a short list of mixed items — a few names, a few marks — a Python list is simpler and perfectly suitable. For a large set of numbers that has to be measured or calculated with, an array is the right tool.

Curiosity Corner

Q4. Why is a NumPy array better than an ordinary Python list for working with numbers?

A. Because every value in an array is of the same type, NumPy can store them close together in memory and apply one operation to all of them at once. This makes the code shorter — marks + 2 instead of a loop — and much faster on large sets of data. NumPy also brings ready-made functions such as mean, median and std, which a list does not have.

19.5 Pandas

19.5.1 The Series

NumPy is happy as long as the data is plain numbers. Real records are rarely so tidy. A school’s marks register has names, subjects, sections and cities alongside the numbers, arranged in rows and columns — the shape of a spreadsheet, not the shape of an array.

Pandas is the library built for that kind of data. It gives you two containers. The simpler one is the Series.

A Series is a single column of values with a label attached to each one. Those labels together are called the index. If you do not supply labels, Pandas numbers the values 0, 1, 2 and so on.

Definition

(A Series is a one-dimensional set of values in Pandas, where every value carries a label, called its index.)

Program

a Series of subject marks Problem: Store one student’s marks in four subjects, first without labels and then labelled by subject, and read out the Maths mark.

import pandas as pd

marks = pd.Series([82, 76, 91, 68])
print(marks)

subjects = pd.Series([82, 76, 91, 68],
                     index=["English", "Hindi", "Maths", "Science"])
print(subjects)
print(subjects["Maths"])

Output:

0    82
1    76
2    91
3    68
dtype: int64
0
English    82
Hindi      76
Maths      91
Science    68
dtype: int64
91

The last line of each printed Series, dtype: int64, is Pandas telling you what kind of values it is holding — here, whole numbers.

Notice the difference the labels make. Without them you must remember that position 2 is Maths. With them you simply ask for subjects["Maths"].

19.5.2 The DataFrame

One column is rarely enough. A DataFrame puts several Series side by side, sharing one index, which gives you a table with named columns and numbered rows. Fig. 19.5-1 shows the relation between the two.

Diagram

Fig. 19.5-1 – on the left, a single vertical strip of four boxes with labels beside them, headed “Series (one labelled column)”. On the right, a table of four rows and three columns with the column headings “Name”, “Marks”, “City” along the top and the row numbers 0 to 3 down the left, headed “DataFrame (several Series sharing one index)”. A dashed arrow runs from the left strip to the “Marks” column of the table.

Definition

(A DataFrame is a two-dimensional table in Pandas, made of labelled columns and an index of rows, where each column may hold a different kind of value.)

The usual way to build one is from a dictionary. Each key becomes a column heading, and the list under it becomes that column’s values. All the lists must be the same length, because they have to fill the same number of rows.

Program

a DataFrame of student records Problem: Build a table of four students holding their name, marks and city, and print the table along with the number of rows and columns.

import pandas as pd

data = {
    "Name": ["Aarti", "Farhan", "Ishita", "Rohit"],
    "Marks": [82, 76, 91, 68],
    "City": ["Nagpur", "Kochi", "Patna", "Surat"]
}

students = pd.DataFrame(data)

print(students)
print(students.shape)

Output:

     Name  Marks    City
0   Aarti     82  Nagpur
1  Farhan     76   Kochi
2  Ishita     91   Patna
3   Rohit     68   Surat
(4, 3)

The numbers 0 to 3 down the left are the index. Pandas added them because none were given. shape reports (4, 3) — four rows and three columns, in that order.

19.5.3 Selecting and Adding Columns

A table is only useful if you can pull pieces out of it. Naming one column in square brackets gives you that column back as a Series. Naming a list of columns gives you a smaller DataFrame.

Program

picking columns out of a table Problem: From the student table, print only the names, and then only the names with their marks.

print(students["Name"])
print(students[["Name", "Marks"]])

Output:

0     Aarti
1    Farhan
2    Ishita
3     Rohit
Name: Name, dtype: object
     Name  Marks
0   Aarti     82
1  Farhan     76
2  Ishita     91
3   Rohit     68

The single pair of brackets gives a Series, and the line Name: Name, dtype: object tells you its name and that it holds text. The double pair gives a DataFrame, which is why it prints as a table.

Adding a column is just as direct. Assign a single value and every row receives it. Assign a list and the values are taken in order. Assign a calculation on an existing column and Pandas works it out for the whole column at once, exactly as NumPy does with arrays.

Program

adding columns to a table Problem: Add a Class column holding 10 for every student, and an Adjusted column holding each student’s marks plus five grace marks.

students["Class"] = 10
students["Adjusted"] = students["Marks"] + 5

print(students)

Output:

     Name  Marks    City  Class  Adjusted
0   Aarti     82  Nagpur     10        87
1  Farhan     76   Kochi     10        81
2  Ishita     91   Patna     10        96
3   Rohit     68   Surat     10        73

Rows can be picked out too, by writing a condition inside the brackets. Pandas checks the condition for every row and keeps only the rows where it is true.

print(students[students["Marks"] > 80])

Output:

     Name  Marks    City  Class  Adjusted
0   Aarti     82  Nagpur     10        87
2  Ishita     91   Patna     10        96

The index numbers 0 and 2 come along unchanged. That is deliberate — it tells you which rows of the original table these were.

19.5.4 Reading Data from CSV and JSON Files

Typing data into a program is fine for four students. For four thousand, the data has to come from a file.

The commonest such file is the CSV file, short for comma-separated values. It is an ordinary text file in which each line is one row of the table and the values in that row are separated by commas. The first line usually holds the column headings. Fig. 19.5-2 shows a CSV file beside the table it stands for.

Diagram

Fig. 19.5-2 – on the left, a plain text file icon showing four lines: “Item,Price,Sold”, “Notebook,45,30”, “Pen,10,120”, “Eraser,5,60”. On the right, the same information drawn as a three-column table with the headings Item, Price and Sold. A wide arrow labelled “pd.read_csv()” points from left to right.

Pandas can write a DataFrame out to a CSV file and read one back in. The program below does the writing.

Program

saving a stationery sales table to a CSV file Problem: Build a table of six stationery items with their price in rupees and the number sold, and save it as stationery.csv.

import pandas as pd

data = {
    "Item": ["Notebook", "Pen", "Eraser", "Scale", "Pencil", "Sharpener"],
    "Price": [45, 10, 5, 15, 8, 12],
    "Sold": [30, 120, 60, 25, 90, 40]
}

sales = pd.DataFrame(data)
sales.to_csv("stationery.csv", index=False)

Output: nothing is printed. A file named stationery.csv appears in the same folder as the program. index=False tells Pandas not to write the row numbers into the file, since they are not part of the data.

Reading it back needs one function.

Program

reading the sales file and looking at the top of it Problem: Read stationery.csv into a DataFrame, print its first five rows, and print how many rows and columns it holds.

import pandas as pd

sales = pd.read_csv("stationery.csv")

print(sales.head())
print(sales.shape)

Output:

       Item  Price  Sold
0  Notebook     45    30
1       Pen     10   120
2    Eraser      5    60
3     Scale     15    25
4    Pencil      8    90
(6, 3)

head() shows the first five rows, which is usually enough to check that the file has loaded correctly. Pass it a number, as in head(3), to see a different count, and use tail() for the last rows instead. A third method, info(), prints a short report of the column names, how many values each holds and what type they are.

There is a second file format worth knowing. JSON, short for JavaScript Object Notation, stores data as labelled pairs rather than as rows of commas, which makes it the usual format for sending data between a website and the program behind it. Pandas reads it with pd.read_json("stationery.json") and writes it with to_json(), and the DataFrame you get back behaves exactly the same afterwards.

19.5.5 Handling Missing Data

Real data has holes in it. A student was absent on the day of the test. A form was submitted with the city left blank. Pandas marks every such hole with a special value called NaN, short for “not a number”.

The first job is to find out how many holes there are and where. isnull() checks every cell and answers true or false, and adding those answers up column by column gives you a count.

Program

finding the missing values in a report table Problem: Build a table of four students in which one mark and one city are missing, print it, and count the missing values in each column.

import pandas as pd
import numpy as np

data = {
    "Name": ["Aarti", "Farhan", "Ishita", "Rohit"],
    "Marks": [82, np.nan, 91, 68],
    "City": ["Nagpur", "Kochi", np.nan, "Surat"]
}

report = pd.DataFrame(data)

print(report)
print(report.isnull().sum())

Output:

     Name  Marks    City
0   Aarti   82.0  Nagpur
1  Farhan    NaN   Kochi
2  Ishita   91.0     NaN
3   Rohit   68.0   Surat
Name     0
Marks    1
City     1
dtype: int64

Notice that the marks are now printed as 82.0 rather than 82. NaN is a decimal value, so the whole column has become a column of decimals.

Once you know where the holes are, there are two things you can do about them.

Drop the rows. dropna() removes every row that has a missing value anywhere in it. It is the honest choice when the missing entries are few and the rest of the data is complete.

print(report.dropna())

Output:

    Name  Marks    City
0  Aarti   82.0  Nagpur
3  Rohit   68.0   Surat

Half the class has disappeared, which shows the cost of this method. Dropping is safe on a large table and wasteful on a small one.

Fill the holes. fillna() puts a value of your choosing into each gap. A missing mark might be filled with the average of the marks that are present; a missing city might be filled with a plain label.

report["Marks"] = report["Marks"].fillna(round(report["Marks"].mean(), 1))
report["City"] = report["City"].fillna("Not given")

print(report)

Output:

     Name  Marks       City
0   Aarti   82.0     Nagpur
1  Farhan   80.3      Kochi
2  Ishita   91.0  Not given
3   Rohit   68.0      Surat

The average of 82, 91 and 68 is 241 ÷ 3 = 80.33, which rounds to 80.3, and that is what has gone into Farhan’s row. Pandas also offers ffill(), which copies the value from the row above into a gap — useful for readings taken at regular intervals, where the previous reading is a reasonable stand-in.

Ethics Watch

Filling a gap invents a value that was never measured. Farhan’s 80.3 is not his mark; it is a guess made from other students’ marks, and a report card built on it would be unfair to him. Whenever missing values are filled in, say so alongside the results, and never fill in data that will be used to make a decision about a particular person.

Common Error

expecting fillna to change the table by itself What goes wrong: A student writes report.fillna(0), prints the table, and finds the gaps still there. The fix: Pandas returns a corrected copy rather than altering the original. Store the result back, as in report = report.fillna(0), or assign it to the column you were correcting.

19.6 Matplotlib

19.6.1 Line Plots, Labels and Titles

Here are the numbers of books issued from a school library in the first six months of a year: 120, 145, 130, 160, 155, 180. Read them again and try to say, without counting, whether borrowing is rising or falling and where it dipped.

It is hard work, and there are only six numbers. Draw them as a line and the answer takes half a second. Numbers tell you the amounts; a chart tells you the shape.

Matplotlib is the library that draws those charts. Almost all of its drawing tools live in one module inside it, pyplot, which is why the import line reaches in and gives that module the short name plt.

Building a chart follows the same four steps every time: give Matplotlib the data, label the picture, and tell it to display.

Algorithm

drawing any Matplotlib chart Input: Two lists — what goes along the bottom, and the values to be plotted. Output: A chart on the screen. Steps:

1. Import matplotlib.pyplot with the short name plt.
2. Put the data into lists.
3. Call the drawing function for the chart you want, such as plt.plot.
4. Add labels for the two axes and a title.
5. Call plt.show() to display the finished chart.

Why it works: each plt call adds one more instruction to a picture Matplotlib is holding in memory; plt.show() is the moment it puts that picture on the screen.

Program

a line chart of library books issued Problem: Draw a line chart of the books issued each month from January to June, with both axes labelled and a title.

import matplotlib.pyplot as plt

months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun"]
books = [120, 145, 130, 160, 155, 180]

plt.plot(months, books, color="green", marker="o")
plt.xlabel("Month")
plt.ylabel("Books issued")
plt.title("Library books issued each month")
plt.show()

Output: nothing is printed. A window opens showing the chart in Fig. 19.6-1 — a green line rising overall from January to June, with a small round marker at each month and a dip at March.

Diagram

Fig. 19.6-1 – line chart from the code above; months Jan to Jun on the x-axis labelled “Month”, books issued on the y-axis labelled “Books issued”, a green line with round markers, titled “Library books issued each month”

The color and marker settings are optional. Leave them out and Matplotlib draws a plain blue line with no markers. Two more optional calls are worth knowing: plt.grid() adds faint grid lines behind the chart so values are easier to read off, and plt.legend() adds a small box naming each line, which matters once a chart carries more than one.

Common Error

writing colour instead of color What goes wrong: plt.plot(months, books, colour="green") looks right in Indian English, but Matplotlib does not recognise the word and the program stops. The fix: Matplotlib uses the American spelling, color. Keyword names in a library must be typed exactly as the library spells them, whatever the spelling in the rest of your writing.

Curiosity Corner

Q5. How does a program turn a plain table of readings into a graph on the screen?

A. The library does the drawing. You hand Matplotlib two lists — the labels along the bottom and the values — and it works out how tall the chart must be, where each point sits, and how to space the marks along each axis. Each plt call adds one more instruction to a picture held in memory, and plt.show() puts that finished picture on the screen.

19.6.2 Bar Graphs and Histograms

A line chart suits a value that changes over time. When the data falls into separate groups instead, a bar graph is the right picture: one bar for each group, its height showing the count.

Program

a bar graph of games chosen by a class Problem: Forty students in a class have each chosen one game. Draw a bar graph of how many chose each.

import matplotlib.pyplot as plt

games = ["Kabaddi", "Cricket", "Kho-kho", "Badminton"]
students = [14, 12, 5, 9]

plt.bar(games, students, color="steelblue")
plt.xlabel("Game")
plt.ylabel("Number of students")
plt.title("Games chosen by Class 10")
plt.show()

Output: a window opens showing four upright bars, the tallest being Kabaddi at 14 students and the shortest Kho-kho at 5, as in Fig. 19.6-2.

Diagram

Fig. 19.6-2 – bar chart from the code above; four blue bars labelled Kabaddi, Cricket, Kho-kho and Badminton on the x-axis labelled “Game”, counts on the y-axis labelled “Number of students”, titled “Games chosen by Class 10”

A histogram looks similar and means something different. It is used when the data is one long list of measurements rather than a set of named groups. Matplotlib sorts the measurements into equal ranges, called bins, and each bar shows how many measurements fell into that range.

Take the marks of twenty students in a test out of 100:

34, 41, 45, 47, 52, 55, 56, 58, 61, 63, 64, 66, 68, 71, 73, 75, 78, 82, 88, 94

Asking for five bins spreads them from the lowest mark, 34, to the highest, 94 — a spread of 60 marks, which splits into five ranges of 12 marks each. Counting by hand gives Table 19.4.

Table

Table 19.4 – mark ranges and the number of students in each

Mark range	Number of students
34 to 46	3
46 to 58	4
58 to 70	6
70 to 82	4
82 to 94	3

Program

a histogram of test marks Problem: Draw a histogram of the twenty marks using five bins, so that the shape of the class’s performance can be seen.

import matplotlib.pyplot as plt

marks = [34, 41, 45, 47, 52, 55, 56, 58, 61, 63,
         64, 66, 68, 71, 73, 75, 78, 82, 88, 94]

plt.hist(marks, bins=5, color="skyblue", edgecolor="black")
plt.xlabel("Marks")
plt.ylabel("Number of students")
plt.title("Spread of marks in the class test")
plt.show()

Output: a window opens showing five bars with black outlines, matching the counts in Table 19.4 — the tallest in the middle at six students, and three students at each end, as in Fig. 19.6-3.

Diagram

Fig. 19.6-3 – histogram from the code above; five light blue bars with black outlines, marks on the x-axis labelled “Marks”, counts on the y-axis labelled “Number of students”, titled “Spread of marks in the class test”

The picture says something the list of marks does not: most of the class sits in the middle, and few are at either extreme. Change bins to 10 and the same marks tell a rougher, more detailed story. The bin count is a choice you make, and it changes what the reader sees, so choose it to show the data honestly rather than to make a point.

Try Yourself

Run the histogram program again with bins=3 and then with bins=10. Note down how the shape changes, and decide which of the three pictures describes the class most fairly.

19.6.3 Scatter Plots and Pie Charts

Sometimes you have two measurements for each person or thing, and the question is whether they move together. A scatter plot answers that. Each point carries one measurement across and the other up, so the pattern of the cloud of points is the answer.

Program

a scatter plot of study hours against marks Problem: Ten students have reported the hours they studied for a test, and their marks are known. Draw a scatter plot of hours against marks.

import matplotlib.pyplot as plt

hours = [1, 2, 2, 3, 3, 4, 4, 5, 6, 6]
marks = [35, 42, 48, 52, 60, 58, 68, 72, 80, 85]

plt.scatter(hours, marks, color="purple", marker="o")
plt.xlabel("Hours studied")
plt.ylabel("Marks scored")
plt.title("Study hours and marks")
plt.show()

Output: a window opens showing ten purple dots that climb from the lower left towards the upper right, as in Fig. 19.6-4.

Diagram

Fig. 19.6-4 – scatter plot from the code above; ten purple round points, hours on the x-axis labelled “Hours studied”, marks on the y-axis labelled “Marks scored”, titled “Study hours and marks”

Read that chart carefully. It shows that in this small group, more hours went with higher marks. It does not show that the hours caused the marks — a student who finds the subject easy may both study comfortably and score well. A scatter plot shows a pattern; it never proves a cause.

A pie chart answers a different question: not how two things move together, but how a whole is divided up. It works only when the parts add up to one complete whole.

Program

a pie chart of how students travel to school Problem: Show how a group of 100 students reaches school, with the percentage marked on each slice.

import matplotlib.pyplot as plt

modes = ["Walk", "Bus", "Cycle", "Car"]
counts = [40, 30, 20, 10]

plt.pie(counts, labels=modes, autopct="%1.1f%%")
plt.title("How students travel to school")
plt.show()

Output: a window opens showing a circle cut into four slices, the largest marked 40.0% for walking and the smallest 10.0% for car, as in Fig. 19.6-5.

Diagram

Fig. 19.6-5 – pie chart from the code above; four slices labelled Walk, Bus, Cycle and Car with the percentages 40.0%, 30.0%, 20.0% and 10.0% printed on them, titled “How students travel to school”

The setting autopct="%1.1f%%" tells Matplotlib to write each slice’s share on it, to one decimal place. The doubled %% at the end is how you ask for a plain percent sign to be printed.

19.6.4 Several Plots in One Figure

Two charts are easier to compare when they sit side by side. plt.subplot(rows, columns, position) divides the window into a grid and says which cell the next chart should go into. Positions are counted from 1, left to right and then down.

Program

two charts side by side Problem: In one window, show the school’s daily attendance percentage through the week as a line chart, and the number of absentees in each section as a bar graph.

import matplotlib.pyplot as plt

days = ["Mon", "Tue", "Wed", "Thu", "Fri"]
attendance = [92, 95, 89, 94, 90]

sections = ["A", "B", "C", "D"]
absentees = [3, 1, 5, 2]

plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.plot(days, attendance, marker="o", color="green")
plt.xlabel("Day")
plt.ylabel("Attendance (%)")
plt.title("Attendance through the week")

plt.subplot(1, 2, 2)
plt.bar(sections, absentees, color="orange")
plt.xlabel("Section")
plt.ylabel("Number of students")
plt.title("Absentees by section")

plt.tight_layout()
plt.show()

Output: a single wide window opens with two charts in it, the line chart on the left and the bar graph on the right, as in Fig. 19.6-6.

Diagram

Fig. 19.6-6 – two charts side by side from the code above; left, a green line with markers for Mon to Fri titled “Attendance through the week”; right, four orange bars for sections A to D titled “Absentees by section”

Two extra calls make this work comfortably. plt.figure(figsize=(10, 4)) asks for a window ten units wide and four tall, so the two charts are not squeezed. plt.tight_layout() adjusts the spacing so that the titles and axis labels of one chart do not run into the other.

Common Error

forgetting plt.show() What goes wrong: The program runs, no error appears, and no chart is seen. The fix: Matplotlib holds the chart in memory until it is told to display it. Every plotting program ends with plt.show().

19.7 SciPy

At the end of Section 19.4 one summary figure was left unfinished. NumPy gives you the mean, the median, the variance and the standard deviation, but it has no function for the mode — the value that occurs most often. That gap is not an oversight. NumPy’s job is to hold numbers and do arithmetic on them quickly; the heavier scientific work is left to a second library built on top of it.

That library is SciPy, short for Scientific Python. It takes NumPy arrays as its input and returns NumPy arrays as its output, so the two are used together as a matter of course.

Definition

(SciPy is a library of scientific and mathematical tools built on top of NumPy, covering work such as statistics, solving equations and numerical calculations.)

SciPy is organised into modules, each covering one field of work:

scipy.stats — statistics, including the mode and many statistical tests.
scipy.linalg — linear algebra, such as solving sets of equations.
scipy.optimize — finding the smallest or largest value a formula can take.
scipy.integrate — numerical integration.
scipy.signal — filtering and processing signals.
scipy.ndimage — working with images held as arrays of numbers.

You import only the module you need, rather than the whole library. Fig. 19.7-1 shows where SciPy sits among the libraries of this chapter.

Diagram

Fig. 19.7-1 – four stacked layers drawn as wide horizontal bars. The bottom bar is labelled “Python”. Above it, a bar labelled “NumPy – arrays and fast arithmetic”. Above that, three separate bars side by side labelled “Pandas – tables of data”, “Matplotlib – charts” and “SciPy – scientific calculations”, each with a short upward arrow resting on the NumPy bar.

Finding the mode

The mode is the value that appears most often in a set of readings. A tailor stitching school shirts cares about it far more than about the average, because the average shirt size may be a size nobody actually wears.

Program

the most common shirt size in a class Problem: Ten school shirts have been issued in the sizes 38, 40, 40, 42, 40, 44, 42, 38, 40 and 42. Find the size issued most often, and how many times it was issued.

import numpy as np
from scipy import stats

sizes = np.array([38, 40, 40, 42, 40, 44, 42, 38, 40, 42])

result = stats.mode(sizes)

print("Most common size:", result.mode)
print("Number of times issued:", result.count)

Output:

Most common size: 40
Number of times issued: 4

Counting by hand confirms it: size 38 appears twice, size 40 four times, size 42 three times and size 44 once. Where two values tie for the most appearances, stats.mode reports the smaller of them.

Solving a pair of equations

Two unknowns and two facts about them make a pair of simultaneous equations. You have solved these by hand in mathematics; SciPy solves them in one line, which matters when there are not two unknowns but twenty.

Suppose a shop sells only pens and notebooks. Two pens and three notebooks cost ₹130. Four pens and one notebook cost ₹110. Let \(p\) be the price of one pen and \(n\) the price of one notebook, both in rupees. Then

\(2p + 3n = 130\)

\(4p + 1n = 110\)

Example 19.2

the price of a pen and a notebook

Given: \(2p + 3n = 130\) and \(4p + n = 110\), with \(p\) and \(n\) in rupees.

Find: The price of one pen and one notebook.

Working:

From the second equation, \(n = 110 - 4p\).

Put this into the first: \(2p + 3(110 - 4p) = 130\).

Expanding: \(2p + 330 - 12p = 130\), so \(-10p = -200\) and \(p = 20\).

Then \(n = 110 - 4 \times 20 = 110 - 80 = 30\).

Check: \(2 \times 20 + 3 \times 30 = 40 + 90 = 130\), and \(4 \times 20 + 30 = 110\). Both hold.

A pen costs ₹20 and a notebook costs ₹30.

To hand the same problem to SciPy, the equations are split into two arrays. One holds the numbers in front of the unknowns, taken row by row; the other holds the values on the right-hand side.

Program

solving the pen and notebook prices Problem: Use SciPy to find the price of a pen and a notebook from the two equations above.

import numpy as np
from scipy.linalg import solve

coefficients = np.array([[2, 3],
                         [4, 1]])
totals = np.array([130, 110])

prices = solve(coefficients, totals)

print("Pen:", prices[0])
print("Notebook:", prices[1])

Output:

Pen: 20.0
Notebook: 30.0

The answers match the hand working, and they are printed as 20.0 and 30.0 rather than 20 and 30 because solve always works in decimal numbers.

SciPy has an honest limit, and it is the same one that applies to every library in this chapter. It does not decide what is worth calculating, and it cannot tell you that your data was collected badly. For everyday work with a few hundred numbers, NumPy alone is usually enough; SciPy earns its place when the calculation itself is the hard part.

19.8 Summary

A module is a single Python file, saved with a .py extension, that holds code such as functions and variables so it can be used again in other programs.
Any Python file is a module. The word describes how the file is used, not how it is written.
A module is brought into another program with an import statement, and its contents are reached using dot notation, as in marks_tools.total().
The import statement has three common forms: importing the whole module, importing one name from it with from, and giving the module a short nickname with as.
Built-in modules such as math, random, datetime and os arrive with Python and together form the standard library. User-defined modules are the ones you write yourself.
Modules give code reuse, readable projects, separate namespaces so names do not clash, and repairs that need to be made in only one place.
A package is a folder of related modules, marked as a package by a file named __init__.py, which may be empty.
A package may contain a sub-package, which is simply another package folder inside it, with its own __init__.py.
Packages give a project a visible shape, keep names apart, and let a whole group of modules be reused together.
A library is a body of ready-written code shared for others to use. Some libraries come with Python; third-party libraries are collected on the Python Package Index and installed with pip.
pip commands are typed at the command prompt or terminal, never inside a Python program, and a library is installed once per computer.
The four libraries of this chapter are usually imported with the same short names: np for NumPy, pd for Pandas, plt for Matplotlib’s pyplot module, and named modules from SciPy.
A module is a file, a package is a folder of files, and a library is a body of work shared with the world.
A NumPy array holds many values of the same type and allows one operation to be applied to all of them at once, which is called an element-wise operation.
Arrays can be built with np.array, np.zeros, np.ones and np.arange, and reshaped into rows and columns with reshape. Indexing and slicing work as they do for lists.
NumPy supplies the mean, median, variance, standard deviation and range of a set of values, each as a single function call.
Compared with a Python list, a NumPy array is limited to one type of value and to a fixed size, but is faster on large sets of numbers and carries far more ready-made functions.
Pandas handles data arranged in rows and columns. A Series is one labelled column; a DataFrame is a table of several such columns sharing one index.
Columns of a DataFrame can be selected, added, calculated from other columns, and used to filter rows.
A CSV file stores a table as plain text with commas between values and one row on each line. Pandas reads one with read_csv and writes one with to_csv; JSON files are handled in the same way with read_json and to_json.
Missing values are marked as NaN. They can be counted with isnull, removed with dropna, or filled with fillna — and filling one invents a value that was never measured, so it must always be declared.
Matplotlib draws charts through its pyplot module. A line chart suits values changing over time, a bar graph suits separate named groups, a histogram suits one long list of measurements sorted into bins, a scatter plot shows whether two measurements move together, and a pie chart shows how a whole is divided.
A scatter plot shows a pattern between two measurements. It never proves that one caused the other.
Several charts can be placed in one window with plt.subplot, and every plotting program ends with plt.show().
SciPy is built on top of NumPy and covers heavier scientific work, including the mode through scipy.stats and the solving of simultaneous equations through scipy.linalg.

19.9 Key Terms

Built-in module – a module that comes with every Python installation and needs no downloading, such as math or random.
CSV file – comma-separated values file; a plain text file in which each line is one row of a table and the values in that row are separated by commas.
DataFrame – a two-dimensional Pandas table made of labelled columns and an index of rows, where each column may hold a different kind of value.
Element-wise operation – an operation applied to every value of an array at once, without writing a loop.
Histogram – a chart that sorts a list of measurements into equal ranges, called bins, and shows how many fell into each range.
Import statement – the Python instruction, beginning with import, that brings a module or library into the current program.
JSON – JavaScript Object Notation; a text format that stores data as labelled pairs, commonly used to send data between a website and the program behind it.
Library – a collection of ready-written code, usually made of packages and modules, shared so that other programmers need not write it themselves.
Matplotlib – the Python library used to draw charts and graphs, whose drawing tools are in its pyplot module.
Module – a single Python file, saved with a .py extension, that holds code such as functions and variables so it can be used again in other programs.
Namespace – the private list of names belonging to one module or package, which keeps its names from clashing with names elsewhere.
NaN – not a number; the special value Pandas uses to mark a missing or undefined entry.
NumPy – Numerical Python; the library that provides fast arrays and mathematical functions for large sets of numbers.
NumPy array – a container holding many values of the same type, arranged in one or more dimensions, on which one operation can act on all values at once.
Package – a folder of related modules, marked as a package by a file named __init__.py.
Pandas – the library used to work with data arranged in rows and columns, like a spreadsheet.
pip – the tool that comes with Python for installing third-party libraries; typed at the command prompt, not inside a program.
Python Package Index (PyPI) – the online collection from which pip downloads third-party libraries.
Scatter plot – a chart in which each point carries one measurement across and another up, used to see whether two measurements move together.
SciPy – Scientific Python; a library built on top of NumPy for scientific work such as statistics, equations and numerical calculations.
Series – a one-dimensional set of values in Pandas in which every value carries a label, called its index.
Standard library – the whole set of modules that arrives with Python itself.
Sub-package – a package folder placed inside another package, with its own __init__.py.
Third-party library – a library written outside the Python installation, which must be installed before it can be imported.
User-defined module – a module written by you rather than supplied with Python. ## 19.10 Objective Questions
Which file must be present in a folder for Python to treat it as a package?
1. __main__.py
2. __init__.py
3. package.py
4. setup.py
Fill in the blank: The tool that comes with Python for installing third-party libraries from the Python Package Index is called ____________.
State whether the following statement is true or false: A NumPy array can hold whole numbers and text side by side, just as a Python list can.
Which line correctly brings in Matplotlib’s plotting module under the short name most programmers use?
1. import matplotlib as plt
2. from matplotlib import plt
3. import matplotlib.pyplot as plt
4. import pyplot as plt
Fill in the blank: In a Pandas DataFrame, an entry that has no value is shown as ____________.
Match each library in the left column with the work it is used for in the right column.

Library	Used for
a. NumPy	i. drawing charts and graphs
b. Pandas	ii. scientific calculations built on arrays
c. Matplotlib	iii. fast work with large sets of numbers
d. SciPy	iv. handling data arranged in rows and columns

Write the output of the following code.

import numpy as np

arr = np.arange(2, 12, 2)
print(arr[1:4])

Which of these is not a benefit of splitting a large program into modules?
1. The same code can be reused without copying it.
2. A correction needs to be made in only one place.
3. Names in one module cannot clash with names in another.
4. The program will run correctly even if the module has a mistake in it.

19.11 Concept Questions

Define a module. Explain why any Python file can be described as one.
Differentiate between a package and a library, giving one example of each.
Give one reason why a NumPy array works faster than a Python list on a long set of numbers, and describe one situation in which a list would still be the better choice.
A student saves her practice code in a file named random.py. Later, in the same folder, she writes import random and calls random.randint(1, 6), and the program fails. Explain what has gone wrong and how she should correct it.
Write a Python program that stores the six readings 12, 15, 11, 18, 14 and 16 in a NumPy array and prints their mean, median and range.
A shopkeeper draws a scatter plot of daily ice-cream sales against the day’s temperature and says it proves that hot weather causes higher sales. Explain what the chart actually shows and why his claim goes further than the evidence allows.

19.12 Case-Based Questions

A school canteen keeps a record of what it sold during one week. The price of each item and the number sold are given below.

Table

Table 19.5 – canteen items with their price and the number sold in a week

Item	Price (₹)	Sold
Samosa	15	80
Idli	20	45
Poha	25	30
Juice	30	25
Biscuit	10	60

Write the Python lines that build a Pandas DataFrame from this record and then add a column named Revenue holding the price multiplied by the number sold for each item.
The canteen manager wants a single picture showing what share of the week’s total revenue came from each item. Name the chart you would draw and the Matplotlib function that draws it, and give one reason for your choice.
The manager also wants to compare the number of items sold, item by item. State which chart suits this better, and explain why it is not the same choice as in part (b).

The weather club of a school notes the midday temperature every day. Two members were absent, so two readings were never taken.

Table

Table 19.6 – midday temperature over ten days, with two readings missing

Day	Temperature (°C)
1	31
2	33
3	not taken
4	34
5	32
6	30
7	not taken
8	35
9	33
10	32

When this record is loaded into a Pandas DataFrame, what value appears in place of the two missing readings, and which method would you use to count how many are missing in the column?
The club decides to fill both gaps with the average of the eight readings that were taken. Work out that average, correct to one decimal place, and state one problem with putting it into the table.
The club wants one chart showing how the temperature changed from day 1 to day 10. Name the chart you would draw and give one reason for choosing it over a bar graph.