Introduction¶

What are we aiming for?¶

By the end of this course:

• You should be able to identifiy problems that can be solved with the help of a program.
• You should understand the basics of programming to design reasonably efficient and maintainable programs.
• You should know enough about Python's syntax to implement programs in it.
• You should be able to find help and access documentation.
• You should have an idea of some libraries that might be useful for your work.

The very Basics¶

Programming is telling a machine what to do. This has been done even before computers existed, e.g. for mechanical looms. For computers, programming can be done at different levels of abstraction. We could write the very instructions the hardware expects - but nobody does this. Instead, we will work with a high level language.

High Level Languages¶

There are many different high-level langages with different levels of abstraction availabel (C/C++, Java, Python, R, Haskell, ...). They fall into two main groups:

• compiled: Commands are translated into harware instructions once before running. This usually runs faster.
• interpreted: Commands are translated at runtime. This is usually faster to develop, when you don't exactly know yet what you want (e.g. exploring data).

Python¶

• an interpreted langauge
• free software
• extensive documentation
• runs almost everywhere
• large collection of libraries
• large user base among scientists
• many nice tools that make life easier

Course Design¶

The Problem¶

Learning the basics of a language is "easy". Learning an additional langauge is very easy. The problems usually are:

• understanding the task clear enough
• splitting the task into managable chunks
• developing a clear idea of what you want the computer to do

I will try to teach you the basics and give a lot of tipps and hints on how to solve the three points above. I will also be happy to help when you struggle with a problem. But you really need as much hands on experience as possible.

Schedule, Slides and Exercises¶

Online at https://hsg.crbn.ch.

The Basics¶

Starting¶

There is different ways to start Python. For the first part, we will work with interactive python shell (in Spyder, Thonny or a Terminal).

Make sure everyone has a running python shell

Simple Calculations¶

You can use python as a simple calculator.

Note: we got a float (floating point number, a number containing a fractional part) above. Floats and ints (integers) are two different things in computers. They are stored completely differently.

Note 2: Floating point numbers have a limited precision.

Powers and operator precedence work as well

Errors¶

If we do something wrong, Python will tell us with an Error message

If you get an error, make sure to really look at the message. Python tries to help you figure out what's wrong. It tells you:

• in which file and on what line the error occured
• what this line looks like
• where in the line it thinks the error is
• what type of error it is, including a short explanation

Variables¶

Variables are one of the core concepts. Similar to what you are used to in math they represent a "value" and can be used in a formulas.

It's easy now to repeat the calculation with a different value of the variables

You can also assign the result of a calculation to a variable. The calculation can even include this or another variable - as long as they already exist.

Naming¶

Please use speaking variable names! (but try to keep it simple)

(i*0.01+1)**y*v

Scripts¶

You might want to repeat this calculation tomorrow. If you do not want to type everything in again, you want to write a script.

• Open a text file in your favourite editor
• Copy the lines into the file and save as my_script.py
• run python3 my_script.py

You will not see any output. The most important difference between scripts and interactive sessions is, that scripts only print output if you explicitly tell them so.

Basic Types¶

So far we have seen two different types of numbers, ints and floats. You can ask python the type of a value or a variable with type.

There are many more types, but to get started I will just introduce the most basic ones here.

Text¶

You probably will want to work with text at some point. Text is stored in Strings.

(Note the limited precision of floats)

Lists¶

We will soon start to work with more then one value at once. There most basic type for this is lists. They are created with [].

You can access the elements with [] -- Warning: indexing starts at 0!

(This code is is not really nice, but we will discuss this later)

You can assign each entry of the list to a different variable in one go (called unpacking)

(Unpacking is not so useful here, but will be a very common pattern later.)

If you want to know how many elements are in a list, use len()

Accessing elements, unpacking and counting elements works for all data types containing elements too. Another example of suche a type are strings:

Booleans¶

Booleans are either True or False and can be used in logical operations. We will talk about this more later.

Calling Functions¶

We've seen the print()- and the type()- Function already.

There are many others, and you will soon be defining your own funtions. They are the main building-blocks of programs. The syntax to call a function is with round brackets containing the argument(s).

Calling a function tells python to run through the code contained in the function definition. So in a first approximation, a function is a block of code, stored somewhere else and using its own variables.

Remember: You call a function with () you access elements in a collection with []

Some functions give back values we can store

others don't

None is the value indicating, that print() did not return anything

On the other hand some functions don't have input.

Modules¶

Python has a bunch of functions like round and print built-in. But the real power is all the additional collections of functions available. These collections are often called libraries or modules in Python.

You add them with the import statement.

Once imported, you can use module.function to use the new functions.

People who do not like to type so much, can use the following shorthand.

There are other ways to load things, but for now this syntax is enough.

Hands On 1¶

Questions¶

• How do you import a module in python?
• How do you assign a value to a variable?
• What is the difference between a = 42 and a == 42
• How do you call a function?
• How do you call a function from a module?
• How do you define a list?
• What is the difference between res = thing(1) and res = thing[1]?

Tasks¶

• repeat the things we did, type in the commands yourself and experiment with them.
• What does the following program print a = -1 b = 2 c, d = b, a print(c > 0)
• Implement a simple formula from your own domain as a python script.
• Calculate the mean of the following numbers: 2,3,5,7,11,13
  • manually by tipping the calculation
  • using a list and numpy.mean

Looking at Refugee Data¶

As a toy study for this tutorial, we will look at the development of the refugee populations in the different EU countries. I downloaded the data files from the World Bank. We will work with these later. But to make things easier for the start, I created a few seperate files in a format directly understood by Python.

If the data has the optimal format, we can load it simply by calling numpy.loadtxt. This returns an array with the data from the file, skipping rows starting with #.

The output looks like we got a list of lists. However an array is a different type, introduced by numpy. It is heavily optimised to work with data and used by many scientific modules.

Like with lists we can access a single element (a full row in our case) with square brackets

Of course we want picutes...

Intermezzo: Objects¶

An object is a programming concept, grouping data and code together. We will discuss why this is a good idea and how to do this yourself in more detail on Wednesday (object oriented programming).

Here we just want to understand how to use existing objects in Python. Objects have attributes:

• data attributes (variables, accessed without brackets)
• methode attributes (functions, called with round brackets)

We access the attributes with the syntax object.attribute. If this seems familiar, you are right. In python everything is an object. In the code above, the plot function is a methode attribute of the imported matplotlib.pyplot module-object.

The easiest way to get an overview of all available attributes is to use tab-completion in an interactive session.

Some example of more things that are objects¶

Knowing that everything is an object, we can have a second look at variables as well.

All objects exist somewhere in the memory of our computer (cf. memory-address above), and variables are just labels we attach to this objects ("pointers to these memory locations").

This is important to understand what happens here:

This does not happen with numbers or strings, as you can not change the content of a these objects (they are immutable).

So when ever you change such a variable, the label must point to some new location in memory.

Note: In Thonny you can make this visible by showing Variables and Heap (both in the View-Menu)

Nice Plots¶

I hope you all would recject a plot like the one above, if you got that from a student. Let's turn this into a useful plot.

Let's discuss this in details

• we create a list with all the years in our data by copy-paste from the data-file (the colum-labels are a comment, because we do not want to mix different types of data)
• we create a new figure-object
• we add an axes-object to that figure that will hold our plot (will look at the details later)
• we call the plot methode from our axes object it works identical to the one we called before
  • if passed a singel list these are the y-values and they will be continously numbered for x
  • if passed two arguments these are x- and y-values
  • you can pass many more arguments to configure the line (see the documentation)
• We then use three methodes from the axes object to set x- and y-labels as well as a title of this object. (and maybe the range)
• Finally we call the show-function from the pyplot module to display the plot we just created.

Finding Help¶

You find a lot of documentation online, but normally the documentation is even availabel directly in Python.

Comments¶

You might want to add comments to your code as a note to your future self or other people that might need to understand your program. There are two options:

(This acutally creates a multi-line string-object, that you could asign to a variable )

A Closer Look at Arrays¶

So far we have just loaded data from a file into an array and then accessed individual rows from it. But arrays are a lot more powerful.

To look closer at arrays, we first create a simple test-array:

Make sure you understand the meaning of round and square brackets

Note: We passed the array-Function a list containing lists here. Lists can contain all kind of things, so you can create arbitrary complex constructs. (but please aviod that :-)

Accessing Elements¶

As we have seen, we can access elements with [] as well. (Remember: indexing starts at 0!)

You can also put the different dimensions in one square bracket, separated by comma. As with the separate bracktes, the order is row, column.

You might want to choose only a slice of an array. For this you can use the syntax lower:upper.

Note: The upper bound is not included!

If you leave a boundary empty, it will select everything until the edge of the array

You can also index from the end using negativ indices.

And you can even choose a stepsize

Such selections are called slices. You can store them in a new variable.

And you can assign to them

but be aware, a slice is not a separate array

Ohter Ways to Access Elements¶

For arrays, there are two more useful ways to access elements.

• You can pass a list of indices to an array. Numpy will then build tuples from the different lists and access the corresponding elements.

Doing this, numpy is quiet smart. You can define a new shape for the result and it expands missing values when possible.

• You can pass an "array" of the same shape, containing booleans. This array then works as a mask. Numpy will then select all elements where the mask contains True

Attributes of Arrays¶

Of course arrays are objects too. They have many useful attributes. Type test_array. then press tab.

For example, you can ask the array for its shape or aggregate the values along a given axis.

Math with Arrays¶

One of the major advantages of arrays is, that math involving arrays is done elementwise.

Doing the same with lists, would not work as expected.

Question: Using arrays, how would you simplify the calculation of the compound interest for 3 different interest rates?

Hands On 2¶

Questions¶

• What does plot([0, 0, 1, 1], [0, 1, 0, 1]) display?
• What kind of attributes does an object have in python? What is the difference when accessing them?
• Explain in your own words what this code prints and why: my_list = [1,2,3] my_other_list = my_list my_other_list[1] = 0 print(my_list)
• What methods are available to get help?
• How can you add a comment to python code?
• Slicing also works on lists and strings. Given element = 'oxygen' what do the following commands return:
  • element[:4]
  • element[4:]
  • element[:]
  • element[-1]
  • element[-2]
  • element[1:-1]
• What is the result of the two print statments in the following code: my_list = [1,2,3] my_array = np.array([1,2,3]) print(my_list * 2) print(my_array * 2)

Tasks¶

• Repeat the things we did, type in the commands yourself and experiment with them.
• Replace refugees-europe_0.txt with refugees-europe_1.csv, adapt the call to loadtxt according to the new input format.
• Get a general idea of the functions provided by numpy. (use tab-completion and the help-system)
• Plot the refugee population of Germany over the periode from 2000 unitl 2010.
• Use the axes.set_ylim and array.max to make sure the y axis of your plot starts at 0 and includes all data.
• Draw a horizontal line into your plot, indicating the mean.
• Use numpy.diff to plot the change in refugees instead of the absolut value.

Repeating Things¶

We do have many different datasets. And most likely we will want to include all of them in whatever analysis we run. Let's start with creating the plot above for several countries.

How not to do it¶

The first option is of course to copy-paste the code and adapt it. Let's just add Germany.

Why no to do this¶

• Are you looking forward to adding the other 26 countries?
• Once you are done, I'll tell you that I want the Axis labels properly capitalized.

For-Loops¶

To repeat parts of the code, all programming languages offer loops. The easiest to understand is a for-loop. A for-loop allows us to repeat a block of code for every item in a sequence.

If we want to ask for different types of cheese, instead of writing several almost identical print statements

we can write a for-loop.

Note: The code block is defined by the indentation.

Iterating works for many different types of sequences

see what's happening¶

Sometimes it is really useful to visualise what happens. This is easiest using Thonny. We will look at the default debugger later in this course as well - and finally you could use this website:

http://www.pythontutor.com

(Unfortunately the website does not support numpy)

Useful helpers¶

Python has a few helper functions, that are especially useful when working with for-loops.

range([min=0], sup, [step=1])¶

Returns the numbers from min to sup (excluding sup) with step increments.

So we could write the last loop as

You do not need to use the loop variable inside the loop. This is also useful to run some code N-times.

enumerate(sequence)¶

Enumerate returns each item in a sequence together with its position (remember unpacking):

zip(...)¶

Zip allows to iterate over two (or more) lists (or other iterables) in parallel.

Plotting in a Loop¶

With this new knowledge of loops, let's rewirte the plotting code.

Of course we would like to use the proper country names. For the moment, we pick them from the file "manually"

This is a lot more efficient. We could still imporve a few things, but first we look at some other things.

More Control Flow Tools¶

We might want to limit our analysis on countries that have a large absolut number of refugees. To achieve this, we want to check that the average number of refugees is above 10000.

Of course we could manually create a list of these countries, but we will much rather use python.

If Statement¶

A Complex Example¶

Note:

• == comparison; = assignement
• you can compare against a range

More:

• is / is not
• in / not in
• boolean operations
• float comparison

Defining Functions¶

As mentioned at the beginning, once your program gets longer, you should splitt it into small, managable chunks.

In our example, there is at least three tasks:

• defining the data variables
• plotting a single country
• managing several plots

We followed this split when we developed our code. But we still put everything into one big code block. The next step is to define new functions for the spearate chunks. The obvious candidate for a function is the plotting for a single country.

A function has a fixed structure:

• The keyword def introduces a function definition.
• It must be followed by the function name (choose something helpful)
• and the list of formal parameters in round brackets.
• the definition line is then ended by a :
• The statements that form the body of the function start at the next line, and must be indented.

Functions define a scope for variables. This means:

• Variables we define inside a function (including the parameters) only exist inside the function.
• On the other hand, variables defined in the global scope will exist inside all functions.

Our first examples do not return anything.¶

But like the predefined functions we have used so far, our own functions can return results too. The keyword for this is return.

Make sure you understand the difference between print and return

Obviously this code is not very elegant. Once you understand the difference between print and return, you could get rid of the info function completely.

User Input¶

Of course we can hard-code the balance in our last exampe (put the number into the code). But it would be a lot nicer, if our script would ask the user for the balance to use. The easiest way to achieve this is, to ask the user during execution. To do this, Python offers the input function.

The return value of the input function is always a string, no matter what you enter:

So if you want to read a number, you will need to convert the output of input. Luckily this is easy¶

With this we can adapt the compound interest script

Hands On 3¶

Questions¶

• What is the basic structure of a for-loop?
• What type of objects can you use a for loop on?
• Explain the functions enumerate and zip.
• How do you use if, elif, else?
• How do you define a function?
• What variables are availabel in the different functions of the last example.
• What is the difference between return and print?
• What is intendation good for?

Tasks¶

• Repeat the things we did, type in the commands yourself and experiment with them.
• Write a loop that calculates 2 ** 8 with simple multiplication.
• Look at the help for range and create a loop that prints a countdown.
• Adapt the compound interest code:
  • let it use a sensible default value for balance and time if these values are not specified.
  • let it calculate the result for a fixed list of interest rates (e.g.: 0.5, 1., 2., 4., 8.) instead of reading a single value from the terminal.
  • calculate the results for each interest up to 15 years and plot the result.
• Define a function that calculates some basic statistics (min, max, mean, standard deviation) for the passed data and returns a list with them.
• Use the function from the previous task to print these statistics for all countries where the mean is above 10000.
• Adapt the refugees code such that the user can choose the time range to plot.