# An Overview of Python¶

In [1]:

    import numpy as np
    # define data variables
    years = range(1990,2023)
    countries = ["Austria", "Belgium", "Bulgaria", "Croatia", "Cyprus", "Czechia", "Denmark", "Estonia", "Finland", "France", "Germany", "Greece", "Hungary", "Ireland", "Italy", "Latvia", "Lithuania", "Luxembourg", "Malta", "Netherlands", "Poland", "Portugal", "Romania", "Slovak Republic", "Slovenia", "Spain", "Sweden", "United Kingdom",]
    refugees = np.loadtxt("data/refugees-europe_0.txt") # data from World Bank

## More Containers¶

So far we have used the builtin container `list`. Additionally we used arrays from numpy. Next we will look at some additional builtin containers. Choosing the right container can simplify a problem considerably. And of course you can combine containers to get more complex data structures.

But first, I would like to introduce a very useful shorthand and a caveat with arrays:

### List comprehension¶

In [2]:

    [x**2 for x in range(10)]  #  if x % 3 == 0

Out[2]:

    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

List comprehension is a one-line for-loop creating a list. It can optionally contain an if condition.

### Comparing Arrays¶

Arrays are compared element-wise

In [3]:

    a = np.array([1,2,3])
    b = np.array([1,2,3])
    a == b

Out[3]:

    array([ True,  True,  True])

this means the result of a comparison can not be used directly in a condition

In [4]:

    if a == b:
        print("equal")

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[4], line 1
    ----> 1 if a == b:
          2     print("equal")

    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [5]:

    if (a == b).all():
        print("equal")

    equal

### Tuples¶

Sometimes you want to return several values form a function. This can be done by listing them comma-separated after `return`. Python will then create a `tuple`.

In [6]:

    def country_stats(data):
        min_ = data.min()
        max_ = data.max()
        mean = data.mean()
        std  = data.std()
        return min_, max_, round(mean, 1), round(std, 1)

    ret = country_stats(refugees[10])
    type(ret)

Out[6]:

    tuple

In [7]:

    ret[1]

Out[7]:

    2075445.0

Tuples are similar to lists, but once created they can not be changed. Tuples can be created by listing them with a comma (with or without round brackets):

In [8]:

    my_tuple = (1,"foo",1)
    print(my_tuple)

    (1, 'foo', 1)

In [9]:

    my_tuple[1] = 2

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[9], line 1
    ----> 1 my_tuple[1] = 2

    TypeError: 'tuple' object does not support item assignment

For a tuple with one element only, a comma is mandatory

In [10]:

    one_tuple = 42,
    print(one_tuple)

    (42,)

like lists, tuples can be unpacked in one go

In [11]:

    a, b, c = my_tuple
    print(b)

    foo

When comparing two tuples, python will compare one position after the other, using the specified comparison operator

In [12]:

    (1,2,3) < (1,2,4)

Out[12]:

    True

Note: This will fail, if the tuples contain numpy-arrays.

In [13]:

    (1,2,np.array([1,2])) < (1,2,np.array([1,2]))

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[13], line 1
    ----> 1 (1,2,np.array([1,2])) < (1,2,np.array([1,2]))

    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

### Sets¶

While a tuple is an ordered collection and allows repetitions, a set is an unordered collection of unique elements

In [14]:

    set([1,1,2,3,5])

Out[14]:

    {1, 2, 3, 5}

In [15]:

    s = {5,1,2,3,1,5}
    print(s)

    {1, 2, 3, 5}

### Dictionaries¶

Dictionaries are similar to lists as well, but instead of continuously numbering the entries you can define your own keys.

In [16]:

    def country_stats(data):
        res = {}
        res["std"]  = data.std()
        res["min"] = data.min()
        res["max"] = data.max()
        res["mean"] = data.mean()
        return res

    country_stats(refugees[10])

Out[16]:

    {'std': 377344.61433153594,
     'min': 187545.0,
     'max': 2075445.0,
     'mean': 900264.7878787878}

Note: In older versions of Python (before 3.7) dictionaries do not preserve the order of the elements from assignment.

Accessing an element in a dict, works as expected

In [17]:

    def country_stats(data):
        res = {}
        res["min"] = data.min()
        res["max"] = data.max()
        res["mean"] = data.mean()
        res["std"]  = data.std()
        return res

    stats = country_stats(refugees[10])
    stats["mean"]

Out[17]:

    900264.7878787878

The help shows many ways to define a dictionary:

In [18]:

    dict?

In [19]:

    dict([(1,2), (3,4)])

Out[19]:

    {1: 2, 3: 4}

In [20]:

    dict(a=1, b=2)

Out[20]:

    {'a': 1, 'b': 2}

Dict comprehension exists as well:

In [21]:

    {k: v for k, v in enumerate("abc")}

Out[21]:

    {0: 'a', 1: 'b', 2: 'c'}

A key in a dictionary can be almost anything (anything that is hashable)

In [22]:

    d = {range(5): "this is a range object", print: "the print function"}

In [23]:

    d[print]

Out[23]:

    'the print function'

#### Looping Over Dicts¶

The naive for-loop on a dict iterates over the dict keys:

In [24]:

    for stat in stats:
        print(stat)

    min
    max
    mean
    std

However dicts offer different functions to access the keys, the values or both when iterating:

In [25]:

    for stat in stats.values():
        print(stat)

    187545.0
    2075445.0
    900264.7878787878
    377344.61433153594

In [26]:

    for stat_name, stat_value in stats.items():
        print(stat_name, int(stat_value))

    min 187545
    max 2075445
    mean 900264
    std 377344

As always, tab-completion is very useful:

In [27]:

    #stats.

## More on Functions¶

### optional- and keyword-arguments¶

We can specify default values for the parameters of a function. This turns them into optional-arguments. Optional arguments can be passed either based on their position or on their keyword.

Keyword-arguments:

- must be defined and passed after the positional arguments
- their order is irrelevant
- must use an existing parameter-name

In [28]:

    import numpy as np
    import matplotlib.pyplot as plt

    def plot_refugees(years, data, country, color="r", ylabel="refugees"):
        # create axes object
        figure = plt.figure()
        axes = figure.subplots()
        # plot the data
        axes.plot(years, data, color=color)
        # add labels and title
        axes.set_xlabel("year")
        axes.set_ylabel(ylabel)
        axes.set_title("Refugees in "+country)

    # define data variables
    years = range(1990,2023)
    countries = ["Austria", "Belgium", "Bulgaria", "Croatia", "Cyprus", "Czechia", "Denmark", "Estonia", "Finland", "France", "Germany", "Greece", "Hungary", "Ireland", "Italy", "Latvia", "Lithuania", "Luxembourg", "Malta", "Netherlands", "Poland", "Portugal", "Romania", "Slovak Republic", "Slovenia", "Spain", "Sweden", "United Kingdom",]
    refugees = np.loadtxt("data/refugees-europe_1.csv", delimiter=",") # data from World Bank

    means = refugees.mean(axis=1)
    for row in range(len(countries))[:4]:
        if means[row] > 10000:
            plot_refugees(years, country=countries[row], data=refugees[row], ylabel="number", color="b")

    # display the plot
    plt.show()

Important: You should not use mutable objects as default arguments. They are created once at load-time and not reset for individual calls.

In [29]:

    def append_one(list_=[]):
        """ Do not use mutable default values """
        list_.append(1)
        return list_

    my_list = []
    print(append_one(my_list))
    print(append_one(my_list))
    my_list = append_one()
    print(my_list)
    print(append_one(my_list))
    my_list = append_one()
    print(my_list)

    [1]
    [1, 1]
    [1]
    [1, 1]
    [1, 1, 1]

### Arbitrary Argument Lists¶

In addition to listing individual paraemters, you can also tell python to accept all arguments and/or keywordarguments

In [30]:

    def i_want_it_all(*args, **kwargs):
        print(args)
        print(kwargs)

    i_want_it_all(1,2,3,foo="so long", bar="and thanks for all the fish")

    (1, 2, 3)
    {'foo': 'so long', 'bar': 'and thanks for all the fish'}

As you see, `*args` creates a tuple of all passed positional arguments, while `**kwargs` creates a dict. You can use the same syntax to pass tuples and dicts to the function when calling it:

In [31]:

    i_want_it_all(*("a", "b"), **{"robot": "marvin"})

    ('a', 'b')
    {'robot': 'marvin'}

### Docstrings¶

When defining functions, you should always document them as well. Documentation will help others and your future self.

One very nice feature of python is, the ability to access documentation quickly. A docstring is a string given directly after the function definition. Python will display that string when you ask for help on the function.

In [32]:

    def country_stats(data):
        """Calculate min, max, mean and standard deviation from data

        Arguments
        ---------
        data : numpy-array
            data to calculate statistics on

        Returns
        -------
        stats : dict
            dictionary containing min, max, mean and standard deviation

        """
        res = {}
        res["min"] = data.min()
        res["max"] = data.max()
        res["mean"] = data.mean()
        res["std"]  = data.std()
        return res

    country_stats?

In [33]:

    np.abs?

## More on Loops¶

We looked at the for loop in the first part. We will add a second loop type (the while loop) and look at more tricks avaibale for both loop types.

### Break and continue¶

Break and continue are keywords to interrupt the normal execution of loops. They are quiet similar, so it is important to understand the difference.

`break` exits the loop completely and continues execution after the loop.

`continue` skips the rest of this iteration and moves on to the next round.

In [34]:

    for i in range(30):
        if i % 2 == 0:  # % is modulo division -> i is even
            continue
        elif i > 20:
            break
        print(i)

    print("We are done.")

    1
    3
    5
    7
    9
    11
    13
    15
    17
    19
    We are done.

### else¶

Once you use break, you might want to execute code only if your loop terminates normaly. A typical example is when you search for someting in a list:

In [35]:

    list_ = ["hay", "hay", "hay", "hay", "hay", "needle", "hay", "hay", "hay"]
    #list_ = ["hay", "hay", "hay", "hay", "hay", "hay", "hay", "hay"]
    for value in list_:
        if value == "needle":
            print("found")
            break

    found

You could add an additional variable to track if you found the needle:

In [36]:

    list_ = ["hay", "hay", "hay", "hay", "hay", "needle", "hay", "hay", "hay"]
    list_ = ["hay", "hay", "hay", "hay", "hay", "hay", "hay", "hay"]
    found = False
    for value in list_:
        if value == "needle":
            print("found")
            found = True
            break

    if not found:
        print("not found")

    not found

using `else` no additional variable is necessary:

In [37]:

    list_ = ["hay", "hay", "hay", "hay", "hay", "needle", "hay", "hay", "hay"]
    #list_ = ["hay", "hay", "hay", "hay", "hay", "hay", "hay", "hay"]
    for value in list_:
        if value == "needle":
            print("found")
            break
    else:
        print("not found")

    found

## while¶

We can use a for loop to iterate over a given list of items (or with `range` for a fixed number of iterations). However in many cases we do not know how many iterations we will need, but we know what condition we want to reach. In this case we use a `while`-loop:

In [38]:

    value = 1
    while value < 1000:
        print(value)
        value = value * 2

    1
    2
    4
    8
    16
    32
    64
    128
    256
    512

Again you can use `break`, `continue` and `else`

In [39]:

    tries = 0
    while tries < 3:
        pin = input("Enter PIN: ")
        if pin == "":
            continue
        if pin == "123":
            break
        tries = tries + 1
    else:
        print("Too many bad tries, terminating!")
        #import sys
        #sys.exit()

    print("PIN accepted, welcome back!")

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[39], line 3
          1 tries = 0
          2 while tries < 3:
    ----> 3     pin = input("Enter PIN: ")
          4     if pin == "":
          5         continue

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

### A note on security¶

The code above is of course a bad exampale: You should not store passwords in cleartext. Instead you should use a specialised password-hash function to store a salted and hashed value.

In [40]:

    from passlib.hash import argon2

    tries = 0
    while tries < 3:
        pin = input("Enter PIN: ")
        if pin == "":
            continue
        if argon2.verify(pin, '$argon2id$v=19$m=102400,t=2,p=8$s9a6F4Jwbg0BgBCCsLY2Rg$cpl1xrY0GAc8tAxkFmXN/A'):
            break
        tries = tries + 1
    else:
        print("Too many bad tries, terminating!")
        #import sys
        #sys.exit()

    print("PIN accepted, welcome back!")

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[40], line 5
          3 tries = 0
          4 while tries < 3:
    ----> 5     pin = input("Enter PIN: ")
          6     if pin == "":
          7         continue

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

In [41]:

    #argon2.hash("123")

## More Errors¶

### Understanding Errors¶

Let's look at some typical error messages in more detail.

In [42]:

    import matplotlib.pyplot as plt
    plt.plot([1,2,3],[1])

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[42], line 2
          1 import matplotlib.pyplot as plt
    ----> 2 plt.plot([1,2,3],[1])

    File /usr/lib/python3/dist-packages/matplotlib/pyplot.py:3590, in plot(scalex, scaley, data, *args, **kwargs)
       3582 @_copy_docstring_and_deprecators(Axes.plot)
       3583 def plot(
       3584     *args: float | ArrayLike | str,
       (...)
       3588     **kwargs,
       3589 ) -> list[Line2D]:
    -> 3590     return gca().plot(
       3591         *args,
       3592         scalex=scalex,
       3593         scaley=scaley,
       3594         **({"data": data} if data is not None else {}),
       3595         **kwargs,
       3596     )

    File /usr/lib/python3/dist-packages/matplotlib/axes/_axes.py:1724, in Axes.plot(self, scalex, scaley, data, *args, **kwargs)
       1481 """
       1482 Plot y versus x as lines and/or markers.
       1483
       (...)
       1721 (``'green'``) or hex strings (``'#008000'``).
       1722 """
       1723 kwargs = cbook.normalize_kwargs(kwargs, mlines.Line2D)
    -> 1724 lines = [*self._get_lines(self, *args, data=data, **kwargs)]
       1725 for line in lines:
       1726     self.add_line(line)

    File /usr/lib/python3/dist-packages/matplotlib/axes/_base.py:303, in _process_plot_var_args.__call__(self, axes, data, *args, **kwargs)
        301     this += args[0],
        302     args = args[1:]
    --> 303 yield from self._plot_args(
        304     axes, this, kwargs, ambiguous_fmt_datakey=ambiguous_fmt_datakey)

    File /usr/lib/python3/dist-packages/matplotlib/axes/_base.py:499, in _process_plot_var_args._plot_args(self, axes, tup, kwargs, return_kwargs, ambiguous_fmt_datakey)
        496     axes.yaxis.update_units(y)
        498 if x.shape[0] != y.shape[0]:
    --> 499     raise ValueError(f"x and y must have same first dimension, but "
        500                      f"have shapes {x.shape} and {y.shape}")
        501 if x.ndim > 2 or y.ndim > 2:
        502     raise ValueError(f"x and y can be no greater than 2D, but have "
        503                      f"shapes {x.shape} and {y.shape}")

    ValueError: x and y must have same first dimension, but have shapes (3,) and (1,)

As explained in the introduction, we can learn a lot from this output:

- The traceback tells us that the error occured deep inside the matplotlib module (5 function calls)
- More precisely the error happend when matplotlib wanted to read the x and y coordinates.
- But the top of the traceback also tells us which line of our code caused the problem.
- This is a `ValueError`, one of the values we passed is probably wrong.
- The final line tells us that we have a problem with the dimensions of our x and y lists.

### Common Errors when Passing Values¶

In addition to the value error above, there is also a `TypeError`, indidcating that we passed the wrong type:

In [43]:

    int(["1"])

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[43], line 1
    ----> 1 int(["1"])

    TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'

If you screw up when accessing an element in a container you usually get a `IndexError` or a `KeyError`

In [44]:

    l = [1,2,3]
    l[3]

    ---------------------------------------------------------------------------
    IndexError                                Traceback (most recent call last)
    Cell In[44], line 2
          1 l = [1,2,3]
    ----> 2 l[3]

    IndexError: list index out of range

In [45]:

    d = {"a" : 1}
    d["b"]

    ---------------------------------------------------------------------------
    KeyError                                  Traceback (most recent call last)
    Cell In[45], line 2
          1 d = {"a" : 1}
    ----> 2 d["b"]

    KeyError: 'b'

### Common Errors in Syntax¶

In [46]:

    123a

      Cell In[46], line 1
        123a
          ^
    SyntaxError: invalid decimal literal

`SyntaxError` means that your words do not make any sense...

In [47]:

    print("Hallo welt"
    a = 123

      Cell In[47], line 1
        print("Hallo welt"
             ^
    SyntaxError: '(' was never closed

In [48]:

    if True:
    print("Better")
        print("Faster")

      Cell In[48], line 2
        print("Better")
        ^
    IndentationError: expected an indented block after 'if' statement on line 1

`IndentationErrors` show up, when you forget to indent or mess up the alignment of your code.

Note: Do not mix Tabs and Spaces

In [49]:

    if True:
        print("Better")
        print("Faster")
    	print("Stronger")

      File <string>:4
        print("Stronger")
                         ^
    TabError: inconsistent use of tabs and spaces in indentation

In [51]:

    variable_with_a_complicated_name = 1
    print(variable_with_complicated_name)

    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    Cell In[51], line 2
          1 variable_with_a_complicated_name = 1
    ----> 2 print(variable_with_complicated_name)

    NameError: name 'variable_with_complicated_name' is not defined

A `NameError` is raised when you use a name that Python does not know about. Often this means you made a typo. (To avoid these, use tab-completion.) Or we just forgot to define a variable or function.

In [52]:

    function_we_never_defined()

    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    Cell In[52], line 1
    ----> 1 function_we_never_defined()

    NameError: name 'function_we_never_defined' is not defined

When accessing attributes from an object, the same problem usually triggers an `AttributeError`

In [53]:

    my_dict = {}
    my_dict.append(1)

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Cell In[53], line 2
          1 my_dict = {}
    ----> 2 my_dict.append(1)

    AttributeError: 'dict' object has no attribute 'append'

### Handling Errors¶

In some cases, you might expect an error and know how to handle it. A typical example is when handling user input:

In [54]:

    int(input("Enter an integer: "))

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[54], line 1
    ----> 1 int(input("Enter an integer: "))

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

In [55]:

    try:
        print(int(input("Enter an integer: ")))
    except ValueError:
        print("This is not an integer")

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[55], line 2
          1 try:
    ----> 2     print(int(input("Enter an integer: ")))
          3 except ValueError:
          4     print("This is not an integer")

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

Hints:

- The easiest way to determine which error to except is to trigger it.
- Try keep the code inside the try-block minimal. (Otherwise the chances for errors in there rise)
- You can have several except clauses following one try block.

In [56]:

    input_ = input("Enter an integer: ")
    try:
        print(1/int(input_))
    except ValueError:
        print("This is not an integer")
    except ZeroDivisionError:
        print("Division by Zero")

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[56], line 1
    ----> 1 input_ = input("Enter an integer: ")
          2 try:
          3     print(1/int(input_))

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

If you have a block of code you want to execute even if an error occured, you can use `fianlly`:

In [57]:

    input_ = input("Enter an integer: ")
    try:
        print(int(input_))
    finally:
        print("Goodbye")

    ---------------------------------------------------------------------------
    StdinNotImplementedError                  Traceback (most recent call last)
    Cell In[57], line 1
    ----> 1 input_ = input("Enter an integer: ")
          2 try:
          3     print(int(input_))

    File /usr/lib/python3/dist-packages/ipykernel/kernelbase.py:1281, in Kernel.raw_input(self, prompt)
       1279 if not self._allow_stdin:
       1280     msg = "raw_input was called, but this frontend does not support input requests."
    -> 1281     raise StdinNotImplementedError(msg)
       1282 return self._input_request(
       1283     str(prompt),
       1284     self._parent_ident["shell"],
       1285     self.get_parent("shell"),
       1286     password=False,
       1287 )

    StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

This is most useful when you need to clean up something. (eg. close an open file or disconnect from a database)

## Reading and Writing Files¶

So far we used `numpy.loadtxt` to read files. This is a good choice for simple data files. In addition we will look at other modules for reading data from files later. Here I want to introduce the most basic file access available in Python directly.

The concept to keep in mind when working with files directly is, that files are a long chain of symbols. When you open them, the operating system puts a pointer onto the first symbol and then advances the pointer as you read the file.

In [58]:

    f = open("data/countries-europe.csv")
    f.read(1)

Out[58]:

    '#'

In [59]:

    for _ in range(6):
        print(f.read(1))

    p
    y
    t
    h
    o
    n

if you repeat the last cell it will keep reading new symbols until you reach the end of the file. Then it will keep returning empty strings, as the pointer is now stuck at the end of the file.

If you want to change the position of the pointer you can use `seek`. A common use case is to reset the pointer to the beginning of a file without opening the file again:

In [60]:

    f.seek(0)
    for _ in range(7):
        print(f.read(1))

    #
    p
    y
    t
    h
    o
    n

In most cases you are probably not interested in individual symbols but in full lines. Python provides two useful functions for this:

- `readline()` to read until the next line break.
- `readlines()` to read all remaining lines in a file, store as list

In [61]:

    f.readline()

Out[61]:

    ' row index,country\n'

In [62]:

    f.readlines()

Out[62]:

    ['0,Austria\n',
     '1,Belgium\n',
     '2,Bulgaria\n',
     '3,Croatia\n',
     '4,Cyprus\n',
     '5,Czechia\n',
     '6,Denmark\n',
     '7,Estonia\n',
     '8,Finland\n',
     '9,France\n',
     '10,Germany\n',
     '11,Greece\n',
     '12,Hungary\n',
     '13,Ireland\n',
     '14,Italy\n',
     '15,Latvia\n',
     '16,Lithuania\n',
     '17,Luxembourg\n',
     '18,Malta\n',
     '19,Netherlands\n',
     '20,Poland\n',
     '21,Portugal\n',
     '22,Romania\n',
     '23,Slovak Republic\n',
     '24,Slovenia\n',
     '25,Spain\n',
     '26,Sweden\n',
     '27,United Kingdom\n']

`readlines` is usually the easiest way to work with files. However for very large files this needs a lot of memory. Using `readline` can avoid this problem.

Once you are done working with a file you should close it

In [63]:

    f.close()

### Using a Context Manager¶

As mentioned in the last section, we can wrap our file handling code in `try...finally...` if we want to make sure the file is closed properly even in case of an error.

However, since this is a very common pattern, there is an even easier solution:

In [64]:

    with open("data/countries-europe.csv") as f:
        lines = f.readlines()
    print(lines[4])
    #f.seek(0)

    3,Croatia

`with` uses a Context Manager to execute a block of code. A context manager defines `try...except...finaly...` blocks and wraps them around a given code-block.

### Writing Files¶

Writing a file is not much more difficult then reading it:

In [65]:

    with open("data/tmp.txt", "w") as f:
        for i in range(10):
            f.write("Yes!\n")
    with open("data/tmp.txt", "a") as f:
        f.write("NO!\n")

In [66]:

    !cat data/tmp.txt

    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    Yes!
    NO!

The second argument to `open` defines the mode how the file should be opened. The default is for reading text, the most common other modes are:

- `w` for writing a file (deleting content if the file exists)
- `a` for appending to a file

For more details, see the documentation.

## Strings in Depth¶

Strings have several more complex topics as well.

### String functions¶

Strings come with a large number of helpful functions: https://docs.python.org/3/library/stdtypes.html#string-methods

- you can search and replace in the string
- you can split the string on given characters
- you can strip characters from its ends
- you can change the case
- ...

In [67]:

    "This is an Example String".find("is") #vs " is "; vs `in`

Out[67]:

    2

In [68]:

    "This is an Example String".replace("Example", "Demo")

Out[68]:

    'This is an Demo String'

In [69]:

    "This is an Example String".split() # vs. split("a")

Out[69]:

    ['This', 'is', 'an', 'Example', 'String']

In [70]:

    print("    This is an Example String   \n".strip())

    This is an Example String

In [71]:

    "this and that".strip("t") # vs lstrip, rstrip

Out[71]:

    'his and tha'

In [72]:

    "This is an Example String".lower() # upper, title

Out[72]:

    'this is an example string'

### Encoding¶

Internally a computer stores strings as a sequence of numbers. What letters the individual numbers represent is defined by the encoding. Unfortunately the information what encoding was used is often not provided with the string. This can cause a lot of annyoing problems.

Python3 strings always use "utf-8" encoding and avoids many problems. However input you receive might still have been stored in some other encoding. A wrong encoding can result in strange characters displayed or in error messages.

Different operating systems use different default encodings for text files and Python tries the default encoding when opening a file.

In [73]:

    with open("data/preamble.txt") as file_: # , encoding="iso-8859-1"
        for l in file_.readlines()[:5]:
            print(l)

    ---------------------------------------------------------------------------
    UnicodeDecodeError                        Traceback (most recent call last)
    Cell In[73], line 2
          1 with open("data/preamble.txt") as file_: # , encoding="iso-8859-1"
    ----> 2     for l in file_.readlines()[:5]:
          3         print(l)

    File /usr/lib/python3.12/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
        319 def decode(self, input, final=False):
        320     # decode input (taking the buffer into account)
        321     data = self.buffer + input
    --> 322     (result, consumed) = self._buffer_decode(data, self.errors, final)
        323     # keep undecoded input until the next call
        324     self.buffer = data[consumed:]

    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 2: invalid continuation byte

When reading data with a different encoding you need to tell Python what encoding your input is. Python will then covert this to utf-8.

If for some reason you need to output text in some other encoding, you will have to convert again. Luckily this is trivial.

In [74]:

    "Grüäzi".encode("iso-8859-1")

Out[74]:

    b'Gr\xfc\xe4zi'

`str.encode` returns a bytes-Object. This is similar to a string, but not considered text by python. Instead bytes objects actually behave like immutable sequences of integers but are displayed in a strange way:

- integers representing "useful" symbols in ASCII (American Standard Code for Information Interchange) encoding, are displayed as that symbol
- all other integers are given in HEX, prepended with `\x`

(If you did not understand a lot, you don't miss much)

In [75]:

    bytes.fromhex('00 01 02 03 41 61 30 2B')

Out[75]:

    b'\x00\x01\x02\x03Aa0+'

To turn bytes into strings, you can use decode:

In [76]:

    "Grüäzi".encode("iso-8859-1").decode("iso-8859-1")

Out[76]:

    'Grüäzi'

In [77]:

    ## writing files with a different encoding
    #with open("data/tmp.txt", "w", encoding="iso-8859-1") as f:
    #    f.write("Grüäzi\n")
    #
    #with open("data/tmp.txt", "w+b") as f:
    #    f.write("Grüäzi\n".encode("iso-8859-1"))

### Raw Strings¶

By convention a backslash is used in most string processing to introduce special character combinations.

In [78]:

    print("One\n\tTwo")

    One
    	Two

Of course, it might be that you need such a combination in your string without it's special meaning (for example if you want to use LaTeX syntax in your plot-labels). In this case you have two options:

- you can escape the backslash with an additional backslash
- you can prefixed your string with an `r`, turning it into a raw-string

In [79]:

    print("One\\n\\tTwo")

    One\n\tTwo

In [80]:

    print(r"One\n\tTwo")

    One\n\tTwo

### String Formatting¶

Rather sooner than later, you will want to construct more complex strings. There are several different options for this. We will look at the most modern only.

We return to our compound interest code and change it such that it prints a nicer message.

In [81]:

    def with_interest(value, rate, years):
        return value*(1+rate/100)**years

    def compound_interest(value, rate, years):
        return with_interest(value, rate, years) - value

    balance = 1000
    interest = compound_interest(balance, 0.5, 5)
    print(f"The interest earned on {balance} CHF over 5 years will be {interest}")

    The interest earned on 1000 CHF over 5 years will be 25.25125312812429

An f-string starts with `f"` (or `f'` or `f"""` or ...) and can contain variables enclosed by curly-brackets `{}`. Python will look for the referenced variables in the current scope:

In [82]:

    print(f"At {rate} percent, the interest earned on {balance} CHF over 5 years will be {interest}")

    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    Cell In[82], line 1
    ----> 1 print(f"At {rate} percent, the interest earned on {balance} CHF over 5 years will be {interest}")

    NameError: name 'rate' is not defined

Note: `rate` exists inside the two functions, but not outside of them.

If we insert numbers into a string, we can even specify how they are displayed:

In [83]:

    balance = 1000
    rate = 0.5
    duration = 5
    interest = compound_interest(balance, rate, duration)
    print(f"At {rate:.2f} percent, the interest earned on \
    {balance:.2f} CHF over {duration} years will be {interest:.2f}")

    At 0.50 percent, the interest earned on 1000.00 CHF over 5 years will be 25.25

For details on number-formatting, have a look at https://docs.python.org/3/library/string.html#formatspec. (`.format` works similar to f-strings, but needs variables passed explicitly.)

There are a lot more things you can do with f-strings, try to avoid making things too complex.

## Command Line Arguments¶

So far we started all our scripts from the command line with `python3 interest.py`

and used `input()` if we wanted to obtain additional input from the user.

We could let the user pass additional input directly on the command line `python interest.py 1000 5 .5`

Of course we need to adapt our programm for this to work.

The simplest approach is to use the argument vector provided by the `sys` module. The argument vector is a list with the name of the script running at postion 0. This is followed by any other "word" given on the command line.

In [84]:

    ## this code does not work in a notebook
    #import sys
    #
    #def with_interest(value, rate, years):
    #    return value*(1+rate/100)**years
    #
    #def compound_interest(value, rate, years):
    #    return with_interest(value, rate, years) - value
    #
    # print(sys.argv)
    #balance = float(sys.argv[1])
    #duration = float(sys.argv[2])
    #rate = float(sys.argv[3])
    #res = compound_interest(balance, rate, duration)
    #print("At {rate} percent, the interest earned on {balance} CHF over {duration} years will be {interest}".format(
    #        rate=rate, balance=balance, duration=duration, interest=res))

A more elegant solution is to use the `argparse` module. This allows you to create more complex commandline interfaces easily.

In [85]:

    #import argparse
    #
    #def with_interest(value, rate, years):
    #    return value*(1+rate/100)**years
    #
    #def compound_interest(value, rate, years):
    #    return with_interest(value, rate, years) - value
    #
    #parser = argparse.ArgumentParser(description='calculate compound interest')
    #parser.add_argument('balance', type=float, help='the starting balance')
    #parser.add_argument('duration', type=int, help='the number of years to accumulate interest for')
    #parser.add_argument('--rate', type=float, default=0.5, help='the interest rate in percent')
    #args = parser.parse_args()
    #
    #res = compound_interest(args.balance, args.rate, args.duration)
    #print("At {rate} percent, the interest earned on {balance} CHF over {duration} years will be {interest}".format(
    #        rate=args.rate, balance=args.balance, duration=args.duration, interest=res))

The help of `argparse` is rather extenisve and it is easy to find examples online.

## Writing modules¶

Once your code gets bigger, it is a good idea to split it into several smaler files.

The easiest way to do this is to store all files in the same folder. In Python you can then import functions from other files just like you do this with system modules

In [86]:

    !cat interest.py

    def with_interest(value, rate, years):
        return value*(1+rate/100)**years

    def compound_interest(value, rate, years):
        return with_interest(value, rate, years) - value

In [87]:

    from interest import compound_interest

In [88]:

    compound_interest(1000,.5,10)

Out[88]:

    51.140132040789695

It is not much more difficult to organise your files into a folder structure. All you need to do is to create a file `__init__.py` in each folder of your module. In most cases you can leave `__init__.py` empty.

In [89]:

    from tools.interest import with_interest

In [90]:

    with_interest(100,5,1)

Out[90]:

    105.0

If inside your module folder you want to load another file, you can use the same syntax as you would use outside, giving the absolute module name.

In [91]:

    !cat tools/compound.py

    from tools.interest import with_interest

    def compound_interest(value, rate, years):
        return with_interest(value, rate, years) - value

### Importing Scripts¶

You can import from any python file. However if you have a script with code in the global scope, that code will be executed on import.

In [92]:

    !cat interest_script.py

    def with_interest(value, rate, years):
        return value*(1+rate/100)**years

    def compound_interest(value, rate, years):
        return with_interest(value, rate, years) - value

    print("Result:", compound_interest(1000, 0.5, 10))

In [93]:

    from interest_script import with_interest

    Result: 51.140132040789695

This is usually not what you want. To avoid this you can guard the executable part of your script with a special if statement.

In [94]:

    !cat interest_guarded.py

    def with_interest(value, rate, years):
        return value*(1+rate/100)**years

    def compound_interest(value, rate, years):
        return with_interest(value, rate, years) - value

    if __name__ == "__main__":
        print("Result:", compound_interest(1000, 0.5, 10))

In [95]:

    from interest_guarded import with_interest

## Useful Standard Modules¶

### sys¶

We have already seen the `sys` module. It provides access to some objects used or maintained by the interpreter. One of these (`sys.argv`) we have seen above. For more details, look at the documentation. (https://docs.python.org/3/library/sys.html)

### os¶

OS routines for the system your running on. Especially useful are the `os.path` functions (https://docs.python.org/3/library/os.html)

In [96]:

    import os.path
    os.path.join("dir", "filename.py")

Out[96]:

    'dir/filename.py'

In [97]:

    os.path.basename("dir/filename.py")

Out[97]:

    'filename.py'

In [98]:

    os.path.dirname("dir/subdir/file.py")

Out[98]:

    'dir/subdir'

In [99]:

    for path in os.walk("example_tree"):
        print(path)

    ('example_tree', ['subdir'], ['fileB', 'fileA'])
    ('example_tree/subdir', [], ['file2', 'file1'])

You might also want to use `glob`

In [100]:

    import glob

    matches = glob.glob("example_tree/**/file*", recursive=True)
    print("\n".join(matches))

    example_tree/fileB
    example_tree/fileA
    example_tree/subdir/file2
    example_tree/subdir/file1

### re¶

Support for regular expressions. Especially useful when you need to parse strings. (https://docs.python.org/3/library/re.html)

In [101]:

    import re
    print(re.findall(r"\d+.\d{2}", "At 0.50 percent, the interest earned on 1000.00 CHF over 5 years will be 25.25"))

    ['0.50', '1000.00', '25.25']

Look at the website for many more options and a detailed explanation of the syntax of regular expressions.

### datetime¶

Support for working with dates and time. (https://docs.python.org/3/library/datetime.html)

- Most important class datetime to represent date (year, month, day) and time (hour, minute, second, millisecond)
- `strptime` to load dates from a string
- `strftime` to print dates to a string
- Timezone info encodable via abstract base class of tzinfo, e.g. `pytz`
- `timedelta` is the difference between datetime objects and allows to make calculations

In [102]:

    import datetime as dt

defining a simple time is easy

In [103]:

    t1 = dt.datetime(2017,9,11,13,15)
    print(t1)
    print(t1.utctimetuple().tm_hour) # uses time-zone info to calculate UTC time

    2017-09-11 13:15:00
    13

time zone info can be added using timedelta

In [104]:

    tz_cest = dt.timezone(dt.timedelta(hours=+2))
    t2 = dt.datetime(2017,9,11,13,15,tzinfo=tz_cest)
    print(t2.hour)
    print(t2.utctimetuple().tm_hour) # uses time-zone info to calculate UTC time

    13
    11

create a string from time according to a given format

In [105]:

    output_string = t1.strftime("%d %b %Y %I:%M:%S %p")
    print(output_string)

    11 Sep 2017 01:15:00 PM

extract the date and time info from a string of given format

In [106]:

    input_string = "6 June 2016 8h45'32''"
    t3 = dt.datetime.strptime(input_string,"%d %B %Y %Hh%M'%S''")
    print(t3)

    2016-06-06 08:45:32

`pytz` provides predefined timezone objects that can be used to create datetime-objects with the right timezone info.

In [107]:

    import pytz
    tz_zurich = pytz.timezone("Europe/Zurich")
    # do not pass tz_zurich to the datetime constructor
    # tzinfo can not handle the varying timezone offsets stored by pytz
    t_zh = dt.datetime(2017,9,11,13,15,tzinfo=tz_zurich)
    print("bad: ", t_zh)
    t_zh = tz_zurich.localize(dt.datetime(2017,9,11,13,15))
    print("good:", t_zh)
    t_ny = t_zh.astimezone(pytz.timezone("America/New_York"))
    print(t_ny)

    bad:  2017-09-11 13:15:00+00:34
    good: 2017-09-11 13:15:00+02:00
    2017-09-11 07:15:00-04:00

In [108]:

    tz_kolkata = dt.timezone(dt.timedelta(hours=5, minutes=30))
    t_zh.astimezone(tz_kolkata)

Out[108]:

    datetime.datetime(2017, 9, 11, 16, 45, tzinfo=datetime.timezone(datetime.timedelta(seconds=19800)))

you can do calculations with datetime objects

In [109]:

    tdelta = t1-t3
    print(tdelta)
    print(type(tdelta))

    462 days, 4:29:28
    <class 'datetime.timedelta'>

In [110]:

    tdelta +=dt.timedelta(1)  # add a day
    print(tdelta)
    tdelta+=dt.timedelta(hours=-5) # subtract  5 hours
    print(tdelta)
    print(tdelta+t2) # add the time delta to a datetime object

    463 days, 4:29:28
    462 days, 23:29:28
    2018-12-18 12:44:28+02:00

### CSV¶

If you need to read CSV data, python provides the relevant module. (https://docs.python.org/3/library/csv.html)

In [111]:

    import csv

In [112]:

    data = list()
    with open("data/countries-europe.csv") as csv_f:
        csv_reader = csv.reader(csv_f)
        for row in csv_reader:
            data.append(row)
            print(row)

    ['#python row index', 'country']
    ['0', 'Austria']
    ['1', 'Belgium']
    ['2', 'Bulgaria']
    ['3', 'Croatia']
    ['4', 'Cyprus']
    ['5', 'Czechia']
    ['6', 'Denmark']
    ['7', 'Estonia']
    ['8', 'Finland']
    ['9', 'France']
    ['10', 'Germany']
    ['11', 'Greece']
    ['12', 'Hungary']
    ['13', 'Ireland']
    ['14', 'Italy']
    ['15', 'Latvia']
    ['16', 'Lithuania']
    ['17', 'Luxembourg']
    ['18', 'Malta']
    ['19', 'Netherlands']
    ['20', 'Poland']
    ['21', 'Portugal']
    ['22', 'Romania']
    ['23', 'Slovak Republic']
    ['24', 'Slovenia']
    ['25', 'Spain']
    ['26', 'Sweden']
    ['27', 'United Kingdom']

## Other Useful Modules¶

There are many other useful modules. Based on your answers to my questions, I think the following modules might be of interest to many of you.

### Data Analysis¶

- `scipy` - the main module for data-analysis functionality
- `statsmodels` - statistical models, statistical tests, data exploration
- `pandas` - powerful data analysis and manipulation library

### Web Scraping¶

- `requests` - to access web-content
- `bs4` - to parse html or xml
- `selenium` - to control your browser

### Web Development¶

- `django` - web development framework
- `flask` - micro web framework

### Machine Learning¶

- `sklearn` - the main machine learning module

### Databases¶

- `sqlite3` - simple file based database

(I have not enough experience with other database-modules to recommend one.)

### Other¶

- `subprocess` - Start other processes and access their input and output
- `asyncio` - to parallelize io-bound code
- `multiprocessing` - to parallelize cpu-bound code