Session Five: Advanced Argument passing, List and Dict Comprehensions, Lambda and Functional programming¶

Review/Questions¶

Review of Previous Class¶

Dictionaries

Exceptions

Files, etc.

Homework review¶

Homework Questions?

My Solutions to the dict/set lab, and some others in the class repo in: Solutions

A few tidbits:

dicts aren’t sorted, so what if you want to do something in a sorted way?

The “old” way:

keys = d.keys()
keys.sort()
for key in keys:
    ...

collections.OrderedDict

sorted()

(demo)

Advanced Argument Passing¶

Keyword arguments¶

When defining a function, you can specify only what you need – in any order

In [151]: def fun(x,y=0,z=0):
        print x,y,z
   .....:
In [152]: fun(1,2,3)
1 2 3
In [153]: fun(1, z=3)
1 0 3
In [154]: fun(1, z=3, y=2)
1 2 3

A Common Idiom:

def fun(x, y=None):
    if y is None:
        do_something_different
    go_on_here

Can set defaults to variables

In [156]: y = 4
In [157]: def fun(x=y):
    print "x is:", x
   .....:
In [158]: fun()
x is: 4

Defaults are evaluated when the function is defined

In [156]: y = 4
In [157]: def fun(x=y):
    print "x is:", x
   .....:
In [158]: fun()
x is: 4
In [159]: y = 6
In [160]: fun()
x is: 4

Function arguments in variables¶

function arguments are really just

a tuple (positional arguments)
a dict (keyword arguments)

def f(x, y, w=0, h=0):
    print "position: %s, %s -- shape: %s, %s"%(x, y, w, h)

position = (3,4)
size = {'h': 10, 'w': 20}

>>> f( *position, **size)
position: 3, 4 -- shape: 20, 10

Function parameters in variables¶

You can also pull the parameters out in the function as a tuple and a dict:

def f(*args, **kwargs):
    print "the positional arguments are:", args
    print "the keyword arguments are:", kwargs

In [389]: f(2, 3, this=5, that=7)
the positional arguments are: (2, 3)
the keyword arguments are: {'this': 5, 'that': 7}

Passing a dict to the `string.format()` method¶

Now that you know that keyword args are really a dict, you can do this nifty trick:

The format method takes keyword arguments:

In [24]: u"My name is {first} {last}".format(last=u"Barker", first=u"Chris")
Out[24]: u'My name is Chris Barker'

Build a dict of the keys and values:

In [25]: d = {u"last":u"Barker", u"first":u"Chris"}

And pass to format()``with ``**

In [26]: u"My name is {first} {last}".format(**d)
Out[26]: u'My name is Chris Barker'

LAB¶

Let’s do this right now:

keyword arguments

Write a function that has four optional parameters (with defaults):
- foreground_color
- background_color
- link_color
- visited_link_color
Have it print the colors (use strings for the colors)
Call it with a couple different parameters set
Have it pull the parameters out with *args, **kwargs

A bit more on mutability (and copies)¶

mutable objects¶

We’ve talked about this: mutable objects can have their contents changed in place.

Immutable objects can not.

This has implications when you have a container with mutable objects in it:

In [28]: list1 = [ [1,2,3], ['a','b'] ]

one way to make a copy of a list:

In [29]: list2 = list1[:]

In [30]: list2 is list1
Out[30]: False

they are different lists.

What if we set an element to a new value?

In [31]: list1[0] = [5,6,7]

In [32]: list1
Out[32]: [[5, 6, 7], ['a', 'b']]

In [33]: list2
Out[33]: [[1, 2, 3], ['a', 'b']]

So they are independent.

But what if we mutate an element?

In [34]: list1[1].append('c')

In [35]: list1
Out[35]: [[5, 6, 7], ['a', 'b', 'c']]

In [36]: list2
Out[36]: [[1, 2, 3], ['a', 'b', 'c']]

uuh oh! mutating an element in one list mutated the one in the other list.

Why is that?

In [38]: list1[1] is list2[1]
Out[38]: True

The elements are the same object!

This is known as a “shallow” copy – Python doesn’t want to copy more than it needs to, so in this case, it makes a new list, but does not make copies of the contents.

Same for dicts (and any container type)

If the elements are immutable, it doesn’t really make a differnce – but be very careful with mutable elements.

The copy module¶

most objects have a way to make copies (dict.copy() for instance).

but if not, you can use the copy module to make a copy:

In [39]: import copy

In [40]: list3 = copy.copy(list2)

In [41]: list3
Out[41]: [[1, 2, 3], ['a', 'b', 'c']]

This is also a shallow copy.

But there is another option:

In [3]: list1
Out[3]: [[1, 2, 3], ['a', 'b', 'c']]

In [4]: list2 = copy.deepcopy(list1)

In [5]: list1[0].append(4)

In [6]: list1
Out[6]: [[1, 2, 3, 4], ['a', 'b', 'c']]

In [7]: list2
Out[7]: [[1, 2, 3], ['a', 'b', 'c']]

deepcopy recurses through the object, making copies of everything as it goes.

I happened on this thread on stack overflow:

http://stackoverflow.com/questions/3975376/understanding-dict-copy-shallow-or-deep

The OP is pretty confused – can you sort it out?

Make sure you understand the difference between a reference, a shallow copy, and a deep copy.

Mutables as default arguments:¶

Another “gotcha” is using mutables as default arguments:

In [11]: def fun(x, a=[]):
   ....:     a.append(x)
   ....:     print a
   ....:

This makes sense: maybe you’d pass in a list, but the default is an empty list.

But:

In [12]: fun(3)
[3]

In [13]: fun(4)
[3, 4]

Huh?!

Remember that that default argument is defined when the function is created: there will be only one list, and every time the function is called, that same list is used.

The solution:

The standard practice for such a mutable default argument:

In [15]: def fun(x, a=None):
   ....:     if a is None:
   ....:         a = []
   ....:     a.append(x)
   ....:     print a
In [16]: fun(3)
[3]
In [17]: fun(4)
[4]

You get a new list every time the function is called

List and Dict Comprehensions¶

List comprehensions¶

A bit of functional programming

consider this common for loop structure:

new_list = []
for variable in a_list:
    new_list.append(expression)

This can be expressed with a single line using a “list comprehension”

new_list = [expression for variable in a_list]

What about nested for loops?

new_list = []
for var in a_list:
    for var2 in a_list2:
        new_list.append(expression)

Can also be expressed in one line:

new_list =  [exp for var in a_list for var2 in a_list2]

You get the “outer product”, i.e. all combinations.

(demo)

But usually you at least have a conditional in the loop:

new_list = []
for variable in a_list:
    if something_is_true:
        new_list.append(expression)

You can add a conditional to the comprehension:

new_list = [expr for var in a_list if something_is_true]

(demo)

Examples:

In [341]: [x**2 for x in range(3)]
Out[341]: [0, 1, 4]

In [342]: [x+y for x in range(3) for y in range(5,7)]
Out[342]: [5, 6, 6, 7, 7, 8]

In [343]: [x*2 for x in range(6) if not x%2]
Out[343]: [0, 4, 8]

Remember this from last week?

[name for name in dir(__builtin__) if "Error" in name]
['ArithmeticError',
 'AssertionError',
 'AttributeError',
 'BufferError',
 'EOFError',
 ....

Set Comprehensions¶

You can do it with sets, too:

new_set = { value for variable in a_sequence }

same as for loop:

new_set = set()
for key in a_list:
    new_set.add(value)

Example: finding all the vowels in a string...

In [19]: s = "a not very long string"

In [20]: vowels = set('aeiou')

In [21]: { let for let in s if let in vowels }
Out[21]: {'a', 'e', 'i', 'o'}

Side note: why did I do set('aeiou') rather than just aeiou ?

Dict Comprehensions¶

Also with dictionaries

new_dict = { key:value for variable in a_sequence}

same as for loop:

new_dict = {}
for key in a_list:
    new_dict[key] = value

Example

In [22]: { i: "this_%i"%i for i in range(5) }
Out[22]: {0: 'this_0', 1: 'this_1', 2: 'this_2',
          3: 'this_3', 4: 'this_4'}

(not as useful with the dict() constructor...)

Anonymous functions¶

lambda¶

In [171]: f = lambda x, y: x+y
In [172]: f(2,3)
Out[172]: 5

Content can only be an expression – not a statement

Anyone remember what the difference is?

Called “Anonymous”: it doesn’t need a name.

It’s a python object, it can be stored in a list or other container

In [7]: l = [lambda x, y: x+y]
In [8]: type(l[0])
Out[8]: function

And you can call it:

In [9]: l[0](3,4)
Out[9]: 7

Functions as first class objects¶

You can do that with “regular” functions too:

In [12]: def fun(x,y):
   ....:     return x+y
   ....:
In [13]: l = [fun]
In [14]: type(l[0])
Out[14]: function
In [15]: l[0](3,4)
Out[15]: 7

Functional Programming¶

map¶

map “maps” a function onto a sequence of objects – It applies the function to each item in the list, returning another list

In [23]: l = [2, 5, 7, 12, 6, 4]
In [24]: def fun(x):
             return x*2 + 10
In [25]: map(fun, l)
Out[25]: [14, 20, 24, 34, 22, 18]

But if it’s a small function, and you only need it once:

In [26]: map(lambda x: x*2 + 10, l)
Out[26]: [14, 20, 24, 34, 22, 18]

filter¶

filter “filters” a sequence of objects with a boolean function – It keeps only those for which the function is True

To get only the even numbers:

In [27]: l = [2, 5, 7, 12, 6, 4]
In [28]: filter(lambda x: not x%2, l)
Out[28]: [2, 12, 6, 4]

reduce¶

reduce “reduces” a sequence of objects to a single object with a function that combines two arguments

To get the sum:

In [30]: l = [2, 5, 7, 12, 6, 4]
In [31]: reduce(lambda x,y: x+y, l)
Out[31]: 36

To get the product:

In [32]: reduce(lambda x,y: x*y, l)
Out[32]: 20160

Comprehensions¶

Couldn’t you do all this with comprehensions?

Yes:

In [33]: [x+2 + 10 for x in l]
Out[33]: [14, 17, 19, 24, 18, 16]
In [34]: [x for x in l if not x%2]
Out[34]: [2, 12, 6, 4]

(Except Reduce)

But Guido thinks almost all uses of reduce are really sum()

Functional Programming¶

Comprehensions and map, filter, reduce are all “functional programming” approaches}

map, filter and reduce pre-date comprehensions in Python’s history

Some people like that syntax better

And “map-reduce” is a big concept these days for parallel processing of “Big Data” in NoSQL databases.

(Hadoop, MongoDB, etc.)

A bit more about lambda¶

Can also use keyword arguments}

In [186]: l = []
In [187]: for i in range(3):
    l.append(lambda x, e=i: x**e)
   .....:
In [189]: for f in l:
    print f(3)
1
3
9

Note when the keyword argument is evaluated: this turns out to be very handy!

Homework¶

List comprehensions¶

Note: this is a bit of a “backwards” exercise – we show you code, you figure out what it does.

As a result, not much to submit – but so we can give you credit, submit a file with a solution to the final problem.

>>> feast = ['lambs', 'sloths', 'orangutans', 'breakfast cereals', 'fruit bats']

>>> comprehension = [delicacy.capitalize() for delicacy in feast]

What is the output of:

>>> comprehension[0]
???

>>> comprehension[2]
???

(figure it out before you try it)

>>> feast = ['spam', 'sloths', 'orangutans', 'breakfast cereals',
            'fruit bats']

>>> comprehension = [delicacy for delicacy in feast if len(delicacy) > 6]

What is the output of:

>>> len(feast)
???

>>> len(comprehension)
???

(figure it out first!)

>>> list_of_tuples = [(1, 'lumberjack'), (2, 'inquisition'), (4, 'spam')]

>>> comprehension = [ skit * number for number, skit in list_of_tuples ]

What is the output of:

>>> comprehension[0]
???

>>> len(comprehension[2])
???

>>> list_of_eggs = ['poached egg', 'fried egg']

>>> list_of_meats = ['lite spam', 'ham spam', 'fried spam']

>>> comprehension = [ '{0} and {1}'.format(egg, meat) for egg in list_of_eggs for meat in list_of_meats]

What is the output of:

>>> len(comprehension)
???

>>> comprehension[0]
???

>>> comprehension = { x for x in 'aabbbcccc'}

What is the output of:

>>> comprehension
???

>>> dict_of_weapons = {'first': 'fear',
                       'second': 'surprise',
                       'third':'ruthless efficiency',
                       'forth':'fanatical devotion',
                       'fifth': None}
>>> dict_comprehension = \
{ k.upper(): weapon for k, weapon in dict_of_weapons.iteritems() if weapon}

What is the output of:

>>> 'first' in dict_comprehension
???
>>> 'FIRST' in dict_comprehension
???
>>> len(dict_of_weapons)
???
>>> len(dict_comprehension)
???

`dict` and `set` comprehensions¶

Let’s revisiting the dict/set lab – see how much you can do with comprehensions instead.

Specifically, look at these:

First a slightly bigger, more interesting (or at least bigger..) dict:

food_prefs = {"name": u"Chris",
              u"city": u"Seattle",
              u"cake": u"chocolate",
              u"fruit": u"mango",
              u"salad": u"greek",
              u"pasta": u"lasagna"}

1. Print the dict by passing it to a string format method, so that you get something like:

“Chris is from Seattle, and he likes chocolate cake, mango fruit,

greek salad, and lasagna pasta”

2. Using a list comprehension, build a dictionary of numbers from zero to fifteen and the hexadecimal equivalent (string is fine).

Do the previous entirely with a dict comprehension – should be a one-liner

4. Using the dictionary from item 1: Make a dictionary using the same keys but with the number of ‘a’s in each value. You can do this either by editing the dict in place, or making a new one. If you edit in place, make a copy first!

5. Create sets s2, s3 and s4 that contain numbers from zero through twenty, divisible 2, 3 and 4.

Do this with one set comprehension for each set.

What if you had a lot more than 3? – Don’t Repeat Yourself (DRY)

create a sequence that holds all three sets

loop through that sequence to build the sets up – so no repeated code.

Extra credit: do it all as a one-liner by nesting a set comprehension inside a list comprehension. (OK, that may be getting carried away!)

lambda and keyword argument magic¶

Write a function that returns a list of n functions, such that each one, when called, will return the input value, incremented by an increasing number.

Use a for loop, lambda, and a keyword argument

( Extra credit ):

Do it with a list comprehension, instead of a for loop

Not clear? here’s what you should get

In [96]: the_list = function_builder(4)
### so the_list should contain n functions (callables)
In [97]: the_list[0](2)
Out[97]: 2
## the zeroth element of the list is a function that add 0
## to the input, hence called with 2, returns 2
In [98]: the_list[1](2)
Out[98]: 3
## the 1st element of the list is a function that adds 1
## to the input value, thus called with 2, returns 3
In [100]: for f in the_list:
    print f(5)
   .....:
5
6
7
8
### If you loop through them all, and call them, each one adds one more
to the input, 5... i.e. the nth function in the list adds n to the input.

Functional files¶

Write a program that takes a filename and “cleans” the file be removing all the leading and trailing whitespace from each line.

Read in the original file and write out a new one, either creating a new file or overwriting the existing one.

Give your user the option of which to perform.

Use map() to do the work.

Write a second version using a comprehension.

sys.argv hold the command line arguments the user typed in. If the user types:

$ python the_script a_file_name

Then:

import sys
filename = sys.argv[1]

will get filename == "a_file_name"

Session Five: Advanced Argument passing, List and Dict Comprehensions, Lambda and Functional programming¶

Review/Questions¶

Review of Previous Class¶

Homework review¶

Advanced Argument Passing¶

Keyword arguments¶

Function arguments in variables¶

Function parameters in variables¶

Passing a dict to the string.format() method¶

LAB¶

A bit more on mutability (and copies)¶

mutable objects¶

The copy module¶

Mutables as default arguments:¶

List and Dict Comprehensions¶

List comprehensions¶

Set Comprehensions¶

Dict Comprehensions¶

Anonymous functions¶

lambda¶

Functions as first class objects¶

Functional Programming¶

map¶

filter¶

reduce¶

Comprehensions¶

Functional Programming¶

A bit more about lambda¶

Homework¶

List comprehensions¶

dict and set comprehensions¶

lambda and keyword argument magic¶

Functional files¶

Recommended Reading¶

Passing a dict to the `string.format()` method¶

`dict` and `set` comprehensions¶