Python Decorators

Functions are "first class citizens" in Python—they are objects just like everything else and can be passed as arguments to other functions. This means we can define a function call_fun which takes in a function, pass in fun as a parameter, and call fun during the execution of call_fun.

def call_fun(fun):
    print('This is call_fun, which calls fun()...')
    fun()

print("Executing 'call_fun(fun)'...")
call_fun(fun)

Executing 'call_fun(fun)'...
This is call_fun, which calls fun()...
This is fun

We can use the idea from call_fun, of passing functions through arguments, to augment the behavior of fun. We create a function with three specific traits and then reassign fun to be this function.

The three traits are:

1. The function takes fun as an argument,
2. it defines another function within its definition that, among other things, calls fun, and
3. returns the inner function.

Let's name the function extend_fun and observe that it has the three traits. Let's also note the memory signature of fun:

print('fun:', fun)

fun: <function fun at 0x7f6f37b291b0>

Now we define the function with the three traits:

# 1. Define a function which takes a function, fun, as an argument
def extend_fun(fun):
    """Use argument to define an extended function.

    Arguments
    ---------
    fun : callable

      Function to be augmented.

    Returns
    -------
    callable

      Function which, among other things, calls the function which was
      passed in.

    """

    # 2. Define another function within its definition that, among
    #    other things, calls fun
    def extension():
        """Call fun and do something else."""
        print('This is the extension function')
        fun()  # this will be fun

    # 3. Return the inner function
    return extension

# reassign fun to be the inner function
fun = extend_fun(fun)
print('fun: ', fun)

print('Calling fun after it has been reassigned to extend_fun(fun)...')
fun()

fun:  <function extend_fun.<locals>.extension at 0x7f6ef82cbbe0>
Calling fun after it has been reassigned to extend_fun(fun)...
This is the extension function
This is fun

See how the memory signature changes when fun is reassigned? This tells us that fun no longer refers to what it used to.

However, notice that the original behavior of fun is preserved. (i.e. the message "This is fun" still prints.) extend_fun is a decorator. It extends the behavior of fun without altering the source code definiton of fun.

Decorators allow us to extend the functionality of extant functions, including those not explicitly defined by us, such as functions provided by third-party libraries. Decorators are useful for modifing the behavior of functions without altering their API signatures.

You might be wondering, why doesn't the extension function take an argument? Why don't we pass fun into it? The simple answer is we don't need to. fun is a valid name within the scope of extend_fun. We can use it right away.

Looking at the how the code is executed may make it clearer why extension doesn't need an argument. Let's redefine extend_fun with an additional print and go through the same sequence again.

def fun():
    print('This is fun')

print("Observing the execution order:\n")

def extend_fun(fun):
    print('Note that extend_fun is called during assignment')

    def extension():
        print("Note that extension is called only when fun is called")
        fun()  # this will be fun at runtime

    return extension

print('Reassigning fun as extension...')
fun = extend_fun(fun)

print('Calling fun (which has been reassigned to extension)...')
fun()

Observing the execution order:

Reassigning fun as extension...
Note that extend_fun is called during assignment
Calling fun (which has been reassigned to extension)...
Note that extension is called only when fun is called
This is fun

Observe that when we ressign fun as extend_fun, the extension function isn't called, even though the extend_fun code is processed. It's not until fun is called that extension is executed. This is because the extend_fun only defines extension. It never calls extension. It's only when fun is called after reassignment that the extension code is executed. It's also worth noting that the extend_fun code executes only once, during reassignment of fun.

Remember that function calls are made in Python by putting parentheses after a function object:

def returnsastring():
    return "this is the string"

print(returnsastring)
print(returnsastring())

<function returnsastring at 0x7f6ef82cbbe0>
this is the string

The function returnsastring returns a string when called. The characters "returnsastring" refers to the function object. Putting parentheses after "returnsastring" executes the function, returning the string. (Note that if this were run in interactive mode, you wouldn't need the calls to print.)

It might have been better to name extension as captivate. First-class functions enable closures, or captured lexical scopes. The term 'scope' describes where name references, such as fun, are valid. 'Lexical scope' means a name is valid depending on where in the source text the name is referenced.

To illustrate, we define a global variable wont_be_clobbered. The function definition of will_it_clobber provides a new scope. The scope only applies within the definition (one indentation). See how the global scope and the function scope each contain references for the name wont_be_clobbered? Because Python uses lexical scoping, the wont_be_clobbered inside will_it_clobber is, in fact, not clobbered.

wont_be_clobbered = True

def will_it_clobber():
    wont_be_clobbered = "Within a function == different lexical scope"
    print(wont_be_clobbered)

print("Global scope's 'wont_be_clobbered': ", wont_be_clobbered)
print('Calling will_it_clobber...')
will_it_clobber()
print("Global scope's 'wont_be_clobbered': ", wont_be_clobbered)
print()

Global scope's 'wont_be_clobbered':  True
Calling will_it_clobber...
Within a function == different lexical scope
Global scope's 'wont_be_clobbered':  True

To clobber wont_be_clobbered within will_it_clobber, we'd need to declare wont_be_clobbered as global:

wont_be_clobbered = False

def will_it_clobber():
    global wont_be_clobbered  # will totally be clobbered
    wont_be_clobbered = "global keyword binds the function's 'wont_be_clobbered' to the global 'wont_be_clobbered'"
    print(wont_be_clobbered)

print("Global scope's 'wont_be_clobbered': ", wont_be_clobbered)
print('Calling will_it_clobber...')
will_it_clobber()
print("Global scope's 'wont_be_clobbered': ", wont_be_clobbered)
print()

Global scope's 'wont_be_clobbered':  False
Calling will_it_clobber...
global keyword binds the function's 'wont_be_clobbered' to the global 'wont_be_clobbered'
Global scope's 'wont_be_clobbered':  global keyword binds the function's 'wont_be_clobbered' to the global 'wont_be_clobbered'

A closure captures state within a particular scope. Decorators work by using closures.

We can see that extend_fun defines a closure if instead of redefining fun in terms of the extend_fun, we redefine it as something else. Because extend_fun is a closure, changing fun outside of extend_fun won't change the call to fun made by extension.

Since we previously redefined fun, let's reset it to its original definition.

def fun():
    print('This is fun')

Let's also reset the definition of extend_fun to remove the extra print call.

def extend_fun(fun):
    def extension():
        print('This is the extension function')
        fun()  # this will be fun at runtime
    return extension

Now, let's see how extend_fun is a closure:

is_closure = extend_fun(fun)  # will be extension

print('Calling extension() as is_closure()...')
is_closure()
print()

print('Redefining fun as unfun...')
def unfun():
    print('This is actually unfun')

fun = unfun

print('Calling fun()...')
fun()
print()

print('Calling extension() as is_closure()...')
is_closure()

Calling extension() as is_closure()...
This is the extension function
This is fun

Redefining fun as unfun...
Calling fun()...
This is actually unfun

Calling extension() as is_closure()...
This is the extension function
This is fun

The call to fun made from inside is_closure didn't change when we redefined fun as unfun. This is because the lexical environment of extend_fun was preserved. Specifically, when fun was passed as the 'fun' argument, the fun reference in extend_fun remained as fun within the extend_fun scope whereas fun became unfun in the global scope.

So, what about passing in fun as an argument? We've seen that it's not needed, but can we do it? Yes, we can. The key is observing that the result of extend_fun is the extension function. If extension takes an argument, then whatever we assign the result to must also take an argument:

def fun():
    print("this is fun")

def extend_fun(fun):
    def extension(fun):
        print("this is the extension function")
        fun()
    return extension

print(extend_fun(fun))
extend_fun(fun)(fun)

<function extend_fun.<locals>.extension at 0x7f6ef82cbc70>
this is the extension function
this is fun

The first call, extend_fun(fun) returns the extension function. The second call calls extension with fun as a parameter.

It might be easier to see if written with extra parentheses:

(extend_fun(fun))(fun)

this is the extension function
this is fun

A decorator (noun) is a function which augments another function through use of a closure.

def decorator(fun):
    """Use argument to define an extended function.

    Arguments
    ---------
    fun : callable

      Function to be augmented.

    Returns
    -------
    callable

      Function which, among other things, calls the function which was
      passed in.

    """

    def closure():
        """Extend the function 'fun'.

        The function is extended by doing something before or after it
        is called.  Actions taken beforehand can be passed into the
        decorator function.  Return values can be used afterward.  In
        this way, the function's (apparent) return value can be changed
        entirely.

        """

        print('Doing something before calling fun')
        result = fun()
        print('Doing something after calling fun')

        # return a different result entirely, if you want
        return result + " plus modification"

    return closure

None

To "decorate" (verb) a function, replace the original reference with the closure extension (i.e. the decorator's return value):

def fun():
    return "fun return"

extended_fun = decorator(fun)
fun = extended_fun
print(fun())

Doing something before calling fun
Doing something after calling fun
fun return plus modification

or simply

def fun():
    return "fun return"

fun = decorator(fun)
print(fun())

Doing something before calling fun
Doing something after calling fun
fun return plus modification

Python gives a "shorthand" for the decoration pattern. Instead of

def fun():
    return "fun return"

def decorator(fun):
    def closure():
        print('Doing something before calling fun')
        result = fun()
        print('Doing something after calling fun')

        # return a different result entirely, if you want
        return result + " plus modification"

    return closure

fun = decorator(fun)
print(fun())

Doing something before calling fun
Doing something after calling fun
fun return plus modification

you can write

def decorator(fun):
    def closure():
        print('Doing something before calling fun')
        result = fun()
        print('Doing something after calling fun')

        # return a different result entirely, if you want
        return result + " plus modification"

    return closure

@decorator
def fun():
     return "fun return"

print(fun())

Doing something before calling fun
Doing something after calling fun
fun return plus modification

The shorthand syntax can be "stacked":

def inner(fun):
    def closure():
        print('Inner')
        result = fun()
        # return a different result entirely
    return closure

def outer(fun):
    def closure():
        print('Outer')
        result = fun()
        # return a different result entirely
    return closure

@outer
@inner
def my_func():
    pass

my_func()

Outer
Inner

Although people say "stacked", it'd be more accurate to say "composed":

def inner(fun):
    def closure():
        print('Inner')
        result = fun()
        # return a different result entirely
    return closure

def outer(fun):
    def closure():
        print('Outer')
        result = fun()
        # return a different result entirely
    return closure

def my_func():
    pass

my_func = outer(inner(my_func))
my_func()

Outer
Inner

When decorators take arguments, what's happening is we're defining a function which takes an argument and returns a decorator. Such functions, which themselves return functions, are often called "factories".

In the following example, we define a decorator "factory" create_decorator_using_arg. It takes an arg and uses that to define a decorator. The decorator is returned. We then pass in the function to be decorated.

def create_decorator_using_arg(arg):
    def decorator(fun):
        def extension():
            print('this is the extension with arg:', arg)
            fun()
        return extension
    return decorator

def fun():
    print("fun return")

custom_decorator = create_decorator_using_arg("banana")
fun = custom_decorator(fun)
fun()

this is the extension with arg: banana
fun return

The "shorthand" syntax looks like:

def create_decorator_using_arg(arg):
    def decorator(fun):
        def extension():
            print('this is the extension with arg:', arg)
            fun()
        return extension
    return decorator

@create_decorator_using_arg("banana")
def fun():
    print("fun return")

fun()

this is the extension with arg: banana
fun return