SECRET OF CSS

All the Ways To Introspect Python Objects at Runtime | by Martin Heinz | Oct, 2022


Tips and tricks for inspecting Python objects and getting information about your code at runtime

1*lVp3cIBHPb3t0r9erlhJkQ
Photo by Jesse Cason on Unsplash

Python provides a lot of ways to ask questions about your code. Whether it’s basic things like help() function, builtin functions like dir() or more advanced methods in inspect module — the tools are there to help you find the answers to your questions.

Let’s find out what kinds of questions about our own code can Python answer for us and how it can help us during debugging sessions, dealing with type annotations, validating inputs, and much more.

As already mentioned — there are a couple of categories of introspection tools/functions in Python — let’s start with the most basic one, which is the built-in functions.

Python includes a basic set of built-in functions, most of which we already know — such as len(), range() or print(). There are however a couple of obscure ones that can help us answer some questions about our code:

The locals(), globals() and hasattr(), can help us find out whether local/global variable or class instance attribute exists.

Furthermore, we can use built-in functions to also check whether a variable is a function:

There are a couple of ways to do that — the best option is to use callable(), if you’re however running Python 3.1, then you’d have to check for the presence of __call__ attribute using hasattr function instead. The last option would be to use isfunction() from inspect module, be careful with that one though, as it will return False for builtin functions such as sum, len or range because these are implemented in C, so they’re not Python functions.

Another thing you might want to check is whether a variable is a list (sequence) or a scalar:

The “obvious” solution is to use isinstance to check whether the variable is an instance of abstract class Sequence. That however won’t work with non-builtin types such as NumPy arrays. It also considers strings a sequence that is correct but might not be desirable. The alternative is to use hasattr to check whether the variable has __len__ attribute, this will work for NumPy arrays, but will also return True for dictionaries. The last option is to pass a tuple of types to isinstance to customize the behavior to your needs.

We’ve already seen how we can use globals() to check if a variable exists, we can however use it also to call a function by a string:

A similar strategy can be used for object (class) attributes with getattr(instance, "func_name")().

The last example with built-in functions uses locals(). Let’s say you have a function that takes a lot of arguments, all of which need to be passed to another function, and you don’t want to write them all out. Well, you can simply use locals() with the ** (destructuring) operator, like so:

The above should give you a decent idea of what you can do with the built-in functions, it’s however not an exhaustive list. There are couple more functions, such as dir() or vars(), so be sure to check out the docs to get a full picture.

If the above built-ins aren’t giving you all the answers, then we can dig a little deeper. Every object in Python has quite extensive set of attributes that can tell us more about a particular object.

These are essentially the values used to construct the output of help(object), so anything you find in the output of help(object) can be pulled from object attribute. For example, you can find their object’s doc string (.__doc__), function’s source code (.__code__), or traceback/stack information (e.g. .tb_frame).

Messing with attributes is not ideal though, but there’s a better way. Let’s take a look at inspect modul.

The inspect module leverages all the above attributes (and some more) to allow us to introspect our code more efficiently.

You might have already used the built-in dir() method to get all attributes of an object. inspect module has a similar one called getmembers():

inspect.getmembers() has the advantage of providing a second argument which can be used for filtering the attributes. Here we use it to filter out the variable attributes and functions respectively.

Another great use case for inspect module is debugging. You can, for example, use it to debug the state of a generator:

Here we define a dummy generator by putting yield in a body of a function. We can then test whether it’s really a generator using isgeneratorfunction(). We can also check its state using getgeneratorstate() which will return one of GEN_CREATED (not yet executed), GEN_RUNNING, GEN_SUSPENDED (waiting at yield) or GEN_CLOSED (consumed).

With help of inspect.signature you can also debug things related to function signature, such as mutable default arguments:

It’s common knowledge that you should not use mutable types for argument defaults, such as list, because they will get modified (mutated) during each function execution. Inspecting the function signature with inspect.signature() makes it very clear here.

While on the topic of argument defaults, we can also use the signature() function to read them:

This can be helpful if you want to pass the defaults of one function to another one. The above example shows that you can either iterate over all the arguments and pick out the ones that have a non-empty default value, or if you know the argument name then you can query it directly.

Some more advanced uses of signature() function include injecting extra arguments to a function using a decorator:

The above snippet defines a decorator called optional_debug which injects debug argument when applied to a function. It does this by first inspecting the function signature and checking for the presence of debug argument. If not present, it then replaces the original function signature with a new one with a keyword-only argument appended.

Now, let’s say you have a function that only declares *args and **kwargs for arguments. This makes the function very “general-purpose”, but makes parameter checking quite messy. We can solve this with Signature and Parameter classes from inspect module:

First, we define the expected parameters ( params) using the Parameter class, specifying their types and defaults, after which we create signature from them. Inside the “general-purpose” function, we use bind method of signature to bind the provided *args and **kwargs to the prepared signature. If the parameters don’t satisfy the signature, we receive an exception. If all is good, we can proceed to access the bound parameter using the return value of bind method.

The above worked for validating parameters inside a basic function, but what if we wanted to validate that a generator gets all the necessary parameters from a decorated function?

The above snippet shows an implementation of an authentication decorator that expects username parameter to be passed to the decorated function. To validate that the parameter is present, we use from_callable() method of Signature class which pulls the supplied parameters from the function. We then check whether username is in the returned tuple before performing any actual logic.

Moving on from the inspect.signature, you can also use the inspect module to introspect your source code files:

Here we simply look up the location (file) of some_func function using getfile. The same also works for builtin functions/objects, demonstrated above with datetime.date.

The inspect module also includes helpers for working with tracebacks. You can use it, for example, to find a source file and line that’s being currently executed:

There are couple of ways of doing that, the most straightforward is using getframeinfo(currentframe()), you can however also access the stack frames through stack() function. In that case, you will have to index into the stack to find the correct frame.

Another option is to use currentframe() directly, in that case, you will have to access the f_back and f_lineno to find the correct frame and line respectively.

The next thing you can do with traceback helpers, is to access the caller object:

To do so, we use inspect.currentframe().f_back.f_locals['self'] which gives us access to the “parent” object. This should only ever be used for debugging – in case you need to access the caller object, you should pass it into the function as an argument, rather than crawling through the stack.

Last but not least, if you’re using type hints in your code, then you might be familiar with typing.get_type_hints which helps you inspect type hints. This function however will often throw NameError (especially when called on a class), so if you’re on Python 3.10 you should probably switch to using inspect.get_annotations which handles many edge cases for you:

For more details about accessing annotations in Python 3.10 and beyond, see docs.

As we’ve seen, there are more than enough tools to introspect and interrogate your Python code. Some of these are quite obscure, and you might never need to use them, but it’s good to be aware that they exist, especially the ones that can be used for debugging.

If however the above isn’t enough, or you just want to dive a bit deeper, then you might want to take a look at ast, which can be used to traverse and inspect the syntax tree or even the dis module used for disassembling Python bytecode.



News Credit

%d bloggers like this: