Unpickling throws ValueError

[tkram01]

This example code throws a ValueError when unpickling.

import pickle
import sys
from enum import Enum
from functools import partial

from tblib import pickling_support

pickling_support.install()


def my_function():
    return 5


class MyEnum(Enum):
    MY_CONSTANT = partial(my_function)


try:
    properties = {}
    for prop in MyEnum:
        properties[prop.name] = MyEnum[prop].value()
    print(properties)
except:
    s1 = pickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)

    try:
        tb = pickle.loads(s1)
    except ValueError as e:
        print("Shouldn't throw")
        raise e

Results in:

Shouldn't throw
Traceback (most recent call last):
  File "/Users/kramer/Documents/tblib-example/example.py", line 22, in <module>
    properties[prop.name] = MyEnum[prop].value()
  File "/anaconda/envs/tblib/lib/python3.6/enum.py", line 327, in __getitem__
    return cls._member_map_[name]
KeyError: <MyEnum.MY_CONSTANT: functools.partial(<function my_function at 0x106bf4a60>)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kramer/Documents/tblib-example/example.py", line 31, in <module>
    raise e
  File "/Users/kramer/Documents/tblib-example/example.py", line 28, in <module>
    tb = pickle.loads(s1)
  File "/anaconda/envs/tblib/lib/python3.6/enum.py", line 291, in __call__
    return cls.__new__(cls, value)
  File "/anaconda/envs/tblib/lib/python3.6/enum.py", line 533, in __new__
    return cls._missing_(value)
  File "/anaconda/envs/tblib/lib/python3.6/enum.py", line 546, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: functools.partial(<function my_function at 0x106bf4a60>) is not a valid MyEnum

Process finished with exit code 1

[jamadden]

The outer try/except block is raising a KeyError, because Enums are indexed by strings, but they iterate their (object) members:

>>> for prop in MyEnum:
...     MyEnum[prop]
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py", line 327, in __getitem__
    return cls._member_map_[name]
KeyError: <MyEnum.MY_CONSTANT: functools.partial(<function...>)>

The first argument to the KeyError is the invalid key, which was MyEnum.MY_CONSTANT. When you pickle the exception, you pickle that argument.

But that object itself can't be (un)pickled; that's the source of the ValueError (not tblib):

>>> import pickle
>>> pickle.loads(pickle.dumps(MyEnum.MY_CONSTANT))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py", line 291, in __call__
    return cls.__new__(cls, value)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py", line 533, in __new__
    return cls._missing_(value)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py", line 546, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: functools.partial(<function...>) is not a valid MyEnum

I don't know the rules for pickling/unpickling Enum instances, but this code seems to break them :)

[ionelmc]

I think this issue boils down to unpickleable exceptions (the KeyError(garbage)).

Note that there's no handling in tblib for exception objects (tblib only handles the traceback objects). So I think this problem is outside tblib's intended purpose.

Try only doing pickle.dumps(sys.exc_info()[2]).