tcti/py: initial commit #2749

williamcroberts · 2024-01-14T20:00:27Z

Implmenet a TCTI module that will allow folks to write TCTIs in Python3. This will allow for more crazy and complex TCTI scenarios without having to re-write everyones stacks to use tpm2-pytss.

TODO:

Check error paths
Add a test
doc cleanups and add examples

Example Commandline:

PYTHONPATH=$HOME tpm2_getcap --verbose --tcti=$(realpath ./src/tss2-tcti/.libs/libtss2-tcti-py.so.0.0.0):pytcti:$TPM2TOOLS_TCTI properties-fixed

Example TCTI:
For the ultimate in mind-bending: C -> Python -> C and back for efficiency :-p.

from tpm2_pytss import TCTILdr

class MyPyTCTI(object):
    def __init__(self, args: str):
        c = args.split(":", maxsplit=1)
        print(f"PYTHON: Initializing TCTI Ldr with mod: {c[0]} args: {c[1]}")
        self._tcti = TCTILdr(c[0], c[1])

    @property
    def magic(self):
         # Optional Method
         print("PYHTON magic")
         return 42

    def receive(self, timeout: int) -> bytes:
        print("PYTHON receive")
        return self._tcti.receive(timeout=timeout)

    def transmit(self, data:bytes):
        print("PYTHON transmit")
        self._tcti.transmit(data)

def tcti_init(args: str) -> MyPyTCTI:
    print(f"PYTHON tcti_init called with: {args}")
    return MyPyTCTI(args)

williamcroberts · 2024-01-14T20:20:30Z

Ahh I'm not supposed to link explicitly to libpython I guess: https://peps.python.org/pep-0513/#libpythonx-y-so-1

No just gotta do it right... Using python3-confifg tool

joholl · 2024-01-15T14:36:17Z

Woah, exciting! I was working on something similar, an out-of-tree Rust framework for writing TCTIs. It is already fully functional.

However, it escalated a little, so I am currently working on marshalling/unmarshalling stuff...

pub mod lib {
    use tss2_tcti::define_api_symbols;
    use tss2_tcti::tcti::error::TctiError;
    use tss2_tcti::tcti::tcti::{Api, Info, State, Tcti, TctiLib};
    use tss2_tcti::tctildr::tcti_loader::TctiLoader;
    use tss2_tcti_sys::tpm2_tss;

    #[repr(C)]
    #[derive(Debug)]
    pub struct TctiFoobar {
        api: Api,
        state: State,
        child_tcti: Option<TctiLoader>,
    }

    impl TctiLib for TctiFoobar {
        const INFO: Info<'static> = Info {
            name: b"tpm2_tcti-foobar\0",
            description: b"A demo tcti written in Rust.\0",
            config_help: b"Child TCTI config string as expected by TctiLdr.\0",
        };
        const MAGIC: u64 = 0x44a50b8745675fe5;

        fn new(conf: &str) -> Result<Self, TctiError> {
            let mut tcti = Self {
                api: Self::get_api_static(),
                state: State::NotInitialized,
                child_tcti: None,
            };

            tcti.init(conf)?;

            Ok(tcti)
        }

        fn init_inner(&mut self, conf: &str) -> Result<(), TctiError> {
            self.api = TctiFoobar::get_api_static();
            self.child_tcti = Some(<TctiLoader as Tcti>::new(conf)?);
            self.state = State::Transmit;
            Ok(())
        }

        fn transmit_inner(&mut self, command: &[u8]) -> Result<(), TctiError> {
            println!("TX: {:?}", command);

            if let Some(child_tcti) = self.child_tcti.as_mut() {
                return child_tcti.transmit(command);
            }

            todo!()
        }

        fn receive_inner(&mut self) -> Result<Vec<u8>, TctiError> {
            if let Some(child_tcti) = self.child_tcti.as_mut() {
                return child_tcti.receive();
            }

            todo!()
        }

        fn get_state(&self) -> Option<State> {
            Some(self.state)
        }
        fn set_state(&mut self, state: State) {
            self.state = state;
        }
    }

    define_api_symbols!(TctiFoobar);
}

joholl · 2024-01-19T13:48:11Z

src/tss2-tcti/tcti-py.c

+        _global.refcnt = new_cnt;
+        if (new_cnt == 1) {
+            /* See: https://stackoverflow.com/questions/60719987/embedding-python-which-uses-numpy-in-c-doesnt-work-in-library-dynamically-loa */
+            _global.dlhandle = dlopen ("libpython3.so", RTLD_LAZY|RTLD_GLOBAL);


Why do we not link libpython3.so as a shared library? That would lead to compile-time errors if the libpython dependency is not installed (instead of runtime errors). Also, it would simplify the code.

EDIT: If i understand correctly, this is to enable multiple instances of tcti-py, right? In that case we would need dlopen()...

We actually do link it, but if other bindings are in the link chain, like CFFI, it will resolve the symbols for that and leave some of the symbols unresolved, so we add GLOBAL and LAZY to force the resolution of undefined symbols at run time.

That comment above is what led me down this path, it seems to work as expected. I didn't find any other way to work around it.

Hm, I'm not sure I have fully understood the issue. You mean because the python module uses tctildr?
Another thought: we need to get the pkg-config library name (for me "libpython3.10.so") via define, I think. For me, I had to change the hardcoded shared library name.

Hm, I'm not sure I have fully understood the issue. You mean because the python module uses tctildr? Another

No it has to do with symbol resolution and the dynamic linker loader, not anything tss2 specific.

we need to get the pkg-config library name (for me "libpython3.10.so") via define

Did that in the newest spin

whooo · 2024-01-28T14:19:48Z

src/tss2-tcti/tcti-py.c

+        _global.refcnt = new_cnt;
+        if (new_cnt == 1) {
+            /* See: https://stackoverflow.com/questions/60719987/embedding-python-which-uses-numpy-in-c-doesnt-work-in-library-dynamically-loa */
+            _global.dlhandle = dlopen ("libpython3.so", RTLD_LAZY|RTLD_GLOBAL);


On my system (debian unstable) libpython3.so doesn't exist, but one for each installed python version (libpython3.11.so.1.0, etc), so tests fails with:

WARNING:tcti:src/tss2-tcti/tcti-py.c:442:Tss2_Tcti_Py_Init() Could not dlopen libpython3.so, some things may not work: libpython3.so: cannot open shared object file: No such file or directory ModuleNotFoundError: No module named 'pytcti' Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL) Python runtime state: finalizing (tstate=0x00007ff374562978)

Same... see my comment above.

Should work now, I parse out the library name, the proper package to have is python3-embed, figuring that out for pkg-config helped a bit to simplify things.

whooo · 2024-01-28T14:23:46Z

src/tss2-tcti/tcti-py.c

+py_error:
+    PyErr_Print();
+    SAFE_DECREF(py_buf);


This isn't really important, but have you looked at using the tss logging infrastructure for the output of tracebacks? Or providing some API for the python

It would take a lot of work to plumb everything I think, far more work than it's worth IMO. Unless I am missing something.

whooo · 2024-01-28T14:49:47Z

src/tss2-tcti/tcti-py.h

+        PyObject *receive;
+        PyObject *make_sticky;
+    } methods;


make_sticky doesn't seem to be used and is marked as unsupported.
Also, would be set_locality be useful?

Well I dropped these do to a weird bug, but I spent all day figuring it out, It was all from not clearing the error handler when get_attr() in the C code failed. Later I would get errors when calling the next Py_ but it was related to something N calls ago :-p. It was wierd that when you called transmit, it would say, no attribute named 'make_sticky'.

joholl · 2024-01-29T10:40:03Z

Works for me!

LD_LIBRARY_PATH=:../tpm2-tss/src/tss2-tcti/.libs PYTHONPATH=/path/to/python/script/ TSS2_LOG=tcti+trace tpm2 startup -c --tcti="py:foobar:libtpms"

When my python script errors during init (e.g. because I call it with args=None which the example script cannot handle), I get a seg fault, probably because we call tcti_py_finalize() on the not-yet-initialized context.

joholl · 2024-01-29T14:23:46Z

Also, I made tpmstream (master branch, not released yet) a tcti like in your example. So you can do the following now:

LD_LIBRARY_PATH=:../tpm2-tss/src/tss2-tcti/.libs  tpm2 startup -c --tcti="py:tpmstream:libtpms"

src/tss2-tcti/tctildr-dl.c

williamcroberts · 2024-02-14T00:24:23Z

Works for me!
LD_LIBRARY_PATH=:../tpm2-tss/src/tss2-tcti/.libs PYTHONPATH=/path/to/python/script/ TSS2_LOG=tcti+trace tpm2 startup -c --tcti="py:foobar:libtpms"
When my python script errors during init (e.g. because I call it with args=None which the example script cannot handle), I get a seg fault, probably because we call tcti_py_finalize() on the not-yet-initialized context.

Fixed that, it was from calling Py_Finalize() followed by a Py_XDECREF()

Implmenet a TCTI module that will allow folks to write TCTIs in Python3. This will allow for more crazy and complex TCTI scenarios without having to re-write everyones stacks to use tpm2-pytss. Example Commandline: ```bash PYTHONPATH=$HOME tpm2_getcap --verbose --tcti=py:pytctitabrmd properties-fixed ``` Example TCTI: For the ultimate in mind-bending: C -> Python -> C and back for efficiency :-p. ```python3 from tpm2_pytss import TCTILdr from typing import Union class MyPyTCTI(object): def __init__(self, args: str): c = args.split(":", maxsplit=1) mod = c[0] args = c[1] if len(c) > 1 else "None" print(f"PYTHON: Initializing TCTI Ldr with mod: {mod} args: {args}") self._tcti = TCTILdr(mod, args) @Property def magic(self) -> Union[bytes, int]: # Optional Method print("PYTHON magic") return 42 def receive(self, timeout: int) -> bytes: print("PYTHON receive") return self._tcti.receive(timeout=timeout) def transmit(self, data: bytes) -> None: print("PYTHON transmit") return self._tcti.receive(timeout=timeout) def tcti_init(args: str) -> MyPyTCTI: print(f"PYTHON tcti_init called with: {args}") return MyPyTCTI(args) ``` Signed-off-by: Bill Roberts <bill.c.roberts+gh@gmail.com>

williamcroberts · 2024-02-14T00:36:20Z

Building Fedora-34 with python3-devel, so once that is published I can add a test to the ci:

fedora-34: add python3-devel for pytcti tests tpm2-software-container#82

williamcroberts requested a review from whooo January 14, 2024 20:00

williamcroberts force-pushed the add-python-tcti branch 3 times, most recently from e970577 to c8feee6 Compare January 15, 2024 01:39

williamcroberts changed the title ~~[RFC] tcti/py: initial commit~~ tcti/py: initial commit Jan 15, 2024

joholl reviewed Jan 19, 2024

View reviewed changes

whooo suggested changes Jan 28, 2024

View reviewed changes

joholl mentioned this pull request Jan 29, 2024

Add tcti as used by tpm2-tss tcti-py joholl/tpmstream#21

Merged

joholl reviewed Feb 13, 2024

View reviewed changes

src/tss2-tcti/tctildr-dl.c Outdated Show resolved Hide resolved

williamcroberts force-pushed the add-python-tcti branch from c8feee6 to dbb61bd Compare February 14, 2024 00:17

williamcroberts force-pushed the add-python-tcti branch from dbb61bd to 5cc69ec Compare February 14, 2024 00:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tcti/py: initial commit #2749

tcti/py: initial commit #2749

williamcroberts commented Jan 14, 2024

williamcroberts commented Jan 14, 2024 •

edited

Loading

joholl commented Jan 15, 2024 •

edited

Loading

joholl Jan 19, 2024 •

edited

Loading

williamcroberts Jan 24, 2024

joholl Jan 29, 2024

williamcroberts Feb 13, 2024

williamcroberts Feb 14, 2024

whooo Jan 28, 2024

joholl Jan 29, 2024

williamcroberts Feb 14, 2024

whooo Jan 28, 2024

williamcroberts Feb 14, 2024

whooo Jan 28, 2024

williamcroberts Feb 14, 2024

joholl commented Jan 29, 2024 •

edited

Loading

joholl commented Jan 29, 2024

williamcroberts commented Feb 14, 2024

williamcroberts commented Feb 14, 2024

tcti/py: initial commit #2749

Are you sure you want to change the base?

tcti/py: initial commit #2749

Conversation

williamcroberts commented Jan 14, 2024

williamcroberts commented Jan 14, 2024 • edited Loading

joholl commented Jan 15, 2024 • edited Loading

joholl Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joholl commented Jan 29, 2024 • edited Loading

joholl commented Jan 29, 2024

williamcroberts commented Feb 14, 2024

williamcroberts commented Feb 14, 2024

williamcroberts commented Jan 14, 2024 •

edited

Loading

joholl commented Jan 15, 2024 •

edited

Loading

joholl Jan 19, 2024 •

edited

Loading

joholl commented Jan 29, 2024 •

edited

Loading