Skip to content

Commit

Permalink
Use base TextIOWrapper for stdout and stderr
Browse files Browse the repository at this point in the history
Continuation of #24
providing an approximate replica of sys.stdout and sys.stderr.
Also allow the files to be left open, and allow contents to
be obtained after the stream is detached or closed.

Closes #3
  • Loading branch information
jayvdb committed Aug 3, 2019
1 parent 18771de commit 951002c
Show file tree
Hide file tree
Showing 2 changed files with 199 additions and 21 deletions.
117 changes: 97 additions & 20 deletions src/stdio_mgr/stdio_mgr.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,57 @@
"""

import sys
from contextlib import contextmanager
from io import BufferedReader, BytesIO, StringIO, TextIOBase, TextIOWrapper
from contextlib import contextmanager, ExitStack, suppress
from io import BufferedRandom, BufferedReader, BytesIO, TextIOBase, TextIOWrapper

import attr


class _PersistedBytesIO(BytesIO):
"""Class to persist the stream after close.
The persisted stream is available at _closed_buf.

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

@jayvdb Perhaps revise the docstring to explicitly note that _closed_buf is a bytes copy of the stream contents, not a duplicate reference to the actual BytesIO?

"""

def close(self):
self._closed_buf = self.getvalue()
super().close()


class RandomTextIO(TextIOWrapper):
"""Class to capture writes to a buffer even when detached.

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

@jayvdb Is this accurate? It doesn't capture the writes; it just provides a means for accessing a copy of the buffer after detach?

This comment has been minimized.

Copy link
@jayvdb

jayvdb Aug 3, 2019

Author Contributor

No. This is the wrapper class. Inside is the reader/writer. Inside the reader/writer is the buffer.

The detach() decouples the reader/writer from the wrapper, often so it can be wrapped with something else.

The reader/writer is still a living useful object.

The copy of the bytes in the BytesIO subclass is only so that it can be accessed after .close() because close usually destroys everything. We could very easily not have _PersistedBytesIO, but having it means that a .close() (at any level) inside the context doesnt destroy the objects which are being monitored.

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

Right, but the way I read this docstring header, I would expect the following to successfully write foo down into the buffer:

>>> rtio= RandomTextIO(...)
>>> rtio.detach()
>>> rtio.write('foo')

But, .write() is just using the standard TextIOWrapper.write() machinery, which IIUC does not have this behavior.

Subclass of :cls:`~io.TextIOWrapper` that utilises an internal
buffer defaulting to utf-8 encoding.
All writes are flushed to the buffer.
This class provides :meth:`~RandomTextIO.getvalue` which emulates the
behavior of :meth:`~io.StringIO.getvalue`, decoding the buffer
using the :attr:`~io.TextIOWrapper.encoding`. The value is available
even if the stream is detached or closed.
"""

def __init__(self):
"""Initialise buffer with utf-8 encoding."""
self._stream = _PersistedBytesIO()
self._encoding = "utf-8"
self._buf = BufferedRandom(self._stream)
super().__init__(self._buf, encoding=self._encoding)

def write(self, *args, **kwargs):
"""Flush after each write."""
super().write(*args, **kwargs)
self.flush()

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

@jayvdb I'm guessing this matches the behavior of stdout/stderr?

This comment has been minimized.

Copy link
@jayvdb

jayvdb Aug 3, 2019

Author Contributor

Not really. It is needed to replicate the behaviour of StringIO.getvalue() with BufferedRandom, otherwise the stdin tee writes are not flushed through to the other-side of the BufferedRandom, and dont appear in StringIO.getvalue() unless there is code in the context to do the flushes. At some point we may need to replace BufferedRandom with BufferedRWPair, and re-implement getvalue() to do a look-ahead somehow (tell, flush, & seek back to original position)


def getvalue(self):
"""Obtain buffer of text sent to the stream."""
if self._stream.closed:
return self._stream._closed_buf.decode(self.encoding)
else:
return self._stream.getvalue().decode(self.encoding)


@attr.s(slots=False)
class TeeStdin(TextIOWrapper):
"""Class to tee contents to a side buffer on read.
Expand Down Expand Up @@ -152,13 +197,44 @@ def getvalue(self):
return self.buffer.peek().decode(self.encoding)


class _SafeCloseIOBase(TextIOBase):
"""Class to ignore ValueError when exiting the context.
Subclass of :cls:`~io.TextIOBase` that disregards ValueError which can
occur if the file has already been closed.
"""

def __exit__(self, exc_type, exc_value, traceback):
"""Suppress ValueError while exiting context.
ValueError may occur when the underlying

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

@jayvdb Please change to :exc:`ValueError` in the docstring body, here.

This module is getting substantial enough that I may want to stand up proper Sphinx docs for it.

OTOH, probably stdio_mgr should be the only piece of the public API, right?

In which case, maybe all of the other classes should be 'privatized'...? (I guess, I've assumed that the classes you've marked with leading underscores are meant to be private... is that the case? Or, did you do that to mark them as abstract?)

This comment has been minimized.

Copy link
@jayvdb

jayvdb Aug 3, 2019

Author Contributor

This module is getting substantial enough that I may want to stand up proper Sphinx docs for it.

Yes please, even if it is only to check the syntax, and not published anywhere, so it kicks me in the butt when I use the wrong syntax.

_SafeCloseIOBase and _PersistedBytesIO are very much private IMO.

The one use of RandomTextIO and TeeStdin being public is so they can be used with isinstance inside the caller logic to detect if it is operating under stdio_mgr or the real sys io streams.

OTOH, it is very likely that their constructors will see breaking changes as features are added.

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

OTOH, it is very likely that their constructors will see breaking changes as features are added.

Exactly. What about converting to _RandomTextIO and _TeeStdin, and then creating a new stub superclass (StdioMgrStream, or somesuch) for user isinstance checks?

buffer is detached or the file was closed.
"""
with suppress(ValueError):
super().__exit__(exc_type, exc_value, traceback)


class SafeCloseRandomTextIO(_SafeCloseIOBase, RandomTextIO):
"""Class to capture writes to a buffer even when detached and safely close.
Subclass of :cls:`~_SafeCloseIOBase` and :cls:`~TeeStdin`.

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

Typo: RandomTextIO, not TeeStdin

"""


class SafeCloseTeeStdin(_SafeCloseIOBase, TeeStdin):
"""Class to tee contents to a side buffer on read and safely close.
Subclass of :cls:`~_SafeCloseIOBase` and :cls:`~TeeStdin`.
"""


@contextmanager
def stdio_mgr(in_str=""):
def stdio_mgr(in_str="", close=True):
r"""Subsitute temporary text buffers for `stdio` in a managed context.
Context manager.
Substitutes empty :cls:`~io.StringIO`\ s for
Substitutes empty :cls:`~io.RandomTextIO`\ s for
:cls:`sys.stdout` and :cls:`sys.stderr`,
and a :cls:`TeeStdin` for :cls:`sys.stdin` within the managed context.
Expand All @@ -181,22 +257,32 @@ def stdio_mgr(in_str=""):
out_
:cls:`~io.StringIO` -- Temporary stream for `stdout`,
:cls:`~io.RandomTextIO` -- Temporary stream for `stdout`,
initially empty.
err_
:cls:`~io.StringIO` -- Temporary stream for `stderr`,
:cls:`~io.RandomTextIO` -- Temporary stream for `stderr`,
initially empty.
"""
if close:
out_cls = SafeCloseRandomTextIO
in_cls = SafeCloseTeeStdin
else:
out_cls = RandomTextIO
in_cls = TeeStdin

old_stdin = sys.stdin
old_stdout = sys.stdout
old_stderr = sys.stderr

new_stdout = StringIO()
new_stderr = StringIO()
new_stdin = TeeStdin(new_stdout, in_str)
with ExitStack() as stack:
new_stdout = stack.enter_context(out_cls())
new_stderr = stack.enter_context(out_cls())
new_stdin = stack.enter_context(in_cls(new_stdout, in_str))

close_files = stack.pop_all().close

sys.stdin = new_stdin
sys.stdout = new_stdout
Expand All @@ -208,14 +294,5 @@ def stdio_mgr(in_str=""):
sys.stdout = old_stdout
sys.stderr = old_stderr

try:
closed = new_stdin.closed
except ValueError:
# ValueError occurs when the underlying buffer is detached
pass
else:
if not closed:
new_stdin.close()

new_stdout.close()
new_stderr.close()
if close:
close_files()
103 changes: 102 additions & 1 deletion tests/test_stdiomgr_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@
"""


import io
import warnings

import pytest

from stdio_mgr import stdio_mgr

Expand Down Expand Up @@ -114,8 +115,108 @@ def test_repeated_use():
test_capture_stderr()


def test_manual_close():
"""Confirm files remain open if close=False after the context has exited."""
with stdio_mgr(close=False) as (i, o, e):
test_default_stdin()
test_capture_stderr()
assert not i.closed
assert not o.closed
assert not e.closed

i.close()
o.close()
e.close()


def test_manual_close_detached_fails():
"""Confirm files remain open if close=False after the context has exited."""

This comment has been minimized.

Copy link
@bskinn

bskinn Aug 3, 2019

Owner

@jayvdb Needs to be updated after copy-paste

with stdio_mgr(close=False) as (i, o, e):
test_default_stdin()
test_capture_stderr()
i.detach()
o.detach()
e.detach()

with pytest.raises(ValueError):
i.close()
with pytest.raises(ValueError):
i.closed
with pytest.raises(ValueError):
o.close()
with pytest.raises(ValueError):
o.closed
with pytest.raises(ValueError):
e.close()


def test_stdin_detached():
"""Confirm stdin's buffer can be detached within the context."""
with stdio_mgr() as (i, o, e):
print("test str")

f = i.detach()

assert "test str\n" == o.getvalue()

print("second test str")

assert "test str\nsecond test str\n" == o.getvalue()

assert not f.closed
assert o.closed
assert e.closed


def test_stdout_detached():
"""Confirm stdout's buffer can be detached within the context.
Like the real sys.stdout, writes after detach should fail, however
writes to the detached stream should be captured.
"""
with stdio_mgr() as (i, o, e):
print("test str")

f = o.detach()

assert f is o._buf
assert f is i.tee._buf

assert "test str\n" == o.getvalue()

with pytest.raises(ValueError):
print("anything")

f.write("second test str\n".encode("utf8"))
f.flush()

assert "test str\nsecond test str\n" == o.getvalue()

assert not f.closed
assert i.closed
assert e.closed


def test_stdout_detached_and_closed():
"""Confirm stdout's buffer can be detached within the context.
Like the real sys.stdout, writes after detach should fail, however
writes to the detached stream should be captured.
"""
with stdio_mgr() as (i, o, e):
print("test str")

f = o.detach()

assert isinstance(f, io.BufferedRandom)
assert f is o._buf

assert "test str\n" == o.getvalue()

with pytest.raises(ValueError):
print("anything")

f.write("second test str\n".encode("utf8"))
f.close()

assert "test str\nsecond test str\n" == o.getvalue()

0 comments on commit 951002c

Please sign in to comment.