Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions Lib/encodings/base64_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,20 @@
import codecs
import base64

### Codec Helpers

def _assert_strict(errors):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe _check_strict from the original patch would be better?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer assert_strict since it is closer to what was there before, and it is more obvious, to me at least, that it will fail if errors != 'strict', whereas check_strict, seems like something that will enable more verbose errors or the like.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, it's not fully correct to name method assert_something and don't make any assert checks or raising an AssertionError.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malemburg which do you prefer? _check_strict or _assert_strict

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a preference. Both are fine.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is duplicated in six modules. Since all of them already import codecs, would not it be better to add a shared implementation in codecs? Change the name to be clear that it asserts/checks that the error handler is strict only.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; this helper should live in codecs.py.

if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"error handling mode" does not occur anywhere in the code or documentation. Only "error handling"/"error handler".

idna raises f"Unsupported error handling: {errors}". This is not a strong precedent, and ValueError is more appropriate than UnicodeError, but take this into account.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"error handling parameter" would be clearer.


### Codec APIs

def base64_encode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, now that I see where you are using the helper, I'm not so sure that this is the right approach anymore.

"assert" only works when Python is run without -O, so the helper does change semantics. I'm also worried that the extra function call will make things slower.

The change in semantics should be documented and I would prefer to have the checks inlined in all these cases. Yes, it's duplication, but those are only two lines of code and the error message could additionally include information about which encoding caused the problem.

return (base64.encodebytes(input), len(input))

def base64_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (base64.decodebytes(input), len(input))

class Codec(codecs.Codec):
Expand All @@ -26,12 +32,12 @@ def decode(self, input, errors='strict'):

class IncrementalEncoder(codecs.IncrementalEncoder):
def encode(self, input, final=False):
assert self.errors == 'strict'
_assert_strict(self.errors)
return base64.encodebytes(input)

class IncrementalDecoder(codecs.IncrementalDecoder):
def decode(self, input, final=False):
assert self.errors == 'strict'
_assert_strict(self.errors)
return base64.decodebytes(input)

class StreamWriter(Codec, codecs.StreamWriter):
Expand Down
14 changes: 10 additions & 4 deletions Lib/encodings/bz2_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,20 @@
import codecs
import bz2 # this codec needs the optional bz2 module !

### Codec Helpers

def _assert_strict(errors):
if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

### Codec APIs

def bz2_encode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (bz2.compress(input), len(input))

def bz2_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (bz2.decompress(input), len(input))

class Codec(codecs.Codec):
Expand All @@ -28,7 +34,7 @@ def decode(self, input, errors='strict'):

class IncrementalEncoder(codecs.IncrementalEncoder):
def __init__(self, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
self.errors = errors
self.compressobj = bz2.BZ2Compressor()

Expand All @@ -44,7 +50,7 @@ def reset(self):

class IncrementalDecoder(codecs.IncrementalDecoder):
def __init__(self, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
self.errors = errors
self.decompressobj = bz2.BZ2Decompressor()

Expand Down
14 changes: 10 additions & 4 deletions Lib/encodings/hex_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,20 @@
import codecs
import binascii

### Codec Helpers

def _assert_strict(errors):
if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

### Codec APIs

def hex_encode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (binascii.b2a_hex(input), len(input))

def hex_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (binascii.a2b_hex(input), len(input))

class Codec(codecs.Codec):
Expand All @@ -26,12 +32,12 @@ def decode(self, input, errors='strict'):

class IncrementalEncoder(codecs.IncrementalEncoder):
def encode(self, input, final=False):
assert self.errors == 'strict'
_assert_strict(self.errors)
return binascii.b2a_hex(input)

class IncrementalDecoder(codecs.IncrementalDecoder):
def decode(self, input, final=False):
assert self.errors == 'strict'
_assert_strict(self.errors)
return binascii.a2b_hex(input)

class StreamWriter(Codec, codecs.StreamWriter):
Expand Down
12 changes: 10 additions & 2 deletions Lib/encodings/quopri_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,23 @@
import quopri
from io import BytesIO

### Codec Helpers

def _assert_strict(errors):
if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

### Codec APIs

def quopri_encode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
f = BytesIO(input)
g = BytesIO()
quopri.encode(f, g, quotetabs=True)
return (g.getvalue(), len(input))

def quopri_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
f = BytesIO(input)
g = BytesIO()
quopri.decode(f, g)
Expand Down
10 changes: 8 additions & 2 deletions Lib/encodings/uu_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,16 @@
import binascii
from io import BytesIO

### Codec Helpers

def _assert_strict(errors):
if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

### Codec APIs

def uu_encode(input, errors='strict', filename='<data>', mode=0o666):
assert errors == 'strict'
_assert_strict(errors)
infile = BytesIO(input)
outfile = BytesIO()
read = infile.read
Expand All @@ -35,7 +41,7 @@ def uu_encode(input, errors='strict', filename='<data>', mode=0o666):
return (outfile.getvalue(), len(input))

def uu_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
infile = BytesIO(input)
outfile = BytesIO()
readline = infile.readline
Expand Down
14 changes: 10 additions & 4 deletions Lib/encodings/zlib_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,20 @@
import codecs
import zlib # this codec needs the optional zlib module !

### Codec Helpers

def _assert_strict(errors):
if errors != 'strict':
raise ValueError(f'Unsupported error handling mode: "{errors}" - must be "strict"')

### Codec APIs

def zlib_encode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (zlib.compress(input), len(input))

def zlib_decode(input, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
return (zlib.decompress(input), len(input))

class Codec(codecs.Codec):
Expand All @@ -26,7 +32,7 @@ def decode(self, input, errors='strict'):

class IncrementalEncoder(codecs.IncrementalEncoder):
def __init__(self, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
self.errors = errors
self.compressobj = zlib.compressobj()

Expand All @@ -42,7 +48,7 @@ def reset(self):

class IncrementalDecoder(codecs.IncrementalDecoder):
def __init__(self, errors='strict'):
assert errors == 'strict'
_assert_strict(errors)
self.errors = errors
self.decompressobj = zlib.decompressobj()

Expand Down
14 changes: 14 additions & 0 deletions Lib/test/test_codecs.py
Original file line number Diff line number Diff line change
Expand Up @@ -3129,6 +3129,20 @@ def test_uu_invalid(self):
# Missing "begin" line
self.assertRaises(ValueError, codecs.decode, b"", "uu-codec")

def test_invalid_error_input(self):
for encoding in bytes_transform_encodings:
with self.subTest(encoding=encoding):
encoder = codecs.getencoder(encoding)
decoder = codecs.getdecoder(encoding)

self.assertRaises(ValueError, encoder, 'in', errors='notstrict')
self.assertRaises(ValueError, decoder, 'in', errors='notstrict')
Comment on lines +3138 to +3139

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Input should be bytes, not str.


incdev = codecs.getincrementaldecoder(encoding)
if encoding not in ('base64_codec', 'uu_codec', 'quopri_codec', 'hex_codec'):
Comment thread
StanFromIreland marked this conversation as resolved.
self.assertRaises(ValueError, incdev, errors='notstrict')



# The codec system tries to add notes to exceptions in order to ensure
# the error mentions the operation being performed and the codec involved.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
The ``base64_codec``, ``uu_codec``, ``quopri_codec``, ``hex_codec``,
``zlib_codec`` and ``bz2_codec`` now raise a :exc:`ValueError` when their
decoder/encoder is provided an *errors* parameter that is not equal to
``'strict'``.
Loading