Regular expression: Is this a mistake or not?

[JurySpb]

import re
re.sub(b"\x00", b"", b"\x00\x01")
>>> b'\x00\x01'

Deletion does not occur.

On "big" python it works.
Other values ​​- no problem.

re.sub(b"\x01", b"", b"\x00\x01")
>>> b'\x00'

(MicroPython v1.23.0-220.g6927639898 on 2024-07-11; LOLIN_S2_MINI with ESP32-S2FN4R2)

[sophiedegran]

Yes same thing on micropython 1.23 stable...
but
re.sub (b'x00', b'', b'x00\x01')
return b'\x01'.. so it's seem the first \ causing problem when it is b'\x00' because
re.sub (b'\x01', b'', b'\x00\x01') return ok (b'\x00')

[GitHubsSilverBullet]

\x00 is the string terminator in ›C‹
Wouldn't be surprised if the internal handling isn't quite right.

[sophiedegran]

Yes you're right but it's work under python! ☺

[JurySpb]

The user does not know about the end of the str. This is an internal matter of python.

bytes("abc","utf8")
>>> b'abc'

[JurySpb]

Yes, so too

re.sub (b'x00', b'', b'\x01x00')
>>> b'\x01'

Unfortunately, this is an artificial situation.

[robert-hh]

It stops at the NULL byte. Example:

>>> import re
>>> re.sub (b'a\x00b', b'', b'a\x00bc')
b'\x00bc'

[JurySpb]

On "big" python:

re.sub (b'a\x00b', b'', b'a\x00bc')
>>>  b'c'

and

re.sub (b'[a-b\x00]', b'', b'a\x00bc')
>>>  b'c'

Micropython:

re.sub (b'[a-b\x00]', b'', b'a\x00bc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: error in regex

and

re.sub (b'[a-b]', b'', b'a\x00bc')
b'\x00c'

[scruss]

MicroPython's re module is very limited. Remember the Micro- part of MicroPython. It doesn't have to do everything that unskimmed Python does.

First off, MicroPython regular expressions can't handle any escape sequences outside the short list given in the documentation. The list doesn't include \x. But MicroPython does have a specific hate for zero characters in strings, no matter how you represent them. This doesn't work:

re.sub(chr(0), "", chr(0)+"\x01")

Secondly, don't use re for bytes. Regular expressions are for strings. Zero bytes don't belong in strings. While you can do it in CPython, you probably shouldn't.