Update to work with Perl 5.31.7
This deprecated module was using deprecated macros that recently have
been removed. Any use of them should have generated a warning for the
past two Perl releases. Nonetheless, no one apparently reported this to
the module's bg tracker, and so, the issue was raised only when things
stopped working.
The code contains the file 'stolen_chunk_of_toke.c'. And this is as
scary as it sounds. There was a similar situation a while back with
B::Hooks::Parser, and that was solved by making public a few functions
in toke.c that should be used outside the perl core only by that module.
It makes sense for that module to have access to these, given its
purpose, and it becomes innoculated against most future changes to the
parser.
Devel::Declare, unfortunately is using an earlier version of toke.c, and
so the above approach doesn't work, because the parameters to the
functions have changed, and the code is highly entwined with various
interpreter level lexing variables, expecting them to work in the way it
has coded for them, which may not be the case now or at some point in
the future if it used the real toke.c functions. So its best to try to
get this stolen chunk to work.
A problem is that there are bugs in the way toke.c worked at the time
this was stolen. I have fixed only the most obvious.
There are two main issues.
The easiest is that this won't compile because of security-related
changes in blead. This is why most of these macros were deprecated:
they allow a potential read past the end of the buffer. That means an
extra parameter must be passed, giving the upper limit for the buffer.
There have been versions of the macros available for several releases
that takes the extra parameter, so all we have to do is convert to use
those versions when available, and the old versions when not.
The harder is that these use is_utf8_mark() to find the ends of
identifiers being parsed. This is long obsolete, valid mostly only for
Western Latin-based languages. Unicode came up with a better scheme
many versions ago, which is to define a property that indicates if a
character continues an identifier or not. Perl 5.31.7 finally removed
the old deprecated function. I have tried to change the code here that
used it without changing the results when using inputs that used to be
valid. But there was a bug in the ordering of this. The UTF-8 case
must come first, so I reordered it. Also, the new toke.c code has a
do-while loop for the non-UTF8 case. I copied that too, without
investigating why that change was made in the modern toke.c.
This now passes all its tests on blead. I did find some omissions in
blead that I fixed to get this to pass.