=head1 Description
-Starting in 5.90080 L<Catalyst> will enable UTF8 encoding by default for
+Starting in 5.90080 L<Catalyst> will enable UTF8 encoding by default for
text like body responses. In addition we've made a ton of fixes around encoding
and utf8 scattered throughout the codebase. This document attempts to give
an overview of the assumptions and practices that L<Catalyst> uses when
and explain where we may have been exceptional to respect our commitment
to backwards compatibility.
-=head1 UTF8 in Controller Actions
+=head1 UTF8 in Controller Actions
Using UTF8 characters in your Controller classes and actions.
package MyApp::Controller::Root;
- use uf8;
+ use utf8;
use base 'Catalyst::Controller';
sub heart_with_arg :Path('♥') Args(1) {
The first one is a classic Path type action and the second uses Chaining, and
spans three actions in total. As you can see, you can use unicode characters
-in your Path and PartPart attributes (remember to use the C<utf8> pragma to allow
+in your Path and PathPart attributes (remember to use the C<utf8> pragma to allow
these multibyte characters in your source). The two constructed matchable routes
would match the following incoming URLs:
Again, remember that we are display the unicode character and using it to match actions
containing such multibyte characters BUT over HTTP you are getting these as URL encoded
-bytes. For example if you looked at the L<PSGI> C<$env> value for C<REQUEST_URI> you
-would see (for the above request)
+bytes. For example if you looked at the L<PSGI> C<$env> value for C<REQUEST_URI> you
+would see (for the above request)
REQUEST_URI => "/base/%E2%99%A5/%E2%99%A5/%E2%99%A5/%E2%99%A5"
use utf8;
my $url = $c->uri_for( $c->controller('Root')->action_for('arg'), ['♥','♥']);
-When you stringyfy this object (for use in a template, for example) it will automatically
+When you stringify this object (for use in a template, for example) it will automatically
do the right thing regarding utf8 encoding and url encoding.
http://localhost/base/%E2%99%A5/%E2%99%A5/%E2%99%A5/%E2%99%A5
=head2 UTF8 in URL query and keywords
-The same rules that we find in URL paths also cover URL query parts. That is if
-one types a URL like this into the browser (again assuming a modernish UI that
-allows unicode)
+The same rules that we find in URL paths also cover URL query parts. That is
+if one types a URL like this into the browser
http://localhost/example?♥=♥♥
Again, remember that values of your parameters are now decode into Unicode strings. so
for example you'd expect the result of length to reflect the character length not
-the byte length.
+the byte length.
Just like with arguments and captures, you can use utf8 literals (or utf8
strings) in $c->uri_for:
use utf8;
my $url = $c->uri_for( $c->controller('Root')->action_for('example'), {'♥' => '♥♥'});
-When you stringyfy this object (for use in a template, for example) it will automatically
+When you stringify this object (for use in a template, for example) it will automatically
do the right thing regarding utf8 encoding and url encoding.
http://localhost/example?%E2%99%A5=%E2%99%A5%E2%99%A5
Since again what you want is a properly url encoded version of this. Ultimately what you want
-to send over the wire via HTTP needs to be bytes (not unicode characters).
+to send over the wire via HTTP needs to be bytes (not unicode characters).
Remember if you use any utf8 literals in your source code, you should use the
C<use utf8> pragma.
+B<NOTE:> Assuming UTF-8 in your query parameters and keywords may be an issue if you have
+legacy code where you created URL in templates manually and used an encoding other than UTF-8.
+In these cases you may find versions of Catalyst after 5.90080+ will incorrectly decode. For
+backwards compatibility we offer three configurations settings, here described in order of
+precedence:
+
+C<do_not_decode_query>
+
+If true, then do not try to character decode any wide characters in your
+request URL query or keywords.
+
+C<default_query_encoding>
+
+This setting allows one to specify a fixed value for how to decode your query, instead of using
+the default, UTF-8.
+
+C<decode_query_using_global_encoding>
+
+If this is true we decode using whatever you set C<encoding> to.
+
=head2 UTF8 in Form POST
In general most modern browsers will follow the specification, which says that POSTed
As a result since L<Catalyst> now serves UTF8 encoded responses by default, this means that
you can mostly rely on incoming form POSTs to be so encoded. L<Catalyst> will make this
assumption and decode accordingly (unless you explicitly turn off encoding...) If you are
-running Catalyst in developer debug, then you will see the correct unicode characters in
+running Catalyst in developer debug, then you will see the correct unicode characters in
the debug output. For example if you generate a POST request:
use Catalyst::Test 'MyApp';
And if you had a controller like this:
package MyApp::Controller::Example;
-
+
use base 'Catalyst::Controller';
sub posted :POST Local {
use Encode 2.21 'decode_utf8';
is decode_utf8($req->content), 'hearts => ♥';
-In this case we decode so that we can print and compare strings with multibyte characters.
+In this case we decode so that we can print and compare strings with multibyte characters.
B<NOTE> In some cases some browsers may not follow the specification and set the form POST
encoding based on the server response. Catalyst itself doesn't attempt any workarounds, but one
a number of approaches. Hopefully as more compliant browsers become popular these edge cases
will fade.
+B<NOTE> It is possible for a form POST multipart response (normally a file upload) to contain
+inline content with mixed content character sets and encoding. For example one might create
+a POST like this:
+
+ use utf8;
+ use HTTP::Request::Common;
+
+ my $utf8 = 'test ♥';
+ my $shiftjs = 'test テスト';
+ my $req = POST '/root/echo_arg',
+ Content_Type => 'form-data',
+ Content => [
+ arg0 => 'helloworld',
+ Encode::encode('UTF-8','♥') => Encode::encode('UTF-8','♥♥'),
+ arg1 => [
+ undef, '',
+ 'Content-Type' =>'text/plain; charset=UTF-8',
+ 'Content' => Encode::encode('UTF-8', $utf8)],
+ arg2 => [
+ undef, '',
+ 'Content-Type' =>'text/plain; charset=SHIFT_JIS',
+ 'Content' => Encode::encode('SHIFT_JIS', $shiftjs)],
+ arg2 => [
+ undef, '',
+ 'Content-Type' =>'text/plain; charset=SHIFT_JIS',
+ 'Content' => Encode::encode('SHIFT_JIS', $shiftjs)],
+ ];
+
+In this case we've created a POST request but each part specifies its own content
+character set (and setting a content encoding would also be possible). Generally one
+would not run into this situation in a web browser context but for completeness sake
+Catalyst will notice if a multipart POST contains parts with complex or extended
+header information and in those cases it will not attempt to apply decoding to the
+form values. Instead the part will be represented as an instance of an object
+L<Catalyst::Request::PartData> which will contain all the header information needed
+for you to perform custom parser of the data.
+
=head1 UTF8 Encoding in Body Response
When does L<Catalyst> encode your response body and what rules does it use to
my ($self, $c) = @_;
$c->response->content_type('text/html');
$c->response->write("<p>This is stream_write action ♥</p>");
- }
+ }
sub stream_write_fh :Local {
my ($self, $c) = @_;
application/xml
application/vnd.user+xml
-You should set your content type prior to header finalization if you want L<Catalyst> to
+You should set your content type prior to header finalization if you want L<Catalyst> to
encode.
B<NOTE> We do not attempt to encode C<application/json> since the two most commonly used
content type carefully to determine if the body needs encoding).
B<NOTE> If you set the character set of the response L<Catalyst> will skip encoding IF the
-character set is set to somethng that doesn't match $c->encoding->mime_name. We will assume
+character set is set to something that doesn't match $c->encoding->mime_name. We will assume
if you are setting an alternative character set, that means you want to handle the encoding
yourself. However it might be easier to set $c->encoding for a given response cycle since
you can override this for a given response. For example here's how to override the default
requirement and $c->encoding is set (which it is, as long as you did not change it).
B<NOTE> If you try to change the encoding after you start the stream, this will invoke an error
-reponse. However since you've already started streaming this will not show up as an HTTP error
+response. However since you've already started streaming this will not show up as an HTTP error
status code, but rather error information in your body response and an error in your logs.
The second way to stream a response is to get the response writer object and invoke methods
$writer->close;
}
-This can be used just like the C<write> method, but typicallty you request this object when
+This can be used just like the C<write> method, but typically you request this object when
you want to do a nonblocking style response since the writer object can be closed over or
sent to a model that will invoke it in a non blocking manner. For more on using the writer
object for non blocking responses you should review the C<Catalyst> documentation and also
a encoding on the IO layer level so that we can be sure to properly encode at body finalization.
So this is still an edge case we are writing test examples for. But for now if you are returning
a filehandle like response, you are expected to make sure you are following the L<PSGI> specification
-and that unencoded bytes are returned.
+and return raw bytes.
=head2 Override the Encoding on Context
Where C<MyApp> is your L<Catalyst> subclass.
+If you do not wish to disable all the Catalyst encoding features, you may disable specific
+features via two additional configuration options: 'skip_body_param_unicode_decoding'
+and 'skip_complex_post_part_handling'. The first will skip any attempt to decode POST
+parameters in the creating of body parameters and the second will skip creation of instances
+of L<Catalyst::Request::PartData> in the case that the multipart form upload contains parts
+with a mix of content character sets.
+
If you believe you have discovered a bug in UTF8 body encoding, I strongly encourage you to
report it (and not try to hack a workaround in your local code). We also recommend that you
regard such a workaround as a temporary solution. It is ideal if L<Catalyst> extension
-authors can start to count on L<Catalyst> doing the write thing for encoding
+authors can start to count on L<Catalyst> doing the write thing for encoding.
=head1 Conclusion
projects / catagits/Catalyst-Runtime.git / blobdiff