[Bug]: UTF-8 Encoding issue (on Windows)

[Shadow-Devil]

Description

The following UTF-8 sequence is not correctly handled and it is also handled differently if it is received as a payload from http or put into ballerina source code:

import ballerina/http;
import ballerina/io;

service / on new http:Listener(9090) {
    resource function post test(@http:Payload string payload) returns string {
        io:println("Explicit: Kolkāta");
        io:println("From Payload: ", payload);
        return payload;
    }
}

The resulting output in the console and in the API response look like this:

Explicit: Kolk─üta
From Payload: Kolk├ä´┐¢ta

Steps to Reproduce

No response

Affected Version(s)

2201.10.1

OS, DB, other environment details and versions

Windows 11

Related area

-> Runtime

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[Shadow-Devil]

Maybe this is windows related, since on Choreo it is working as expected.

[ThisaruGuruge]

Hi @Shadow-Devil

I think this happens because the JVM uses a different encoding in Windows.

Try this out:

Add a new variable named JAVA_TOOL_OPTIONS in your environment variables, where the value should be -Dfile.encoding=UTF8.

Check this StackOverflow answer as well.