A simple function to get a char from a string based on the byte position

[mzaks]

Strings in Mojo are backed by a utf-8 buffer and the subscription into a string might be confusing as it works on byte level, where a char in utf-8 can be represented in up to 4 bytes. This function will return a char even if the provided byte position is in the middle of the char code.

from String import String
from Bit import ctlz

fn get_char_at_byte(s: String, i: Int) -> String:
    if i < 0 or i >= len(s):
        return ""
    let code = s.buffer.data.load(i)
    let byte_length_code = ctlz(~code).to_int()
    if byte_length_code == 0:
        return s[i]
    if byte_length_code > 1:
        return s[i: i+byte_length_code]
    var correct_offset = i-1
    while correct_offset >= 0:
        let code = s.buffer.data.load(correct_offset)
        let byte_length_code = ctlz(~code).to_int()
        if byte_length_code == 0:
            return s[correct_offset]
        if byte_length_code > 1:
            return s[correct_offset: correct_offset + byte_length_code]
        correct_offset -= 1
    return ""

let string = "hällo ß 🔥👍👍🏿 "
for i in range(len(string)):
    print(get_char_at_byte(string, i))

[yakupc55]

[mzaks]

Thanks! Note to my self, do not edit the code last minute, without checking it properly. 😅

[ego]

I have a question for skilled devs and/or for Mojo team, what will be cheaper: Mojo implementation or __mlir_op external_call to libc mbstowcs or whatever? It is general question not only for this case.

[mzaks]

There are always two dimensions to consider, space and time. mbstowcs is a memcopy with "widening". So in regards to space it is not really cheap, given a very large text corpus you would need to perform it in chunks. Regarding the time characteristics, I am not sure, memcopies are generally considered expensive, but IMHO only proper profiling can give a definitive answer.

[strangemonad]

@mzaks this is cool. Are you expecting it to always return a valid code point though? Seems like this might get tripped up between pre-composed / decomposed where you're treating $i$ as the code point offset (e.g if ... > len(s) and the byte offset (e.g. s.buffer.data.load(i))? Maybe I'm missing something?

Also, just a naming nit-pick. I'd consider naming this something like get_codepoint_at_index?

I'm definitely looking forward to an Iterator trait that allows iterating over a string a code point at a time

[mzaks]

Hi @strangemonad. Yes you re absolutely right, this solution only returns a valid char on byte level but not on grapheme cluster (user perceived) level.
This is why the print out for "hällo ß 🔥👍👍🏿 " is:

h
ä
ä
l
l
o
 
ß
ß
 
🔥
🔥
🔥
🔥
👍
👍
👍
👍
👍
👍
👍
👍
🏿
🏿
🏿
🏿

👍🏿 Grapheme is stretched over 8 bytes, where the first 4 bytes are the thumbs up grapheme and last 4 bytes are the skin tone. Same would happen for pre-composed / decomposed chars as they are also considered to be a grapheme cluster

[strangemonad]

@mzaks thanks for clarifying. That makes total sense. That said, I can't wait for traits to be elaborated a bit more. This would be a great case for allowing strings expose either

• an iter trait (iterating over a string as code-points)
• a view or slice that with either interpolating or decimating index access as you've done here.

I've seen other applications that need some flavor of this sort of flexible indexing:

e.g. 1 FLIR infrared camera video file processing:
FLIR infrared video files have a concept called "Super-framing" used to increase the dynamic range. Depending on properties of the HW, LUT (raw vs temperature units), and how you're querying the frames, when you fetch a frame you have to do something like for following.

file.nearest_frame_number(
            desired_frame=desired_frame_number,
            preset=fnv.Preset.ALL,
        )

file.nearest_frame_number(
            desired_frame=desired_frame_number,
            preset=fnv.Preset.SUPERFRAME,
        )

e.g. 2 Querying time series from a time series DB
You often find yourself wanting to interpolate or decimate in time and space.

@lattner would it make sense to create a github label that could be applied to start collecting use cases for core traits. I'm curious if it makes sense to start collecting concrete use cases that might drive out some of the design of traits. Serialization / Deserialization is another one I've started creating personal notes on that are still a bit too rough to share