UTF-16 surrogate pairs mishandled when processing non-BMP characters (e.g., 𣄃) in string-to-byte conversions

[yoyo837]

Bug description

Summary

When passing a non-BMP character such as '𣄃' (U+23103) into an AssemblyScript function, the string gets split into two UTF-16 code units (a surrogate pair), which is expected. However, this becomes problematic when attempting to encode the string into a UTF-8 byte stream for hashing (e.g., MD5), as AssemblyScript provides no built-in way to handle surrogate pairs as a single Unicode code point.

This leads to incorrect behavior, especially in cryptographic applications (e.g., generating an MD5 hash of a string containing such characters), where string encoding must exactly match UTF-8 as produced by JavaScript's TextEncoder.

Steps to Reproduce

export function debugUtf16(str: string): void {
  for (let i = 0; i < str.length; i++) {
    trace("charCodeAt", 1, str.charCodeAt(i));
  }
}

Would it be possible to:

Provide built-in String.codePointAt(i: i32): i32 in stdlib to make proper iteration over Unicode strings easier?

Offer utility to convert a string to a proper UTF-8 encoded Uint8Array, respecting surrogate pairs.

Possibly integrate this into String.UTF8.encode() or add a safer variant.

Steps to reproduce

None

AssemblyScript version

0.28.2

[dcodeIO]

Isn't

• Provide built-in String.codePointAt(i: i32): i32 in stdlib to make proper iteration over Unicode strings easier?

already the case? There is

assemblyscript/std/assembly/string.ts

Lines 75 to 83 in 3defefd

	codePointAt(pos: i32): i32 {
	let len = this.length;
	if (<u32>pos >= <u32>len) return -1; // (undefined)
	let first = <i32>load<u16>(changetype<usize>(this) + (<usize>pos << 1));
	if ((first & 0xFC00) != 0xD800 || pos + 1 == len) return first;
	let second = <i32>load<u16>(changetype<usize>(this) + (<usize>pos << 1), 2);
	if ((second & 0xFC00) != 0xDC00) return first;
	return (first - 0xD800 << 10) + (second - 0xDC00) + 0x10000;
	}

in stdlib, and during iteration one would increment by one or two chars depending on the last codepoint's value.

[yoyo837]

Sorry, I found it.