user86667: TLDR
I need to convert text from a website into UTF-16LE format so that I can get the proper MD5 checksum but can't figure out how to go about doing that. This is all happening on an Arduino to log in to a router.
Background
I want to read values from a smart home device connected to a Fritz!Box router using an Arduino with an ethernet connection. The router has an open API and I'm using HTTP (EN|DE) to get the values. C++ programming is not my strong point.
I have been basing my attempts on BASH and javascript examples. For MD5 calculation I'm using tzikis' ArduinoMD5 library.
Steps
Based on the manual, you need to do the following:

Contact the router to get a challenge string (router sends XML)
Generate a response by calculating an MD5 checksum using the challenge string and password: <challenge>-<password>
Send the response, <challenge>-<response>, via POST
Receive the XLM answer from the router containing the SID
Use SID to request data or control a smart home device

I'm stuck on step 2. I have the challenge but cannot calculate the correct checksum.
The manual states:

The MD5 hash is generated from the byte sequence of the UTF-16LE
coding of this string (without BOM and without terminating 0 bytes).

Attempts
So far I can request the challenge: 70067288 for example. The reponse appears to always be alphanumeric. My password is also alphanumeric.
unsigned char* hash2 = MD5::make_hash("1234567z-äbc");
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

This was my attempt at confirmation. The manual gave 1234567z as a challenge example with äbc (the a in the example does indeed have umlauts) as a password. The response shown for this example was 1234567z-9e224a41eeefa284df7bb0f26c2913e2 but my checksum from the above code was 935fe44e659beb5a3bb7a4564fba0513.
I tried "manually" generating UTF-16LE and calculating the MD5 but that didn't work either.
byte rawTest[] = {0x3100, 0x3200, 0x3300, 0x3400, 0x3500, 0x3600, 0x3700, 0x7a00, 0x2d00, 0xe400, 0x6200, 0x6300};
char buffer[12] = {};
unsigned char* hash2 = MD5::make_hash(&buffer[0]);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

It gave me d41d8cd98f00b204e9800998ecf8427e.
I am not entirely sure the example given was correctly calculated. Using their example in the API documentation with BASH code from the site mentioned above, I get a different answer: 1234567z-bb6e5b7c7d4d485590f4e084ad3da989.
#!/bin/bash
# -----------
# definitions
# -----------
FBF="http://192.168.178.1/"
USER="root"
PASS="äbc"
AIN="AINOFYOURFRITZDECTDEVICE"
# ---------------
# fetch challenge
# ---------------
CHALLENGE="1234567z"
# -----
# login
# -----
MD5=$(echo -n ${CHALLENGE}"-"${PASS} | iconv -f ISO8859-1 -t UTF-16LE | md5sum -b | awk '{print substr($0,1,32)}')
RESPONSE="${CHALLENGE}-${MD5}"
echo $RESPONSE

Any help getting it working is greatly appreciatet.

Working Example (with caveats)
Requirements:

Password using only characters found in ASCII
Password at least 17 characters long

The password must use only ASCII characters due to the simple way it's being converted to UTF-16LE.
I can't explain why the password must be at least 17 characters long though. My code checks the length of the actual password programmatically.
With a password at least 17 characters long I get something like this:

Challenge: 6c607ee5
Challenge Pass (26): 6c607ee5-01234567890123456
Challenge Pass Preallocation Length: 26
MD5 (UTF-16LE): 3b26241ae4aab8eaf71d5c1599932178

Any password short and I get something like this:

Challenge: 621611e1
Challenge Pass (26): 621611e1-0123456789012345�
Challenge Pass Preallocation Length: 25
MD5 (UTF-16LE): 30a163ab8a267adfbb524b2b7412e81b

Please excuse any poor coding, C++ is not my strongpoint.
void fritzboxSID() {
  EthernetClient fritzBox;
  const char* pass = "01234567890123456";  // 17 ASCII character minimum
  const char* user = "fritz3456";  // Account on router, randomly generated or user created
  const size_t passLength = strlen(pass);  // Password length
  char c;
  uint8_t xml = 0;
  char challenge[9] = {0};  // Challenge
  char challengePass[8 + 1 + passLength];  // challenge-password
  uint8_t challengeCount = 0;
  
  if (fritzBox.connect("192.168.200.1", 80)) {
    // Send login request
    fritzBox.println("GET /login_sid.lua HTTP/1.1");
    fritzBox.println("Host: 192.168.200.1");
    fritzBox.println("Connection: close");
    fritzBox.println();
    
    while (fritzBox.connected()) {
      while (fritzBox.available()) {
        c = fritzBox.read();  // Read a single character from the router response
        if (c == '>') {
          xml++;
        }
        if (xml == 5) {
            // Challenge starts after the fifth >
            if (challengeCount > 0 & challengeCount < 9) {
                // Save the challenge, character by character
                challenge[challengeCount-1] = c;
                challengePass[challengeCount-1] = c;
            }
            challengeCount++;
        }
        Serial.print(c);  // Print HTTP response
      }
    }
    challengePass[8] = '-';
    for (size_t i = 0;i < passLength; i++){
      // Copy password into the combined challenge-password string
      challengePass[i+9] = pass[i];
    }
    
    // Show challenge and pass, checking lengths
    //      If the strlen(challengePass) doesn't equal the preallocation length then the MD5 will be incorrect
    Serial.println("");
    Serial.print("Challenge: ");
    Serial.println(challenge);
    Serial.print("Challenge Pass (");
    Serial.print(strlen(challengePass));
    Serial.print("): ");
    Serial.println(challengePass);
    Serial.print("Challenge Pass Preallocation Length: ");
    Serial.println(8 + 1 + passLength);

    // Convert to UTF-16LE (only works for standard ASCII characters)
    const size_t length = strlen(challengePass);
    char buffer[2*length];
    for (size_t i = 0; i < length; i++) {
      buffer[2*i] = challengePass[i];
      buffer[2*i+1] = 0;
    }

    // Generate MD5
    unsigned char* hash = MD5::make_hash(buffer, 2*length);
    char *md5str = MD5::make_digest(hash, 16);
    free(hash);
    Serial.print("MD5 (UTF-16LE): ");
    Serial.println(md5str);
  } else {
    Serial.println("Cannot connect to 192.168.200.1");
  }
}

If it helps, the response from the router for the initial login request is this:
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/xml
Date: Fri, 16 Sep 2022 08:21:27 GMT
Expires: -1
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; connect-src 'self'; font-src 'self'; frame-src https://service.avm.de https://help.avm.de https://www.avm.de https://avm.de https://assets.avm.de https://clickonce.avm.de http://clickonce.avm.de http://download.avm.de https://download.avm.de 'self'; img-src 'self' https://tv.avm.de https://help.avm.de/images/ http://help.avm.de/images/ data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'; media-src 'self'

<?xml version="1.0" encoding="utf-8"?><SessionInfo><SID>0000000000000000</SID><Challenge>621611e1</Challenge><BlockTime>0</BlockTime><Rights></Rights><Users><User last="1">fritz3456</User><User>UserName</User></Users></SessionInfo>


user7508: You have to convert the string "<challenge>-<password>" to
UTF-16LE. If the whole string is plain ASCII, the conversion is trivial:
you just have to add a zero byte after each ASCII byte.
// ASCII string to be hashed.
const char* text = "1234567z-abc";  // umlaut removed

// Convert to UTF-16LE.
const size_t length = strlen(text);
char buffer[2*length];
for (size_t i = 0; i < length; i++) {
    buffer[2*i] = text[i];
    buffer[2*i+1] = 0;
}

// Compute and print the hash.
unsigned char* hash2 = MD5::make_hash(buffer, 2*length);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

Note that I use the two-parameter version of MD5::make_hash() in order
to pass the length of the buffer. The one parameter version of this
method would stop at the first zero byte.

Edit: Regarding your extended question (the new section “Working
Example”), this is a problem completely unrelated to the previous one.
It is about string termination.
In C and C++, strings are arrays of characters terminated with a NUL
ASCII character (numeric value zero). The terminating NUL is not part of
the string per se, but it should be stored in the array holding the
string. String literals are interpreted by the compiler as
NUL-terminated arrays of characters. Every function that expects a
string as an input expects it to be NUL-terminated.
In this particular instance, you failed to NUL-terminate
challengePass. When this array is given to strelen(), this function
scans the memory until it finds a zero byte. If that zero byte happens
to be right after the end of the array, your code will work as expected.
Otherwise, any non-zero byte coming after challengePass in memory will
be interpreted as being part of the string. The program behavior is
unpredictable, because you do not know what will be stored in memory
right after challengePass.
The solution is to NUL-terminate the string. Start by allocating one
more byte:
char challengePass[8 + 1 + passLength + 1];  // challenge-password

Then, right after copying the password, add a zero byte to terminate the
string:
challengePass[8 + 1 + passLength] = '\0';  // terminate the string

Note that '\0' means “the character with numeric value zero”.
Alternatively, you can terminate the string right after adding the dash,
then use strcat() to concatenate the password. strcat() will take
care of terminating the string:
challengePass[8] = '-';
challengePass[9] = '\0';  // temporarily terminate the string
strcat(challengePass, pass);  // append password


86667: <h2>TLDR</h2>
<p>I need to convert text from a website into UTF-16LE format so that I can get the proper MD5 checksum but can't figure out how to go about doing that. This is all happening on an Arduino to log in to a router.</p>
<h2>Background</h2>
<p>I want to read values from a smart home device connected to a Fritz!Box router using an Arduino with an ethernet connection. The router has an open <a href="https://avm.de/service/schnittstellen/" rel="nofollow noreferrer">API</a> and I'm using HTTP (<a href="https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID_english_2021-05-03.pdf" rel="nofollow noreferrer">EN</a>|<a href="https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID_deutsch_2021-05-03.pdf" rel="nofollow noreferrer">DE</a>) to get the values. C++ programming is not my strong point.</p>
<p>I have been basing my attempts on <a href="https://raspberrypiandstuff.wordpress.com/2017/08/03/reading-the-temperature-from-fritzdect-devices/" rel="nofollow noreferrer">BASH</a> and <a href="https://github.com/andig/fritzapi/blob/master/index.js" rel="nofollow noreferrer">javascript</a> examples. For MD5 calculation I'm using tzikis' <a href="https://github.com/tzikis/ArduinoMD5" rel="nofollow noreferrer">ArduinoMD5 library</a>.</p>
<h2>Steps</h2>
<p>Based on the manual, you need to do the following:</p>
<ol>
<li>Contact the router to get a challenge string (router sends XML)</li>
<li>Generate a response by calculating an MD5 checksum using the challenge string and password: <code>&lt;challenge&gt;-&lt;password&gt;</code></li>
<li>Send the response, <code>&lt;challenge&gt;-&lt;response&gt;</code>, via POST</li>
<li>Receive the XLM answer from the router containing the SID</li>
<li>Use SID to request data or control a smart home device</li>
</ol>
<p>I'm stuck on step 2. I have the challenge but cannot calculate the correct checksum.</p>
<p>The manual states:</p>
<blockquote>
<p>The MD5 hash is generated from the byte sequence of the UTF-16LE
coding of this string (without BOM and without terminating 0 bytes).</p>
</blockquote>
<h2>Attempts</h2>
<p>So far I can request the challenge: 70067288 for example. The reponse appears to always be alphanumeric. My password is also alphanumeric.</p>
<pre class="lang-cpp prettyprint-override"><code>unsigned char* hash2 = MD5::make_hash(&quot;1234567z-äbc&quot;);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print(&quot;MD5: &quot;);
Serial.println(md5str2);
</code></pre>
<p>This was my attempt at confirmation. The manual gave <code>1234567z</code> as a challenge example with <code>äbc</code> (the a in the example does indeed have umlauts) as a password. The response shown for this example was <code>1234567z-9e224a41eeefa284df7bb0f26c2913e2</code> but my checksum from the above code was <code>935fe44e659beb5a3bb7a4564fba0513</code>.</p>
<p>I tried &quot;manually&quot; generating UTF-16LE and calculating the MD5 but that didn't work either.</p>
<pre class="lang-cpp prettyprint-override"><code>byte rawTest[] = {0x3100, 0x3200, 0x3300, 0x3400, 0x3500, 0x3600, 0x3700, 0x7a00, 0x2d00, 0xe400, 0x6200, 0x6300};
char buffer[12] = {};
unsigned char* hash2 = MD5::make_hash(&amp;buffer[0]);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print(&quot;MD5: &quot;);
Serial.println(md5str2);
</code></pre>
<p>It gave me <code>d41d8cd98f00b204e9800998ecf8427e</code>.</p>
<p>I am not entirely sure the example given was correctly calculated. Using their example in the API documentation with BASH code from the site mentioned above, I get a different answer: <code>1234567z-bb6e5b7c7d4d485590f4e084ad3da989</code>.</p>
<pre class="lang-bash prettyprint-override"><code>#!/bin/bash
# -----------
# definitions
# -----------
FBF=&quot;http://192.168.178.1/&quot;
USER=&quot;root&quot;
PASS=&quot;äbc&quot;
AIN=&quot;AINOFYOURFRITZDECTDEVICE&quot;
# ---------------
# fetch challenge
# ---------------
CHALLENGE=&quot;1234567z&quot;
# -----
# login
# -----
MD5=$(echo -n ${CHALLENGE}&quot;-&quot;${PASS} | iconv -f ISO8859-1 -t UTF-16LE | md5sum -b | awk '{print substr($0,1,32)}')
RESPONSE=&quot;${CHALLENGE}-${MD5}&quot;
echo $RESPONSE
</code></pre>
<p>Any help getting it working is greatly appreciatet.</p>
<hr />
<h2>Working Example (with caveats)</h2>
<p>Requirements:</p>
<ul>
<li>Password using only characters found in ASCII</li>
<li>Password at least 17 characters long</li>
</ul>
<p>The password must use <em>only</em> ASCII characters due to the simple way it's being converted to UTF-16LE.</p>
<p>I can't explain why the password must be at least 17 characters long though. My code checks the length of the actual password programmatically.</p>
<p>With a password at least 17 characters long I get something like this:</p>
<blockquote>
<p>Challenge: 6c607ee5</p>
<p>Challenge Pass (26): 6c607ee5-01234567890123456</p>
<p>Challenge Pass Preallocation Length: 26</p>
<p>MD5 (UTF-16LE): 3b26241ae4aab8eaf71d5c1599932178</p>
</blockquote>
<p>Any password short and I get something like this:</p>
<blockquote>
<p>Challenge: 621611e1</p>
<p>Challenge Pass (26): 621611e1-0123456789012345�</p>
<p>Challenge Pass Preallocation Length: 25</p>
<p>MD5 (UTF-16LE): 30a163ab8a267adfbb524b2b7412e81b</p>
</blockquote>
<p>Please excuse any poor coding, C++ is not my strongpoint.</p>
<pre class="lang-cpp prettyprint-override"><code>void fritzboxSID() {
  EthernetClient fritzBox;
  const char* pass = &quot;01234567890123456&quot;;  // 17 ASCII character minimum
  const char* user = &quot;fritz3456&quot;;  // Account on router, randomly generated or user created
  const size_t passLength = strlen(pass);  // Password length
  char c;
  uint8_t xml = 0;
  char challenge[9] = {0};  // Challenge
  char challengePass[8 + 1 + passLength];  // challenge-password
  uint8_t challengeCount = 0;
  
  if (fritzBox.connect(&quot;192.168.200.1&quot;, 80)) {
    // Send login request
    fritzBox.println(&quot;GET /login_sid.lua HTTP/1.1&quot;);
    fritzBox.println(&quot;Host: 192.168.200.1&quot;);
    fritzBox.println(&quot;Connection: close&quot;);
    fritzBox.println();
    
    while (fritzBox.connected()) {
      while (fritzBox.available()) {
        c = fritzBox.read();  // Read a single character from the router response
        if (c == '&gt;') {
          xml++;
        }
        if (xml == 5) {
            // Challenge starts after the fifth &gt;
            if (challengeCount &gt; 0 &amp; challengeCount &lt; 9) {
                // Save the challenge, character by character
                challenge[challengeCount-1] = c;
                challengePass[challengeCount-1] = c;
            }
            challengeCount++;
        }
        Serial.print(c);  // Print HTTP response
      }
    }
    challengePass[8] = '-';
    for (size_t i = 0;i &lt; passLength; i++){
      // Copy password into the combined challenge-password string
      challengePass[i+9] = pass[i];
    }
    
    // Show challenge and pass, checking lengths
    //      If the strlen(challengePass) doesn't equal the preallocation length then the MD5 will be incorrect
    Serial.println(&quot;&quot;);
    Serial.print(&quot;Challenge: &quot;);
    Serial.println(challenge);
    Serial.print(&quot;Challenge Pass (&quot;);
    Serial.print(strlen(challengePass));
    Serial.print(&quot;): &quot;);
    Serial.println(challengePass);
    Serial.print(&quot;Challenge Pass Preallocation Length: &quot;);
    Serial.println(8 + 1 + passLength);

    // Convert to UTF-16LE (only works for standard ASCII characters)
    const size_t length = strlen(challengePass);
    char buffer[2*length];
    for (size_t i = 0; i &lt; length; i++) {
      buffer[2*i] = challengePass[i];
      buffer[2*i+1] = 0;
    }

    // Generate MD5
    unsigned char* hash = MD5::make_hash(buffer, 2*length);
    char *md5str = MD5::make_digest(hash, 16);
    free(hash);
    Serial.print(&quot;MD5 (UTF-16LE): &quot;);
    Serial.println(md5str);
  } else {
    Serial.println(&quot;Cannot connect to 192.168.200.1&quot;);
  }
}
</code></pre>
<p>If it helps, the response from the router for the initial login request is this:</p>
<pre><code>HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/xml
Date: Fri, 16 Sep 2022 08:21:27 GMT
Expires: -1
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; connect-src 'self'; font-src 'self'; frame-src https://service.avm.de https://help.avm.de https://www.avm.de https://avm.de https://assets.avm.de https://clickonce.avm.de http://clickonce.avm.de http://download.avm.de https://download.avm.de 'self'; img-src 'self' https://tv.avm.de https://help.avm.de/images/ http://help.avm.de/images/ data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'; media-src 'self'

&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;&lt;SessionInfo&gt;&lt;SID&gt;0000000000000000&lt;/SID&gt;&lt;Challenge&gt;621611e1&lt;/Challenge&gt;&lt;BlockTime&gt;0&lt;/BlockTime&gt;&lt;Rights&gt;&lt;/Rights&gt;&lt;Users&gt;&lt;User last=&quot;1&quot;&gt;fritz3456&lt;/User&gt;&lt;User&gt;UserName&lt;/User&gt;&lt;/Users&gt;&lt;/SessionInfo&gt;
</code></pre>
 7508: <p>You have to convert the string &quot;&lt;challenge&gt;-&lt;password&gt;&quot; to
UTF-16LE. If the whole string is plain ASCII, the conversion is trivial:
you just have to add a zero byte after each ASCII byte.</p>
<pre class="lang-cpp prettyprint-override"><code>// ASCII string to be hashed.
const char* text = &quot;1234567z-abc&quot;;  // umlaut removed

// Convert to UTF-16LE.
const size_t length = strlen(text);
char buffer[2*length];
for (size_t i = 0; i &lt; length; i++) {
    buffer[2*i] = text[i];
    buffer[2*i+1] = 0;
}

// Compute and print the hash.
unsigned char* hash2 = MD5::make_hash(buffer, 2*length);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print(&quot;MD5: &quot;);
Serial.println(md5str2);
</code></pre>
<p>Note that I use the two-parameter version of <code>MD5::make_hash()</code> in order
to pass the length of the buffer. The one parameter version of this
method would stop at the first zero byte.</p>
<hr />
<p><strong>Edit</strong>: Regarding your extended question (the new section “Working
Example”), this is a problem completely unrelated to the previous one.
It is about string termination.</p>
<p>In C and C++, strings are arrays of characters terminated with a NUL
ASCII character (numeric value zero). The terminating NUL is not part of
the string per se, but it should be stored in the array holding the
string. String literals are interpreted by the compiler as
NUL-terminated arrays of characters. Every function that expects a
string as an input expects it to be NUL-terminated.</p>
<p>In this particular instance, you failed to NUL-terminate
<code>challengePass</code>. When this array is given to <code>strelen()</code>, this function
scans the memory until it finds a zero byte. If that zero byte happens
to be right after the end of the array, your code will work as expected.
Otherwise, any non-zero byte coming after <code>challengePass</code> in memory will
be interpreted as being part of the string. The program behavior is
unpredictable, because you do not know what will be stored in memory
right after <code>challengePass</code>.</p>
<p>The solution is to NUL-terminate the string. Start by allocating one
more byte:</p>
<pre class="lang-cpp prettyprint-override"><code>char challengePass[8 + 1 + passLength + 1];  // challenge-password
</code></pre>
<p>Then, right after copying the password, add a zero byte to terminate the
string:</p>
<pre class="lang-cpp prettyprint-override"><code>challengePass[8 + 1 + passLength] = '\0';  // terminate the string
</code></pre>
<p>Note that <code>'\0'</code> means “the character with numeric value zero”.</p>
<p>Alternatively, you can terminate the string right after adding the dash,
then use <code>strcat()</code> to concatenate the password. <code>strcat()</code> will take
care of terminating the string:</p>
<pre class="lang-cpp prettyprint-override"><code>challengePass[8] = '-';
challengePass[9] = '\0';  // temporarily terminate the string
strcat(challengePass, pass);  // append password
</code></pre>
