WARNING: if the bug you are reporting has or may have security implications,
we ask that you report it privately to
[email protected]
so that we can prepare and release a fix before publishing the details.
See SECURITY.md.

Summary

System information

Mbed TLS version 3.6.4 (c765c83)
Operating system and version: Ubuntu Linux 22.04 in Docker
Configuration (if not default, please attach mbedtls_config.h): MBEDTLS_MD_C is enabled
Compiler and options (if you used a pre-built binary, please indicate how you obtained it): IAR BX ARM 9.60, building in big-endian mode with --endian=big
Additional environment information:

Expected behavior

When built using IAR compiler and big endian enabled, I expect the generated code to function correctly.

Actual behavior

In a test using a SHA256 test vector from RFC 4634, the wrong SHA256 digest is generated.

*WARNING: if the actual behaviour suggests memory corruption (like a crash or an error
from a memory checker), then the bug should be assumed to have security
implications (until proven otherwise), and we ask what you report it privately,
see the note at the top of this template.

Steps to reproduce

Additional information

The underlying cause is an incorrect assumption being made in alignment.h, reproduced here:

#if defined(__LITTLE_ENDIAN__)
/* IAR defines __xxx_ENDIAN__, but not __BYTE_ORDER__ */
#define MBEDTLS_IS_BIG_ENDIAN 0
#elif defined(__BIG_ENDIAN__)
#define MBEDTLS_IS_BIG_ENDIAN 1
#else
static const uint16_t mbedtls_byte_order_detector = { 0x100 };
#define MBEDTLS_IS_BIG_ENDIAN (*((unsigned char *) (&mbedtls_byte_order_detector)) == 0x01)
#endif

The assumption in the comment here is incorrect. The IAR compiler does not define __xxx_ENDIAN__ based on the endian mode; instead it always defines __LITTLE_ENDIAN__ and its value is 1 (little endian) or 0 (big endian).

The relevant section from the IAR compiler manual on the definition of __LITTLE_ENDIAN__ is:

An integer that reflects the setting of the compiler option --endian and is defined to 1 when the byte order is little-endian. The symbol is defined to 0 when the byte order is big-endian.

Because __LITTLE_ENDIAN__ is still defined when big endian mode is enabled, MBEDTLS_IS_BIG_ENDIAN is being set to 0 in alignment.h and the wrong SHA256 digest comes out in my test.

The following modification to the code in alignment.h appears to fix the issue for me:

#if defined(__LITTLE_ENDIAN__)
#if __LITTLE_ENDIAN__
/* IAR always defines __LITTLE_ENDIAN__, but not __BYTE_ORDER__ */
#define MBEDTLS_IS_BIG_ENDIAN 0
#else
#define MBEDTLS_IS_BIG_ENDIAN 1
#endif
#else
static const uint16_t mbedtls_byte_order_detector = { 0x100 };
#define MBEDTLS_IS_BIG_ENDIAN (*((unsigned char *) (&mbedtls_byte_order_detector)) == 0x01)
#endif