DeserializeValue "Reached end of array" error when compiling IL2CPP on Android.

[John McCormac]

I believe there is a bug with DeserializeValue<> using Dictionaries and MemoryStreams after upgrading to Unity 2022.2.7f1 and Odin 3.1.10.0 when using IL2CPP for android. It works fine on the PC and when using the Mono backend. I have managed to recreate the issue with a small standalone code snippet which causes the error when compiled with IL2CPP (it works correctly otherwise):

[System.Serializable]
public class AssetSaveData {
    public Dictionary<string, MemoryStream> serializedData = new Dictionary<string, MemoryStream>();
}

public void TestDeserializeValue() {
    var testData = new AssetSaveData();
    testData.serializedData = new Dictionary<string, MemoryStream>();
    testData.serializedData["key1"] = new MemoryStream();
    Sirenix.Serialization.SerializationUtility.SerializeValue<string>("value1", testData.serializedData["key1"], DataFormat.Binary);
    var testSaveStream = new MemoryStream();
    Sirenix.Serialization.SerializationUtility.SerializeValue<AssetSaveData>(testData, testSaveStream, dataFormat);
    var loadedTestData = Sirenix.Serialization.SerializationUtility.DeserializeValue<AssetSaveData>(testSaveStream.ToArray(), DataFormat.Binary);
}

When run on the device this results in the following two errors:

AndroidPlayer "Google_Pixel 6a@XXXXXXXXX:0" Reached end of array after 18 elements, when 498216206354 elements were expected.
OdinSerializer.MemoryStreamFormatter:DeserializeImplementation(MemoryStream&, IDataReader)
Sirenix.Serialization.BaseFormatter`1:Deserialize(IDataReader)
Sirenix.Serialization.ComplexTypeSerializer`1:ReadValue(IDataReader)
Sirenix.Serialization.DictionaryFormatter`2:DeserializeImplementation(Dictionary`2&, IDataReader)
Sirenix.Serialization.BaseFormatter`1:Deserialize(IDataReader)
Sirenix.Serialization.AnySerializer:ReadValueWeak(IDataReader)
Sirenix.Serialization.ReflectionFormatter`1:DeserializeImplementation(T&, IDataReader)
Sirenix.Serialization.BaseFormatter`1:Deserialize(IDataReader)
Sirenix.Serialization.AnySerializer:ReadValueWeak(IDataReader)
Sirenix.Serialization.SerializationUtility:DeserializeValue(IDataReader)
Sirenix.Serialization.SerializationUtility:DeserializeValue(Stream, DataFormat, DeserializationContext)
Sirenix.Serialization.SerializationUtility:DeserializeValue(Byte[], DataFormat, DeserializationContext)
GameStateSaver:TestDeserializeValue()

AndroidPlayer "Google_Pixel 6a@XXXXXXXXX:0" Reached end of array after 1 elements, when 502511173633 elements were expected.
Sirenix.Serialization.DictionaryFormatter`2:DeserializeImplementation(Dictionary`2&, IDataReader)
Sirenix.Serialization.BaseFormatter`1:Deserialize(IDataReader)
Sirenix.Serialization.AnySerializer:ReadValueWeak(IDataReader)
Sirenix.Serialization.ReflectionFormatter`1:DeserializeImplementation(T&, IDataReader)
Sirenix.Serialization.BaseFormatter`1:Deserialize(IDataReader)
Sirenix.Serialization.AnySerializer:ReadValueWeak(IDataReader)
Sirenix.Serialization.SerializationUtility:DeserializeValue(IDataReader)
Sirenix.Serialization.SerializationUtility:DeserializeValue(Stream, DataFormat, DeserializationContext)
Sirenix.Serialization.SerializationUtility:DeserializeValue(Byte[], DataFormat, DeserializationContext)
GameStateSaver:TestDeserializeValue()

Hopefully the code snippet is able to repreduce the same error on your side, please let me know if not!

‌

[Elijah James]

Having the same issue after upgrading to 2022 LTS

[Nikunj Rola]

Having Same issue with android and iOS builds (Works fine with Editor)
Unity Version : 2022.3.4f1 (LTS)
Odin Inspector Version : 3.1.13.0

[Tor Esa Vestergaard]

So I have attempted to replicate this and I’m afraid it does not reproduce. I had to modify the given example, because MemoryStream does not support being serialized at all, on any platform - so I changed it to use byte[] instead, and display a comparison result so I can see if serialization worked properly. Here’s the script I’ve been testing:

// DoTheTest.cs
using Sirenix.OdinInspector;
using Sirenix.Serialization;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using UnityEngine;

public class DoTheTest : MonoBehaviour
{
    private bool hasDoneTheThing;
    private bool result;

    [System.Serializable]
    public class AssetSaveData
    {
        public Dictionary<string, byte[]> serializedData = new Dictionary<string, byte[]>();

        public static bool Compare(AssetSaveData a, AssetSaveData b)
        {
            if (a == null && b == null)
                return true;
            else if ((a == null) != (b == null)) 
                return false;

            if ((a.serializedData == null) != (b.serializedData == null))
                return false;

            if (a.serializedData != null)
            {
                if (a.serializedData.Count != b.serializedData.Count)
                    return false;

                foreach (var key in a.serializedData.Keys)
                {
                    if (!b.serializedData.ContainsKey(key))
                        return false;

                    var aStream = a.serializedData[key];
                    var bStream = b.serializedData[key];

                    if (aStream.Length != bStream.Length)
                        return false;

                    var aBytes = aStream;
                    var bBytes = bStream;

                    for (int i = 0; i < aBytes.Length; i++)
                    {
                        if (aBytes[i] != bBytes[i])
                            return false;   
                    }
                }

                foreach (var key in b.serializedData.Keys)
                {
                    if (!a.serializedData.ContainsKey(key))
                        return false;

                    var aStream = a.serializedData[key];
                    var bStream = b.serializedData[key];

                    if (aStream.Length != bStream.Length)
                        return false;

                    var aBytes = aStream;
                    var bBytes = bStream;

                    for (int i = 0; i < aBytes.Length; i++)
                    {
                        if (aBytes[i] != bBytes[i])
                            return false;
                    }
                }
            }

            return true;
        }
    }

    [Button]
    public bool TestDeserializeValue()
    {
        var testData = new AssetSaveData();
        testData.serializedData = new Dictionary<string, byte[]>();
        var saveStream = new MemoryStream();
        Sirenix.Serialization.SerializationUtility.SerializeValue<string>("value1", saveStream, DataFormat.Binary);
        testData.serializedData["key1"] = saveStream.ToArray();
        var testSaveStream = new MemoryStream();
        Sirenix.Serialization.SerializationUtility.SerializeValue<AssetSaveData>(testData, testSaveStream, DataFormat.Binary);
        var loadedTestData = Sirenix.Serialization.SerializationUtility.DeserializeValue<AssetSaveData>(testSaveStream.ToArray(), DataFormat.Binary);
        var result = AssetSaveData.Compare(testData, loadedTestData);
        Debug.LogError("THE RESULT IS: " + result);
        return result;
    }

    // Start is called before the first frame update
    void Start()
    {
        result = TestDeserializeValue();
        hasDoneTheThing = true;
    }

    private void OnGUI()
    {
        if (hasDoneTheThing)
        {
            GUILayout.Label("The result is: " + result);
        }
    }
}

I have confirmed that this code runs correctly and passes that compare check on Windows Mono, Windows IL2CPP, Android Mono and Android IL2CPP. For Android, specifically, I tested on a Samsung Galaxy s10+. Odin 3.1.14.1 in source mode build (we’ve made no serializer changes that would affect this since), in Unity 2022.3.4f1 (I didn’t have the earlier version of Unity installed - but if that matters, this is a Unity IL2CPP/runtime error.) I’m downloading Unity 2022.2.21f1 to quickly test there, just to be on the safe side, I’ll update here how that pans out.

[Tor Esa Vestergaard]

I’ve also tested on Unity 2022.2.21f1 now, and it still serializes and compares successfully. So I’d need a more reliably reproducing setup to be able to move forward with this issue report, I’m afraid.

Edit: Oh, and to clarify, I have been testing in both development and release mode builds.

[Nikunj Rola]

Hello Again,
Yeah we are using 2022.2 unity and it is woking fine in that (As mention in your above message)
But now we wants to upgrade unity to 2022.3.4f1 (LTS Version) but we faced this issue in new unity LTS
So We have created minimal reproduction demo for this test case. Could you please find attached project from your side to reproduce it.

Version Detail
Unity Version : 2022.3.4f1 (LTS)
Odin Inspector Version : 3.1.14.0

Cases we have tried
We have tried it with android mono and IL2CPP and found that in mono it is working but in IL2CPP it does not, and in iOS IL2CPP face the same issue
We are assuming that there is some issue with Dictionary deserialisation when using enum as key in Dictionary. (If this hint can help you to find cause the issue)

Minimal reproduction demo

[Tor Esa Vestergaard]

I’m afraid I still haven’t managed to make this reproduce with your reproduction project - I open it in 2022.3.4f1, build it to an Android device attached over USB, and I’m afraid it works fine and I can click both buttons and get all expected logs through logcat just fine (I turned them into error logs to make them easier to find). There are no serialization errors anywhere to be found.

‌

[Nikunj Rola]

Thank you for quick reply.
Can you please once check that you have enabled ARM64 architecture?
If not could you please try once again by enabling both “Target Architecture” (ARMv7 and ARM64) from player settings. As we need both architecture for our game.

‌

[Tor Esa Vestergaard]

I was doing my testing with both of those architectures enabled, yes. I have only been running the tests on a Samsung Galaxy s10+, though.

‌

[Nikunj Rola]

This is so strange.
I have mailed on “[e-mail]“ Could you please do reply in that.

[Maulik Kaloliya]

Hello, Tor

I think it’s working for you because you did not generate AOT DLL

I tested with these test cases

Case 1: Generated AOT DLL - then got crashed

Case 2: I cleared out generate AOT DLL and cleared every serialized support type and then run build and surprisingly it worked

and throw this error as well

Reached end of array after 3 elements, when 4294967299 elements were expected.

It’s strange that how without AOT generation it’s working in IL2CPP

[Nikunj Rola]

Hello Tor,
Could you please confirm that in minimal repo have you scan AOT and generated DLL
Cos in our test case with minimal repo if we didn’t Scan and generate DLL it’s working as expected, but if will do scan project and generate DLL we face issues
Our live project is huge and i think it is compulsory to scan and generate DLL.
Could you please check and confirm that case.
Thanks.

[Tor Esa Vestergaard]

Hello, I’m back from holidays and ready to get started on this issue again :)

I just ran the test with AOT generated, and the issue still did not replicate. I have discovered a little bit about what is wrong, however - in each case, it seems that when reading the long containing the expected length of the collection we’re about to read, the 5th byte of 8 is somehow wrong. In your case of a collection that is length 3 but reads a long saying there are 4294967299 expected elements, for example, the correct byte sequence for the long is “03 00 00 00 00 00 00 00”, but we’ve read a long 4294967299 which has the byte sequence “03 00 00 00 01 00 00 00”. In every reported case (people have also reported this on Discord), the 5th read byte is wrong, meaning the code has somehow managed to read a wrong value from the buffer, or somewhere else - somewhere, this byte is sneaking in for unknown reasons.

By now it is entirely clear that this is obviously a newly introduced Unity error in how IL2CPP compiles the serializer’s code, since this code works fine in other versions of Unity. However, we’re still interested in figuring out a way to mitigate the issue, if it’s possible at all.

Nobody has as of yet been able to provide us with a project that reproduces for us, which leads me to believe this might be hardware related. As such, the hardware both compiling the build for Android, as well as the phone hardware and OS running it, are under suspicion as possibly being relevant factors - if you could please tell us the exact hardware setup on which this happens, that might be helpful in figuring out what the common factors are. I think the next step is for me to construct a debugging build of Odin that people with the issue can run locally and then provide me with the logs. I’ll add in a ton of logs and memory dumps that trigger when this issue happens.

[Elijah James]

I build my project on an apple silicon mac

[Tor Esa Vestergaard]

Okay everyone, I have a very aggressively logging debug build of Odin that I can provide. It will (hopefully) detect the issue when it happens, and log a LOT of useful info so I can attempt to work out which changes might be necessary to fix this. Anybody running this build should provide me with the ENTIRE Unity log file for that run, as Unity itself logs a lot of valuable info about the hardware and OS setup it’s running on that might be relevant. The logs will include info about which code path is running, a lot of memory dumps and logged values, results of potential alternative deserialization methods, as well as a hex dump of the entire data buffer we are deserializing from. This should indicate if the buffer data itself is bad (meaning the building machine is at fault), or if it’s just the method being used to read from the buffer that’s bad (indicating the issue happens on the Android phone itself).

If you want access to this debugging build, please contact me on Discord (invite link: https://discord.gg/WTYJEra) and I will send it to you there. Please don’t hesitate - the more data we can collect, the better!

@Elijah James Which Android phone model is the error occurring on, and which Android OS version is it running?

‌

[Elijah James]

I have a lot of beta testers on my game and they all experienced the problem, my personal device that experienced it was a Pocophone F1 on Android 12

[Tor Esa Vestergaard]

I have been getting back some logs from people running the build, and it appears that the actual built serialization data is faulty, and Android is in fact deserializing it correctly. So this is somehow happening in the editor during the build process, but it seems only when building for Android, and probably only on specific hardware/OS. I will probably have to construct a new logging build that also has a bunch of logs during serialization/build time to identify where it is going wrong, now.

[Jonell Bagayan]

Hi, I’ve also came across with this issue but not on dictionary and not with “reached end of array”. Mine is on a nested list with error “Invalid array length: 0”. It happened after upgrading to:

Unity version: 2022.3.6f1
Odin: 3.1.14.1

Tested on:
Working
Samsung Galaxy Tab A Android 11
ASUS Zenphone Max M1 Pro Android 9
Not working
Rog Phone 7 Android 13
Mi 11 Lite Android 13
Mi Note 10 Pro Android 11

Xiaomi 9T Pro Android 11

‌

Tested scenarios:
1. with FileStream

2. with File.ReadAllBytes
3. System.Serializable

---

Sample Code:

ClinkAssetManifestData

public class ClinkAssetManifestData
{
  public List<ClinkAssetCatalogData> Catalogs;
  public List<ClinkAssetRecordData> Assets;

  Dictionary<string, ClinkAssetCatalogData> _catalogDictionary;
  Dictionary<string, ClinkAssetRecordData> _assetDictionary;

  [OnDeserialized]
  protected virtual void DidDeserialize(StreamingContext sc)
  {
    _catalogDictionary = new Dictionary<string, ClinkAssetCatalogData>();
    _assetDictionary = new Dictionary<string, ClinkAssetRecordData>();

    _catalogDictionary = Catalogs.ToDictionary(catalog => catalog.Key);
    _assetDictionary = Assets.ToDictionary(asset => asset.Key);
  }
}

ClinkAssetCatalogData

public class ClinkAssetCatalogData
{
  public string Key;
  public long Version;
  public List<ClinkAssetRecordData> AssetRecords;

  Dictionary<string, ClinkAssetRecordData> _assetDictionary;

  [OnDeserialized]
  protected virtual void DidDeserialize(StreamingContext sc)
  {
    _assetDictionary = new Dictionary<string, ClinkAssetRecordData>();
    _assetDictionary = AssetRecords.ToDictionary(asset => asset.Key);
  }
}

ClinkAssetRecordData

public class ClinkAssetRecordData
{
  public string Key;
  public string RemotePath;
  public string RemoteHash;
  public long RemoteVersion;
  public long MinVersion;
  public string DownloadPath;
  public string LocalPath;
  public string LocalHash;
  public long LocalVersion;
  public bool Verified = false;
}

Saving

ClinkAssetManifestData assetManifest = new ClinkAssetManifestData(embeddedManifest);

string filePath = Path.Combine(Application.persistentDataPath, assetPath);
FileInfo file = new(filePath);
file.Directory.Create();

using (FileStream stream = file.Open(FileMode.OpenOrCreate))
{
    SerializationUtility.SerializeValue(assetManifest, stream, DataFormat.Binary);
}

Loading

ClinkAssetManifestData assetManifest = new();
bool success = false;

string filePath = Path.Combine(Application.persistentDataPath, assetPath);
FileInfo file = new(filePath);
if (file.Exists)
{
    using (FileStream stream = file.Open(FileMode.Open))
    {
        assetManifest = SerializationUtility.DeserializeValue<ClinkAssetManifestData>(stream, DataFormat.Binary);

        if (assetManifest != null)
        {
            Debug.Log("LOADED MANIFEST");
            foreach (var catalog in assetManifest.Catalogs)
            {
                Debug.Log($"CATALOG: {catalog.Key}");
                foreach (var asset in catalog.AssetRecords)
                {
                    // Only the first catalog will have asset entries, the rest has no entry (which is wrong)
                    Debug.Log($"ASSET: {asset.Key}");
                }
            }
            success = true;
        }
        else
        {
            assetManifest = new();
        }
    }
}

‌

[Tor Esa Vestergaard]

I believe I have figured out the issue, thanks to some very helpful customers who’ve been running several test and logging builds as I narrow down the possible causes. It seems to be an inlining-related compiler error for unsafe code on IL2CPP targeting Android in Unity 2022.3 and above, specifically when it is translating pre-compiled assemblies compiled by Roslyn; the same error does not occur when the source is compiled by Unity itself. My best theory is that the inlining doesn't assign and clear sufficient stack space for certain variables in the inlined code, and so other garbage values in uncleared stack memory leak into the serialized long values.

Regardless, it looks like adding a no inlining flag to the method in question resolves the problem - however I am still in the process of double checking and verifying this by having more affected customers test the fix. If anybody would like a preview build of this fix to verify that it works before we put it into an actual release, please contact me on Discord and I’ll get it to you promptly.

[Tor Esa Vestergaard]

This issue has been resolved as of patch 3.1.14.3 which should be live on our website within a few hours, and live on the Asset Store within a few days.

Thanks to everyone who helped me debug and work through this very annoying and complicated to diagnose issue. Errors in the compiler as this turned out to be are never fun - especially when related to something as ephemeral as code inlining which is affected by attempts to add debugging statements to the code (either affecting whether a method is inlined or not, as well as the size/layout of the stack memory).

[work phynic]

Hello, I need some help. Our unity game has upgraded Odin to version 3.1.14.3, but after packaging, we still see many "Error: Reached end of array after X elements, when Y elements were expected." Is there any additional setup I need to do to fix this issue? Our unity version is 2022.3.22f1, and the platform with the problem is iOS. The IL2CPP Code Generation setting is Faster (smaller) builds.

@Tor Esa Vestergaard

[work phynic]

3.3.0.0 seems to fix this issue