Building with Flutter + Unity (AR Experience Toolkit)

April 13, 2021 Article

Wallace & Gromit: The Big Fix Up launched in January 2021, thanks to the collaboration between multi-award-winning independent animation studio Aardman and storytelling venture Fictioneers (a consortium of Potato, Tiny Rebel Games, and Sugar Creative).

Available on iOS and Android devices in the UK, US, and Canada, the free app creates a narrative-driven experience, taking the user through AR gameplay, CG animations, in-character phone calls, extended reality (XR) portals, and comic strips. In the new storyline, inventor Wallace and his dog Gromit’s new business venture takes on a contract to “Fix-Up” the city of Bristol, UK.

It’s been three months since The Big Fix Up app was made public and it has gained a lot of attention. This AR masterpiece, which was built on Flutter with Unity, was a first of its kind and raised a lot of interesting questions about our approach to building this real-time storytelling experience:

Why Flutter?
Embedding Unity3D app in Flutter?
Communication between Unity and Flutter
Performance implications for embedding Unity with Flutter,
App bundle size, e.t.c.

Here we’ll break down how we answered these questions and eventually solved the problem of embedding Unity in Flutter. The result was a Flutter Unity View plugin which is published opensource for the community here: https://github.com/juicycleff/flutter-unity-view-widget.

Embedding Unity3D app in Flutter

We needed to export the Unity project as a native iOS and Android project so we can convert it into a native package that can be added to the native codebase of our Flutter app. The problem was that Unity at the time only had support for exporting Unity projects as an app instead of a package (This was pre UaaL). This required modification to the actual Unity project which heavily involves modifying our exported Unity app source and converting it into a library. But today with support for Unity as a library (UaaL), which now does support a lot better for reusing Unity inside native apps, there was little modification to be done, but we still needed a few like;

Notifying Flutter when Unity player is ready to be used in Flutter. For instance, we added a new notification event to the Unity iOS post-build processors by extending the codebase just like so.

/// <summary>
/// Edit 'UnityAppController.mm': triggers 'UnityReady' notification after Unity is actually started.
/// </summary>
private static void EditUnityAppControllerMM(string path)
{
    var inScope = false;
    var markerDetected = false;

    EditCodeFile(path, line =>
    {

        inScope |= line.Contains("- (void)startUnity:");
        markerDetected |= inScope && line.Contains(TouchedMarker);

        if (inScope && line.Trim() == "}")
        {
            inScope = false;

            if (markerDetected)
            {
                return new string[] { line };
            }
            else
            {
                return new string[]
                {
                    "    // Modified by " + TouchedMarker,
                    @"    [[NSNotificationCenter defaultCenter] postNotificationName: @""UnityReady"" object:self];",
                    "}",
                };
            }
        }

        return new string[] { line };
    });

}

We also had to modify the Unity project Android export to work with Flutter seamlessly by modifying the Gradle build script automatically by replacing all Gradle android application-related codes to an android library like so;

public static void DoBuildAndroid()
{
    ...

    // Modify build.gradle
    var build_file = Path.Combine(androidExportPath, "build.gradle");
    var build_text = File.ReadAllText(build_file);
    build_text = build_text.Replace("com.android.application", "com.android.library");
    build_text = build_text.Replace("bundle {", "splits {");
    build_text = build_text.Replace("enableSplit = false", "enable false");
    build_text = build_text.Replace("enableSplit = true", "enable true");
    build_text = build_text.Replace("implementation fileTree(dir: 'libs', include: ['*.jar'])", "implementation(name: 'unity-classes', ext:'jar')");
    build_text = Regex.Replace(build_text, @"\n.*applicationId '.+'.*\n", "\n");
    File.WriteAllText(build_file, build_text);

    // Modify AndroidManifest.xml
    var manifest_file = Path.Combine(androidExportPath, "src/main/AndroidManifest.xml");
    var manifest_text = File.ReadAllText(manifest_file);
    manifest_text = Regex.Replace(manifest_text, @"<application .*>", "<application>");
    Regex regex = new Regex(@"<activity.*>(\s|\S)+?</activity>", RegexOptions.Multiline);
    manifest_text = regex.Replace(manifest_text, "");
    File.WriteAllText(manifest_file, manifest_text);

    ...
}

Knowing we can automate the export from Unity and modify Unity export on the fly, Unity was ready to be embedded into Flutter. This then was a good foundation to enable Flutter to render unity. With the highly performant native extension support in Flutter, we were able to use Flutter Native API to achieve rendering Unity on Flutter, some of which are;

These APIs allowed us to attach the Unity native view in Flutter for both Android and iOS platforms. With rendering out of the way, we needed to have full bidirectional communication between Flutter and Unity.

Communication between Unity and Flutter

Flutter does support a high-performance messaging transport from Flutter to the native side with the MethodChannel API support, one good reason Flutter outperforms other cross-platform frameworks. The MethodChannel API gives us means for communication with the native code however, Unity as a Library only allows one-way communication, which was from Flutter to Unity only and so we had to figure out how Unity can message Flutter. Luckily Unity has support for calling native code classes and static methods from within Unity and this led to writing the Unity Message Manager whose job is to call Kotlin/Java/Obj C/Swift static class methods, which in turn notifies or calls a Flutter method. The Unity message manager looks like this;

public class NativeAPI
{
    #if UNITY_IOS && !UNITY_EDITOR

    [DllImport("__Internal")]
    public static extern void OnUnityMessage(string message);
    #endif


    public static void SendMessageToFlutter(string message)
    {
        #if UNITY_ANDROID
        try
        {
            AndroidJavaClass jc = new AndroidJavaClass("com.xraph.plugin.flutter_unity_widget.UnityPlayerUtils");
            jc.CallStatic("onUnityMessage", message);
        }
        catch (Exception e)
        {
            Debug.Log(e.Message);
        }
        #elif UNITY_IOS && !UNITY_EDITOR
        NativeAPI.OnUnityMessage(message);
        #endif
    }
}

Here we invoke the native Android class UnityPlayerUtils method UnityPlayerUtils and also an ObjectiveC static method NativeAPI.OnUnityMessage(message) while using CSharp preprocessor directives. On the Flutter native end, the methods look like this for Android Kotlin;

class UnityPlayerUtils {
  ...  /**
  * Invoke by unity C#
  */
  @JvmStatic
  fun onUnityMessage(message: String) {
    for (listener in mUnityEventListeners) {
        try {
            listener.onMessage(message)
        } catch (e: Exception) {
            e.message?.let { Log.e(LOG_TAG, it) }
        }
    }
  }  ...
}

and on iOS as simple as this;

void OnUnityMessage(const char* message)
{
    [UnityPlayerUtils unityMessageHandler:message];
}

and it calls the Flutter MessageChannel instance to notify flutter;

@objc public class UnityPlayerUtils: UIResponder, UIApplicationDelegate, UnityFrameworkListener {  @objc
  public static func unityMessageHandler(_ message:        UnsafePointer<Int8>?) {
    if let strMsg = message {        globalChannel?.invokeMethod("events#onUnityMessage", arguments: String(utf8String: strMsg))    } else {        globalChannel?.invokeMethod("events#onUnityMessage", arguments: "")    }
  }}

With these, we had bidirectional communication between Flutter and Unity through the MethodChannel API with very impressive performance, which was vital to our choosing Flutter in the first place.