*AITRIOS and AITRIOS logos are the registered trademarks or trademarks of Sony Group Corporation or its affiliated companies.
This article is based on the Open Source Summit Japan presentation by Kenji Shimizu and Munehiro Shimomura (Sony Semiconductor Solutions), titled “A Next-Generation IoT Platform for Edge AI Apps Leveraging Sensors and Wasm.”
The Edge Device and Edge AI We’re Driving Forward
Edge AI is a transformative technology enabling artificial intelligence to operate directly on edge devices such as smartphones, cameras, drones, IoT sensors, and industrial machinery.
Unlike traditional AI systems, which depend on cloud computing for data processing and analytics, Edge AI processes data locally on the device. This localized approach significantly reduces latency, improves response times, enhances data privacy, and minimizes reliance on constant internet connectivity.
Edge AI technology operates on those edge devices, which include three core components:
- Sensing/Input: Collects raw data from the environment via sensors like cameras, microphones, and GPS. This stage is essential for gathering information for the following AI processing.
- Processing: Using embedded processors or AI chips, the device processes data locally, reducing cloud dependence and latency—vital for real-time tasks such as autonomous driving and predictive maintenance, for instance.
- Actuating/Output: Based on the resulting output of processed data, the device can act autonomously, triggering alerts, moving components, or adjusting settings to respond instantly to environmental changes.
The Importance of Collaboration between Edge AI and Sensors
The collaboration between Edge AI and sensors is essential for real-time data processing and decision-making.
Sensors: Capturing Events as ‘Data’
Sensors play a pivotal role in converting real-world events into structured, actionable data that Edge AI can process immediately, right where it’s needed.
Importance of data
In today’s digital landscape, data-driven decision-making is indispensable, particularly for Edge AI. Accurate, high-quality data from sensors forms the backbone of intelligent decision-making at the edge, enabling systems to process and act on information in real time.
Sensor data, such as that from image sensors, captures detailed visual insights about the surrounding environment, offering multi-dimensional information beyond the capabilities of most other sensor types. For instance, cameras and LIDAR sensors in autonomous vehicles provide precise environmental data to ensure safe driving and efficient navigation.
The Value of Image Data
Image data is uniquely powerful, often providing information that is “worth a thousand words.” The richness of image data enhances the accuracy and reliability of AI-driven systems, adding a crucial layer of depth that supports more informed and intelligent decision-making.
In short, the collaboration between Edge AI and sensors is essential, with image data playing a particularly valuable role in advancing reliable, responsive, and safe AI applications.
Diverse Types of Sensors
There are various types of sensors, each tailored to specific characteristics and applications. To begin with, when considering their imaging and sensing capabilities, we can categorize them as follows:
- Image Sensor:
- For smartphones
- For security cameras
- Vision Sensor:
- Event-based
- Image sensor + Stabilize LSI
- ToF sensor for smartphones
- SPAD for LiDAR (in-vehicle)
- SWIR sensor
- Polarization sensor
Following that, we can present an extensive list of sensor types, which includes more than just the ten examples mentioned:
Multi-modal AI and Sensing
As AI applications evolve, they increasingly demand diverse data inputs for complex problem-solving. This shift highlights the necessity of multi-modal sensing in Edge AI, where combining various types of data—such as text, images, and audio—enhances inference accuracy and allows for more robust decision-making.
Increasingly Advanced AI Use Cases
AI is being used for increasingly complex and advanced applications that require more diverse data inputs.
Combining Multiple Data Modalities
High-accuracy AI requires a combination of different types of data, such as text, images, and audio.
Example: Generative AI leverages multiple data modalities (text + images + audio) to enhance inference quality.
The Role of Multi-modal Sensing in Edge AI
This concept applies equally to Edge AI, where we handle an increasing variety of data sources. This requires precise control of sensor timing and data quality, demanding a seamless, comprehensive approach to multi-modal sensing.
The Direction and Complexity of Edge Devices and AI
The increasing complexity of AI and IoT systems presents several challenges, including intricate system architecture, control management, and diverse data handling. To build a robust AI IoT framework that integrates sensors and Edge AI, comprehensive software solutions and open-source support are essential for ensuring performance and security.
Various complexities are increasing:
System architecture complexity
How we connect and configure numerous devices and sensors.
Control complexity
Effectively managing each device and AI application.
Handling diverse and large-scale data
Managing vast amounts of various data types is challenging.
Optimizing AI configurations
Selecting and implementing the best AI setup for system performance.
Security complexity
As the system grows, so does the importance of strong security measures.
To effectively develop a robust AI IoT system that leverages sensors and edge AI, comprehensive software solutions are essential. Additionally, open-source support plays a critical role in facilitating this process, providing the necessary resources and flexibility for development.
End-to-End System from Edge to Cloud
For a company aiming to develop solutions that leverage Edge AI and IoT, establishing a cohesive platform that seamlessly connects devices to the cloud is essential. This Edge AI system can predominantly be constructed using open-source software, ensuring flexibility and customization throughout the stack.
By packaging the necessary software elements into an Edge System Framework, organizations can simplify the development of Edge Devices. Key components of Such an Edge Device’s stack include AI and sensor control, and application lifecycle management. By making these elements accessible as open-source, we can enhance the utilization of Edge AI and foster its widespread adoption.
Key Components Powering the Future of Edge AI IoT and sensor network
To drive the future of Edge AI in the IoT landscape, we think the following several components are keys that enhance functionality and performance.
ByteHive™ (tentative name) enables comprehensive life-cycle management for on-device AI models and applications, while SensCord™ facilitates seamless access to diverse sensor data. Additionally, technologies like WAMR (WebAssembly) runtime and WASI enhance security and virtualize edge resources, allowing for better abstraction of underlying hardware and software. Together, these innovations enable efficient AI processing even for constrained Edge devices.
Deploying Edge AI Apps using Wasm for edge virtualization
Generally, the development and deployment of IoT solutions based on resource-constrained and tiny IoT devices, can pose several challenges. Below, you will find more detailed information on these pain points:
Development Process
- Embedded development is difficult – mostly implemented in C and C++
- Difficult or impossible to change functionality after deployment widely.
- No standard component model, so customization cost is high.
- No isolation => rigid QA process.
Portability
- Not platform specific (Can be used on ARM, RISC-V, x86, Xtensa, etc).
Safety & Security
- Memory Isolation on MCU is difficult or impossible (no MMU).
- RTOS not designed for multi-user thus syscalls are not secured well.
- Vulnerabilities cannot all be found before shipping.
To overcome these pain points, we take advantage of Wasm as an orchestration-capable application runtime for resource-constrained devices .
The Rising Prominence of WebAssembly: The Future of Computation and Virtualization (for the Edge)
Up to now, virtualization has evolved across various domains: from desktop virtualization for personal computers to server-side and cloud virtualization, and even on-premise data center virtualization within the edge.
Technologies like hypervisor-based virtual machines and container orchestration with Kubernetes have been developed to support these needs.
However, achieving end-to-end virtualization across diverse IoT devices requires different approaches. This is because some devices operate on systems that don’t support containers or lack high-performance processors capable of handling the overhead associated with virtualization.
We are focusing on the applicability of WebAssembly for constrained-device-edge virtualization. This approach offers several advantages, including enhanced security, a small footprint, fast execution, portability and reusability.
Unleashing the Power of Diverse Sensors: SensCord in Action
SensCord is a framework library designed for developing sensing applications. It provides a unified API that facilitates easy access to sensor data, enhancing the development experience for users. The API specification is adaptable and can be utilized across various sensors, resolving discrepancies in API encountered in different sensor SDK development projects.
Widely used for many of Sony’s sensors, SensCord offers common features that streamline development. Its architecture effectively connects sensor software developers with application developers.
SensCord: Features
Supporting multi languages
- C/C++, Python, JavaScript, Java and C#.
- Provide common control sequence.
- Improving the portability of software.
Users can integrate custom sensing functions in SensCord as SPL(Sensing Process Layer)/PPL(Perception Process Layer).
Supporting multi platform
- Windows, Linux(ubuntu) and Android.
Common functions
- Data structure for each sensor type
- Properties
- Grabber (frame buffering)
- Recording and playback (raw, .skv, .bin)
Lower the hurdle by using AITRIOS Local Edition
When developing IoT solutions, it’s crucial to reduce the barriers to setting up a developer environment and to begin with a small-scale Proof of Concept (PoC).
Even with a small-scale PoC, developers often face the need to connect to the cloud and integrate with cloud-based device management features, which can delay their focus on developing applications for the devices themselves. Additionally, the resources required for building business logic can be scattered, preventing concentrated efforts where they are needed most.
Therefore, providing a development environment that enables local, small-scale prototyping without the need for cloud integration or network connections is highly beneficial for developers.
Against this backdrop, AITRIOS Local Edition was created. AITRIOS is an edge AI development environment designed to facilitate easy initiation while offering scalability and expansion capabilities. With the Local Edition, you can start on a small scale, and if you wish to perform large-scale deployment through cloud integration or utilize more advanced online AI features, you can migrate and expand to the cloud-integrated version.
Additionally, some of the software components in the Local Edition will be released as open source, easily accessible, and allows developers to extend their functions.
OSS Projects Plan
Here you have the OSS projects available or to be available soon:
Local Console (Available)
A tool for deploying edge applications and AI models to devices, managing their lifecycle, monitoring and visualizing telemetries. The Local Console simplifies the deployment process and provides visibility into application status across different hardware and software platforms.
ByteHive (tent.) (TBD)
The lightweight runtime is similar to ContainerD but designed for the tiny edge. It manages the execution of WebAssembly (Wasm) modules.
SensCord (TBD)
A high-performance sensor and Image Signal Processor (ISP) framework supporting a variety of sensors, including the IMX500 intelligent image sensor. SensCord is seamlessly integrated with Wasm modules via WASI, and a libcamera2 compatibility layer for broader application support is also planned.
Edge App SDK (TBD)
Used to build edge AI applications, or “post-processing” modules. This SDK abstracts the complexities of hardware integration and simplifies development.
The currently supported edge devices include AITRIOS-compliant edge devices, such as AI cameras, which are tiny, low-power consumption, but resource-constrained. Such device runs the NuttX RTOS and is equipped with an IMX500 intelligent image sensor for on-sensor AI inference. Furthermore, we will soon support the Raspberry Pi (RPi), paired with the Raspberry Pi AI Camera. This edge device runs Linux and is also equipped with an IMX500 intelligent image sensor.
OSS Local Console in the Workflow
The Local Console connects to edge devices via a local IP network and manages the devices, AI models running on them, and edge AI applications (Wasm).
- To Device:
- Establishment of an MQTT connection, and get ready for the data retrieval.
- Deployment of a Wasm module that post processes an AI inference data.
- Python is also supported.
- Deployment of an AI model that executes AI inference on the device edge.
- Start/stop streaming, and set Region of Interest of an image data.
- From Device:
- Retrieval of a telemetry data(an AI inference result) via MQTT Messages.
- Streaming of images via HTTP.
OSS Projects in the Workflow Using Local Edition (2/2)
To generate Wasm applications and AI models that run on edge devices, you can use the commercial product, Local Studio, or you can develop wasm-based edge applications by using Edge App SDK to be released as OSS, as well..
Building Together with Communities
We aim to create a flexible and user-friendly development environment for edge AI IoT and sensor networks in collaboration with the open communities. Our goal is to build a thriving ecosystem for edge AI applications using WebAssembly (Wasm).
As follows, we’re discussing, for example, the types of development we’d like to pursue with the communities, but it’s not limited to those alone. We hope to expand the ecosystem through discussions and collaboration with everyone in the communities.
Edge App SDK
For those who want an easy way to create AI applications for edge devices, our expectation includes:
- Support for modern high-level languages, even those that are not currently supported.
- User-friendly APIs and libraries to streamline development.
- A rich collection of use cases and sample Wasm modules, which can be shared among open communities.
- Reusable Wasm components to enhance productivity.
Local Console
For developers who want the management and visualization tool flexible enough for various edge devices and AI applications. Our expectation includes:
- Expansion of the range of manageable edge devices.
- Extensions of manageable types of applications and runtimes.
- Enhanced visualization methods for better insights.
ByteHive Agent / SensCord
For those who want to support a variety of sensors alongside specific image sensors,
- Porting of the Agent to support various edge devices which are not currently supported.
- Expansion for a wide range of sensors through SensCord, enabling a versatile sensor network.
Summary
In Edge AI IoT and sensor network technology, diverse configurations exist for each user. It’s essential to break through device stack fragmentation.Because of the diversity, collaboration with open-source communities is even more crucial than with cloud technologies.We believe WebAssembly for Edge would be one key to making this possible.
We are introducing open-source projects for end-to-end use in the world of Edge AI IoT and sensor network technologies. The first step is the Local Console. Please, be sure to check out LocalConsole, and stay tuned for updates on our upcoming OSS projects.
