[Writing paper]SENSEBOX

A Multimodal Computation Framework for Behavioral Recognition and Interaction for Mixed Reality Workspace

Keywords:

Human-Computer Interaction

Mixed Reality

Multimodal Behavioral Recognition

Affective Computing

Workspace

User Experience

interaction part: Ruyi Yang, Menghang Liu, Luqi Feng

construction part: Jieshi Chen, Jingtong Li, Tianyu Zhang

instructor: Professor Chao YAN, Tongji University

source code for interaction part: https://github.com/Emmaruyi/Humanizing-MR

SENSEBOX: A humanized phygital Workspace

This project aims to create a workspace that emphasizes human interaction, exploring the relationship between people and space in work scenarios. The workspace is constructed with a minimalist and aesthetically pleasing three-ring spiral design using steel structures, combining bundling and intersections. By utilizing deep data collection and behavioral analysis, the workspace dynamically adapts in a mixed reality environment according to participants’ behaviors (interaction patterns, task processing modes, attention states, emotional states). The space adjusts in real-time, altering aspects like spatial enclosure, openness, regularity, opposite-side height, and color to create a comfortable workspace for interaction and learning.

最终最终成果（加长版）.mp4

Construction

Framework Design: The workspace box (3m x 3m x 2.5m) features a spiral tri-ring design with a core steel pipe framework. Secondary lines are added and secured with bundling straps, then rotated 180 degrees to create a symmetrical structure, forming the final plan and renderings.
Bundling and Node Development: Various bundling methods were explored, resulting in specific bundled nodes for the final design. Each pipe was bent, numbered, and bundled in sequence to form six main components.
Construction and Assembly: The assembly process involved bundling parts together, installing depth cameras, and attaching acrylic seat and table surfaces, completing the Sense Box construction.

Interaction

Utilized mixed reality technology to assist in bending steel pipes for constructing a 3x3x3m installation space prototype.
Acquired human data using ZED camera and headset data, with body landmark detection and emotion recognition conducted.
Analyzed human data using qualitative and quantitative methods, and formulated rules based on distances, orientations, postures, head states, and expressions to derive interaction, task, attention, and emotional patterns.
Implemented real-time spatial changes based on human data, affecting spatial enclosure, height, openness, regularity, and color, using Grasshopper, Unity, and Python. Integrated with physical spaces to create dynamic virtual environments that adapt to human work states.

▪利用混合现实技术辅助弯曲钢管，构建 3*3*3m 的安装空间原型；

▪部署ZED 摄像头和VR头显设备获取人体数据，实现精准的人体关键点检测和情绪识别；

▪综合运用定性和定量分析方法，构建空间感知规则库（距离、方向、姿势、头部状态和表情），建立人机交互模型（互动、任务、注意力和情绪模式）；

▪研发多模态空间响应系统，整合Python数据处理、Grasshopper参数化建模、Unity实时渲染技术栈，根据人类数据实时实现空间围合、高度、开放度、规则性和颜色的动态响应，并与物理空间集成，创建适应人类工作状态的动态虚拟环境

Page updated

Google Sites

Report abuse