the ERT application CONCIERGE

[left] client
[middle] concierge
[right] external service

communication model between VR Unity client, python concierge managing communication, and external serive, using JSON based defined ERT Protocol.

The gaming industry has catalyzed the fast-paced advancement of immersive Virtual Reality (VR) technology. However, there exists somewhat untapped educational benefits in the realm of VR.

A VR classroom can create a shared virtual space that mimics in-person teacher guidance and student collaboration without the obstacle of physical distance. Additionally, the VR classroom creates an opportunity for a new brand of "lab" exercises only feasible in a virtual, immersive 3D space.

Using low-level networking interfaces, I have also created a server-client protocol in order to incorporate external application functionality into the VR space and create a lab exercise for visualizing multivariable mathematical functions. Through the generic plugin architecture of the server's design, this project allows intuitive integration of new labs with extensions in Python, MATLAB or other third-party services, facilitating future collaboration with educators.

_ about

Quad Mesh (surface) generated with vertices and normals from external Matlab service, and sent back to client alcove.

The ERT (Enhanced Reality Teaching) classroom is built on the Unity game engine, and written in C#, using the packages SteamVR and Photon Unity Networking (PUN). In 2019, it was updated to work with the latest version of Unity, SteamVR, and PUN in order to ensure future functionality.

ERT has been extended to allow users to easily integrate their own 'labs' (as applications or simulations) directly into the classroom space. Two example labs have been tested: a planetary simulation and a MATLAB calculator. These labs use socket communication in order to sync information with ERT through a CONCIERGE which handles both continuous updates (in the planetary simulation) and large packets of information (MATLAB calculator).

"The ability to have a flexible collaborative open-source Virtual Reality application that can also easily communicate with any outside application and exchange any input or output (text, geometry, or raw data) is a game-changer!"

From our research we were able to reveal and then focus on the following common educational issues:

Enabling ways for teachers and researchers to easily share common and synchronized results from simulations or code with students.
Enabling simulations or complex computational codes to easily stream their results into our VR collaborative space in realtime.

Developing for Virtual Reality, as well as most Mixed Reality platforms, entails creating game like applications, that are highly dependent on specific harware architectures. Platforms like Steam, have generalized the interface between an application developement space (such as Unity) and a general approach to managing a VR/MR/AR headset and the inputs.

However, applications are still highly dependent on creating a monolothic tool/executable. We can adjust to this reality, by allowing ecosystems that could benefit from a VR display, by having them act as a service. With the creating of a middle service manger --a service concierge-- we can easily incorporate applications into an evironment where input and output is relayed via standard protocols (JSON based).

# actionable insights

Interacting with an external application can be done through specific client designed user interface elements, or by interacting with objects in the scene that are equipped with sensors/colliders. ERT Concierge is designed to send small continous updates (such as in the case of the Python based Planetary Simulation) as we as large comprensive data (as in the case of results from the Matlab Calculator).

input from the front end client is wrapped in generic JSON, and concierge transmits it to the registered service. understanding how to apply the information is not at the service side through call to provided/supported APIs.

[currently supported] python and matlab

#input relay

Communications begin once the server is setup on a specific host and port, listening for new socket connections. The server then automates the execution of the MATLAB script. In ERT, a master Unity client, such as the individual designated the "teacher" in the classroom, would also be tasked with running and server as well as MATLAB.

The master client would not need to manually run the MATLAB script, instead they would just need to hit play for ERT in Unity. Upon the execution of the MATLAB script, the MATLAB client would use TCP/IP functions to connect to the Unity server, thereafter sending the handshake-identification message of "MATLAB" to the server.

The JSON data format also facilitates client collaboration functionality with two JSON fields: Main_ID and Group_IDs. With these fields, the server can manage which clients should receive relays from MATLAB and create PUN interest groups that allow the instantiation of the final graph object only for clients that are part of the group.

#matlab|interface

[left] student, in alcove, changes value of mass for center sun

[right] synchronized simulation results for all participants

#python|interface

Set of handshakes and communication examples between a python based service
_planetary simulation
_the ERT Concierge and
_the ERT classroom.

SERVICE SIDE | PYTHON PROTOCOL

_ HANDSHAKE PROTOCOL

{
    "PROTOCOL": "ERT.CONCIERGE.0.1.2019",
    "TARGET":   "ERT-CLIENT",
    "CONTEXT":  "CONTINOUS",
    "ID":       "PLANETARY.SIMULATION.01.by.ALISON.NOYES",
    "ENVIRONMENT": {
	"GRAVITATIONAL-CONSTANT":"6.67408E-11",
...
    },
    "DATA": {
	"ENTITY": {
		"ID": "0001",
		"CATEGORY": "PLANET",
		"MASS": "120E12"
		"POSITION": {
			"X":"10E5",
			"Y":"0",
			"Z":"0",
		}
...
	}
    }
}

_ UPDATE PROTOCOL

SERVICE SIDE | MATLAB PROTOCOL

MIDDLE CONCIERGE | SERVICE REGISTRATION PROTOCOL

CLIENT SIDE | UNITY PROTOCOL

#protocol

_ project lead : santiago v lombeyda
_ surf 2019 | network services & python realtime app synchronization : alison noyes
_ surf 2019 | network services & matlab large data app communication : sasha fomina
_ center director : george djorgovski

_ team

_ george djorgovski
_ jim barry
_ mathieu desbrun

_ collaborators

PROJECT BORN from CALTECH's OVRAS LAB
_ with support from the CENTER for DATA DRIVEN DISCOVERY
_ with support from Caltech's SUMMER FELLOWSHIP PROGRAM
_ with donations from MICROSOFT
_ with donations from LOGITECH
_ with donations from NVIDIA
_ with donations from HTC

_ acknowledgements

ERT+CONCIERGE

OVRAS | an application service relay for ERT

_ HANDSHAKE PROTOCOL

_ UPDATE PROTOCOL

ERT+CONCIERGE OVRAS \| an application service relay for ERT
about protocol gallery downloads team ERT
[left] client [middle] concierge [right] external service communication model between VR Unity client, python concierge managing communication, and external serive, using JSON based defined ERT Protocol. The gaming industry has catalyzed the fast-paced advancement of immersive Virtual Reality (VR) technology. However, there exists somewhat untapped educational benefits in the realm of VR. A VR classroom can create a shared virtual space that mimics in-person teacher guidance and student collaboration without the obstacle of physical distance. Additionally, the VR classroom creates an opportunity for a new brand of "lab" exercises only feasible in a virtual, immersive 3D space. Using low-level networking interfaces, I have also created a server-client protocol in order to incorporate external application functionality into the VR space and create a lab exercise for visualizing multivariable mathematical functions. Through the generic plugin architecture of the server's design, this project allows intuitive integration of new labs with extensions in Python, MATLAB or other third-party services, facilitating future collaboration with educators.	_ about
Quad Mesh (surface) generated with vertices and normals from external Matlab service, and sent back to client alcove. The ERT (Enhanced Reality Teaching) classroom is built on the Unity game engine, and written in C#, using the packages SteamVR and Photon Unity Networking (PUN). In 2019, it was updated to work with the latest version of Unity, SteamVR, and PUN in order to ensure future functionality. ERT has been extended to allow users to easily integrate their own 'labs' (as applications or simulations) directly into the classroom space. Two example labs have been tested: a planetary simulation and a MATLAB calculator. These labs use socket communication in order to sync information with ERT through a CONCIERGE which handles both continuous updates (in the planetary simulation) and large packets of information (MATLAB calculator).
"The ability to have a flexible collaborative open-source Virtual Reality application that can also easily communicate with any outside application and exchange any input or output (text, geometry, or raw data) is a game-changer!"
From our research we were able to reveal and then focus on the following common educational issues: Enabling ways for teachers and researchers to easily share common and synchronized results from simulations or code with students. Enabling simulations or complex computational codes to easily stream their results into our VR collaborative space in realtime. Developing for Virtual Reality, as well as most Mixed Reality platforms, entails creating game like applications, that are highly dependent on specific harware architectures. Platforms like Steam, have generalized the interface between an application developement space (such as Unity) and a general approach to managing a VR/MR/AR headset and the inputs. However, applications are still highly dependent on creating a monolothic tool/executable. We can adjust to this reality, by allowing ecosystems that could benefit from a VR display, by having them act as a service. With the creating of a middle service manger --a service concierge-- we can easily incorporate applications into an evironment where input and output is relayed via standard protocols (JSON based).	# actionable insights

Interacting with an external application can be done through specific client designed user interface elements, or by interacting with objects in the scene that are equipped with sensors/colliders. ERT Concierge is designed to send small continous updates (such as in the case of the Python based Planetary Simulation) as we as large comprensive data (as in the case of results from the Matlab Calculator).
input from the front end client is wrapped in generic JSON, and concierge transmits it to the registered service. understanding how to apply the information is not at the service side through call to provided/supported APIs. [currently supported] python and matlab	#input relay
Communications begin once the server is setup on a specific host and port, listening for new socket connections. The server then automates the execution of the MATLAB script. In ERT, a master Unity client, such as the individual designated the "teacher" in the classroom, would also be tasked with running and server as well as MATLAB. The master client would not need to manually run the MATLAB script, instead they would just need to hit play for ERT in Unity. Upon the execution of the MATLAB script, the MATLAB client would use TCP/IP functions to connect to the Unity server, thereafter sending the handshake-identification message of "MATLAB" to the server. The JSON data format also facilitates client collaboration functionality with two JSON fields: Main_ID and Group_IDs. With these fields, the server can manage which clients should receive relays from MATLAB and create PUN interest groups that allow the instantiation of the final graph object only for clients that are part of the group.	#matlab\|interface
[left] student, in alcove, changes value of mass for center sun [right] synchronized simulation results for all participants		#python\|interface
Set of handshakes and communication examples between a python based service _planetary simulation _the ERT Concierge and _the ERT classroom.	SERVICE SIDE \| PYTHON PROTOCOL _ HANDSHAKE PROTOCOL { "PROTOCOL": "ERT.CONCIERGE.0.1.2019", "TARGET": "ERT-CLIENT", "CONTEXT": "CONTINOUS", "ID": "PLANETARY.SIMULATION.01.by.ALISON.NOYES", "ENVIRONMENT": { "GRAVITATIONAL-CONSTANT":"6.67408E-11", ... }, "DATA": { "ENTITY": { "ID": "0001", "CATEGORY": "PLANET", "MASS": "120E12" "POSITION": { "X":"10E5", "Y":"0", "Z":"0", } ... } } } _ UPDATE PROTOCOL SERVICE SIDE \| MATLAB PROTOCOL MIDDLE CONCIERGE \| SERVICE REGISTRATION PROTOCOL CLIENT SIDE \| UNITY PROTOCOL	#protocol
_ project lead : santiago v lombeyda _ surf 2019 \| network services & python realtime app synchronization : alison noyes _ surf 2019 \| network services & matlab large data app communication : sasha fomina _ center director : george djorgovski	_ team
_ george djorgovski _ jim barry _ mathieu desbrun	_ collaborators
PROJECT BORN from CALTECH's OVRAS LAB _ with support from the CENTER for DATA DRIVEN DISCOVERY _ with support from Caltech's SUMMER FELLOWSHIP PROGRAM _ with donations from MICROSOFT _ with donations from LOGITECH _ with donations from NVIDIA _ with donations from HTC	_ acknowledgements
ovras \| OVS \| art by jimBARRY \| CD3 \| caltech