the ERT application CONCIERGE

ERT CONCIERGE @ VIRTUAL WORLDS BEST PRACTICES IN EDUCATION

[left] clients
[middle] concierge
[right] external services

The gaming industry has catalyzed the fast-paced advancement of immersive Virtual Reality (VR) technology. However, there exists somewhat untapped educational benefits in the realm of VR.

A VR classroom can create a shared virtual space that mimics in-person teacher guidance and student collaboration without the obstacle of physical distance. Additionally, the VR classroom creates an opportunity for a new brand of "lab" exercises only feasible in a virtual, immersive 3D space.

_ about

[left]
ERT WEB +
ERT CONCIERGE +
RUST GAME

[right]
ERT WEB +
ERT CONCIERGE +
PLANETARY SIMULATION

#ertweb

Using low-level networking interfaces, I have also created a server-client protocol in order to incorporate external application functionality into the VR space and create a lab exercise for visualizing multivariable mathematical functions. Through the generic plugin architecture of the server's design, this project allows intuitive integration of new labs with extensions in Python, MATLAB or other third-party services, facilitating future collaboration with educators.

Quad Mesh (surface) generated with vertices and normals from external Matlab service, and sent back to client alcove.

The ERT (Enhanced Reality Teaching) classroom was originally built on the Unity game engine, and written in C#, using the packages SteamVR and Photon Unity Networking (PUN). In 2019, it was updated to work with the latest version of Unity, SteamVR, and PUN in order to ensure future functionality. In 2020 the ERT Concierge that allowed it to communicate with outside services was updated, and a new range of clients were added to the ecosystem.

In fact, ERT has been extended to allow users to easily integrate their own 'labs' (as applications or simulations) directly into the classroom space. Two example labs have been tested: a planetary simulation and a MATLAB calculator. These labs use socket communication in order to sync information with ERT through a CONCIERGE which handles both continuous updates (in the planetary simulation) and large packets of information (MATLAB calculator).

"The ability to have a flexible collaborative open-source Virtual Reality application that can also easily communicate with any outside application and exchange any input or output (text, geometry, or raw data) is a game-changer!"

From our research we were able to reveal and then focus on the following common educational issues:

Enabling ways for teachers and researchers to easily share common and synchronized results from simulations or code with students.
Enabling simulations or complex computational codes to easily stream their results into our VR collaborative space in realtime.

Developing for Virtual Reality, as well as most Mixed Reality platforms, entails creating game like applications, that are highly dependent on specific harware architectures. Platforms like Steam, have generalized the interface between an application developement space (such as Unity) and a general approach to managing a VR/MR/AR headset and the inputs.

However, applications are still highly dependent on creating a monolothic tool/executable. We can adjust to this reality, by allowing ecosystems that could benefit from a VR display, by having them act as a service. With the creating of a middle service manger --a service concierge-- we can easily incorporate applications into an evironment where input and output is relayed via standard protocols (JSON based).

# actionable insights

Interacting with an external application can be done through specific client designed user interface elements, or by interacting with objects in the scene that are equipped with sensors/colliders. ERT Concierge is designed to send small continous updates (such as in the case of the Python based Planetary Simulation) as we as large comprensive data (as in the case of results from the Matlab Calculator).

input from the front end client is wrapped in generic JSON, and concierge transmits it to the registered service. understanding how to apply the information is not at the service side through call to provided/supported APIs.

[currently supported] python and matlab

#input relay

Communications begin once the server is setup on a specific host and port, listening for new socket connections. The server then automates the execution of the MATLAB script. In ERT, a master Unity client, such as the individual designated the "teacher" in the classroom, would also be tasked with running and server as well as MATLAB.

The master client would not need to manually run the MATLAB script, instead they would just need to hit play for ERT in Unity. Upon the execution of the MATLAB script, the MATLAB client would use TCP/IP functions to connect to the Unity server, thereafter sending the handshake-identification message of "MATLAB" to the server.

The JSON data format also facilitates client collaboration functionality with two JSON fields: Main_ID and Group_IDs. With these fields, the server can manage which clients should receive relays from MATLAB and create PUN interest groups that allow the instantiation of the final graph object only for clients that are part of the group.

#matlab|interface

[left] student, in alcove, changes value of mass for center sun

[right] synchronized simulation results for all participants

#python|interface

_GENERAL CLIENT PROTOCOL

In 2020 we updated ERT Concierge to efficiently tie in clients (of different types) with a range of services. Both clients and services connect to ERT CONCIERGE with a common socket and simmilar protocols, but depending on the registration and context will have different protocols/payloads available. SEE PAYLOAD PROTOCOL

Furthermore, in order to support larger packet/data transfer, we created a file system which stores data with disticint indentifiers and is accessible to registered clients via standard http protocol. SEE FILESYSTEM USAGE

#protocol

_ project lead : santiago v lombeyda
_ surf 2020 | efficient network services & new web base client : an tran
_ surf 2019 | network services & python realtime app synchronization : alison noyes
_ surf 2019 | network services & matlab large data app communication : sasha fomina
_ center director : george djorgovski

_ team

_ george djorgovski
_ jim barry
_ mathieu desbrun

_ collaborators

PROJECT BORN from CALTECH's OVRAS LAB
_ with support from the CENTER for DATA DRIVEN DISCOVERY
_ with support from Caltech's SUMMER FELLOWSHIP PROGRAM
_ with donations from MICROSOFT
_ with donations from LOGITECH
_ with donations from NVIDIA
_ with donations from HTC

_ acknowledgements

ERT+CONCIERGE

OVRAS | an application service relay for ERT

_GENERAL CLIENT PROTOCOL

ERT+CONCIERGE OVRAS \| an application service relay for ERT
about protocol gallery previous downloads team ERT
ERT CONCIERGE @ VIRTUAL WORLDS BEST PRACTICES IN EDUCATION
[left] clients [middle] concierge [right] external services The gaming industry has catalyzed the fast-paced advancement of immersive Virtual Reality (VR) technology. However, there exists somewhat untapped educational benefits in the realm of VR. A VR classroom can create a shared virtual space that mimics in-person teacher guidance and student collaboration without the obstacle of physical distance. Additionally, the VR classroom creates an opportunity for a new brand of "lab" exercises only feasible in a virtual, immersive 3D space.	_ about
[left] ERT WEB + ERT CONCIERGE + RUST GAME [right] ERT WEB + ERT CONCIERGE + PLANETARY SIMULATION		#ertweb
Using low-level networking interfaces, I have also created a server-client protocol in order to incorporate external application functionality into the VR space and create a lab exercise for visualizing multivariable mathematical functions. Through the generic plugin architecture of the server's design, this project allows intuitive integration of new labs with extensions in Python, MATLAB or other third-party services, facilitating future collaboration with educators.
Quad Mesh (surface) generated with vertices and normals from external Matlab service, and sent back to client alcove. The ERT (Enhanced Reality Teaching) classroom was originally built on the Unity game engine, and written in C#, using the packages SteamVR and Photon Unity Networking (PUN). In 2019, it was updated to work with the latest version of Unity, SteamVR, and PUN in order to ensure future functionality. In 2020 the ERT Concierge that allowed it to communicate with outside services was updated, and a new range of clients were added to the ecosystem. In fact, ERT has been extended to allow users to easily integrate their own 'labs' (as applications or simulations) directly into the classroom space. Two example labs have been tested: a planetary simulation and a MATLAB calculator. These labs use socket communication in order to sync information with ERT through a CONCIERGE which handles both continuous updates (in the planetary simulation) and large packets of information (MATLAB calculator).
"The ability to have a flexible collaborative open-source Virtual Reality application that can also easily communicate with any outside application and exchange any input or output (text, geometry, or raw data) is a game-changer!"
From our research we were able to reveal and then focus on the following common educational issues: Enabling ways for teachers and researchers to easily share common and synchronized results from simulations or code with students. Enabling simulations or complex computational codes to easily stream their results into our VR collaborative space in realtime. Developing for Virtual Reality, as well as most Mixed Reality platforms, entails creating game like applications, that are highly dependent on specific harware architectures. Platforms like Steam, have generalized the interface between an application developement space (such as Unity) and a general approach to managing a VR/MR/AR headset and the inputs. However, applications are still highly dependent on creating a monolothic tool/executable. We can adjust to this reality, by allowing ecosystems that could benefit from a VR display, by having them act as a service. With the creating of a middle service manger --a service concierge-- we can easily incorporate applications into an evironment where input and output is relayed via standard protocols (JSON based).	# actionable insights

Interacting with an external application can be done through specific client designed user interface elements, or by interacting with objects in the scene that are equipped with sensors/colliders. ERT Concierge is designed to send small continous updates (such as in the case of the Python based Planetary Simulation) as we as large comprensive data (as in the case of results from the Matlab Calculator).
input from the front end client is wrapped in generic JSON, and concierge transmits it to the registered service. understanding how to apply the information is not at the service side through call to provided/supported APIs. [currently supported] python and matlab	#input relay
Communications begin once the server is setup on a specific host and port, listening for new socket connections. The server then automates the execution of the MATLAB script. In ERT, a master Unity client, such as the individual designated the "teacher" in the classroom, would also be tasked with running and server as well as MATLAB. The master client would not need to manually run the MATLAB script, instead they would just need to hit play for ERT in Unity. Upon the execution of the MATLAB script, the MATLAB client would use TCP/IP functions to connect to the Unity server, thereafter sending the handshake-identification message of "MATLAB" to the server. The JSON data format also facilitates client collaboration functionality with two JSON fields: Main_ID and Group_IDs. With these fields, the server can manage which clients should receive relays from MATLAB and create PUN interest groups that allow the instantiation of the final graph object only for clients that are part of the group.	#matlab\|interface
[left] student, in alcove, changes value of mass for center sun [right] synchronized simulation results for all participants		#python\|interface
_GENERAL CLIENT PROTOCOL In 2020 we updated ERT Concierge to efficiently tie in clients (of different types) with a range of services. Both clients and services connect to ERT CONCIERGE with a common socket and simmilar protocols, but depending on the registration and context will have different protocols/payloads available. SEE PAYLOAD PROTOCOL Furthermore, in order to support larger packet/data transfer, we created a file system which stores data with disticint indentifiers and is accessible to registered clients via standard http protocol. SEE FILESYSTEM USAGE	#protocol
_ project lead : santiago v lombeyda _ surf 2020 \| efficient network services & new web base client : an tran _ surf 2019 \| network services & python realtime app synchronization : alison noyes _ surf 2019 \| network services & matlab large data app communication : sasha fomina _ center director : george djorgovski	_ team
_ george djorgovski _ jim barry _ mathieu desbrun	_ collaborators
PROJECT BORN from CALTECH's OVRAS LAB _ with support from the CENTER for DATA DRIVEN DISCOVERY _ with support from Caltech's SUMMER FELLOWSHIP PROGRAM _ with donations from MICROSOFT _ with donations from LOGITECH _ with donations from NVIDIA _ with donations from HTC	_ acknowledgements
ovras \| OVS \| art by jimBARRY \| CD3 \| caltech