Applying AR into Education Domain from Youku
Project on Github
The main target of this project is to explore the AR technology and apply it in early education area. As we all know, interest is the best teacher for children’s learning. However, traditional education methods are inevitable to be boring and tedious for children, making knowledge learning a burden. So find a way to make the learning procedure more interesting and vivid is really of great sense.
So, what is always interesting, especially for school children? That must be game playing. If we can combine the game elements and knowledge together, making study like playing game, that must be a really cool thing.
Here this project will design a game style application to help children learn English during their early study period.
Existing AR products
Thousand of applications for early education are available for smart phone users and AR technology is applied in these mobile device software.
It is an education software using the basic technology of AR, identify the different single cards and display the corresponding 3d animal models, let the child can see vivid animals and learn the English name and English pronunciation. Single-target and multi-target identification technology of AR are applied.
2.AR magic cards
A virtual English school based on AR technology, combining AR technology with children‘s English education. AR is applied to mobile multimedia interactive learning software, turning the graphic image into a sound, action, interactive 3d animation, multilingual standard pronunciation, with abundant knowledge and education. It aims to stimulate a child’s curiosity, take the initiative to use eyes to see, use hand to touch, use the ear to listen, thinking in the brain to explore, to help the children to know the world in the joy of the game , and cultivate their active learning interest.
3.AR early education garden
This is a fresh, early education software that uses AR technology to show dynamic 3d cartoon characters interacting with children so that children can learn during play. Learning modules include: puzzle game, English letter learning, dinosaur world, world classic and transportation, etc. This is a nice application with many good ideas.
4.AR story robot
Using AR technology to display interactive robot, can tell children stories, children’s ballads, three character primer, life common sense, traffic rules and so on. It is the embryo of the future artificial intelligence early teaching robot.
The target of this project is to develop a game software with Augmented Reality elements to help children learning English with interest and some demanding and ideas are proposed.
1.Word spelling and pronunciation learning, plants and animals identification
Students scan different cards to display different cartoon animals or plants in the scene, and display the corresponding English name, play the pronunciation of the words, and can repeat the pronunciation. This simple way of learning can improve children‘s English primary vocabulary and their knowledge of the real world.
2.Basic spoken English learning
Children can use the device to scan the card to appear the virtual person, and begins a daily oral conversation, helps the student to practice English listening and speaking. The students can click the button to display the English text of the dialogue or the Chinese translation, to help them learn the dialogue more accurately.
Elementary school students are learning English and lack of vocabulary. They often encounter strange words in textbooks or extra-curricular materials. In order to improve students‘ motivation and interest in learning new words, the function of automatic identification of English words in physical paper materials is realized by AR technology. The students use the camera to scan the text, the system automatically recognizes the corresponding words, and identifies them on the screen, displays the corresponding Chinese meaning, and a example sentence.
4.AR English story
In order to stimulate a child‘s imagination, here use AR technology to display virtual scene of fairy tales, and interesting English stories, let the children to be immersed in the scene, enhance children’s English stories impression and experience.
5.English test challenge game
Through the game, the children‘s learning results are tested to improve the children’s learning quality.
However the design and implementation of this part is quite impossible for a personal developer, instead, I developed some mini-games for study encouragement. Children have to complete some learning tasks before being able to play these mini-games.
6.Interesting interface themes
To improve students’ imagination and software’s appeal to students, an interesting cartoon theme can be used on the interface.
Here the Halloween theme and scenario is adopted. That lets the student experience a Halloween adventure, immersion, and it is also a learning process to the knowledge of a foreign language and a impressive childhood memory of the students.
AR, augmented reality, it is a new technology origin from virtual reality technology to combine virtual objects and our real world together, which means it can help people see some digital virtual objects in our real life. Augmented reality is also called mixed reality(MR).
- Integration of the real world and virtual world;
- Real-time interactive;
- Locate virtual objects’ 3d location and render them.
Difference between AR and VR
VR technology aims to help people be immersed in virtual world. It is a way to bring people form real world to virtual world. The user usually have to wear a head device to see the virtual world.
By contrast, AR is a technology that aims to bring virtual objects to our real life to augment the reality.
AR tracking technology development
From the view of computer vision, three-dimensional registration is the core technology of augmented reality. The purpose of registration in AR is to make a geometric understanding of the image data. This determines the location of the data to be superimposed.
- QR CODE
- 2D image
- Three-dimensional objects
- 3D environment
Game development technological support
Unity3d and Xcode are the two editors to use for game development and publish in this project. The game will be developed as a 3d game, we can easily construct a 3d game scene in Unity3d and create some effects efficiently with shader scripts or some nice third-party plugins. Besides, it’s more efficient to develop a game that contains complex logics with script languages in a game engine because we do not need to recompile the whole project after making some minor modification of the code.
On the other hand, here the target platform is iOS, specifically iPhone and iPad. So we have to use Xcode editor to complete the final version of the project.
Introduce Plugins(challenging work)
Communication between Unity3d(C#) and native platforms(iOS,Android)
In the progress of game development with unity, some third-party SDKs or self-built SDKs will often be introduced as plugins into unity projects. Those SDKs are encapsulated as black-box toolkits with native languages like C or C++ and can be reused by many other developers in different projects.
This technology has made a big difference in my product: the self-developed login SDK, Xunfei voice synthesis SDK etc.
Game reverse engineering(challenging work)
There is an advanced technology which is also helpful in this project. In commercial area, reverse engineering can be an illegal action if the programmer use other game apps’ resource for commercial use.
The main purpose to master game reverse engineering technology is to prevent it, preventing others use reverse technology doing some illegal actions on our game products, and also it is used for game performance test.
Here in this project, my demo is not for commercial use so I get some UI resource and learn some theory with the help of this technology.
For example: With game reverse engineering technology, I can grad some draw calls data and reconstruct some game models.
There are also many other technologies and theory used in this project like game UI, design patterns, Unity scene management and so on. Here will not introduce all of them.
The whole project is composed of the server, the unity game project, the self-developed login SDK client and some third-party services.
The final product of this project is an iOS App, which is compiled by a Xcode project and the Xcode project is generated from Unity project.
The server is developed with PHP language and running on apache server. Mysql database is also used here.
Xunfei SDK is used for voice synthesis and Vuforia is used for AR technology implementation in Unity game.
According to the previous objectives, several functions were designed and implemented in our game product.
They are AR Zoo, AR School, AR Text and so on, we will show them later in the video demo.