Project Report
Project Report
BACHELOR OF TECHNOLOGY
IN
COMPUTER SCIENCE AND ENGINEERING
at
Before reading the file to your PC, copy the save.txt file below to IoT Paper and
stop writing mode.
When reading an InkML file manually, copy this save.txt to the root folder \ after
you finish writing. Write mode is stopped and the orange LED turns on. You can
read that file.
In that case, click the "Redetected IoT Paper" button and select "IoT Paper 10.3"
to connect to your IoT Paper device.
To create an InkML file, launch this tool and select the monochrome BMP file
from the “Browse…” button, or drag the BMP file to the gray area, then the
image will be displayed in thegray area at that time. It will not be displayed if the
conditions (size and color depth) are not met.
Even if you specify the BMP file to be converted as a parameter from the
command prompt and start it, an InkML file will be created in the same way.
(Command prompt description example)
>InkmlConverter.exe Brank10inch.bmp
When this program is listed in Explorer, you can also convert it to an InkML file by
draggingthe BMP file onto this list.
Page. 6 Kamlesh Kumar
The pursuit of efficient and accurate assessment tools in education has led to the
exploration of innovative technologies. This project addresses the digitalization of
educational workbooks, employing electronic paper to enable the automatic scoring
of handwritten answers. The significance of this endeavor lies in its potential to
revolutionize assessment processes, offering a seamless transition from traditional
manual scoring to an automated and more scalable solution. Electronic paper with an
electronic pen can be utilized to display questions and to capture handwritten
answers in the form of online trajectory patterns, i.e., digital ink. Digital ink records
the process through which each answerer answers each question, which can be
analyzed to determine how the answerer has arrived at the answer. This approach is
useful if the answer is wrong. Digital ink can be converted into an image so that both
digital ink recognition (online recognition) and image recognition (offline recognition)
can be used to produce reliable recognition.
Databases of handwritten character patterns in many languages have contributed to
the progress of handwritten character recognition. Thus, a database of handwritten
answers in the form of digital ink for actual workbook questions by many answerers
should be prepared to develop automatic scoring, which includes correct and various
incorrect answers.
The remainder of this workshop paper is organized as follows. Section 2 describes the
related work. Section 3 introduces the device and data format. Section 4 presents
prototypes of automatic recognizers and scorers. Section 5 describes prototypes of
user interfaces for answerers and human scorers.
This paper presents three main contributions: a process for collecting handwritten
answers using electronic paper, a prototype system of handwritten answer
recognizers and scorers, and a prototype of user interfaces for answerers and scorers.
Digitalization Process
The cornerstone of this project is the meticulous digitalization of a set of educational
workbooks. Leveraging InkML, a markup language designed for recording digital ink
data, provides a robust foundation for preserving the authenticity of handwritten
responses. InkML captures the nuances of the writing process, such as pen pressure
and stroke dynamics, ensuring that the digital representation closely mirrors the
original analog input. This step is fundamental to the subsequent phases of the
project, laying the groundwork for advanced processing and analysis.
Collected data
Page. 7 Kamlesh Kumar
To digitalize the workbooks as described in Section 2, we prepared an electronic
paper device. This device displays each page of the workbooks and captures
handwritten answers in digital ink. The details are provided in Subsections 3.1–3.4.
Device for collecting handwritten answers
The electronic paper employed in this study is equipped with an electronic pen, an
eraser, and a compass, as shown in Figure 1, with a high sampling rate of 480 Hz. It
records not only the pen/eraser/compass-tip coordinates, but also the writing
pressure, tilt, and how far the tip is from the surface (up to 10 mm). Its dimensions
are 209 mm×157 mm, which is close to those of an A5 sheet, and its thickness is 3
mm
The electronic paper connects to a host device (PC, smartphone, etc.) to display a
page in one of the workbooks with a paper-like appearance and enables the user to
answer with the pen or the compass and erase the trajectory with the eraser, which is
recorded in digital ink. The host device receives the recorded digital ink.
For each page of the workbooks and each answerer, the image, coordinates of the
answer areas, correct answer examples, information about the answerer, and manual
scoring results are recorded in XML. For the convenience of machine learning, we
shows the overall structure of the XML and InkML descriptions in a handwritten
answers log file, and provides an example of digital ink recorded in InkML and a
correct answer description in XML.
XML Structure
Figure: Data structure for handwritten answers.
Page. 9 Kamlesh Kumar
Prototypes Development
The development of prototypes marks a critical phase in the project's evolution.
Handwriting recognizers and automatic scorers have been meticulously crafted for
Japanese, mathematics, and English answers – key subjects that are integral to
educational curricula. These prototypes showcase the project's commitment to
inclusivity, aiming to cover a broad spectrum of subjects to cater to diverse
educational needs.
The handwriting recognizers are designed to interpret and understand the intricacies
of handwritten responses, translating them into a format that the system can analyze
and score. Machine learning algorithms, trained on diverse datasets that encompass
various writing styles, have been employed to enhance the recognizers' accuracy and
adaptability.
Simultaneously, the automatic scorers function as the brain of the system, employing
sophisticated algorithms to evaluate and assign scores to handwritten answers. The
prototypes have undergone rigorous testing to ensure their accuracy and reliability,
and continuous refinement is an ongoing process.
User interfaces have been developed for both answerers and human scorers, focusing
on creating intuitive and user-friendly experiences. Answerers are provided with a
platform that allows them to input handwritten responses seamlessly, while human
scorers have access to tools that facilitate their role in the quality assurance process
Automatic scoring was applied after recognition without any correction of
misrecognition or manual labeling. For short answers with one or a few characters,
we used perfect matching to provide the scores. For long answers, such as sentences,
we employed a scorer that was based on DNNs and the Bidirectional Encoder
Representations from Transformers (BERT) model.
We performed a preliminary evaluation of automatic Math scoring on the collected
dataset of 1st-grade elementary school students. The evaluated dataset consisted of
23,848 samples. We employed precision, recall, and f-measure as the metrics for
evaluation, which are defined as follows
Page. 10 Kamlesh
Kumar
Discussion
The goal of using image processing techniques exploited by IoTSs is to have such
feasibility of contributing to generating energy-efficient and secure IoTSs. Hence,
there are two major considerations which are: (1) the purpose of using image
processing techniques exploited by IoTSs, (2) the possibility that image processing
techniques could contribute to the production of IoT applications and systems that
are more energy-efficient and secure. It is important to note that IoTSs have been
applying image-processing techniques to use to accomplish a wide range of goals.
Image processing techniques, particularly those that have the potential to increase
the amount of energy that is used by IoTSs, need to be investigated. Using image
processing to help reduce power consumption and ensure data security will be
covered in this section.
6.1. Image Processing Techniques’ Roles to Address Energy Consumption for IoTSs
In relation to this, there are several examples that demonstrate successful
implementations of associated image-processing algorithms with several IoTSs. These
examples show that the energy consumption rates of these IoTSs are superior to
those of other applications, resulting in greater energy savings. There are several
instances examined in this article that proved that image processing techniques had
helped design IoTSs that utilize less energy and more security. When it comes to
information processing, volume and energy use are closely related. Therefore, the
cost of storing in the cloud will impact energy use. The following are some of the
roles that image processing techniques have had in enhancing and decreasing the
rate of energy consumption produced by IoTSs:
1. Images will be uploaded to the cloud if they include information that is useful.
That minimizes energy use by reducing transmission time spent [45];
2. Because of advancements in image processing, transmission times have
become significantly shorter. If all of the obtained multi-frame images are
transmitted, it will be significantly less time than that;
3. With the aid of image processing, a mathematical model for the intensity of
compressed pixels such as CA has been developed, enabling the development
of energy-efficient IoTSs that may minimize energy usage [45,49];
Page. 11 Kamlesh
Kumar
4. Lightweight security measures may be applied before images are sent to the
IoT cloud via image processing [57]. As a direct result of this, a significant
amount of processing time and available computer resources will be preserved.
Thus, achieving long-term energy security is a goal.
6.2. Image Processing Techniques’ Roles to Address Security for IoTSs
6.2.1. Security of Networks
1. Image processing has proposed various methods [32,68,69] to address
concerns about the safety of the communication channel that runs between
the image processing and the final destination. It is said that image processing
techniques have a role in strengthening the security of communication
channels [68].
2. The technique of matching images [29,55] has been used in cloud-stored
images, where it has been shown to be effective in maintaining the
confidentiality of data storage and communication networks. It has helped to
ensure the security and confidentiality of the image that was retrieved from its
storage in the cloud. As a result of the matching procedure, communications
have been made more secure [29].
3. Numerous IoTSs have established a connection between image encryption,
image transfer, and cloud storage [67,70]. There has been an increase in the
level of security provided to the proposed communication channels [50].
6.2.2. Security of Contents
Image processing techniques have greatly improved the security of contents and
channels required for transmission. Image processing techniques have contributed to
the development of a variety of security procedures for IoTSs, which we examine.
1. Zigzag image encryption may be more resistant to data attacks [66];
2. As a result of advancements in image processing, IoT devices and networks are
now able to transmit data safely. The current conventional security scheme, in
particular the two-step authentication, has been proposed to be replaced by a
biometric encryption scheme that uses image processing [48];
3. IoTSs stored on a server and hence subject to attack at the server level benefit
from image block splitting as a security measure. Increased security and
Page. 12 Kamlesh
Kumar
decreased attack likelihood are two benefits of block-based image encryption
schemes [68];
4. Encryption of images may improve the safety of the contents of images.
However, improvements to this method are currently being made to enhance
the functionality of privacy protection for contents of images connected to
IoTSs [50];
5. The matching procedure helps to increase the safety of image data associated
with the IoT [29].
7. A Suggested Mind Map
In this section, a mind map mentioning and highlighting the suitable image processing
techniques applied by such an IoT application will be suggested and graphically
represented. For each IoT application (i.e., scenario), there will be a number of image
processing technique(s) that might be applied and can produce a high level of energy-
efficient and secure IoT system/application. Shown in Figure 15 is the suggested mind
map.
Figure 15. A suggested mind map highlighting the concern, which is the scenario (IoT
application marked in a red-colored font) and the corresponding and most suitable
solution (image processing technique marked in a blue-colored font) for such a
Page. 13 Kamlesh
Kumar
concern. Starting from the scenario, applying a solution, aiming to attain an energy-
efficient and secure IoTS.
In Figure 15, a number of solutions (image processing techniques) are proposed to
enhance the scenario (IoT application) by increasing the security and reducing the
energy consumption by such an IoTS. To read and understand this figure, suppose
that an IoT system is needed to be proposed for the purpose of security where
images are a core element in this system; it is suitable to apply face recognition,
biometrics encryption, or image matching for the purpose of verification, depending
on the nature of the application and the necessity of that IoT system/application. One
important thing to consider in this supposed scenario is that it is a must to reach a
high level of energy-use reduction and security. This could lead to contribute to
interested researchers getting benefits for designing and establishing such a potential
IoTS.
Figure 15 also shows that there can be a use of more than an “image processing
technique” with two different “IoT scenarios.” For example, in monitoring IoT
applications, Cloud-in image processing can be used, while this technique can also be
used with some other IoT applications, such as IoT security applications. Similarly, IoT
security and monitoring applications can use, for example, biometrics and face
recognition for almost the same purpose, which is the increase of protection of digital
content. In addition, images can be analyzed using two IoT applications which are IoT
security and location detection applications. In some other scenarios (i.e., IoT
applications), there might be a difference in using such a solution (image processing
technique).
There are, however, commonalities between these techniques utilizing solutions,
where they can be structured as briefly highlighted in Table 3.
Table 3. Commonalities between solutions and scenarios.
Page. 14 Kamlesh
Kumar
depends on the nature of the design of the IoT system. For more clarification, shown
in Figure 16 is a graphical representation highlighting these commonalities.
Page. 17 Kamlesh
Kumar
in the encrypted medical image, the authorized person may verify the
embedded OTP by comparing it to another OTP that was transmitted via a
secure channel.
Conclusion
The progress made in the digitalization of educational workbooks for automatic
scoring of handwritten answers represents a significant stride toward the realization
of a transformative assessment tool. The integration of InkML, coupled with the
association of questions with correct answers, sets the stage for the creation of a
sophisticated scoring system. The prototypes were developed to showcase the
project's commitment to addressing the complexities inherent in diverse subjects and
question formats.
This paper has looked at one of the common problems with IoTSs, which is security.
Image processing techniques have played a significant role in providing solutions.
Several alternative techniques of image processing have been investigated to
determine whether or not they could contribute to the solution of such issues. This
review’s primary goal is to help researchers in related domains to design an energy-
efficient and secure IoTS. It has been shown in these publications that image-
processing techniques may contribute to IoTSs.
In this section, two main sub-sections are discussed. Firstly, detailed concluding notes
on papers evaluated, in which the benefits and drawbacks of systems and papers are
discussed. Other important IoT-related challenges, such as energy usage and security,
have been discussed. The final sub-section lists a number of limitations related to this
article
As the project advances, continuous refinement, and testing will be conducted to
achieve the overarching goal of seamlessly scoring handwritten answers across
various educational domains. The potential impact of this project extends beyond the
realm of assessment, reaching into the heart of education technology, where
innovation converges with the timeless art of handwriting. The journey continues
with a vision of automating assessment processes while preserving the rich and
personal nature of handwritten responses
Page. 18 Kamlesh
Kumar
import shutil
import datetime
filepath = "/run/user/1004/gvfs/mtp:host=Wacom_IoT_Paper_10.3_2KNSE20098/MTP
volume/"
def copy_file():
def move_to_output(filename_var):
############################################
#First need to copy "save" file, so it will terminate session
save_src_path = "Dependencies/save.txt"
save_dst_path = "{}/".format(filepath)
############################################
# Now we need to save basic.inkml file to output folder and renaming the file
shutil.copy("{filepath}/PAGE_001/basic.inkml", "Output/{filename}")
############################################
if os.path.exists(filepath):
os.remove("{filepath}/PAGE_001/basic.inkml")
Page. 19 Kamlesh
Kumar
# Create "Output" folder if it doesn't exist
if not os.path.exists("Output"):
os.makedirs("Output")
# Create window
window = tk.Tk()
window.title("InkML File Selector")
window.geometry("400x200")
# Create buttons
filename_var = tk.StringVar() # create a StringVar to store the filename
btn1 = tk.Button(window, text="New InkML File", command=copy_file)
btn1.place(x=btn1_x, y=btn1_y)
Page. 20 Kamlesh
Kumar
# Reposition buttons
btn1.place(x=btn1_x, y=btn1_y)
btn2.place(x=btn2_x, y=btn2_y)
window.mainloop()
Page. 21 Kamlesh
Kumar
Page. 22 Kamlesh
Kumar
Page. 23 Kamlesh Kumar