MIPRO 2021, September 27 - October 01, 2021, Opatija, Croatia

Automatic, Configurable and Partial Assessment

of Student SQL Queries with Joins and Groupings

Goran Đambić*, Mario Fabijanić*, and Ana Lokas Ćošković*

*
Algebra University College/Software Engineering, Zagreb, Croatia
2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO) | 978-953-233-101-1/21/$31.00 ©202110.23919/MIPRO52101.2021.9596680

[email protected], [email protected], [email protected]

Abstract - Many papers exist that research semi- three schoolwork and an exam. In each of these
automated or fully-automated assessment of SQL queries in assessments, students are required to solve practical
the context of high education, as this approach promises less assignments by writing SQL code against real database. An
load on teachers, more consistent assessment across all average number of students enrolled in the course is around
students and faster feedback to students. Instead of 240 per academic year. Each schoolwork has four
assessment process that results either in all or none marks assignments, each exam has around 25 assignments and
awarded for a query, high education standards are more in there are five exam terms available. Based on these
favor of providing partial marks. This, and the fact that there numbers, it is easy to calculate that in each academic year
should be an easy way for a teacher to customize assessment
teachers have over 30k assignments to assess, in worst-case
rules to adapt them to institution standards, makes
assessment process more complex so many currently
scenario. On average, that number is lower because roughly
available tools fall short. This paper presents a newly 30% of students pass the exam on their first attempt, some
developed fully-automated assessment model and an students pass it on their second attempt, etc. Even so, the
information system that implements it system and explores amount of assignments that teachers have to assess can be
the possibilities of using it for partial assessment in higher measured in thousands. One possible solution would be to
education. This assessment model is designed to support switch some practical assignments to single or multiple-
complex joins and groupings, making it more applicable for choice questions, for which many excellent automatic
real students’ assignments and resulting queries. Students assessment solutions already exist. However, due to the
also get the immediate results and a list of found issues with need for applied skills and knowledge shown in [4], this
their queries, which is very important for them in process of road will not be taken – all exam questions must remain
learning. In this paper, built system is compared with manual practical.
assessments of the same student queries and its shows
promising results, even for very complex joins and groupings. Apart from a great workload put on teachers, there are
three additional issues with manual assessment [1] [5]. The
Keywords – SQL; assessment; fully automated; partial first issue is that the manual process takes longer, so
marks; configurable assessment process; join; group by; higher students have to wait for their results for a couple of days,
education which sometimes leaves them frustrated. The second issue
is the fact that for human beings, it is not easy to apply the
INTRODUCTION same criteria for all students, especially during longer
periods of assessment. Fatigue takes its toll and despite best
Although new database systems, like Not Only SQL efforts, the teacher can sometimes unintentionally apply
Databases (NoSQL), are becoming more and more popular different criteria to different students. Lastly, as students
in recent years, traditional RDBMS systems are still present are aware that teachers might make mistakes, they exercise
and even dominant in many areas [3]. Enterprise-level their right to ask for clarification of given marks, after the
vendors such as Oracle, IBM, and Microsoft, as well as results have been published, further delaying the final
many open-source projects, continue to develop RDBMS results.
products and add new features to them in order to stay
relevant. As shown in [4], skills and knowledge of one or In order to solve all of the above-mentioned issues, it
more of these RDBMS products will make students more would be ideal to create a system with the following
employable in the IT sector [4]. With this in mind, it comes characteristics:
as no surprise that higher education institutions that offer
x It is capable to assess SQL queries automatically.
programs in software engineering, applied computing,
computer science, etc., offer courses that teach students x The assessment process uses provided correct SQL
about relational databases [2]. queries and marking rules and gives partial marks
in range [0, max_marks].
At the Algebra University College, introductory
database course is compulsory for Software Engineering, x Assessment process gives results with the score
System Engineering and Multimedia Computing students and a list of errors made by student.
[1]. The objectives of this course are to teach students how
to design simple relational databases and how to use SQL x It is capable of processing batches of student
language to retrieve, insert, update and delete data from a queries and producing a structured result to be read
relational database [1]. Students’ work is assessed with

837
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.
 by teachers and students. The duration of the entire query and the student query to the actual database, and then
process should be measured in minutes. compare the returned result sets. Some tools give all or
nothing score based on whether the result sets are the same,
By using this ideal system, teachers’ main job would be and some tools give a percentage of points depending on
to prepare student SQL queries in the defined format and the percentage of result sets similarity. This paper finds this
start the process. The assessment would be done very approach valid, but believes it cannot be used by itself
quickly, resulting with a report for teachers and a report for because it is not flexible enough. For example, if we
students. The report for students would contain the score execute the following queries (the first one is a correct
and a list of errors made. Since the marking would be done query, the second is a student query), they will return the
by a system, there would be no ground for complaints. same result set.
The problem is that making such an ideal system is not select l.*, c.Name as City
a trivial task. Many papers have been experimenting with from Location as l
semi-automated and fully automated assessments [1] [6] [7] left join City c on l.CityId = c.CityId
[8] [9] [10] [11] [12] [13] [14]. Many of these approaches
work only in all-or-nothing marking modes – if a student’s select l.*, (select Name from City c
query matches the model query perfectly or returns the where l.CityId = c.CityId) as City
from Location as l
same result set, they award full marks. If not, zero marks
are awarded. A list of made errors and partial marks is For certain assessment goals, this can be considered
rarely provided. This paper specifically builds on research good and a student would receive full marks. However, if
done in [1] because it is the closest match to our ideal the assessment goal is to check whether a student has
system. This paper has learned on problems and mistakes learned how to use joins, then this approach will incorrectly
made in that paper and tries to correct and improve them to award points where points should not be awarded. Another
create a better automatic assessment system. example are the following two queries:
System [1] is capable of partially assessing SELECT, select * from City where Name like 'Tok%'
FROM, WHERE, ORDER BY clauses of SELECT
statement, VALUES clause of INSERT statement and basic select * from City where Name like 'Rok%'
forms of UPDATE and DELETE statements. As this is not Because a student mistyped letter R instead of T, they
enough for our intended purpose, the system proposed in will certainly lose some marks, possibly all of them,
this paper must be able to assess additional SELECT although a student has demonstrated the knowledge of
statement clauses: GROUP BY, HAVING, aggregate filtering.
functions, all types of JOIN and scalar functions.
In order to improve this approach, this paper adopts and
METHODOLOGY improves approach from [1] and uses query text analysis to
gain more insight into students’ work and uses it to improve
This paper suggests a fully automated assessment marking process. By using this analysis on the first
model built on [1]. A system capable of processing multiple example, it would be obvious that a student did not use join
student queries and producing reports was developed based and would receive no marks for that part of the query. In
on this model. This developed system is used to verify the second example, the acceptable Levenshtein’s distance
assumptions and measure how close it is to an ideal system. [15] of 1 would allow a student to receive all marks.
As a first step, the following requirements were put on
the proposed model: B. Requirement 2: Queries must be written according to
a set of guidelines
1. It can execute query against the actual database and
process the results set, and it can parse and analyze SQL is a high-level language and allows the same query
query text. to be expressed in multiple ways. For example, table aliases
are optional; if a student defines a table alias, it is
2. It expects queries to be written by following a set recommended to use it in the rest of the query, but it is also
of guidelines. perfectly valid to define an alias and never use it. Because
of these SQL conveniences, accounting for all possible
3. It must be able to recognize equivalent join
SQL query-writing options is complex and a goal to strive
constructs.
for in the future, but it is not a part of the model proposed
4. Marking should not stop if certain SQL clauses are in this paper. Instead, a set of guidelines are given to
incorrect. students who are expected to follow them. These guidelines
can be summarized as follows:
5. It can give partial marks according to a correct
answer and an extensible set of rules. x Students can choose whether to assign aliases to
tables. If an alias is assigned to one table, it must
6. It must provide a score and a clear list of errors
be assigned to all tables and it must be used in the
made by student.
rest of the query.
A. Requirement 1: Model can execute query against the x If using joins, it is mandatory to use either aliases
database and analyze query text or table names for each column in the query.
When marking a student query, many of the tools in [6] x Aliases for columns are optional. If an alias is
[7] [8] [9] [10] [11] [12] [13] [14] send both the correct assigned, it must be used in the rest of the query.

838
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.
 x In WHERE, column name must always be the first 3. All columns are also alphabetically sorted on the
operand (i.e. Quantity > 0 is correct, 0 < Quantity source column name.
is not).
4. All aliases are expanded into full names.
x In ORDER BY, column names or aliases must be
used, not positions. By following these steps, it is ensured that different
approaches to selecting same data are always canonized in
x All row filtering must be done in WHERE; the same form and can be easily compared. For example,
HAVING can be used only to filter groups. the following queries are all recognized as the same:
x Query must contain only necessary elements (i.e. select * from Invoice r
WHERE 1 = 1 might result in loosing points). inner join Detail s on r.ID=s.InvoiceID
inner join Product a on s.ProductID=a.ID
C. Requirement 3: Model recognizes equivalent joins left outer join CreditCard k
on r.CreditCardID=k.ID
In relational databases, data is stored in its normalized
form [16]. That means that in order to retrieve complex select * from CreditCard k
data, it might be necessary to join the data from multiple right outer join Invoice r
tables. At most times, query writers do not need to worry on r.CreditCardID=k.ID
about optimal order and join types because the optimizer inner join Detail s on r.ID=s.InvoiceID
inner join Product a on s.ProductID=a.ID
takes care of those details and chooses optimal order and
indices to execute the query [17]. This fact can present a select * from Detail s
problem for automatic assessment process because the inner join Product a on s.ProductID=a.ID
correct answer can use one join, and a student can choose inner join Invoice r on r.ID=s.InvoiceID
another one, for example: left outer join CreditCard k
on r.CreditCardID=k.ID
select * from Location l
left join City c on l.CityId = c.CityId In this way, a prerequisite where a model can recognize
equivalent join constructs is satisfied.
select * from City c
right join Location l on l.CityId = c.CityId
D. Requirement 4: Marking should not stop if certain
Both of these joins are equivalent and the student’s SQL clauses are incorrect
answer must be considered as correct. The model must be
Proposed model must give partial marks according to a
able to accommodate all possible join solutions that obey
customizable set of rules. Some of these rules are marked
defined guidelines. In order to achieve that, the first step is
as mandatory, and some as optional. When mandatory rule
to convert joins to canonical form.
is broken, marking process deducts defined number of
From [18], a join of two tables is actually a cross points and stops – no other rules will be validated. In order
product of all rows with a predicate applied, so we can write to achieve partial assessment, the number of mandatory
any join of tables A and B with predicate p like this: rules should be kept small. Usually, only the rule that
checks if the query compiles and executes is marked as
 ( × ) mandatory, all other rules are considered optional.
Because joins are commutative, we can write: Additional important consideration to achieve partial
marking is that each part of SQL should be assessed as
 ( × ) =  ( × ) isolated from other parts of the same SQL as possible. That
means that the student can get 0 points from one part of
SQL (i.e. SELECT if completely wrong) but can get full
Joins are also associative, allowing us to create cross
points for another part of the same SQL (i.e. WHERE is
products of multiple tables in any order we like and then correct). For example, when marking WHERE part, the
apply all predicates at the end: SELECT list written by a student is replaced with * and
  ( × ) ×  = ^ ( ×  × ) executed against the database. The result set is compared
with the correct result set and if they match, full marks are
Two simplifications made in this paper are that only awarded for WHERE. However, if the result sets do not
equi-joins are considered and that the joins are always made match, a query text analysis is performed on WHERE
between primary and foreign keys where both contain only predicates to give partial marks.
one column. Building on [24], the model proposed in this
paper extracts all predicates defined in FROM clause. For Isolated marking can be only implemented with best-
each predicate, source and target tables and columns are effort guarantee. If an error is made in FROM part, it is very
extracted, as well as the type of join (inner, left outer, right likely that other parts will not receive partial marks.
outer or full). Predicates are processed like this:
E. Requirement 5: Model gives partial marks from a
1. All right outer joins are converted to left outer correct answer and an extensible set of rules
joins, with target and source tables swapped.
For each assignment, the teacher is supposed to provide
2. For all inner and outer joins, necessary swaps are three set of values: A) the correct answer, B) what
made to make sure that the source table is always percentage of marks should be given for each query clause,
alphabetically lower than the target table. and C) which rules should be applied. This way, the

839
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.
automatic marking process is completely customizable by And student solution is this:
the teacher and each teacher can have the model resemble
select P.ID, P.Name, K.Name
their marking style. A part of configuration for an from Detail as S inner join Product as P
assignment might be: on S.ProductID = P.ID
{ inner join Invoice as R on S.InvoiceID = R.ID
"correct_answer": "select Id, Name from inner join Employee as K on R.EmployeeID = K.ID
City where Name like 'Tok%'", where K.ID = 275
"max_points": 4, order by P.ID asc
"clause_points_from": 10,
"clause_points_where": 50, The report given to student is like this:
"clause_points_groupby": 0,
"clause_points_having": 0, starting to mark: JaneDoe.sql
"clause_points_select": 40, where section: wrong rows filtered, remaining 90.0 %
"clause_points_orderby": 0, select section: missing column detail.productid.
"parses_and_compiles": { remaining 86.0 %
"deduct": 100, order by section: missing column detail.productid,
"hard_stop": 1 remaining 81.0 %
},
order by section: column 1 should be productid but is id,
"from_each_missing_table": {
"deduct": 50, remaining 76.0 %
}, JaneDoe.sql marking complete, remaining: 76.0 %
"from_each_wrong_join": {
"deduct": 50, The model was built in Python programming language
},
"from_use_specific_join": {
and uses Mozilla SQL Parser1 package for deep query text
"deduct": 100, analysis. At the start-up, the model executes a script to
"specific_join": "left" create a database against which queries will be executed.
}, All queries that change the state of the database run in
… explicit transactions and are rollbacked at the end.
}
This configuration first contains the correct SQL and RESULTS
the maximum number of points for this assignment. Then, In the validation process of the new system, the main
settings with prefix “clause_” define what percentage of goal was to discover the proposed system applicability,
maximum number of points is assigned to each part of the since there is added support for complex joins and
query (i.e. WHERE clause holds 50% of 4 points, which is groupings and grading became consequently more complex
2 points). Then, any amount of rules can be defined for the and challengeable. For that purpose, a total of 100 manually
assignment process. For each rule, the deduction/addition evaluated SELECT statements were used and grades given
percentage is defined and some rules might have additional by teachers were compared with grades given by the
settings. For example, “from_use_specific_join” rule says proposed system. A typical assignment from the earlier
that if the student solved the problem without left join, 0 actual exam term was used (showed earlier in this paper),
points will be awarded for that part. so that the results of the comparison are as realistic as they
Currently, the model supports 17 rules and additional can be. Chosen SELECT statement emphasize a need to use
rules can be embedded in the assessment pipeline with little complex joins, the last added feature to our proposed
effort. system.
Each of 100 solutions was awarded from 0 to 2 points.
F. Requirement 6: Model must provide a score and a Firstly, a rough indicator of the proposed system usability
clear list of errors is the comparison of the difference of total points given by
The result of the marking process for one query is a the proposed system (for all 100 assignments) and total
score and a list of found issues and made deductions. For points given by the teachers that evaluated them manually
example, if correct solution is this: (Fig. 1).

select Detail.ProductID from Detail Another indicator of the proposed system usability is
inner join Invoice the comparison of average points awarded. Fig. 2 shows
that there is a minimal difference in total average points
on Detail.InvoiceID=Invoice.ID
given by the proposed system and those awarded by the
inner join Employee teachers.
on Employee.ID=Invoice.EmployeeID
where Employee.Name='Anna'
and Employee.LastName='Smith'
order by Detail.ProductID

1
https://github.com/mozilla/moz-sql-parser

840
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.
 Another important aspect of validation is the
distribution of given points, since one of the focuses of our
proposed system is the ability of defining precise criteria
for partial points. The earlier version of the proposed
system implemented only plain SELECT statements and
resulted in simpler distribution of points given by the
system, which includes only 0%, 25%, 50%, 75% and
100% of the maximum points [1]. The newly proposed
system shows greater possibilities in giving partial points,
since the criteria created by teachers are much more precise.
Fig. 3 shows the distribution of awarded partial marks. We
can see that the granularity of given grades is noticeably
larger than in the earlier version of the proposed system.
Since teachers tend to evaluate on binary basis and the
Figure 1. Total sum of the points given by proposed system earlier version of the system had only a few scoring levels,
compared with the total sum of points given by teachers we can make an assumption that the development of the
proposed system is headed in the right direction and that it
brings students more precise and fairer evaluation of their
assignments.

Figure 2. Total average points given by proposed system

compared with total average points given by teachers Figure 3. The distribution of given partial points
with total number of valuated assignments

TABLE I. OBSERVED FREQUENCIES OF GRADES

Grades 0% 65% 70% 75% 76% 78% 80% 82% 85% 86% 88% 90% 95% 99% 100% Total
Proportion
of grades
12 0 2 7 0 0 2 0 0 0 6 5 1 0 65 100
evaluated
by teachers
Proportion
of grades
12 2 0 4 1 2 0 2 1 2 0 8 1 1 64 100
evaluated
by system
Total 24 2 2 11 1 2 2 2 1 2 6 13 2 1 129 200

TABLE II. EXPECTED FREQUENCIES OF GRADES

Grades 0% 65% 70% 75% 76% 78% 80% 82% 85% 86% 88% 90% 95% 99% 100% Total
Proportion
of grades
12 1 1 5.5 0.5 1 1 1 0.5 1 3 6.5 1 0.5 64.5 100
evaluated
by teachers
Proportion
of grades
12 1 1 5.5 0.5 1 1 1 0.5 1 3 6.5 1 0.5 64.5 100
evaluated
by system
Total 24 2 2 11 1 2 2 2 1 2 6 13 2 1 129 200

In order to prove our earlier assumptions, we carried out grades given by the proposed system and grades given
statistical hypothesis testing. We used Chi-square test (χ2) manually by the teachers. The observed frequencies of
on the base of our 100 grades and with the null hypothesis grades given by the teachers and grades given by the
that there is only a minor difference when comparing proposed system are shown in Table 1.

841
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.
 To calculate degrees of freedom and χ2 value, we also REFERENCES
need expected frequencies of grades shown in Table 2. The [1] Fabijanić, M., Đambić, G. and Fulanović B. (2020) A Novel System
number of degrees of freedom is: for Automatic, Configurable and Partial Assessment of Student
SQL Queries. 2020 43rd International Convention on Information,
(Number of rows - 1) * (Number of columns - 1) = Communication and Electronic Technology (MIPRO). DOI:
= (2 - 1) * (15 -1) = 1 * 14 = 14 (1) 10.23919/MIPRO48935.2020.9245264.
[2] Mohammad Amin, Pradip Peter Dey and Bhaskar Raj Sinha. (2020)
χ2 value is calculated using the formula: Effectiveness of Online Teaching Paradigm on Learning
 Experiences in Database Courses. The Journal of Computing
(0 −  )2 Sciences in Colleges (papers of the 29th Annual CCSC Rocky
χ2 =  = 22.518 Mountain Conference). Volume 36, Number 2, pg. 88-95.

 [3] Malik, A., Burney, A. and Ahmed, F. (2020) A Comparative Study
(2) of Unstructured Data with SQL and NO-SQL Database
Management Systems. Journal of Computer and Communications,
Our calculated χ2 value is 22.518 (2). It is lower than 8, 59-71.
the critical value for a significance level of 5% and 14 [4] WAGNER, William P.; PANT, Vik. Using virtual servers to teach
degrees of freedom (1) from contingency table, which is the implementation of enterprise-level dbmss: A teaching note.
Journal of Information Systems Education, 2019, 21.4: 1.
23.685. It means that our null hypothesis is not rejected and [5] AL-SALMI, Aisha. Semi-automatic assessment of basic SQL
statements. 2019. PhD Thesis. Loughborough University.
we can conclude that there is no statistical
[6] KLEEREKOPER, Anthony; SCHOFIELD, Andrew. SQL tester: an
significance between grades given by the system and online SQL assessment tool and its impact. In: Proceedings of the
grades given by the teachers. 23rd Annual ACM Conference on Innovation and Technology in
Computer Science Education. 2018. p. 87-92.
DISCUSSION [7] WAGNER, Paul J. The SQL File Evaluation (SQLFE) Tool: A
Flexible and Extendible System for Evaluation of SQL Queries. In:
This paper proposed an upgraded model for the Proceedings of the 51st ACM Technical Symposium on Computer
automatic assessment of SQL statements. The earlier Science Education. 2020. p. 1334-1334.
version fulfilled the obvious goals like dramatically shorter [8] Singporn, Pornwipha, Pakpoom Vichianroj and Areerat
time needed for grading students’ solutions written in SQL, Trongratsameethong. “ASQLAG - Automated SQL Assignment
Grading System for Multiple DBMSs.” (2018).
at the same time providing feedback to students, which is
[9] KLEINER, Carsten; TEBBE, Christopher; HEINE, Felix.
very important for them during the learning process. Automated grading and tutoring of SQL statements to improve
However, the shortage of the earlier version was the student learning. In: Proceedings of the 13th Koli Calling
limitation on simpler SELECT statements, without JOINs, International Conference on Computing Education Research. 2013.
GROUP BY and aggregate functions. This upgraded model p. 161-168.
gives teachers the ability to be more precise when defining [10] BRUSILOVSKY, Peter, et al. An open integrated exploratorium for
input criteria and to pinpoint the main purposes of each database courses. ACM SIGCSE Bulletin, 2008, 40.3: 22-26.
assessment, which now includes the mentioned SELECT [11] DE RAADT, Michael; DEKEYSER, Stijn; LEE, Tien Yu. Do
students SQLify? improving learning outcomes with peer review
statements. and enhanced computer assisted assessment of querying skills. In:
To validate the proposed system, we used 100 student Proceedings of the 6th Baltic Sea conference on Computing
education research: Koli Calling 2006. 2006. p. 101-108.
solutions from an earlier actual exam term, already
[12] RUSSELL, Gordon; CUMMING, A. Automatic checking of sql:
evaluated manually by teachers, and graded them using the Computerised grading. In: Proceedings of the Twelfth International
proposed new system. The teachers defined the criteria they Conference on Learning, Granada, Spain. Published in the
used while manually marking the exams. The results were International Journal of Learning. 2005. p. 127-134.
used to conduct statistical hypothesis testing with Chi- [13] SADIQ, Shazia, et al. SQLator: an online SQL learning workbench.
square test. It showed that there is no statistical In: Proceedings of the 9th annual SIGCSE conference on
significance between the grades given by the proposed Innovation and technology in computer science education. 2004. p.
223-227.
system and the grades given by teachers manually. It means
that our proposed system has become more usable in [14] PRIOR, Julia Coleman. Online assessment of SQL query
formulation skills. In: Proceedings of the fifth Australasian
practice: teachers’ benefit of less time needed for conference on Computing education-Volume 20. Australian
evaluation, students’ benefit of almost immediate results Computer Society, Inc., 2003. p. 247-256.
and useful feedback, which is not only great for the [15] LI, Yujian; LIU Bo. A Normalized Levenshtein Distance Metric.
students, but also for teachers because it eliminates the time In: IEEE Transactions on Pattern Analysis and Machine
needed for commenting each students’ challenged marks. Intelligence, Volume: 29, Issue: 6, 2007. p. 1091-1095.
In addition, the grading consistency improves, thus [16] Vladimir Mateljan, Goran Đambić, Decomposition of relational
eliminating the situations in which teachers, who have more schema without data loss, Odsjek za informacijske znanosti,
Filozofski fakultet, Zagreb, 2007, pg. 335-341.
and more students each year, can make mistakes during
[17] Giakoumakis, Leo, and César A. Galindo-Legaria. "Testing SQL
evaluation. Server's Query Optimizer: Challenges, Techniques and
Further improvements of the system will be directed at Experiences." IEEE Data Eng. Bull. 31.1 (2008): 36-43.
resolving complex tasks like sequential steps during [18] Justin Jaffray. An Introduction to Join Ordering.
https://www.cockroachlabs.com/blog/join-ordering-pt1/
DELETE or INSERT statements, combination of
statements, subqueries, etc. which would make this system
irreplaceable for database courses at Algebra University
College.

842
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:12:57 UTC from IEEE Xplore. Restrictions apply.