Application of the Booking.com Analytics Software Tools in Reliable Processing of Big Data in Hotels Management

 

Abstract

Let's imagine, you are a hotel owner, and your desire is to collect all the statistics needed to make reasonable managerial decisions. What to do? The paper contains results obtained in area of big data analysis for hotel revenue management. Authors challenge the area of hotel management since they have and still improve skills in this area. The paper presents the new results obtained for previously developed Advanced Hotel Management Framework. We use comparatively the new tool “Booking.com Analytics” developed by the company Booking.com B.V. in 2016 for hotels involved in a global partnership program. We learned available features and data from the “Booking.com Analytics”. The performed case study is associated with a minihotel situated in Cambodia. We studied data related with booking percentage depending from tourists’ countries of origin, book window percentage, cancelation of reservation percentage, guests’ review rating, as well as special genius guests’ program percentage. After that we tried to find statistical dependencies between a managed value of room daily rate and available big data. In conclusion, the obtained results are discussed.

Introduction

Tourism including hotel industry consists 10% of the world economy in a sense of GDP (Gross Domestic Product) as it is stated in United Nations World Tourism Organization (UNWTO) Report 2018. Big data collected at booking systems sites could help to organize a competitive strategy for cost-effective hotel management system and hospitality industry development. However, opportunities provided by big data request careful using since any prognosis models have to be completely tested before its implementation for business decision-making. So we need to define reliability of big data in sense of its supporting of hotel business sustainability. The tasks related with revenue management systems have been specified and partly solved before the “big data” term became a hot topic.

The elements of the hotel system include hotel revenue management (RM) centers, data and information, the pricing and non-pricing RM tools, the RM software, and the RM team. The stages of RM process have been identified as goal setting, collection of data and information, data analysis, forecasting, decision making, implementation and monitoring. Special attention was paid to ethical considerations in RM practice, the connections between RM and customer relationship management, and the legal aspect of RM.

Mathematical models include the following: deterministic linear programming, integer programming, dynamic programming, Markov model, bid-price methods, price setting method, expected marginal revenue technique, stochastic programming, probabilistic rule-based framework, Monte Carlo simulation, fuzzy goal programming, and robust optimization. There are some important and successful researches in area of big data using for hotel management.

Whilst the academic literature has shown to place its faith in the increasing automation of revenue management decision-making using big data and analytics which should require no user input, some researches has placed this under a question. There is the advent of the era of large data is the development of the hotel industry transformation opportunities, but also for the construction of the wisdom of the hotel made a series of new problems, the use of large data will fundamentally change the current status of hotel management, the birth of a new model of hotel management. An alternative approach for operational hotel management is based on dependability. Based on the above analysis we can observe the present gap between investigations targeted to design some ideal hotel big data analysis tools and already existing industrial products.

The most impressive results in area of big data analytics for hotel management are obtained by the company Booking.com B.V. In this paper we discuss features of the “Booking.com Analytics” tools provided by the company Booking.com B.V. The main feature of the “Booking.com Analytics” is representation of collected statistic to support decision making in hotel management. To use big data from the “Booking.com Analytics” we hypothesize and check some dependencies between management decisions and rooms revenue amount. We are going to understand opportunities which are provided by modern big data analysis tools in area of hotel revenue management.

The paper objective is to get decision making strategy for hotel management with support of the “Booking.com Analytics” tools. Firstly, we learn features of global reservation services focusing on Booking.com. Features of the “Booking.com Analytics” are analyzed in details. Secondly, we make a case study of a mini-hotel operation in Cambodia. We analyze big data available from the “Booking.com Analytics” and extract statistics which can be directly appropriate for managerial decision making. After that, we hypothesize some dependencies related with decision making (stated room daily rate) and hotel revenue. Another checked hypothesis is about dependency between hotel revenue and number of tourists looking for a hotel in your area. Conclusions made at the final part of the paper are directed to support of hotel managerial decision making and demonstrate some constrains in big data using for reliable and sustainable decisions making.

Hotel Analytics Tools

Booking.com is the global leader in connecting travellers which contains more than one million of properties in one database. “Booking.com Analytics” has been released in 2016. This software tool is based on Big Data collected by Booking.com during many years of operation. “Booking.com Analytics” empowers property owners to grow their businesses with bespoke, actionable data and advice that is delivered in real time, powered by a customizable dashboard. This product is available globally as just one of the standard perks of partnership with Booking.com (as named Booking.com extranet). Other global booking services (for example, AirB&B) provide not so mature platform for Big Data analysis, so in this paper we will consider only “Booking.com Analytics”. “Booking.com Analytics” menu contains the following parts:

• Home includes only the main performance information;
• Rates & Availability supports the calendar with room daily rates broken down into specific dates and types; rooms occupancy and availability also are taken into account;
• Promotions allow to creating discount actions in addition to Rates & Availability;
• Reservations contain details of booking made by the past and future guests;
• Property supports description of hotel in respect with user interface reflected when guests come to your hotel page at the Booking.com system;
• Opportunities should encourage hotels managers to provide some discounts to guests, for example, in the framework of Genius frequent guests program;
• Inbox supports correspondence with guests as well as management of Booking.com;
• Guest Reviews aggregates scores of written guests feedbacks and ratings;
• Finance contains the base for calculation of commission which a hotel has to pay monthly to Booking.com;
• Analytics supports the “Booking.com Analytics” tools which is discussed in details below;
• Marketplace provides opportunities to integrate additional analytic tools, some tools are free, but some tools are not free.

Features of “Booking.com Analytics”

The “Booking.com Analytics” currently features the following parts of menu:

• The Analytics Dashboard aggregates the main performance overview including number of booked room nights, room revenue (the total amount paid by guests), and average daily rate what is room revenue divided to room nights; also the Analytics Dashboard contains links to the main reports briefly discussed below;
• The forward-looking Pace Report enables partners to benchmark their sales on Booking.com versus the previous year and compare their performance with aggregated data from their competitors;
• The Sales Statistics report provides an easy-to-digest snapshot of a property’s sales performance over the past year;
• The Booker Insights provides in-depth insights into country of origin, device used to book, and travel purpose;
• The Bookwindow Information is focusing on how far in advance Booking.com customers are booking their accommodation;
• The Cancelation Characteristics gives details about percentage of canceled guests bookings;
• The Guest Review Score provides data concerning property’s reviews rates written by guests;
• The Manage Competitive Set option allows to choosing up to ten hotel in your region to compare your Key Performance Indicators (KPIs) with the KPIs of the closest competitors;
• The Genius Report demonstrates percentage of booking made in accordance with Booking.com Genius program of frequent travellers;
• The Ranking Dashboard shows how you perform when guests search for properties in your area.

For an even deeper dive into the data, partners can compare their data with the following:

• Last year (own data);
• Peer group including all properties of the same type and star rating (if applicable) in a partner’s destination;
• Competitive set including ten properties of the partner’s choosing;
• Market including all properties in the partner’s destination, regardless of type or star-rating.

Partners can further slice and dice the data for deeper insights on:

• Ranking with their different custom and pre-defined groups of competitors;
• Delta changes over time;
• Date ranges (7, 14, 30, 60, 90, or 365 days).

The next Section provides examples related with analytics reports generation.

Case Study: Details of Big Data Provided by “Booking.com Analytics”

In this Section we consider case study based on the real experience of a mini-hotel “Chateau Puss in Boots” operated in Kep city, Cambodia. The considered hotel started to operate in December 2018, so in March 2019 we have available data only for four months. We do not represent financial data in this paper. It is necessary to introduce several important assumptions to understand the limitations of the study:

• We consider a private mini-hotel with capacity up to 15 room operating activities s (30-40 people), in which there are no corporate procedures, and everything is simplified to the limit in order to reduce overhead costs; therefore, all operating activities are concentrated on the hands of the owners without the participation of any structural units;
• We do not consider the structure of the room daily rate and additional revenue points (bar and restaurant, rent of bicycles and motorbikes, sale of tickets and excursions, spa etc.);
• We do not consider a general approach to hotel management.

Fig. 1 provides a part of the Booker Insights concerning distribution of bookings between tourists’ countries of origin. The studied hotel is operated by owner with Ukrainian citizenship what is important from the point of view of nationals’ distribution.

Fig. 1. Case Study: Booking percentage depending from tourists’ countries of origin (the studied hotel against the market)

Fig. 1 demonstrates countries distribution different from the market. Tourists from Cambodia and France represent about 50% of tourist market in Kep, however, in the considered case study they take only 15% and 13% respectively. It can be explained by conservatism of local Cambodian tourists who like to stay at the hotel operated by local Cambodian owners. The same explains the lower percentage of French tourists many of whom are sensitive form the point of view of French language. Russian tourists also like when hotel staff can speak in Russian and it explains why they submit more than 10% of bookings against 1.4% at the market. Concerning New Zealand (10% of booking against 0.6% at the market) and Swiss (8.7% against 2.4% at the market) tourists the higher percentage can be explained by good value for money since tourists from these countries are conservative from the point of view of unnecessary spending of money. The detailed Booker Insights report provides also information separated by countries concerning average daily rate, average length of stay, and cancelation rate. These data are important for prognosis of behavior of tourists depending from a country of origin.

Fig. 2 represents the Bookwindow Information related with percentage of reservations depending from a book window (how many days in advance a room has been booked). A large book window provides more opportunities from a point of view of a room daily rate specification. Also, room daily rate shall take into account local and global festivals and days off. General analytics say that only few guests make reservation for more than 30 days in advance. A diagram on Fig. 2 shows that about 70% of all bookings have been made just before guests check-in. It is not perfect since risk of empty rooms is increased, but from the other hand it pushes to more precise setting of room daily rate for the actual date. Also, the Bookwindow Information report represents a diagram with average daily rate per a book window.

Fig. 2. Case Study: Book window percentage

An important metric which affects any hotel business is cancelation percentage (see Fig. 3). The last-minute cancelation is usually stressful since it essentially decreases a book window and it increases a risk that a canceled room will not be sold out. Unfortunately for the analyzed case study we have 34% of cancelation rate while cancelation rate for the considered market is 28%. The most part of cancelations is explained by reservation with a book window more than one month. It is difficult to elaborate an effective strategy of cancelations number decreasing. People often change plans, or they can find that some another hotel proposal is more attractive. We try to communicate with guest as soon as we get a reservation but this strategy is also not completely successful.

Fig. 3. Case Study: Cancelation of reservations percentage

A hotel business highly depends from a reputation what is supported at the Booking.com with guest reviews. Guest review is based on rating (from 2.5 to 10) of the following hotel features: cleanliness, comfort, location, facilities, staff, and value for money. The Guest Review Score report contains integrated values of hotel rating (see Fig. 4).

Fig. 4. Case Study: Guests' review rating

Booking.com supports loyalty program for frequent travelers with the name Genius. Registered Genius users of Booking.com get discounts for reservations from 10% and more. To get Genius travellers the hotel has to support this program (see Fig. 5).

Fig. 5. Case Study: Genius guests percentage

But an issue is the hotel has to provide discounts for its own expenses. It means that the price for Genius gest is only 90% (sometimes even 85%) from the stated at the Booking.com daily room rate. From the one hand it entails a general hotel revenue decreasing. But from the other hand many Booking.com users are involved in Genius program and these users appreciate when a hotel supports this program. So involvement of a hotel in Genius program can increase the total hotel revenue despite the fact that daily room rate decreases. What is important to remember, daily room rate shall take into account a risk of 10% or 15% decreasing for Genius guests. For the considered case study (Fig. 5) Genius gests consist about 70% of all guests what is a great amount for reservations.

The Ranking Dashboard  presents the following factors influencing to the hotel revenue score in comparison between your hotel and average rate at the market:

• Conversion is the percentage of the hotel page views that converted into reservations;
• Average Daily Rate combined revenue earned from sold rooms divided by the number of sold rooms;
• Cancelations shows the percentage of all reservations that were canceled (Fig. 3);
• Review Score is calculated using the ratings left by guests (Fig. 4);
• Property Page Score shows how complete the hotel page is in regards to information and pictures;
• Reply Score takes into account the hotel answering guests as soon as possible.

Taking into account the above six factor can affect the hotel revenue amount, it makes a sense to consider the associated dependency. However, cancelation percentage, review score, property page score, and reply score can affect only indirectly to revenue. So it looks impossible to find dependencies between revenue score and indirect factors. What seems prospective for big data analysis are conversion percentage and average daily rate. In the next Section we consider hypothesis related with dependency of the hotel revenue score from conversion rate and daily rate.

Case Study: Checking of Big Data Based Hypothesis

So we can get big data from the “Booking.com Analytics”, and it we would like to understand how Big Date using can help in the statement of the daily room rate. Economic science suggests that there are supply and demand curves, and consequently, there is a certain optimal price that allows you to extract the maximum revenue from the sale of a product or a service. Type I errors (a price increase above the optimum) lead to the refuse of customers to purchase, while type II errors (a price reduction below the optimum) lead to a decrease in a potential revenue amount.

Thus, we state the Hypothesis 1 (H1): There is a relationship between the rooms’ revenue S and daily room rate C.

Formally, for each calendar day for one of the room, this can be described by the following minimax criterion:
S = max(c) ˄ f = 1, where S is room revenue numerically equal to daily room rate c = cmin..cmax} (room daily rate belongs to a certain range);
f = {0;1} is a binary sale indicator: f = 0 if the room is not sold and f = 1 if the number is sold.

If there are several rooms of the same type, then not all rooms can be booked every day, in addition, the daily rate ci for the same rooms may change during the book window, and the minimax criterion. If the hotel has several categories of rooms, then each of them applies the above criterion, and the total rooms’ revenue is formed as the sum of sales of all categories of rooms, or everything can be reduced to a general formula, if you increase the dimension by adding another index. In our study we will focus on double rooms’ cate- gory, as the most representative.

Now let’s analyze the mutual dependence between rooms’ revenue and room daily rate (H1). Linear regression does not make a sense because the higher room daily rate obviously produces the higher revenue. Therefore, we calculate a regression coefficient as a measure of relation between two arrays of values of two random variables (relation of covariance value to product of standard deviations):

Values of daily room rate ci and daily room revenue si are defined for every day during four months (120 pairs of values). Performed calculation gives us r(S,C) = 0.354. It means that there is not a statistical co-relation between revenue amount S and room daily rate C. However, our perception is still there should be a relationship. One more perception is than more guests are looking for a room in your than more guest will you get in your hotel. The “Booking.com Analytics” supplies us with such data (Fig. 6).

Fig. 6. Case Study: Daily search results

Fig. 6 represents numbers of daily search results for Kep (Cambodia). The conversion ratio is 132 / 79 377 =0.16% that means only 16 people booked the hotel from every 10 000 people looking for accommodation.

Let’s state the Hypothesis 2 (H2): There is a relationship between the rooms’ revenue S and daily search results R.

Values of daily search results ri and daily room revenue si are defined for every day during four month (120 pairs of values). Performed calculation gives us r(S,R) = -0.172. It means that there is not a statistical co-relation between revenue amount S and daily search results R.

Conclusions

In this paper we analyzed features and data provided by “Booking.com Analytics” which is modern powerful tools for hotels big data analysis. There are the following important data which can be extracted by Booking.com partners from extranet:

• Daily search results;
• Number of property page views;
• Number of booking;
• Room daily rates and average room daily rate;
• Revenues and average revenue;
• Percentage of booking cancelations.

Aggregated are available for different date ranges (7, 14, 30, 60, 90, or 365 days). Data can be compared with your hotel results during the last year as well as with average market data of average data of the hotels from your nearest pre-defined competitors set. In the paper we tried to get decision making strategy for hotel management with support of the “Booking.com Analytics” tools. In fact, the hotel manager has only one “control button” which is room daily rate on the specific date.

We stated two hypotheses:

• Hypothesis 1 (H1): There is a relationship between the rooms’ revenue S and daily room rate C;
• Hypothesis 2 (H2): There is a relationship between the rooms’ revenue S and daily search results R.

Despite our perception both hypotheses turned out not true, what is confirmed by low values of regression coefficients (r(S,C) = 0.354 and r(S,R) = -0.172). However, Booking.com states, that data contained in the “Booking.com Analytics” is harnessed by a proprietary logic that converts it into a prioritized list of actionable business advice. Also, Booking.com thinks that partner hotels can quickly peruse the opportunities, select the most relevant options for their property, and instantly implement them to enhance their listing and grow their business through Booking.com. After making the suggested adjustments, partners can then return to the analytics dashboard to easily monitor the results of their efforts, track their progress over time, and identify new areas for potential growth. In practice there are not any models which support the above statements.

Our conclusion is that big data for hotel management can be used only as a starting point for basic analysis. Reliable and sustainable decision making strategy shall additionally take into account experience, perception and intuition together with macro and micro economic trends. Discussing the hotel decision making strategy, we should remember the manager has only one the real tool to affect the revenue amount. This tool is the daily rate per room. Of course, there are many issues affecting the daily rate, but all the hotels activities are concentrated in this variable.

Structured Arguments for Assurance Case

 
Abstract

The paper describes an approach to improve Assurance Case applicability through structured argumentation. We started from approach based on use of twice argumentation step including reasoning step and evidential step with structured text support. After that, we improve the existing method with the following issues: 1) a general algorithm for the development of the Assurance Case is proposed; b) relations between the argumentation graph and templates of structured text are explicitly explained; c) structured text is supplied with clear templates. We implement a case study applying the obtained method for arguing functional safety compliance. A general conclusion is this method makes Assurance Case methodology more practical and understandable.

History and Concept of Assurance Case

For safety-critical and security-critical applications we always need to we argue or assert that some system is safe. Obviously, a number of criteria must be introduced for that. However, we need to determine how are reliable our knowledge about the analyzed system. Why can we trust this knowledge? What makes our arguments and reasoning credible? Having delved into such problems, one cannot do without philosophical disciplines such as ontology, epistemology and logic. The next step is to understand how should we justify or assess safety and security in a reasonable and logical way. Such approach is based on the theory of argumentation. The Assurance Case (AC) is a structured argument that some system has some properties we desire; that it is safe, or reliable, or secure against attack.

The British philosopher Stephen Toulmin gave a new impetus to the modern development of argumentation in the work entitled "The Uses of Argument", published in 1958. Tulmin extended the logical implicative inference with additional parameters and proposed to represent this operation in graphical form. Tulmin's notation operates with the following entities: data (D) – initial data for analysis, claim (C) – the purpose of logical implication inference (If D So C), warrant (W) – an additional argument, qualifier (Q) – the degree of confidence in the results of the logical output, rebuttable (R) – additional counterargument. Argument maps were used to visualize reasoning before Toulmin, but it was he who most successfully generalized the structural model for the analysis and verification of arguments. Note that modern argument maps do not use directly the Tulmin's notation, because more simplification.

In the 1990s, researchers continue to seek new approaches to assessing safety. The idea seems to be on the surface: let's develop a special notation to justify compliance with the requirements of man-made objects and systems. Two British university teams took over, including City, University of London, where the spin-off company Adelard was formed, and University of York. Today Adelard and University of York also still occupy leading positions in the promotion of the AC. For the development of notations, the emphasis was placed on the logical reasoning that a property or component of the system meets the stated requirements. The works of Stephen Toulmin, which we have already considered, were chosen as the theoretical basis. As a humanitarian, Toulmin hardly thought about technical systems, however, he went down in history, among other things, as the founder of the argumentation for the AC. As the result, University of York developed Goal Structuring Notation (GSN), while Adelard developed Claim, Argument and Evidence (CAE) notation, as well as a software tool Adelard ASCE (Assurance and Safety Case Environment). Despite all benefits and some successful applications, the AC is well known only for some restricted areas.

Developing evidence to support compliance is a creative process that is highly human-driven. So, what is the most practical and realistic method for developing the AC? Some drawbacks are associated with the lack of argumentation techniques. One of the authors who have attempted to bridge this gap is John Rushby, who proposed a modified GSN approach to structured argument development. In this paper, we adopt structured argument approach as the basis, and go ahead to make it more usable and practical.

Improvement of Argumentation

A new wave of AC researches appeared after some critical notes made in the as named Nimrod Report published in 2009. It became clear, that neither the philosophy literature nor other disciplines that use argument seem to offer a universal theory of knowledge that is applicable to safety arguments. Normative models of informal argumentation do not offer clear guidance on when a argument should cite evidence rather than appeal to a more detailed argument. Therefore, improvement of argumentation stimulated a lot papers devoted to this issue, taking into account there is not any completed agreement which kind of evidence could be sufficient.

Epistemology based approach takes into account the study of the nature of knowledge, justification, and the rationality of belief (“What makes justified beliefs really justified?”). Recognition is a set of rules for what counts as sufficient evidence for a given kind of claim under given circumstances would provide developers, assessors, and regulators with a practical means to make justified decisions about how much detail an argument should have and whether an argument is sufficiently compelling.

Eliminative induction was suggested firstly by Sir Francis Bacon for evaluating confidence in a claim. The idea is, confidence in a hypothesis (or claim) increases as reasons for doubting its truth are identified and eliminated (Baconian confidence). 

Transformation of Typical Arguments in a Structured Argument Form

There are some shortcomings in the existing works, which are due to the lack of satisfactory practical argumentation techniques. Thus, in order to apply the AC methodology, it is necessary to select and improve the appropriate mathematical and methodological approaches for structuring the argumentation. The argumentation in the AC corresponds to the implication in logic, when the truth of the conclusion depends on the truth of the conditions. A logical rule involves a logical multiplication in the form of: SC1 AND SC2… AND SCn IMPLIES C, where SCi are subgoals, which also can be complex expressions.

As noted above, there are some drawbacks in the existing papers that are related to the lack of argumentation techniques. One of the few authors who have attempted to address this gap is John Rushby, who in his technical report offers an approach to developing structured arguments based on a modified GSN. In this section we use and update this approach.

Classical application of GSN (Fig. 1) is characterized by support for argumentation steps (AS) of any claim (C) with both subclaims (SC) and evidences (E). This approach has some drawbacks, which are due to the inability to have always a regular and typical argument structure.

Fig. 1. Transformation of a typical argument form to a structured argument form

Modification of argumentation steps is proposed to reduce them to a typical two-step structure. The first step, called the reasoning step (RS), is an analysis of subgoals that are aimed at achieving the primary goal, but there is no recourse to the evidence at that step. In the second step, called the evidential step (ES), the evidence for supporting the subgoals that was formulated in the previous step is formulated. Thus, the graph of the argumentation structure is transformed as shown in Fig. 1. This allows us to make a connection between the concept of safety and security (goal) and our knowledge of the physical world (evidence).

To further formalize the steps of RS and ES it is suggested to use structured text. This approach is appropriate, but in our opinion, it has a number of opportunities for improvement, such as the following: a) there is not a general algorithm for the development of the Assurance Case; b) relations between the argumentation graph and templates of structured text are not explicitly explained; c) structured text does not have clear templates.

Argumentation Improvement: Hierarchy of Requirements and Templates of Structured Text

In addition, the development of the AC is in many ways a creative process, which many depends on the human factor. The below is an improvement of the approach, which, in our view, will allow us to move further in structuring the arguments of the AC and eliminate the above shortcomings. We demonstrate the opportunity of explicitly combining the AC with structured text components. Let's present a hierarchy of requirements that create the structure of the AC in the form of a pyramid. In most regulatory requirements for control systems, the structure of requirements includes 3 or 4 levels (Fig. 2).

Fig. 2. Hierarchy of requirements to control systems and a relation of requirements with argumentation steps

Zero level is a meta-goal according to which the control system must meet all safety requirements. At the first level, global safety goals are achieved, for example, according to functional safety requirements:

- The safety and security management system shall achieve all safety objectives;

- Safety and security life cycle should be implemented during system development;

- A sufficient set of measures against random failure must be applied to the system;

- A sufficient set of measures against systematic and software failures, including cyberattack defense, must be applied to the system.

The requirements groups contain related requirements and support one or other of the global goals. For example, the requirements for safety and security management in IEC 61508 include requirements to human resource management, configuration management, documentation management, and others.

The structure of the links between the zero, first and second levels is a tree transparent enough and does not require detailed elaboration of the arguments, since these arguments are typical and well tested. However, structured arguments are required when moving from the second level to the lower levels. The requirements of the lower levels may be either composite (such as include a number of separate requirements) or separate. If all requirements are separate, this level becomes third, and then it is directly related to the subgroups of requirements.

Fig. 2 combines the overall structure of the AC and the algorithm for constructing structured arguments. Such arguments should be developed for the second, third and fourth (if any) levels. An approach to argument structure is introduced in Fig. 1. For the lowest level, besides the RS, the ES should also be applied. Since it is not appropriate to add detailed information about the content of the arguments on the graph structure, each of the nodes of the AC, starting with the second level, is marked with an argument description using so-called structured text (ST). Notice, that the AC is not a strict tree because the same evidence can support different arguments or subgoals.

Let's develop a typical structured text configuration for the reasoning and evidential steps using the GSN components. The structured text has a template with a set of fields that are denoted by service words that correspond to the GSN components. We need to provide two templates, for the RS and for the ES (Fig. 3,4). In these templates, the names of the service words are given in bold, and italics provide a brief description of the content that should fill the template fields.

Fig. 3. A template of structured text for a reasoning step

Fig. 4. A template of structured text for a evidential step

Algorithm for the Structured Argumentation Method

Based on the results obtained in the preliminary section, we can draw a formalized algorithm for the structured argumentation method (Fig. 5). For that, we use activity diagram notation of the UML. Steps of the algorithm are related with levels of a hierarchy of requirements that is represented on Fig. 2. The input data for the method application include a database of standards applicable for the domain of the licensed system.

Fig. 5. An algorithm for application of the structured argumentation method

The first step of the method application contains analysis of the standards database. The expected result does extract a general set of requirements which has to support a top level of global goals (GG) for safety and security. A typical set of GG for safety related application includes requirements to management, life cycle, protective measures and assessment. GG can be represented in a view of a simple mind map.

The next step is decomposition of GG to groups of requirements (GR). It contains top-down analysis of all requirements which are related with any specific GG. It is possible to use only one target standard as well as a set of standards specified in the requirements to the licensed system. The expected result has to contain sets of the text fragments which cover GG by GG.

For the first step a separate GR can be represented in a view of a mind map. Later it can be transformed in GSN with use of software tools. It is reasonable to draw the AC graph (GSN graph) for each of the separated set of the group of requirements. However, if any relations between subgoals or evidences of different GRs of one GG are discovered, then the AC graph should be built for the GG in general. The next step is the first RS, which decompose GR to SGs. For this step we use the template of ST (see Fig. 2). An issue is some SGs can be composite, so such SGs requests the future decomposition to separate SGs.

Case Study: Application of the Structured Argumentation Method

Let’s synchronize the AC with the hierarchy of requirements (Fig. 2). For this, we implement the obtained method (Fig. 5). The meta-goal (Level 0) is a compliance of some abstract system with all identified requirements to safety and security. Goals of the Level 1 correspond to the main parts of safety and security issues like concept and functions, standards and regulations, system architecture etc. In this paper we consider the Level 1 goal related with safety & security management and assessment.

The transition from the Level 1 to the Level 2 groups of requirements contains an analysis of existing requirements to safety & security management and assessment like human resource management, configuration management, software tools selection and evaluation etc. Let’s consider documentation management on the Level 2. The goal is documentation management complies with all identified requirements.

The transition from the Level 2 to the Level 3 requirements contains the RS, which is based on an analysis of IEC 61508 requirements to documentation management. Such requirements are contained in IEC 61508, Part 1 “General requirements”, Section 5 “Documentation”. This RS transforms the text of IEC 61508 into a set of subclaims related with the Level 2 claim (documentation management complies with all identified requirements). Also, during the subclaims identification and analysis we shall identify composite requirements for which we need one more level to obtain separate requirements from composite requirements, so more argumentation steps will be performed for transition from the Level 3 to the Level 4. Fig. 6 represents RS for the Level 2, and demonstrates that the most parts of the subclames requirements are separate and the next step for it is ES. Exception are the SC6 and SC10 with are composite requirements, so for them we need one more RS to transit from the Level 3 to separate requirements of the Level 4 (Fig. 6).

Reasoning Step (Documentation Management)

Context

Connection between the group of Documentation Management requirements of the Assurance Case Level 2 and composite and separate requirements of Level 3

Docs

Documentation Management Plan

Claim

Documentation Management complies with IEC 61508 requirements

Subclaim SC1 (IEC 61508-1, 5.2.1), SEPARATE

Documentation supports all phases of safety life cycle

Subclaim SC2 (IEC 61508-1, 5.2.2), SEPARATE

Documentation supports functional safety management

Subclaim SC3 (IEC 61508-1, 5.2.3), SEPARATE

Documentation supports functional safety assessment

Subclaim SC4 (IEC 61508-1, 5.2.4), SEPARATE

Documentation complies with standards

Subclaim SC5 (IEC 61508-1, 5.2.5), SEPARATE

Documents are available

Subclaim SC6 (IEC 61508-1, 5.2.6a,…,d), COMPOSITE

Documents have sufficient quality

Subclaim SC7 (IEC 61508-1, 5.2.7), SEPARATE

Documents have title and content

Subclaim SC8 (IEC 61508-1, 5.2.8), SEPARATE

Documents comply with procedures and practices

Subclaim SC9 (IEC 61508-1, 5.2.9), SEPARATE

Documents have version numbers

Subclaim SC10 (IEC 61508-1, 5.2.10a,b), COMPOSITE

Documents have structure for search support. The last version of documents can be identified

Subclaim SC11 (IEC 61508-1, 5.2.11), SEPARATE

Document control system is implemented

Justification

Structure and content of Documentation Management Plan

END Reasoning Step

Fig. 6. Structured text for the reasoning step of Level 2

The future analysis of the point 5.2.6 of the IEC 61508-1 shows that there is a list with four additional requirements. All these requirements are related with quality of documents so they can be covered with the same ES. The same situation is the point 5.2.10 of the IEC 61508-1.

That case does not affect the structured argument form. We propose an additional operation of convolution for framework of structured argumentation. We can implement the convolution, if separate requirements related with one composite requirement are supported with the same evidence step. Also the convolution entails simplification of the Assurance Case graph in the part of transition between the Level 3 and the Level 4. At the Level 4 we have six more separate SCs (four plus two), so no more decomposition is needed. The next is application of the ES as per the developed template. The results of the ES implementation are given on Fig. 7.

Evidential Step ES1,…,ES11

Context

Connection with the subclaims of the Levels 3 and the Level 4

Docs

Documentation Management Plan; Project Repository

Claim

SC1,…, SC11

Evidence E1

Strategy of documentation for functional safety

Evidence E2

Documents access rights

Evidence E3

Documents preparation review and approval

Evidence E4

Documents list and responsibilities

Evidence E5

Documents format and templates

Evidence E6

Documents version and change control

Evidence E7

Project repository structure

Evidence E8

Document control system

Claim à Evidence

SC1 à E1; SC2 à E1; SC3 à E1; SC4 à E1; SC5 à E2&E3&E4; SC6 à E5; SC7 à E5; SC8 à E1; SC9 à E6; SC10.1 à E5; SC10.2 à E6; SC11 à E7&E8

Justification

Structure and content of E1,…,E11

END Evidential Step

Fig. 7. Structured text for the evidential step

Fig. 8. GSN graph for the Assurance case based on structured argumentation

Conclusion

The analysis of existing approaches to the development of the Assurance Case is conducted. Existing works have some drawbacks due to the lack of satisfactory practical argumentation techniques. One of the few authors who attempted to address this gap is John Rushby, who in his technical report offers an approach to developing structured arguments based on modified GSN and structured text. In this paper, we use and develop this approach.

Thus, in order to apply the methodology of the Assurance Case, a mathematical and methodological apparatus for structuring the argumentation was selected and improved. We obtained the structured argumentation method including the following: the overall algorithm of the Assurance Case development; the proposed structure of the Assurance Case graph, which is based on the typical structure of the arguments and is developed in connection with the structured text of the description of these arguments; improved structured text templates for arguments description. The obtained method can be used as the basis of the appropriated argumentation framework supported with a set of formal operations performed with the Assurance Case graph and supported structural text.

We applied the proposed structured argumentation method for the group of requirements related with documentation management. As the result, we get the template with the Assurance Case graph and structural text related with typical reasoning and evidential steps. The obtained practical and theoretical results may be used for any kinds of safety and security critical systems and applications.

Case-Based and Project-Based Methods for Effective E-learning in ICT Safety and Security

Abstract

Let's dive to the topic of distance learning, which became a hit during 2020. All industries and academias pay now attention to this issue. We would like to focus on the most important and effective aspects of E-learning. The objective of this paper is to develop a practical E-learning framework with implementation of case method and project-based learning. In paper we obtain the following results. Content of the massive open online course (MOOC) “Safety and Security of Control Systems” is analyzed. This MOOC was introduced in 2017 for master students program “Cybersecurity” at National Aerospace University “KhAI” (Kharkiv, Ukraine). A core part of this MOOC is a project devoted to safety and security assessment of real systems and software. Taxonomy for indicators of E-learning effectiveness is proposed. Case study was done in Ukraine between students of Cybersecurity program after finish of study of the MOOC “Safety and Security of Control Systems”. A sample includes 40 master students involved in the course learning during 2017-2019. Case study results confirmed a set of hypotheses related with E-learning effectiveness when case method and project-based learning are implemented.

Introduction

Online learning also named as electronic learning (E-learning) means 100% virtual education via internet media with support of information technologies (IT). E-learning entails a huge transformation of education which becomes more and more distance as well as personalized. A modern approach to E-learning implementation is consists in development of massive open online courses (MOOCs). Despite numerous well known advantages and disadvantages of E-learning we put emphasis on the challenges and opportunities related with application of this relatively new education technology, such as: individual approaches to students with opportunities to build individual learning trajectories based in student-oriented approach; application of Learning Management System (LMS) as centralized environment for administration, documentation, tracking, reporting, and delivery of learning courses; needs in essential scope of relevant studies and data, that would provide a strong background for empirical based analysis; choice of relevant indicators to make qualitative and quantitative assessment of E-learning; high degree of importance of students’ homework which can be organized in both individual and collaborative manner; needs in choice of effective teaching methods depending on features of courses and sciences; in the paper we discuss the case method as well as the project-based learning applied for the course devoted to safety and security of control systems.

It is needed to notice concerning some gaps in researches devoted to the above points. The actual research is directed to fulfill some of these gaps. Based on the above, we develop a concept for our research area (Fig. 1). This concept recognizes dramatic importance of students’ homework for successful implementation of the whole E-learning process. After that, we take into account learning course defendant features and discuss in this paper the course “Safety and Security of Control Systems” which is provided since 2017 by the Department of Computer Systems, Networks and Cybersecurity of National Aerospace University “KhAI” (Kharkiv, Ukraine). Also, we discuss the additional values which are provided by application of the case method as well as project-based learning for E-learning.

Fig. 1. A Concept of the paper: an influence of a learning course, the case method and the project-oriented approach to homework during E-learning

Objective and tasks

We recognize some gaps in area of research related with case-based and project-based E-learning. These gaps are in the area of some lack of empirical data as well as a lack of theoretical basis for development of MOOC supported with PBL and the case method. The objective of this paper is to develop a practical E-learning framework with implementation of case method and PBL. To achieve the paper objective we perform the following research steps:

- Firstly, we analyze theoretical and practical content of the MOOC “Safety and Security of Control Systems”;

- Secondly, we propose taxonomy for indicators of E-learning effectiveness;

- After that, we do setting a field questioner based study of experience of students involved in E-learning of the MOOC “Safety and Security of Control Systems”;

- Finally we develop a practical E-learning framework with implementation of case method and PBL.

Analysis of the MOOC “Safety and Security of Control Systems”

The MOOC “Safety and Security of Control Systems” was introduced in 2017 for master students program “Cybersecurity” by the Department of Computer Systems, Networks and Cybersecurity at National Aerospace University “KhAI” (Kharkiv, Ukraine). The goal was initially to make the course as a problem solving, so it is aimed at solving a practical problem: preparing for safety assessment against the requirements of the standard IEC 61508 “Functional safety of electrical/ electronic/ programmable electronic safety-related systems” or other similar safety standards. Another purpose of this course was to make it fit for industrial training.

Thus, the course participants received the whole set of advantages of MOOC users, like learning with small information blocks, study at a convenient time, if a student did not understand something or missed, then he “scroll through” it again, etc. In the subject area, the benchmark was made primarily for Industrial Control Systems (ICS), but other control system architectures are also considered like embedded systems, and Internet of Things ( IoT). A look at security was directed from the point of view of safety assurance. However, where appropriate, a relation between safety and security is demonstrated, for example, a common life cycle of safety and security. It was considered which techniques of safety assurance can increase the level of security.

The course includes six lectures or, in MOOC terminology, six weeks of study. Primary, the lectures were posted on the YouTube channel in the form of the following playlists (see Fig. 2): Lecture 1. Introduction; Lecture 2. Requirements of safety standards; Lecture 3. Functional safety management; Lecture 4. Safety and security life cycle; Lecture 5. Quantitative assessment of safety; Lecture 6. Techniques and measures of safety assurance.

Fig. 2. Structure of the MOOC “Safety and Security of Control Systems”

Every lecture is supported with a set of 5-10 minutes videos in quantity from four to seven per one lecture as well as with lecture slides and homework items. Students’ homework includes the following items: video lectures and slides of lecture for independent learning of theory; answer to test questions (quiz); reading of recommended literature, including literature included in the curriculum specially for mandatory individual learning; study of additional materials exceeding the scope of the course program; the course project, which is the most important part of the course practicum. The course learning contains the following techniques: asynchronous interactive learning with video lectures during a primary study of theoretical material; self-paced independent learning with slides of lectures and recommended literature during a deep study of theoretical material; collaborative learning during implementation of the course project (Fig. 3); synchronous learning during consultancy leaded by a teacher.

Fig. 3. Structure of the Assurance Case template for performance of the course project

The course project performance is the main objective and result of the course learning. First of all, the project is necessary to solve an applied problem based on the acquired knowledge. The main task of the project is to develop a document covering the analysis and assessment of safety and security related with the lecture material. This document is called the Assurance Case in accordance with the actual practice of safety and security assessment. The development of the Assurance Case is now used in the practice of assessment and certification against safety and security requirements. The components of ICS like controllers, actuators and sensors are offered as subjects of assessment. Students are given the Assurance Case template, and they fill it out in stages, based on the material in each lecture. During the project implementation we assign project teams with two, three or four students depending from their preferences. The structure of the Assurance Case template associated with the lecture content for the course project performance is represented on Fig. 3.

Taxonomy of E-Learning Effectiveness Indicators

The proposed three-level taxonomy of E-learning effectiveness indicators is presented in Table 1. This taxonomy is a good starting point to analyze the known empirical researches, but also the taxonomy can be enriched by results of new experiments.

Table 1. Taxonomy of E-learning effectiveness indicators

Researches, which are analyzed above, deal mainly with the education performance indicators defined by teachers and students in the learning process. It should be noted that assessment of the learning process by students is quite subjective, since students do not fully possess objective information about the actual usefulness of the acquired skills and knowledge. The same applies to some extent to teachers. Complaints about the lag of teachers’ experience from life's realities are justified in many cases. Therefore, if teachers are working in the framework of the curriculum only, they can also not always fully appreciate the objective usefulness of the knowledge and skills taught. Therefore, in our opinion, it is necessary to take into account two more important categories of indicators. The first category is effectiveness indicators determined by graduates during some time after the end of the graduation, and the second category is effectiveness indicators determined by the employer. A challenge is to take valid and relevant data to estimate these kinds of indicators, but modern online technology can help to resolve this issue.

Case Study of E-learning Effectiveness

Our research setting is Ukrainian National Aerospace University “KhAI”. Computer Systems, Networks and Cybersecurity Department introduced a new education program “Cybersecurity” in 2017. During 2017-2019, four groups of master students have been graduated in accordance with the Cybersecurity program. One of the courses of this program, named “Safety and Security of Control Systems” was provided on completely E-learning basis as a MOOC with use YouTube for video files disposition as well as Google Drive for distribution of all other course materials. A non-random sample covers all general population of 40 students, which is presented in Table 2 with gender details. It should be noticed that in 2017 we taught two groups of students because the Cybersecurity program update and shifting of the considered MOOC to other semester. The groups of students were 8 to 12 people, so it was easy to distribute and to collect the study questionnaire since individual communications have been established between teacher and students.

Table 2. A sample of students for study

Study design is based on a set of the following hypotheses related with E-learning effectiveness (see Fig. 4): Hypothesis 1 (H1): Students are satisfied with E-learning process; Hypothesis 2 (H2): Students prefer E-learning format faster than traditional format; Hypothesis 3 (H3): Theoretical E-learning materials can be presented in a form which is understandable for students; Hypothesis 4 (H4): Practical E-learning exercises can be presented in a form which is understandable for students; Hypothesis 5 (H5): E-learning motivates students to get new knowledge.

Fig. 4. A structure of the hypotheses network

All students were asked to fulfill the questionnaire after the end of the course study. The questionnaire contains five parts related with five stated hypotheses. Fivedegree scale is used to estimate students’ agreement with the questionnaire statements. Also we asked students to indicate the positive issues of the course as well as the issues which can be improved. For this part we calculate quantity of students who are agree with one or the other statement. One student can choose more than one statement. Results of the study are presented in Table 3. We also include average exam grades in ECTS points to Table 3.

Table 3. Results of case study

Results of review confirmed the truth of the hypotheses H1-H5. An average satisfaction degree is 4.2, so H1 “Students are satisfied with E-learning process” is confirmed. An average confidence level for E-learning preference is 4.1, so H2 “Students prefer E-learning format faster than traditional format” is confirmed. An average understanding level of the lecture materials is 4.0, so H3 “Theoretical E-learning materials can be presented in a form which is understandable for students” is confirmed. An average understanding level of the project materials is 3.9, so H4 “Practical Elearning exercises can be presented in a form which is understandable for students” is confirmed. We notice, the degree for project materials understanding is the lowest because the project is really the most difficult and challenging part of the course. An average motivation degree is 4.3, so H5 “E-learning motivates students to get new knowledge” is confirmed. All ratings from students are in the range from 3.8 to 4.5 what mean a low level of the dispersion. An average exam degree is 77.7. Three groups of students got high level degree more than 80 on the average, and only the second group at 2017 got 64.3, what affect dramatically the degree for all four groups. The most important issues for students are opportunities to be free in a choice of a place and a time of study (80% of respondents). After that students emphasize the importance of a new view for security issues (62.5%), availability of all the course materials at one place (57.5%), and an opportunity to drive a real world project (47.5%). The main concern of students is a lack of teacher support (90% of respondents). Concerning this point it is worth to notice, that it is the main challenge which face students during E-learning, and also this is the main difference between E-learning and traditional “face-to-face” learning. We understand a lack of the selfdriven behavior is a big challenge during the project performance. It is also highlighted with other students’ issues such as the requests for more detailed steps for the project implementations (82.5%) and for more transparent expectations and acceptance criteria for the project (67.5%).

We continue to work for more transparent and understandable project descriptions. The degree of project material understandability increased from 3.8-3.9 in 2017 to 4.1 in 2019. The same, the number of students who need more explanations for the project steps decreased from 12 (100%) in 2017 to 5 (62.5%) in 2019, and the number of students who expect more transparency and acceptance criteria decreased from 10 (83.3%) in 2017 to 5 (33.3%) in 2019. It is also important for students to have the project subject in the field related with the students’ experience in software engineering (32.5%), including ICS and IoT.

E-learning Framework with Case Method and Project-Based Learning

An integral part of this research joints all outputs in a view of E-learning framework supported with case method and PBL. This framework is presented on Fig. 5 Below we briefly discuss the main parts of the framework.

Fig. 5. A structure of the E-learning framework supported with case method and PBL

PBL principles are related with approach to choose and drive the project as the core practical activity of the course. We adopt the traditional approach with the following principles:

- Projects are the central item of the course and the central teaching strategy, because students learn the course via the project;

- Projects involve students in a special kind of investigation directed to knowledge construction and transformation;

- Students play the main role in projects performance, and projects are realistic.

Principles of case method implementation are close related with PBL approach. The main idea is to propose for students a real software-hardware product and real documents covering safety and security assessment and assurance. Successful case method implementation suggests consideration of all best practices related with PBL and homework organization. Homework organization includes supplying of students with all materials and instructions. Principles of homework are the following: collection of online materials in the LMS area; combination, depending on the material being studied, of various forms of homework (such as classroom individual or group study, extracurricular individual or group study); student-centered approach with the formation of an individual trajectory of the education and homework depending on the individual preferences of students and the recommendations of teachers; focus aimed at the development of creative and research competencies; monitoring by teacher in order to analyze adoption of learning material by students; analysis of the opportunities to improve the homework process; taking into account curriculums of other interrelated academic courses.

MOOCs development and improvement bases on using of the modern design tools and LMS. Other MOOCs related issues lay in students feedback monitoring and continuous improvement. For E-learning effectiveness monitoring teachers should choose a set of appropriate effectiveness indicators. An example of taxonomy is presented in Table 1. Results of effectiveness monitoring are outputs of a case study organized to supply stakeholders with relevant empirical data. Periodical field review can be a method to implement a case study. Also other relevant case study can be included in data collection to support cases meta-analysis.

Conclusions

Twenty years after beginning of intensive implementation of E-learning we continue to follow technologies rather than theory like “the cart has been placed before the horse”. However, pedagogic innovations are such important like technological innovations especially, if a lack of students’ motivation is founded. In this paper we obtain the following results. Content of the MOOC “Safety and Security of Control Systems” is analyzed. This MOOC was introduced in 2017 for master students program “Cybersecurity” by the Department of Computer Systems, Networks and Cybersecurity at National Aerospace University “KhAI” (Kharkiv, Ukraine). A core part of this MOOC is a project devoted to safety and security assessment of real systems and software. Taxonomy for indicators of E-learning effectiveness is proposed. This taxonomy contain three the following levels: stakeholder who defines indicators including teachers, students, graduates and employers, group of indicators and single indicators.

Case study was done in Ukraine between students of Cybersecurity program after finish of study of the MOOC “Safety and Security of Control Systems”. A sample includes 40 master students involved in the course learning during 2017-2019. Case study confirmed a set of hypotheses related with E-learning effectiveness when case method and PBL are implemented. We found the truth of five hypotheses, H1-H5 (see Fig. 4). Also we checked the main issues which increase or decrease quality of E-learning. The students’ opinion is the most important advantage of E-learning is an opportunity to be free in a choice of a place and a time of study (80% of respondents). After that students emphasize the importance of a new view for security issues (62.5%), availability of all the course materials at one place (57.5%), and an opportunity to drive a real world project (47.5%). Students are concerned about a lack of teacher support (90%), needs for more detailed explanation concerning steps of the project implementations (82.5%) and more explanations for the lecture materials (70%), more transparent expectations and acceptance criteria for the project (67.5) and more the project subjects from software engineering area (32.2%).

Finally, we develop a practical E-learning framework with implementation of case method and PBL, which includes the following entities: principles of PBL implementation, principles of case method implementation, principles of homework organization, MOOCs development and improvement, E-learning effectiveness monitoring, and case study collection. A practical value of the paper is determined by development of a framework for effective E-learning implementation based on the principles of PBL and case method. A theoretical novelty of the paper lies in a technique of Elearning effectiveness analysis supported with new experimental data.