Research and application of an evaluation mechanism for enterprise information systems

In this section, we will conduct an in-depth analysis of the key technology model and innovatively propose an information system evaluation mechanism based on the improved normal cloud model. This mechanism organically integrates cutting-edge technologies such as ANP, entropy weight method, game-theory-based combined weights, and hierarchical clustering method, aiming to achieve a comprehensive and in-depth evaluation of the information system from all aspects.

To fully verify the scientificity and effectiveness of this evaluation mechanism, we will carry out detailed comparative analyses from multiple dimensions. On the one hand, we will select representative advanced technology models currently available as references. Through horizontal comparisons, we can highlight the advantages and characteristics of this mechanism in terms of performance and effectiveness. On the other hand, we will conduct vertical comparisons of the evaluation mechanism before and after improvement to visually present the significant enhancements brought about by the improvement measures.

In the subsequent content, we will elaborate in detail and systematically on each technical method and its specific implementation steps, striving to comprehensively and thoroughly analyze the technical details involved, and present a complete and clear technical picture for readers.

Table of Contents

Mechanism structure

Gareth Shepherd et al.²⁹ have posited that the strategic awareness of top leadership and their ability to leverage IT for management are crucial factors influencing enterprise informatization. The primary goal of information system planning is to align enterprise operational strategy with information system strategy. From this, we can infer that while the business functions implemented by various enterprise information systems may differ, their fundamental construction and operational objectives remain consistent. Consequently, the evaluation mechanisms for information systems possess a degree of universality. Enterprise information system construction adheres to principles of comprehensiveness, universality, feasibility, stability, and coordination. This research, taking into account the importance, advancement, and efficiency of daily internal and external business technical support, has developed a comprehensive evaluation mechanism.

The specific evaluation mechanism is shown in Fig. 1, which mainly includes the following steps:

1.

Construction of the information system evaluation index system: This research closely focuses on the essential characteristics of information systems and deeply integrates the actual operation status of enterprises to meticulously establish a comprehensive and detailed information system evaluation index system. This system comprehensively covers four key dimensions: planning, construction, operation, and effectiveness, with a total of 4 first-level indicators, 9 s-level indicators, and 21 third-level indicators. Among them, there are 9 subjective indicators and 11 objective indicators. The combination of subjective and objective indicators is designed to accurately evaluate the information system from multiple perspectives.
2.

Calculation of the weights of subjective and objective indicators: ANP and the entropy weight method are used to accurately calculate the weights of subjective and objective indicators. First, by widely collecting multi-sample data and organizing in-depth discussions among industry experts, all the intricate interdependent relationships among various indicators are comprehensively considered. On this basis, a comprehensive scoring evaluation is carried out. The powerful analysis ability of ANP is utilized to determine the weights of subjective indicators, and the entropy weight calculation is used to scientifically determine the weights of objective indicators, ensuring the rationality and accuracy of weight allocation.
3.

Calculation of the combined weights: The linear weighted method based on game theory is adopted to organically integrate the subjective and objective weights, so as to calculate the combined weights. This approach of combining subjective judgment with objective data effectively circumvents the limitations of a single method, making the entire evaluation method more scientific and reasonable, and enabling it to more truthfully reflect the actual situation of the information system.
4.

Evaluation classification based on the improved normal cloud model: The single-linkage hierarchical clustering method is used to construct the enterprise information system evaluation matrix, deeply analyze and clearly clarify the complex relationships among evaluation indicators, and then derive scientific and reasonable evaluation criteria. For each evaluation indicator, the parameters of the normal cloud model are finely adjusted and optimized by introducing the Gaussian fuzzy factor density function. The powerful learning and adaptive capabilities of the normal cloud model are fully utilized to accurately simulate and efficiently process various uncertain information, while also fully considering human subjective cognitive factors. The cloud models of each indicator are strictly cross-validated to obtain the most accurate membership degree values. Finally, combined with the previously calculated combined weights, a weighted average calculation is carried out to obtain a comprehensive and accurate overall evaluation grade of the enterprise information system.

Construction of the information system evaluation index system

In close alignment with the actual operational circumstances of enterprises, This research undertakes a meticulous process of indicator classification and research design. Leveraging the robust framework of AHP³⁰, the research is solidified through in-depth expert interviews and practical validations. Given the distinct features of high-level business specialization and the targeting of specific user segments, a comprehensive evaluation index system has been architected.

This system encompasses four critical criterion layers: system planning, system construction, system operation, and system effectiveness, housing a total of 21 well-defined indicators. Each indicator within this system is meticulously crafted to be in strict accordance with the strategic development goals and requirements of the enterprise. It adheres scrupulously to the internal standards and specifications governing the procurement, acceptance, and maintenance of informatization projects, ensuring a high level of consistency and quality control.

Furthermore, the system incorporates the authentic usage experiences and candid feedback from both internal and external business users. This not only enriches the evaluation data but also effectively reflects the unique characteristics and operational nuances of the enterprise. For example, when evaluating the indicator of coordinated construction degree, it is imperative to adhere strictly to the enterprise’s strategic development imperatives. Similarly, the assessment of business matching degree must be congruent with the actual business process norms, thus guaranteeing the practical relevance and applicability of the evaluation.

The detailed breakdown of these specific indicators can be found in Table 2, providing a clear and structured overview for further analysis and application.

Table 2 The evaluation index system for enterprise information systems.

Calculation of the weights of subjective and objective indicators

In this section, we introduce the calculation methods of subjective and objective weights. To enhance the scientificity and objectivity of the methods in this paper, we calculate the weights by combining subjective and objective approaches. ANP is selected as the calculation method for subjective weights, and the entropy weight method is chosen as the calculation method for objective weights. Section “ANP calculation method” presents the ANP calculation method, and Section “Entropy weight method calculation method” introduces the entropy weight method calculation method.

ANP calculation method

ANP is a decision-making methodology for non-independent hierarchical structures, introduced by Professor T.L. Saaty of the University of Pittsburgh in 2004³¹. Building on the previously developed AHP³² by Satty in 1994, ANP not only takes into account the hierarchical structure but also comprehensively considers the relationships among criteria. In fact, ANP constructs an intricate network of relationships among various criteria. This unique feature enables ANP to generate more reliable and accurate results, offering a more in-depth and nuanced analysis compared to traditional AHP³³. The calculation of subjective weights for evaluation indicators using ANP encompasses the following steps.

1.

Construction of Pairwise Comparison Matrices.

Conduct pairwise comparisons of sub-criteria under each decision criterion and alternatives under each sub-criterion. Determine comparison values and generate multiple comparison matrices.

2.

Construction of Unweighted Supermatrix.

Compute the maximum eigenvalue and normalize the eigenvector for each comparison matrix to construct the unweighted supermatrix.

The sorting vector is calculated by the characteristic root method. $\left( {\mathop w\nolimits_{{i1}}^{{j1}} } \right., \mathop w\nolimits_{{i2}}^{{j1}} …..\left. {\mathop w\nolimits_{{ini}}^{{j1}} } \right)$ is denoted as ${\text{W}_{ij}}$:

$${\text{Wij}}=\left[ {\begin{array}{*{20}{c}} {\mathop w\nolimits_{{i1}}^{{j1}} }&{\mathop w\nolimits_{{i2}}^{{j2}} }&{…}&{\mathop w\nolimits_{{i1}}^{{jnj}} } \\ {\mathop w\nolimits_{{i2}}^{{j1}} }&{\mathop w\nolimits_{{i2}}^{{j2}} }&{…}&{\mathop w\nolimits_{{i2}}^{{jnj}} } \\ {…}&{…}&{…}&{…} \\ {\mathop w\nolimits_{{ini}}^{{j1}} }&{\mathop w\nolimits_{{ini}}^{{j2}} }&{…}&{\mathop w\nolimits_{{ini}}^{{jnj}} } \end{array}} \right]$$

(1)

The column vectors of ${\text{W}_{ij}}$ are the sorting vectors of the influence degree of evaluation index ${\text{R}}_{ini}$ on evaluation index ${\text{R}_{jnj}}$. If ${\text{R}}_{ini}$ has no influence on ${\text{R}}_{ini}$, then ${\text{W}_{ij}}$ = 0 (i = 1,2,…N; j = 1,2,…N) (;). The unweighted supermatrix for evaluating mutual influence is shown in Eq. (2):

(2)

3.

Construction of Weighted Matrix.

Multiply the normalized matrix of each level by the corresponding weight to calculate the weighted supermatrix. Conduct an importance comparison between the evaluation index layer ${\text{R}}_i$ and the secondary criterion evaluation index layer ${\text{R}_{j}}$. Then, construct the judgment matrix, calculate the eigenvector, and perform a consistency check. The sorting vector of the evaluation index group irrelevant to ${\text{R}}_j$ is set to 0, and the weighted matrix A is obtained:

$${\text{A}}=\begin{array}{*{20}{c}} {\left[ {\begin{array}{*{20}{c}} {a11} \\ {a21} \\ \vdots \\ {ai1} \\ \vdots \\ {aN1} \end{array}} \right.}&{\begin{array}{*{20}{c}} {a12} \\ {a22} \\ \vdots \\ {ai2} \\ \vdots \\ {aN2} \end{array}}&{\begin{array}{*{20}{c}} \cdots \\ \cdots \\ \vdots \\ \cdots \\ \vdots \\ \cdots \end{array}}&{\begin{array}{*{20}{c}} {a1j} \\ {a2j} \\ \vdots \\ {aij} \\ \vdots \\ {aNJ} \end{array}}&{\begin{array}{*{20}{c}} \cdots \\ \cdots \\ \vdots \\ \cdots \\ \vdots \\ \cdots \end{array}}&{\left. {\begin{array}{*{20}{c}} {a1N} \\ {a2N} \\ \vdots \\ {aiN} \\ \vdots \\ {aNN} \end{array}} \right]} \end{array}$$

(3)

Weight the factors in the supermatrix W to obtain the weighted supermatrix:

$${\text{w}}=\begin{array}{*{20}{c}} {\left[ {\begin{array}{*{20}{c}} {a11{\text{w}}11} \\ {a21{\text{w}}12} \\ \vdots \\ {ai1{\text{w}}i1} \\ \vdots \\ {aN1{\text{w}}N1} \end{array}} \right.}&{\begin{array}{*{20}{c}} {a12{\text{w}}12} \\ {a22{\text{w}}22} \\ \vdots \\ {ai2{\text{w}}i2} \\ \vdots \\ {aN2{\text{w}}N2} \end{array}}&{\begin{array}{*{20}{c}} \cdots \\ \cdots \\ \vdots \\ \cdots \\ \vdots \\ \cdots \end{array}}&{\begin{array}{*{20}{c}} {a1j{\text{w}}1j} \\ {a2j{\text{w}}2j} \\ \vdots \\ {aij{\text{w}}ij} \\ \vdots \\ {aNJ{\text{w}}Nj} \end{array}}&{\begin{array}{*{20}{c}} \cdots \\ \cdots \\ \vdots \\ \cdots \\ \vdots \\ \cdots \end{array}}&{\left. {\begin{array}{*{20}{c}} {a1N{\text{w}}1N} \\ {a2N{\text{w}}2N} \\ \vdots \\ {aiN{\text{w}}iN} \\ \vdots \\ {aNN{\text{w}}NN} \end{array}} \right]} \end{array}$$

(4)

4.

Calculation of Indicator Weights.

Calculate the geometric mean of the weighted supermatrix, perform limit processing to obtain the limit supermatrix, and integrate it with the control layer weights to compute the global priority of each indicator $W’=\left( {W11,W12, \cdots ,Wini \cdots {{\left. {WNnN} \right)}^T}} \right.$, yielding the weights of prioritized indicators.

$$\mathop W\nolimits_{i}^{\prime } =\left( {\mathop W\nolimits_{{i1}}^{\prime } ,\mathop W\nolimits_{{i2}}^{\prime } , \cdots {{\left. {\mathop W\nolimits_{{ini}}^{\prime } } \right)}^T}} \right.$$

(5)

5.

Consistency Check.

Perform a consistency ratio (CR) calculation based on the consistency check algorithm using indicator weights. A CR value less than 0.1 indicates passage of the consistency check, validating the final indicator weights.

$$CI=\frac{{\lambda \hbox{max} – n}}{{n – 1}}$$

(6)

$${\text{CR}}=\frac{{CI}}{{RI}}$$

(7)

Where, CI is the consistency index, which is the ratio of the difference between the maximum eigenvalue of the judgment matrix and n (n is the order of the judgment matrix) to (n-1). RI is the average random consistency index of this order.

Entropy weight method calculation method

The entropy weight method is an objective method of assignment, which measures the weight of each indicator by information entropy. The level of indicators is quantified by entropy, and littler information entropy leads to larger discrete degree of the indicators, and the indicators have a bigger impact on the comprehensive evaluation³⁴. The entropy weight method, derived from the concept of information entropy, stands out from traditional weighting approaches. It doesn’t rely on subjective judgment or expert expertise, thus effectively mitigating the influence of subjective bias. Moreover, instead of treating elements as independent variables, it takes into account the correlations and interactions among them. This allows it to accurately capture the level of uncertainty and complexity in the information, providing a more comprehensive evaluation of the significance of decision factors^35,36. The calculation of objective weights for evaluation indicators comprises the following steps.

1.

Construction of Evaluation Matrix and Data Standardization.

Based on the scores given by experts and the data from preliminary research, a unified quantitative value range for each indicator is determined. The proportion results of different types of subjective and objective data are statistically analyzed. According to the proportion, the subjective and objective data are mapped to the determined unified scores to form standardized numerical data.

$$\text{Positive Indicator:}\quad {\text{V}}_{ij}=\frac{{Xij – X_{\hbox{min} }}}{{X_{\hbox{max}} – X_{\hbox{min}} }}$$

(8)

$$\text{Negative Indicator}\quad{\text{V}}ij=\frac{{X_{\hbox{max}} – Xij}}{{X_{\hbox{max}} – X_{\hbox{min} }}}$$

(9)

Where, X_max is the maximum value of the i-th indicator, and X_min is the minimum value of the i-th indicator.

Perform normalization processing to obtain the standard matrix:

$${\text{Pij}}=\frac{{Vij}}{{\sum\nolimits_{{i=1}}^{m} {Vij} }}$$

(10)

Where, V_ij represents the standardized data matrix of positive indicators and negative indicators.

2.

Calculation of Entropy Values and Corresponding Entropy Weights for Each Indicator.

$${\text{ej}}=-{\text{k}}\sum\limits_{{{\text{i}}={\text{1}}}}^{{\text{m}}} {{\text{P}}_{{ij}} \text{lnP}_{ij}}$$

(11)

$${\text{k}}=\frac{1}{{\ln m}}$$

(12)

Where k denotes the calculation coefficient, m represents the number of data points, and P_ij signifies the normalized data matrix.

Calculate the indicator difference coefficient:

$${\text{gj}}=1 – ej$$

(13)

Where ej represents information entropy.

Calculate indicator weights:

$${\text{Sj}}=\frac{{gj}}{{\sum\nolimits_{{j=1}}^{m} {gj} }}$$

(14)

Calculation of the combined weights

The game theory-based combinational weighting method is a technique used to integrate weights derived from multiple methods to determine optimal weight values³⁷. Drawing on the concept of combination weighting from game theory, linearly combine the subjective weights derived from ANP with the objective weights obtained from the entropy weight method to compute the combined weights.

$${{\text{W}}_{\text{i}}}{\text{ = a}}*{\text{ }}{{\text{W}}_{{\text{ANP}}}}+\left( {{\text{1}} – {\text{a}}} \right)*{{\text{W}}_{{\text{Entropy Method}}}}$$

(15)

Where a represents the sensitivity coefficient. Acknowledging that both subjective and objective weights significantly impact the results, a is set to 0.5. W_ANP denotes the indicator weight matrix obtained from ANP, while W_{Entropy Method} signifies the indicator weight matrix derived from the entropy weight method.

Evaluation classification based on the improved normal cloud model

This section is dedicated to an in-depth introduction of the improved normal cloud model. Initially, the hierarchical clustering method is employed to categorize the measurement data samples. This step is crucial as it guarantees the objective determination of the index evaluation criteria within the normal cloud model, thereby enhancing the reliability and validity of the subsequent analysis. Subsequently, the normal cloud model is utilized to compute the index membership values, which play a fundamental role in the comprehensive evaluation process.

Specifically, Section “Construction of index evaluation matrix using hierarchical clustering method” elaborates on the hierarchical clustering method, covering its theoretical basis, operational procedures, and practical applications in the context of this research. Section “Improved normal cloud model” delves into the improved normal cloud model, exploring its innovative features, mathematical formulations, and how it addresses the limitations of the traditional model. This detailed presentation aims to provide readers with a clear understanding of the methodological framework and enable them to apply these techniques effectively in relevant research and practical scenarios.

Construction of index evaluation matrix using hierarchical clustering method

To ensure the objectivity of the predetermined evaluation scale for indicators, This research employs a hierarchical clustering algorithm³⁸. Hierarchical clustering algorithm is a commonly used unsupervised learning algorithm that partitions a dataset into clusters at multiple levels³⁹. The hierarchical clustering algorithm has a distinct feature compared to K-means clustering: it eliminates the necessity of predefining the number of clusters. While K-means clustering demands that the number of clusters be determined beforehand, the hierarchical clustering algorithm is more adaptable as it doesn’t rely on such a pre-set parameter, enabling a more natural and data-centric clustering process⁴⁰. Rather than relying on a pre-determined number of clusters like some other methods, it forges a hierarchical framework through the computation of the similarity or distance among samples. This approach allows the algorithm to gradually group samples based on their inherent relationships, revealing the underlying data structure in a more organic way.

The measurement data samples are clustered into five categories, thereby defining the evaluation grade standards for tertiary indicators as “Excellent,” “Good,” “Average,” “Poor,” and “Very Poor.”

1.

Initialization.

Select one of the tertiary indicators and regard each sample data of it as a separate cluster. If there are n data points, there will be n clusters accordingly.

2.

Calculate the distance.

Compute the distance matrix among all the cluster centers. In the case of Single-linkage, the cluster center is exactly the data point itself.

$${\text{d}}(C1,C2)=\hbox{min} Xa \in C1,Xb \in C2d(Xa,Xb)$$

(16)

Where, d(xₐ, x_β) represents the Euclidean distance between the data points xₐ and x_β.

3.

Merge Clusters.

Locate the two clusters that correspond to the smallest distance within the distance matrix, and then combine these two clusters into a brand-new cluster.

4.

Update the Cluster Matrix.

Remove the rows and columns associated with the two pre-merged clusters from the distance matrix. Then, add the distances between the newly formed cluster and all other existing clusters to the new distance matrix.

$${\text{d}}(C{\text{n}},Cj)=\hbox{min}\; Xc \in Cn,Xd \in Cjd(Xc,Xd)$$

(17)

Where, x^m and xₔ are the data points in C_n and C_j respectively.

5.

Repeat steps 3 and 4 until the number of clusters reaches 5, at which point the loop terminates. The specific evaluation standards are presented in the Table 3.

Table 3 Representative examples of evaluation standards for the third-level indicators.

Improved normal cloud model

The normal cloud model is a qualitative and quantitative uncertainty conversion model proposed on the basis of traditional fuzzy set theory and probability statistics¹⁰. It uniformly characterizes the randomness and fuzziness between uncertain linguistic values and precise numerical values. The normal cloud model is widely used in system evaluation and other fields, such as, in the field of wind turbine state evaluation⁴¹, and it has achieved good results in solving uncertainty problems.

This research enhances the normal cloud model by incorporating the indicator evaluation matrix obtained through hierarchical clustering. For each evaluation indicator, a specified number of cloud drops are generated. The membership degree of each cloud drop is calculated for each evaluation grade. Cross-validation is performed on each indicator’s cloud model, adjusting and optimizing the corresponding normal cloud model parameters. This process ultimately yields precise membership degree values for each evaluation indicator across various evaluation grades. Finally, the overall evaluation grade of the enterprise information system is determined based on the combined weights.

1.

Determining Normal Cloud Parameters: Expectation(Ex), Entropy(En), and Hyper-entrop(He).

The expectation(Ex) is calculated using the mean value, entropy(En) is computed using the probability density function of the Gaussian fuzzy factor, and hyper-entropy (He) is determined using the variance formula for random variables.

(1)

Due to the diversity of the types of tertiary indicators, in order to maintain consistency, the mean value is used to calculate the expected value Ex:

$${\text{Ex}}=\frac{1}{n}\sum\limits_{{i=1}}^{n} {Xi}$$

(18)

Where, Xi represents the i-th collected value in the single indicator dataset, and n represents the total number of samples.

(2)

In order to effectively measure the uncertainty and fuzziness of each index data, the probability density function of the Gaussian fuzzy factor is used to calculate the entropy En:

$${\text{En}}= – \sum\limits_{{i=1}}^{n} {p(Xi)\log bP(Xi)}$$

(19)

Where, ${\text{p}}(xi)$ is the probability density function of the i-th value in the single indicator X set based on the Gaussian fuzzy factor, b represents the base of the logarithm, usually b = 2, and n represents the total number of samples.

(3)

It provides a unified measurement scale for the degree of dispersion of different distributions, which is used to compare the volatility of different datasets or probability distributions. The variance formula of random variables is adopted to calculate the hyper-entropy He:

$${\text{H}}e \approx \sqrt {\sum\limits_{{i=1}}^{n} {p(xi)(xi – Ex\mathop )\nolimits^{2} } }$$

(20)

Where, p(xi) is the probability density function of the i-th value in the single indicator X set based on the Gaussian fuzzy factor, and n represents the total number of samples.

2.

Constructing Normal Cloud Model for Each Indicator.

Based on the determined parameters (Ex, En, He), a normal distribution function is used to generate a specified number of cloud drops for each indicator, thus simulating the uncertainty of that indicator. The cloud drops of each indicator are multiplied by the corresponding evaluation grade membership matrix to obtain the membership degree of each indicator for each evaluation grade.

3.

Cross-validation of Normal Cloud Model.

To enhance the performance of the indicator cloud model, cross-validation is conducted for each indicator’s cloud model, adjusting and optimizing the expectation(Ex), entropy(En), and hyper-entropy(He).

4.

Calculating Membership Degree Values.

The membership degree of each indicator is multiplied by its combined weight to obtain the weighted membership degree. The weighted membership degrees for all evaluation grades are summed to derive the comprehensive evaluation value of the system, representing its membership degree values for the five grades: “Excellent,” “Good,” “Average,” “Poor,” and “Very Poor.” The grade with the highest membership value is designated as the system’s comprehensive rating.

$${\text{Bi}}=w(z1,z2,z3) * \operatorname{Re} i$$

(21)

Where $w(z1,z2,z3)$ represents the combined indicator weight, and $\operatorname{Re} i$ denotes the membership degree of each indicator.

link