Software Evaluation inAI

The Selection Problem

• Recency of products

• Evolving nature of products

• Variety of products

• Lack of standards

Common Approach to Choice

• Select a few seemingly important or complicated functions within the products, and compare the products based on them.

Shortcomings

• It is evaluator-specific and unreliable.

• No product could be judged by only a few attributes.

• It excludes the relevant issues in a particular application.

• It lacks an evaluation measure, reflecting the strength of a product or its ratings compared with its competitors.

A Structured Approach

• Contain all of the important features of AI products

• Make it possible to compare different types of products

• Accommodate the special needs and requirements of AI project

• Summarize the results of evaluation into a quantitative or qualitative measure

Attribute Hierarchy of AI Tools

• Financial Aspects

• Producer Aspects

• Special Aspects

• Hardware Aspects

• Functional Aspects

Financial Aspects

• One-time costs: purchasing cost

• Periodic costs: licensing, training, and maintenance

Producer Aspects

• Reputation

• Length of time in business

• Product line: compatibility issues

• Technical support

Social Aspects

• User group

• Number of users

• Compatible products: technical leadership of a product

Hardware Aspects

• Platform: OS, Networking, Parallel Processing

• I/O Devices

• Resource Requirements: RAM, Disk Storage …

• Efficiency: Response Time

Functional Aspects

• Knowledge Representation

• Inference Engine

• Knowledge Management

• Outside Hooks

Knowledge Representation

• Logic-Based Representation: Rule, Fuzzy Logic• Object-Based Representation: Frame, Semantic

Net• Uncertainty Representation: Bayesian, Fuzzy

Logic, Certainty Factors, User-Defined• Meta-Knowledge Representation• Mathematical Representation: Math Operations,

Math Functions, Variables

Inference Engine

• Chaining: Forward, Backward, Mixed

• Induction: Dec. Tree

• Object-Oriented: Single vs. Multiple Inheritance

• Blackboard

• Conflict Resolution: Recency, Antecedent Ordered, Consequent Ordered, Top-Down

Knowledge Mgt. Tools

• User-Interface Creation: Graphics, Windows, NLP, Voice Input/Output, Help, Animation

• User Interface: How, Why, Graphics, Uncertainty I/O, Help

• Debugging Tools: Tracing, Error Messages

• Knowledge Maintenance: Editor, Menu, Mouse,

Knowledge Mgt. Tools Continued

• System Security: User Access, KB Control

• Integrated Tools: Databases, Spreadsheets, Forms, Files, Prog. Languages

• Developer - User Assistance: On-Line Documents, Off-Line Documents, Tutorials, Error-Message References

Outside Hooks Components

• Access Data: Database, Files, Spreadsheets

• Text Access: Reports, Forms, Word Processing

• Knowledge Base Access: Multiple Access, Concurrent Access to Different KBs

• Language Access: Access to Different Languages

Outside Hooks Components Continued

• Portability: Exporting to Other Platforms, Generating Standard Files, like ASCII

Legal Issues

• Data Processing Services Inc. v. L. H. Smith Oil Corp. : The Indiana Court of Appeals upheld a lower court’s verdict that Data Processing Services was liable for professional malpractice.

Legal Issues

• Quad County Distributing Co. v. Burroughs Corp.: This case is significant because the court held that a computer program is covered by the UCC provisions concerning the sale of products. This means that the injured party does not have to prove negligence on the part of manufacturer to recover damages.

Legal Issues

• Users

• Domain Experts

• Knowledge Engineers

• Seller Organizations

Users

• Who is responsible if a decision maker uses a defective expert system to make a decision that leads to damage? There are cases that the user could be held responsible even though he/she has been unaware of the fault. For example, when a software error caused a machine to dispense a lethal dose of radiation to a patient, the doctor was sued alongside the manufacturer of the machine and the institution where the machine was being used.

Users

• On the other hand, there could be consequences for NOT using an available an available system. Take the case of a nurse who does not have access to a doctor, and chooses not to use an expert system that could save a patient’s life. Is the nurse liable for negligence?

• Need for procedures?

Domain Experts

• Experts who do not have adequate expertise ti stand the test of a court challenge should altogether avoid getting involved in the development of the system.

Knowledge Engineer

• The knowledge engineer could damage the integrity of the knowledge base by his personal biases, negligence, and lack of understanding of the knowledge domain. The documentation process and paper trail of the knowledge engineering process would be of critical importance in auditing the quality of knowledge engineering.

Seller

• If Turbo Tax program is defective and gives wrong tax advice that leads to financial losses, is the company liable?

Seller

• Almost all software companies have limited warranties. As long as a software system is considered a product, such warranties would protect the seller. However, when the system is considered to be providing a service to customers, then such disclaimers could not prevent litigation.

Some Questions...

• Can management force employees to contribute their expertise?

• What is the value of an expert opinion in court when the expertise is encoded in a computer?

• Who is liable for wrong information provided by an ES?

• Who owns the knowledge in KB?