![Page 1: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/1.jpg)
Software Evaluation inAI
![Page 2: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/2.jpg)
The Selection Problem
• Recency of products
• Evolving nature of products
• Variety of products
• Lack of standards
![Page 3: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/3.jpg)
Common Approach to Choice
• Select a few seemingly important or complicated functions within the products, and compare the products based on them.
![Page 4: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/4.jpg)
Shortcomings
• It is evaluator-specific and unreliable.
• No product could be judged by only a few attributes.
• It excludes the relevant issues in a particular application.
• It lacks an evaluation measure, reflecting the strength of a product or its ratings compared with its competitors.
![Page 5: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/5.jpg)
A Structured Approach
• Contain all of the important features of AI products
• Make it possible to compare different types of products
• Accommodate the special needs and requirements of AI project
• Summarize the results of evaluation into a quantitative or qualitative measure
![Page 6: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/6.jpg)
Attribute Hierarchy of AI Tools
• Financial Aspects
• Producer Aspects
• Special Aspects
• Hardware Aspects
• Functional Aspects
![Page 7: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/7.jpg)
Financial Aspects
• One-time costs: purchasing cost
• Periodic costs: licensing, training, and maintenance
![Page 8: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/8.jpg)
Producer Aspects
• Reputation
• Length of time in business
• Product line: compatibility issues
• Technical support
![Page 9: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/9.jpg)
Social Aspects
• User group
• Number of users
• Compatible products: technical leadership of a product
![Page 10: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/10.jpg)
Hardware Aspects
• Platform: OS, Networking, Parallel Processing
• I/O Devices
• Resource Requirements: RAM, Disk Storage …
• Efficiency: Response Time
![Page 11: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/11.jpg)
Functional Aspects
• Knowledge Representation
• Inference Engine
• Knowledge Management
• Outside Hooks
![Page 12: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/12.jpg)
Knowledge Representation
• Logic-Based Representation: Rule, Fuzzy Logic• Object-Based Representation: Frame, Semantic
Net• Uncertainty Representation: Bayesian, Fuzzy
Logic, Certainty Factors, User-Defined• Meta-Knowledge Representation• Mathematical Representation: Math Operations,
Math Functions, Variables
![Page 13: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/13.jpg)
Inference Engine
• Chaining: Forward, Backward, Mixed
• Induction: Dec. Tree
• Object-Oriented: Single vs. Multiple Inheritance
• Blackboard
• Conflict Resolution: Recency, Antecedent Ordered, Consequent Ordered, Top-Down
![Page 14: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/14.jpg)
Knowledge Mgt. Tools
• User-Interface Creation: Graphics, Windows, NLP, Voice Input/Output, Help, Animation
• User Interface: How, Why, Graphics, Uncertainty I/O, Help
• Debugging Tools: Tracing, Error Messages
• Knowledge Maintenance: Editor, Menu, Mouse,
![Page 15: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/15.jpg)
Knowledge Mgt. Tools Continued
• System Security: User Access, KB Control
• Integrated Tools: Databases, Spreadsheets, Forms, Files, Prog. Languages
• Developer - User Assistance: On-Line Documents, Off-Line Documents, Tutorials, Error-Message References
![Page 16: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/16.jpg)
Outside Hooks Components
• Access Data: Database, Files, Spreadsheets
• Text Access: Reports, Forms, Word Processing
• Knowledge Base Access: Multiple Access, Concurrent Access to Different KBs
• Language Access: Access to Different Languages
![Page 17: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/17.jpg)
Outside Hooks Components Continued
• Portability: Exporting to Other Platforms, Generating Standard Files, like ASCII
![Page 18: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/18.jpg)
Legal Issues
• Data Processing Services Inc. v. L. H. Smith Oil Corp. : The Indiana Court of Appeals upheld a lower court’s verdict that Data Processing Services was liable for professional malpractice.
![Page 19: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/19.jpg)
Legal Issues
• Quad County Distributing Co. v. Burroughs Corp.: This case is significant because the court held that a computer program is covered by the UCC provisions concerning the sale of products. This means that the injured party does not have to prove negligence on the part of manufacturer to recover damages.
![Page 20: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/20.jpg)
Legal Issues
• Users
• Domain Experts
• Knowledge Engineers
• Seller Organizations
![Page 21: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/21.jpg)
Users
• Who is responsible if a decision maker uses a defective expert system to make a decision that leads to damage? There are cases that the user could be held responsible even though he/she has been unaware of the fault. For example, when a software error caused a machine to dispense a lethal dose of radiation to a patient, the doctor was sued alongside the manufacturer of the machine and the institution where the machine was being used.
![Page 22: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/22.jpg)
Users
• On the other hand, there could be consequences for NOT using an available an available system. Take the case of a nurse who does not have access to a doctor, and chooses not to use an expert system that could save a patient’s life. Is the nurse liable for negligence?
• Need for procedures?
![Page 23: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/23.jpg)
Domain Experts
• Experts who do not have adequate expertise ti stand the test of a court challenge should altogether avoid getting involved in the development of the system.
![Page 24: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/24.jpg)
Knowledge Engineer
• The knowledge engineer could damage the integrity of the knowledge base by his personal biases, negligence, and lack of understanding of the knowledge domain. The documentation process and paper trail of the knowledge engineering process would be of critical importance in auditing the quality of knowledge engineering.
![Page 25: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/25.jpg)
Seller
• If Turbo Tax program is defective and gives wrong tax advice that leads to financial losses, is the company liable?
![Page 26: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/26.jpg)
Seller
• Almost all software companies have limited warranties. As long as a software system is considered a product, such warranties would protect the seller. However, when the system is considered to be providing a service to customers, then such disclaimers could not prevent litigation.
![Page 27: Software Evaluation in AI. The Selection Problem Recency of products Evolving nature of products Variety of products Lack of standards](https://reader036.vdocuments.site/reader036/viewer/2022070401/56649f1e5503460f94c36353/html5/thumbnails/27.jpg)
Some Questions...
• Can management force employees to contribute their expertise?
• What is the value of an expert opinion in court when the expertise is encoded in a computer?
• Who is liable for wrong information provided by an ES?
• Who owns the knowledge in KB?