Purposes of Follow-Up Monitoring & Evaluation

Evaluation not for measurement of project’s goals,for accountability and improvement of organization

What’s Follow-up Monitoring & Evaluation?

We would like to find the contribution/impacts of projects to society for ACCOUNTABILITY, and seek out points that should be improved in our PROJECT MANAGEMENT, and also seek out needs of follow-up/new projects for POLICY PLANNING!

What’s the subject of it?

All R&D projects are subjects, except for

international cooperation /dissemination of new energy

projects!

American Evaluation Association ConferenceOctober 28, 2005

Hiroshi Sano, sanohrs@nedo.go.jpShuji Yumitori, yumitorisji@nedo.go.jp

Kazuaki Kohmoto, Kohmotokza@nedo.go.jp

New Energy and Industrial Technology Development Organization (NEDO)

http://www.nedo.go.jp/english/index.html

Follow-Up Monitoring & Evaluation Systemsfor National R&D at NEDO 　 Part II

～ Feedback for Management ～

☞ 　 Purpose 　☞ 　 A Need for the New Tool (Follow-up Chart) 　☞ 　 Case Studies

☞ 　 Feedback for Management

☞ 　 Conclusions

Contents

Objective

Derive concrete implications to improve R&D management methods.

Follow major events throughout the projectand understand key points on management or the reason why certain events happened through interviews.

Statistical data didn’t help to derive concrete implications. For example, it would be too boldto conclude from the following graph that we shouldexclude companies whose objective for participation were improvement of their technical level.

A Need for New ToolUnderstanding when certain events happened or the magnitude of each event in detail solely by interviews was particularly difficult.

0% 20% 40% 60% 80% 100%

Matched a purpose of the organization

Challenge a new

technical target

Invitation of other

organization

Matched a purpose of the organization

Challenge a new technical

target

Invitation of other

organization

Improvement of

technology

Improvement of

technology

Create network

Create network

Personnel training

Other

Personnel training

Other

Practical application & commercialization

(n=78)No post-project activity

(n=246)

※ Multiple answers allowed

Objective for participation

Follow-up ChartMethodologies

☞ Conduct a interview to a company about major events on the project and influences of management.☞ Plot the above information in a conceptual chart. ・ The horizontal axis is the time from the beginning　　　 to present. ・ The vertical axis is possibilities or expectation for practical use of technology. (The vertical axis is interviewee’s personal opinion because to measure the possibilities was difficult.)

☞ Determine the form of the chart through the interview.☞ Understand the positive/negative factors which would affect the possibility of practical use of the R&D results through the interview.

Time(-)

Large

Possibilities/expectationfor practical use

Follow-up Chart (Image)～ Status from the beginning ～

Plan-ning Present

1styear

Mid. term Evaluation

Mid.

Small

2styear

3rdyear

4thyear

5thyear

End term Evaluation

Follow the events at the turning points and influ- ence/effect of management for practical use.

: Turning point

◆ It is difficult to find a fixed quantity of possibilities. The vertical axis is based on interviewee’s opinion.

Case Studies

Large

Plan-ning Present

1st year

2nd year 3rd year Final year

④Project goals were high. A list of targets was created to demonstrate what needed to be accomplished. Although there was hope to collect necessary materials, odds for success seemed low due to legal issues.

☆Factors that raised the possibilities of practical use. Red： by NEDO　　 Italic： by except for NEDO　① Development of bold approach. (Relation to )⑤　② Large scale demonstration test.　③ Influential figure.　④ High level targets.　⑤ Old approach was examined thoroughly (Relation to )①　⑥ Government spurred industry.　⑦ Key person who was integrate the R&D/business unit.

①A bold technical approach was developed.(Relation to )⑤

③There was an influential figure in NEDO. This particular individual persuaded the technical committee and the government to adapt to the new technical approach.

②The trial productions were tested by real users, which led to the grater possibilities for commercialization. This is due to a large scale demonstration test that earned user's confidence.

⑥Under the leadership of METI, the demand promotion committee was established, which helped spur the industry.

⑦The R&D went smoothly because there was a liaison personnel who was able to integrate the R&D and business unit.

●It became clear that the method did not go well technically. In addition, the law did not allow the use of materials that they were counting on.

⑤The old technical approach was examined thoroughly for six months. This helped achieve the development of the prominent new approach. . （ Relation to ① ）

Possibilities/expectationfor practical use

Mid.

Small

Time

Case Study A

●Demand has reached its critical level. Measures under government leadership are desirable.

TimePlanning

Present1st year 2nd year 3rd year 4th year Final year End term

evaluation

Various trial & error testing methods were available.

Efficiency and performance increased threefold.

Appropriate evaluation based on the results.Budget allocation based on mid-term evaluation.

Collaboration scheme went well in the basic R&D phase.

Collaboration between basic science, electronic board and device group was good.

Setting goals emphasized on “The World’s first” and “The World’s best” to attempt to be the World’s #1.

Solid relationship with universities.

Universities acquired the same equipments as companies which fostered data analysis and evaluation.

Modification of company organization to identify business opportunities.Company executives attended conferences to present project results, which helped them to create a stronger network with other attendees. This helped win support for the project within the company.

Med. TermEvaluation

Attainment of “World’s best” data. 　　　　　　　　　　　　　

Collaboration scheme is not always beneficial.

Well organized collaboration system in the basic R&D phase became an obstacle in the practical R&D phase.

Emergence of technical problem for practical use. Technology for stable/mass production, cost reduction, patent problems etc.

From research phase to practical phase.

Well planed collaboration scheme (Producer and Consumer, etc.) 　 Important information such as spec or permission range of quality was obtained.

Enhanced business opportunity by human network.

Capitalization on human network acquired from the project. 　　

Positive factor

Negative factor

Color Indication

Human network

Environment was favorable for a company to enter into a new tech. field.Collaboration with universities and/or user companies was useful.

Possibilities/expectationfor practical use

Large

Mid.

Small

Case Study B

Consideration

☞ Positive/negative effects and the magni- tude of events.

☞ Change of personal opinion concerning possibilities for practical use.

☞ Status of R&D, business and social environment.

Follow-up chart shows on a simple peace of paper:

◆ (Attention) - Selection of the items or magnitude of the events that were recorded on the chart is influenced by interviewee’s personal opinions. - Personal opinions determine the shape of the chart.

Feedback for Management (Image)

BudgetRequest

CheckList

Concrete Implications・ Does the project earn user’s confidence? (from case study A)

・ Do you have backup schemes for situations where a company ente-

ring a new business/tech. field joins the project? (from case study B)

GoalSetting

CheckList

Mid. termEvaluation

MemberSetting

CheckList

CheckList

G u i d e l i n e

Conclusions

Follow-up chart is a useful tool for improving management.

Follow-up chart is based on personal opinion. (Further consideration will be needed.)

☞ Follow major events throughout the project and understand key points on management or the reason why certain events happened by hearing.

☞ Derive concrete implications on project management.

OK

OK

Finally

I have introduced how to feedback of follow-up evaluation for management.

Today, I would like to know about your activityabout feedback for management.

I would be happy if I could acquire some suggestions about the vertical axis.

Thank you for your attention.