Purposes of Follow-Up Monitoring & Evaluation
Evaluation not for measurement of project’s goals,for accountability and improvement of organization
What’s Follow-up Monitoring & Evaluation?
We would like to find the contribution/impacts of projects to society for ACCOUNTABILITY, and seek out points that should be improved in our PROJECT MANAGEMENT, and also seek out needs of follow-up/new projects for POLICY PLANNING!
What’s the subject of it?
All R&D projects are subjects, except for
international cooperation /dissemination of new energy
projects!
American Evaluation Association ConferenceOctober 28, 2005
Hiroshi Sano, [email protected] Yumitori, [email protected]
Kazuaki Kohmoto, [email protected]
New Energy and Industrial Technology Development Organization (NEDO)
http://www.nedo.go.jp/english/index.html
Follow-Up Monitoring & Evaluation Systemsfor National R&D at NEDO Part II
~ Feedback for Management ~
☞ Purpose ☞ A Need for the New Tool (Follow-up Chart) ☞ Case Studies
☞ Feedback for Management
☞ Conclusions
Contents
Objective
Derive concrete implications to improve R&D management methods.
Follow major events throughout the projectand understand key points on management or the reason why certain events happened through interviews.
Statistical data didn’t help to derive concrete implications. For example, it would be too boldto conclude from the following graph that we shouldexclude companies whose objective for participation were improvement of their technical level.
A Need for New ToolUnderstanding when certain events happened or the magnitude of each event in detail solely by interviews was particularly difficult.
0% 20% 40% 60% 80% 100%
Matched a purpose of the organization
Challenge a new
technical target
Invitation of other
organization
Matched a purpose of the organization
Challenge a new technical
target
Invitation of other
organization
Improvement of
technology
Improvement of
technology
Create network
Create network
Personnel training
Other
Personnel training
Other
Practical application & commercialization
(n=78)No post-project activity
(n=246)
※ Multiple answers allowed
Objective for participation
Follow-up ChartMethodologies
☞ Conduct a interview to a company about major events on the project and influences of management.☞ Plot the above information in a conceptual chart. ・ The horizontal axis is the time from the beginning to present. ・ The vertical axis is possibilities or expectation for practical use of technology. (The vertical axis is interviewee’s personal opinion because to measure the possibilities was difficult.)
☞ Determine the form of the chart through the interview.☞ Understand the positive/negative factors which would affect the possibility of practical use of the R&D results through the interview.
Time(-)
Large
Possibilities/expectationfor practical use
Follow-up Chart (Image)~ Status from the beginning ~
Plan-ning Present
1styear
Mid. term Evaluation
Mid.
Small
2styear
3rdyear
4thyear
5thyear
End term Evaluation
Follow the events at the turning points and influ- ence/effect of management for practical use.
: Turning point
◆ It is difficult to find a fixed quantity of possibilities. The vertical axis is based on interviewee’s opinion.
Case Studies
Large
Plan-ning Present
1st year
2nd year 3rd year Final year
④Project goals were high. A list of targets was created to demonstrate what needed to be accomplished. Although there was hope to collect necessary materials, odds for success seemed low due to legal issues.
☆Factors that raised the possibilities of practical use. Red: by NEDO Italic: by except for NEDO ① Development of bold approach. (Relation to )⑤ ② Large scale demonstration test. ③ Influential figure. ④ High level targets. ⑤ Old approach was examined thoroughly (Relation to )① ⑥ Government spurred industry. ⑦ Key person who was integrate the R&D/business unit.
①A bold technical approach was developed.(Relation to )⑤
③There was an influential figure in NEDO. This particular individual persuaded the technical committee and the government to adapt to the new technical approach.
②The trial productions were tested by real users, which led to the grater possibilities for commercialization. This is due to a large scale demonstration test that earned user's confidence.
⑥Under the leadership of METI, the demand promotion committee was established, which helped spur the industry.
⑦The R&D went smoothly because there was a liaison personnel who was able to integrate the R&D and business unit.
●It became clear that the method did not go well technically. In addition, the law did not allow the use of materials that they were counting on.
⑤The old technical approach was examined thoroughly for six months. This helped achieve the development of the prominent new approach. . ( Relation to ① )
Possibilities/expectationfor practical use
Mid.
Small
Time
Case Study A
●Demand has reached its critical level. Measures under government leadership are desirable.
TimePlanning
Present1st year 2nd year 3rd year 4th year Final year End term
evaluation
Various trial & error testing methods were available.
Efficiency and performance increased threefold.
Appropriate evaluation based on the results.Budget allocation based on mid-term evaluation.
Collaboration scheme went well in the basic R&D phase.
Collaboration between basic science, electronic board and device group was good.
Setting goals emphasized on “The World’s first” and “The World’s best” to attempt to be the World’s #1.
Solid relationship with universities.
Universities acquired the same equipments as companies which fostered data analysis and evaluation.
Modification of company organization to identify business opportunities.Company executives attended conferences to present project results, which helped them to create a stronger network with other attendees. This helped win support for the project within the company.
Med. TermEvaluation
Attainment of “World’s best” data.
Collaboration scheme is not always beneficial.
Well organized collaboration system in the basic R&D phase became an obstacle in the practical R&D phase.
Emergence of technical problem for practical use. Technology for stable/mass production, cost reduction, patent problems etc.
From research phase to practical phase.
Well planed collaboration scheme (Producer and Consumer, etc.) Important information such as spec or permission range of quality was obtained.
Enhanced business opportunity by human network.
Capitalization on human network acquired from the project.
Positive factor
Negative factor
Color Indication
Human network
Environment was favorable for a company to enter into a new tech. field.Collaboration with universities and/or user companies was useful.
Possibilities/expectationfor practical use
Large
Mid.
Small
Case Study B
Consideration
☞ Positive/negative effects and the magni- tude of events.
☞ Change of personal opinion concerning possibilities for practical use.
☞ Status of R&D, business and social environment.
Follow-up chart shows on a simple peace of paper:
◆ (Attention) - Selection of the items or magnitude of the events that were recorded on the chart is influenced by interviewee’s personal opinions. - Personal opinions determine the shape of the chart.
Feedback for Management (Image)
BudgetRequest
CheckList
Concrete Implications・ Does the project earn user’s confidence? (from case study A)
・ Do you have backup schemes for situations where a company ente-
ring a new business/tech. field joins the project? (from case study B)
GoalSetting
CheckList
Mid. termEvaluation
MemberSetting
CheckList
CheckList
G u i d e l i n e
Conclusions
Follow-up chart is a useful tool for improving management.
Follow-up chart is based on personal opinion. (Further consideration will be needed.)
☞ Follow major events throughout the project and understand key points on management or the reason why certain events happened by hearing.
☞ Derive concrete implications on project management.
OK
OK
Finally
I have introduced how to feedback of follow-up evaluation for management.
Today, I would like to know about your activityabout feedback for management.
I would be happy if I could acquire some suggestions about the vertical axis.
Thank you for your attention.