evaluation plan 2009-2013 (draft) · web viewthese diagrams assist in the identification of key...
TRANSCRIPT
Sheep Industry Business Innovation (SIBI)
Impact Evaluation Plan 2014-2018
ContentsSummary.........................................................................................................................................................1
Abbreviations..................................................................................................................................................1
Project Justification Statement and Goal....................................................................................................2
Project outputs and outcomes.....................................................................................................................3
Practice changes sought..............................................................................................................................5
Program Logic................................................................................................................................................6
Purpose of the evaluation...........................................................................................................................10
Expected impact on target audience: next and end users.....................................................................10
Measurement of adoption...........................................................................................................................14
Goal 1: Dedicated value chains..................................................................................................................14
Goal 2: Business and technical skills...........................................................................................................17
Measurement of projects to support achievement of goals 1 and 2........................................................19
Evaluation activities.....................................................................................................................................20
Monitoring schedule....................................................................................................................................22
Management and Utilisation Strategy.......................................................................................................23
Supporting outcome performance indicators................................................................................................23
Appendix 1 – Estimated value of the sheep industry as at…..........................................................................24
Appendix 2 - A list of product lines and average prices available as at...........................................................24
SummaryThis document describes in detail the activities that the Sheep Industry Business Innovation (SIBI) Project will undertake to evaluate its success or otherwise. In doing so, the Projects outcomes are documented along with the steps the Project will undertake to achieve those outcomes. This document identifies the target audience, the behavioural changes required by the target audience groups to achieve the Projects goals and the baseline data currently known in relation to those behaviours.
While this document covers the evaluation requirements for the life of the Project it is recommended that this document is revised on an annual basis to maintain its relevance to the Project and to ensure that the evaluation of any new research, development and extension activities that are devised as the Project is implemented is also captured.
AbbreviationsASBV Australian Sheep Breeding Values
DRD Department of Regional Development
EQ Evaluation Question
GVAP Gross Value of Agricultural Production
ILS Indigenous Landholder Service (DAFWA Project)
KASA Knowledge, Attitude, Skills and Aspiration
KASAP KASA plus Practice
KEQ Key Evaluation Question
KPI Key Performance Indicator
LS Lamb survival
LTEM Life Time Ewe Management
MLA Meat and Livestock Australia
MR Marking Rate
RfR Royalties for Regions
RIST Rural Industries Skill Training
SBC Sheep Business Centre
Sheep CRC Australian Sheep Industry Cooperative Research Centre
SIBI Sheep Industry Business Innovation
SILC Sheep Industry Leadership Council
SR Stocking rate
SVCTS Sheep Value Chain Transformation Strategy
VCP Value Chain Project
1
Project Justification Statement and Goal
Over-arching project intent: A more internationally competitive WA sheep industry that is growing in value (more rapidly than would otherwise occur)
The Sheep Industry Business Innovation (SIBI) project aims to grow the WA sheep industry through transformational innovation in market development, value chains and production systems. SIBI will address the issues associated with the recent decline and position the industry for export expansion to capture the new market opportunities for sheep meat and live sheep in China, the Middle East and USA.
Ultimate outcome: Increase in Gross Value of Agricultural Product (GVAP) in Western Australia from $925M to $1.05B1 by 2018
The number of sheep in Western Australia has declined steadily from 38 million in 1991 to 15 million head in 2013. While numbers have stabilized somewhat over recent years, there has been little incentive for those currently in the industry to increase their involvement, or for newcomers to become involved.
The increasing demand for red meat protein from Asia and the Middle East now provides that incentive. Accordingly, the peak body of the WA sheep industry – the Sheep Industry Leadership Council (SILC) – has adopted a vision of a robust and vibrant industry, focused on meat more than wool, that will double its annual value of production by 2025.
Intermediate outcome 1: Increase the Western Australian sheep flock from 15M to 16M
Intermediate outcome 2: Increase the Gross Value of Agricultural Product (GVAP) of sheep in Western Australia from $75/head to $80/head2
The work planned is divided into two areas: the Sheep Value Chain Transformation Strategy (SVCTS) that will drive the introduction of new investment, dedicated value chains, business systems and partnerships, and the Sheep Business Centre (SBC) at Katanning that will drive human and physical capacity building. The activities in these two areas are designed to work together to provide the systems needed to build trust, confidence and commitment to grow value across the value chain.
The specific goal for each of the work areas is as follows:
Sheep Value Chain Transformation Strategy (SVCTS) GOAL: to initiate dedicated supply chains based on new markets and to increase the technical and business skills across the market sector
Sheep Business Centre (Katanning) GOAL: To establish the human and physical resources needed to research, develop and demonstrate the elements that comprise a profitable sheep industry
The development of the Sheep Business Centre is a necessary component of the project that will go on to facilitate the activities of the Sheep Value Chain Transformation Strategy. This document outlines how these strategies and their related activities are anticipated to achieve the goals of the project and what data collection and analysis activities will need to be conducted to evaluate the impact of this project. A timeline for data collection is included. As such, this document may be used as an instruction manual for the evaluation of the SIBI project.
1 DAFWA (2014) Business Case Proposal for Sheep Industry Business Innovation Project; page 72 Use of GVAP is consistent with the DAFWA agency level efficiency indicators that are based on ABS GVAP five year rolling averages. However, GVAP for specific agricultural commodities such as sheep meat and wool is less useful in short duration projects because of the lag in reporting by ABS, the impact of fluctuations in commodity prices and the lack of consideration of the effect of changes in the flock size.
Project outputs and outcomesThe Operational Plan for the Sheep Industry Business Innovation project clearly outlines the expected outputs and outcomes expected to be achieved by SIBI. The time frames by which these outputs and outcomes are to be achieved are also documented, along with the performance measures that may be used to demonstrate achievement. This information is replicated here as it articulates key short and long-term evaluation measures.
Table 1 Outcomes and performance measuresOutcomes Performance Measures
1 The WA sheep industry has grown and new markets have been cultivated for sheep meat and live export.
1. Average Gross Value of Agricultural Production (GVAP) per head (meat and wool) increases from $75 to $80.
2. Increase in the production of boxed meat and live export.3. State flock increases from 15 million to 16 million.4. Development of four new dedicated export value chains based on lamb and
hogget carcasses, breeding stock and slaughter ready stock to supply international consumers.
2 Aboriginal participation & economic development.
1. Provision of technical and/or business training opportunities.2. The number of Aboriginal staff placements made.3. Evidence to show improvement in Aboriginal participation in agribusinesses and
related services.
Table 2 Outputs and performance measures
Output Performance Measures
1. Establishment of a Sheep Business Centre (SBC) at Katanning that will serve as a best practice sheep facility demonstration model as well as a training and research facility.
Upgraded meeting rooms, video conferencing facility and information technology systems.
Establishment of a sheep facility which includes labour saving devices, electronic data recording and flock management technologies.
SBC business plan is developed in partnership with industry.
Four SBC staff appointed to provide information, research, training and innovation for the WA sheep industry.
2. Report on the China Plate pilot trial providing guidance to industry and government on value chains development around specific consumer/ customer needs for successful export to new markets.
Two of the five customers identified through either the Asia Market Development/Brand WA project approach or through local/international contacts are secured by 2020.
Customers testify that their involvement was the result of participating in new dedicated value chains based on new business models.
The Report will contain information on: Consumer research; Product innovation;
Value chain design;
New business models; and
Industry capacity for new dedicated value chains.
3. New premium sheep meat products developed.
Genetics resource flock to measure genetic breeding values for sheep meat traits is established.
Number of new products.
Output Performance Measures
4. Development and delivery of a web based portal for the Industry.
Portal includes management, market and value chain information for current and anticipated participants in new value chains.
Portal is utilised by target audience.
Output is delivered on time, to scope on time and within budget.
5. Development and delivery of Lifetime Ewe Management (LTEM) and a strategy to increase the adoption of Australian Sheep Breeding Values.
Increase in stocking rate for participating businesses.
Increase in lamb marking rate for participating businesses.
Increase in number of rams sold by studs using ASBVs.
6. Delivery of the SIBI Communications Plan.
Successful implementation of the SIBI Communications Plan.
Further, the Operational Plan 2014-2015 also outlines what milestones and deliverables the project has agreed to fulfil. These are replicated in the following two tables. While Table 4 is relevant only to the 2014-2015 financial year, it provides a valuable insight into the specific and ongoing activities that will require to be reported on for the life of the project.
Table 3 Major milestones of the project for the life of the project
Main Activities / Milestone Milestone Date
Genetics resource flock established. 31 December 2014
ASBVs strategy developed and implemented. 31 December 2015
LTEM course developed and delivery commenced for mixed farming enterprises. 31 December 2015
Potential partners in China and the Middle East are identified and engaged to participate in dedicated value chains. 30 June 2016
Appointment of SBC leader, industry development and business staff. 30 June 2016
New premium sheep meat product developed. 31 December 2016
China Plate pilot trial Report is finalised. 31 December 2016
Agrifood Business Skills courses, study tours and cultural awareness programs developed and delivered to value chain participants.
30 June 2017
Development and delivery of Lifetime Ewe Management (LTEM) and a strategy to increase the adoption of Australian Sheep Breeding Values
June 2017
Installation of contemporary labour saving and electronic sheep management technologies at Katanning Research Facility. 31 December 2017
Industry Web-portal developed and operational. 31 December 2017
SBC Business Plan developed in partnership with industry. 30 June 2018
Main Activities / Milestone Milestone Date
Development of four new value chains for premium WA sheep meat products, breeding and slaughter ready stock. 30 June 2018
Final report. 31 December 2018
Table 4 Evaluation-relevant milestones of the project 2014-2015Description Completion dateDevelopment of WA LTEM manual for grain and sheep producers and deliverer training/retraining
February 2015
Communication plan developed February 2015Identify options for international partners to host scholarships February 2015Project evaluation plan 26 February 2015Communication log established 26 February 20155 Lamb Survival Groups to commence 27 February 2015Economic analysis completed for the Resource Flock research program March 2015Incorporation of SILC March 2015Develop profiles of new staff at Sheep Business Centre at Katanning March 2015Identify partners for intake of placements March 2015Engage with all major meat processors in WA March 2015Review of literature on yearling meat quality (Sheep CRC) March 2015Pilot Cultural awareness and insights of our key markets workshop with DAFWA staff March 2015Evaluation baseline data report 30 March 2015Design eating quality program for yearling meat with international consumers (Sheep CRC)
June 2015
Plan China sheep meat product specifications project with processors June 2015Develop and begin the implementation of the plan to increase the profile of SILC in the WA sheep industry
June 2015
Report on progress in the implementation of Sheep Industry Strategic Plan which will be undertaken with industry partners.
June 2015
First Feedlotting or Understanding the sheepmeat industry workshop developed in conjunction with Walsh’s
June 2015
Investment and finance models report commissioned June 2015Scoping study of existing databases and stakeholders (for web portal) June 2015Implement internal and external communication of project activities June 2015First market insights of our key markets workshop in conjunction with MLA June 2015Supply chain alignment and production dynamics for V&V Walsh flagship project documented
June 2015
Study tour initiated June 2015Market research on motivational factors commissioned June 2015100 first intake participants enrolled in LTEM course funded by RfR and MLA 30 June 2015Value of ASBVs showcased at 5 events 30 June 2015RamSelect workshops (6) run 30 June 2015Practical application & support of ram breeders 30 June 2015
Practice changes sought Outcome 1 is the key outcomes that all activities and outputs will plan to address. Outcome 2, will be achieved as a result of the way many of the training activities are implemented. The focus of this plan, therefore, will be on how the project will achieve Outcome 1, while specific activities that will aim to achieve Outcome 2 will be highlighted as such.
While the responsibility for achieving the listed Outputs falls discretely into one of the five subprojects within this project, the aims of the subprojects are highly inter-related. The success of
this project will depend on the activities and their desired outputs successfully affecting multiple outcomes. The broad overview of how the activities of the project will impact on the desired outcomes is shown in Diagram 1, while a more detailed depiction of how the tasks of projects will interact to achieve each outcome is shown in Diagrams 2 and 3. These diagrams assist in the identification of key evaluation questions and performance measures.
The development of the Sheep Business Centre (SBC), while a significant project in itself, will facilitate the activities of the project and hence does not, in itself, contribute to the practice changes sought (although activities conducted there will). From this point on, the structure of this document will focus on the goals of the Sheep Value Chain Transformation Strategy (SVCTS), as expressed by Outcome 1, and its associated activities.
SVCTS goals1. To initiate dedicated supply chains based on new markets; and2. To increase the technical and business skills across the market
sector
Due to the breadth of the project’s ambitions there are several target groups that will require a change in behaviour for this project to be successful. Target audience for this project include:
Chinese, and Middle Eastern purchasers of sheep products (importers, retailers and consumers)
Western Australian meat supply chain participants (processors, transporters, exporters, etc.)
Western Australian sheep producers
Specific practice changes sought are outlined in more detail in the section “Target audience: next- and end-users”.
Program LogicThe following three pages diagrammatically describe how the activities of the project link to impact on the industry and create the outcomes expected of the project. The first covers, in general terms, a broader explanation of the project’s activities and desired impact on individuals and industry in a “Bennett’s Hierarchy” format. Diagrams 2 and 3 describe in more detail the actions of the particular activities related to each of the project’s two overarching goals (as stated above). These diagrams aim to concisely explain what industry sectors are affected by SIBI’s activities and to what effect. These diagrams are also colour coded to indicate:
The high-level indicators of project success
The high-level Key Evaluation Questions (KEQs) and medium-scale Evaluation Questions (EQs)
Key Performance Indicators (KPI)
Change expected in target audience
Underlying assumptions that may impact on project success
Activities of SIBI project
Diagram 1 Broad overview of program logic for SIBI
Ultimate performance measure: Increase in Gross Value of Agricultural Product (GVAP) in Western Australia from $858M to $974M by 2018 (leading to an increase in the GVAP per head of WA sheep from $75 to $80)
Key components contributing to project outcome
Expected change to industry
Expected change in target audiences groups
Adoption and communication activities
Research and development activities
Expansion of resources and infrastructure development
Establish Katanning Sheep Business Centre as a credible training, research
and business information centre
Industry placement of four short-term (6-month)
business/industry specialists
Improved producer business and risk management skills – to support new value chains
Producer training in production (e.g.
LTEM) and business skills
Provide information and feedback
services (SBC, web)
Support development and implementation of new value chains
Encourage investments into new value chains
Develop value chain model (Middle East and China)
Other DAFWA sheep, pasture and business projects
Market and economic analysis to show
value of new chains
Industry training in business and technical skills
Improve business skills
of industry
External input
Pilot China Plate model
Increase the average flock
size
Increase the number of flocks/producers
with sheep
Better communication between customer,
producer and industry
Increased confidence and capacity of industry to respond to
changing market demand
Increase the WA sheep flock from 15M to 16M Increase production boxed meat & live export
New business relationships developed
New value chains
developed (4)
New products available (meat, live animals, genetics)
Improve producer skills in managing ewe flock to increase number of
lambs and improve genetics
Genetic resource flock - improve accuracy of genetic information
New meat science and on-farm
technology specialists
Diagram 2 Cause and effect logic check for Goal 1 “To initiate dedicated supply chains based on new markets”
Processors
Producers
Assumption: WA produce will be desirable to OS customers
DAFWA (SIBI)
Performance measure 1: Average Gross Value of Agricultural Production (GVAP) per head (meat and wool) increases from $75 to $80Performance measure 4: Development of four new dedicated export value chains based on lamb and hogget carcasses, breeding stock and
slaughter ready stock to supply international customers
KPI: current supply chains documented
Develop web portal to
support chains
KEQ: To what extent did SIBI contribute to increased value of WA
sheep meat?
EQ: what value did business gain from placement?
KPI: change in total value of
sheep turned off
KEQ: To what extent did SIBI contribute to the development of
four new supply chains?
Change expected in target audience
Processors and others see value in investing in new supply chains
4 industry organisations co-employ new specialists
Assumption: Sheep industry has capacity to attract investment
KPI: Producers understand market specs
improve
International markets: China, Middle East and others
International markets secure greater quantity of higher quality sheep meat from WA
International markets negotiate with WA on product requirements
China: builds new processing
facilities
Others: buyers
diversify demand
EQ: To what extent did SIBI influence knowledge, awareness of investment
opportunities in WA?
KPI: Placements continue to be employed in sector
KPI: # new sheep products developed
KPI: # recommend-ations based
on “China Plate” trial
Middle East: buyers invest in WA sheep
market
KPI: 4 recommended value
chains developed
KPI: international consumers value
WA sheep products
Assess industry info requirements
A4.4
KPI: industry more confident in its own transparency and
stability
Assess effectiveness of
trial consignment to new market
Research, development and extension activities
KPI: # proposed supply chains documented
Identify partners to develop new value chain participants
A4.1
Support relationship
between WA sheep meat
suppliers and international customers
Resource Flock provide information (new value chain products) A1.3
4 new industry placements employed in
partnership with DAFWA A3.3
Conduct analysis of new
value chains
Sheep CRC
Develop technology to
test meat qualityProducers develop capacity and provide improved product to market (see D3.2)
Integrate meat quality testing technologies into facilities (A4.2)
Promote value chains to industry
A4.1, A4.2
Improved capacity to respond to
demand from new value chains
Diagram 3 Cause and effect logic check for Goal 2 “To increase the technical and business skills across the market sector”
Ram breeders
Business skills courses
to be supported by
Boosting Business
Skills project
DAFWA
Ram Breeders submit
production data to SG, provide
ASBVs to clientsKPI: # producers
participating in training
KPI: # rams with
ASBVs
KEQ: To what extent did producers gain new KASAP relating to
genetic selection, higher stocking rates and reproductive rates?
KPI: change in producers KASAP
(as per above)
KEQ: To what extent did SIBI
contribute to the improved business
skills of value chain sector?
KEQ: Has the SBC developed the capacity
to operate independently of DAFWA?
KPI: flock size
Ram breeders register with SG
SBC consultant convinces breeders of value of ASBVs
Producers
More producers willing to increase sheep flock with more profitable sheep businesses
Producers participate in production training
Performance measure 3: State flock increases from 15M to 16M Performance measure 2: Increase in the production of boxed meat and live export Performance measure 5: 80%+ of participants provide evidence of business practice change contributing to increased profitability and sustainability
Outcome 2 performance measure 1: Provision of technical and business training opportunities to indigenous Australians
Agribusiness
SGSG provide ASBVs to breeders
and buyers
Support LTEM & RamSelect
training A3.1, A3.2
Develop and run business skills courses A4.3
KPI: # producers understand and
confident to access new
markets
KPI: ASBV strategy developed
KPI: SBC business
plan agreed, implemented
Development of the Katanning Sheep Business Centre
Leader engaged to
develop facility as a business
Genetics Officer
engaged to promote ASBVs
Comms Officer engaged to
promote facilities
KPI: Communi-cation Plan developed
Resource flock to strengthen ASBV data on key traits
Research social & financial barriers to growing sheep industry A3.4
KPI: strategy for addressing barriers to industry growth
developed
Agribusiness advise clients to attend courses
Producers with more efficient production and business systems
Producers with greater ability to select rams, manage flocks better
Producers with greater
ability to make informed business decisions
Grower groups
Groups support training and Lamb Survival Groups A3.2
Assumption: Industry support
training
Work with ILS to support
indigenous landholder
participation in training
Producers value participation in training Producers participate
in business training
Producers understand
new business models
VCP believe there is value in participating
in training
VCP participate in business
training
VCP understand and implement strategies
to use new investment and finance models
Value chain can
supply markets
Value Chain Participants (processors, exporters, w/salers, etc)
Purpose of the evaluationThe evaluation activities will inform DAFWA and the Western Australian Government of the;
1. success (or otherwise) of the individual adoption and communication activities to build improvement and assess value of each activity (internal formative evaluation),
2. impact of the program on industry, including both next- and end-users (external summative evaluation) using
a. Baseline of current practice for each behaviour targeted by the project so that it can be used to compare change in practice at the end of the project in 2018,
b. Actual extent of adoption for each of the interventions using either direct information on the transformation of the value of the industry (GVAP) or practice change adopted by the producer.
3. Fulfil reporting requirements to stakeholders
Expected impact on target audience: next and end usersThe ‘end-users’ are those people whose behaviours this project intend to influence to achieve its outcomes. The ‘next-users’ are people with whom this project will interact with the intention of having them influence end-users. Next-users are also referred to as intermediaries. End- and next-users are different for each of the activities that this project is implementing to achieve its goals. The following table outlines the target audience for each of the projects activities, segregated according to the goal that the activity is intended to address. For details about the activities please refer to the Operational Plan.
Goal 1: To initiate dedicated supply chains based on new marketsACTIVITY TARGET AUDIENCE EXPECTED IMPACT ON TARGET AUDIENCEActivity 4.1: New supply chains
Expected response by SIBI to this activity
To produce a report that outlines the best-bet activities based on information received from the various analysis being undertakenTo be informed of market opportunities and to develop further activities (China Plate and others) aimed at developing new markets
WA processors To implement processes that facilitate the production of new sheep products to new markets
End-users Desired behavioursSome end-users to be determined
Dependant on results of the investigations and analysis – to be revised July 2015
Customers Believe that WA sheep meat is a desired product among its consumersInvest in Infrastructure and processes to accept sheep or sheep meat from WA
Consumers View WA sheep meat as a product worth purchasing (inferring that is safe and of sufficient value to purchase)
Activity 4.2: Meat product development
Expected response by SIBI to this activity
To better understand the needs of international marketsTo incorporate this knowledge (with outputs of 4.1) into new market development (through 4.2)
Next-users Desired behavioursTraining bodies (eg CRC, Universities, DAFWA, industry placements)
Include content in coursework (for processors and producers) that supports the production of high quality sheep meat
Processors Offer an attractive system and price for producers to supply high quality sheep meat (eg hogget, lamb, mutton)Adopting and communicating new specification that relate to product qualityProvide feedback to producers on quality
End-users Desired behavioursProducers Seek out information and training on improving the
quality of their sheep meatAdopt practices that improve the quality of their sheep meat
International customers
Develop and awareness and appreciation of quality WA product (through product research and subsequent eating quality program)
Activity 4.3: Business skills development
Next-users Desired behavioursTraining organisations To conduct training (for processors and producers)
recommended by SIBI to transform processors and producers businesses to be able to access new markets
End-users Desired behavioursProducers To understand the new investment and financing models
available to them that may assist in accessing new capital and/or marketsTo incorporate this understanding into their business modelMake informed decisions on business management
Industry placements and processors
Understand the cultural framework that new markets operate within
Activity 4.4: Develop a web portal
Next-users Desired behavioursWebsite owner (TBD) To support the presence of a market information portal
End-users Desired behavioursSupply chain parties To access market information via the portal
To make business decisions based on the information available via the portal
Goal 2: To increase the technical and business skills across the market sectorActivity 3.1: Lifetime Ewe Management delivery and lamb survival
Next-users Desired behavioursRIST Develop a version of the LTEM course that is of
particular relevance to WA producersEmploy a process which will increase the number of LTEM trainers in WA
Trainers Conduct training for 600 producers
Consultants To recommend LTEM to their clients
Regional grower groups
To establish and administer participation in the Lamb Survival Initiative from their membershipTo recommend participation in LTEM to their members
End-users Desired behavioursProducers Participate in LTEM course, Lamb Survival Initiative and
100%+ ClubUnderstand the key principles associated with improved reproduction and managementImplement practices that improve stocking rate, lamb survival or marking rates
Activity 3.2: Adoption of genetic technologies
Next-users Desired behavioursAgents, sheep classers, agribusiness, rural media, ram breeders
To promote the use of ASBVsTo promote resources to assist in the adoption of genetic technologies in WA (eg DAFWA staff and training)
End-users Desired behavioursProducers To use genetic information (ASBV and indexes) when
selecting rams and ram sources
Ram breeders To provide ASBVs on more rams (from 27% rams sold to 50%)
Activity 3.3: Industry placement and scholarships
End-users Desired behavioursSupply chain business’ (exporters, retailers, processers and producers)
To view industry placements as valuable to the industry and their organisationsTo invest in placements and provide an appropriate training environment
Industry placements To actively participate in the training processTo enthusiastically contribute to the industry on completion of their training
Activity 3.4: Barriers to adoption
Expected response by SIBI to this activity
To better understand the social / personal / economic barriers impacting on the growth of sheep production in WATo incorporate this knowledge into a strategy for reducing the effect of the key identified barriersTo implement methodologies for removing barriers
Projects to support achievement of goals 1 and 2Activity 1.2: Electronic flock management technologies
Next-users Desired behavioursAgribusiness To promote the value of implementing electronic flock
management technologies
End-users Desired behavioursProducers To implement labour saving or quality control
technology that allows them to increase their flock value, profitability, flock size or turnoff
Activity 1.3: Resource flock
End-users Desired behavioursResearch and development organisations
That they integrate the new information on hard to measure traits (eg Lean Meat Yield and Eating Quality) into their activities
Activity 2.1: Sheep Industry Leadership Council (SILC)
Next-users Desired behavioursDAFWA projects To incorporate information from SILC into activities to
ensure that activities have value to the industry
End-users Desired behavioursSILC To actively communication the needs of the industry
to DAFWATo feel confident that issues affecting the industry are being addressed by R, D and E organisationsTo feedback to industry on progress of R, D and E organisations activities such that confidence in the industry is increased
The following three sections group the expected impacts of all the activities by target audience. The collective impacts on target audience groups are described and current and expected KASAP changes defined. This information assists in identifying the data that needs to be collected and the method for doing so.
Measurement of adoption Goal 1: Dedicated value chainsKey Evaluation Questions:
1. To what extent did SIBI contribute to increased value of WA sheep meat?2. To what extent did SIBI contribute to the development of four new supply chains?
Target audience
Desired impact Baseline measure as at 2014/2015
Target measure 2018 Collection methodology
Leader and date of collection
Processors 1. To incorporate understanding of new value chains into their business model
2. That processors value access to sheep meat information via an online system
3. Adopt and communicate specification requirements, timing, etc (that relate to product requirements) for new products
4. Offer an attractive system for producers to supply high quality sheep meat
5. To view industry interns as valuable to the industry and their organisations
6. To invest in interns and provide an appropriate training environment
1. Future value chain opportunities currently not known or understood
2. Online system not available
3. New products currently unavailable
4. List of product lines and average prices available to be included in Evaluation Plan M2
5 and 6: No sheep industry interns currently exist to value
1. Target measure to be determined by August 2016 (dependant on what new chains look like)
2. a) Processors regularly access online system b) Processors rate system higher than 5/10 for value
3. Processor publications produced
4. a) At least 1 consignment delivered to each new market b) List of product lines to have increasedc) Producers receiving premium prices for products delivered to new value chains
5 and 6: Four new specialists (ex-interns) employed in sheep meat industry
(A) For 1: Targeted semi-structured interview of processors with qualitative results(B) For 2a: Web access data(C) For 2b: Short survey linked to online system for users of system(D) For 3 and 4a, b and c: Collation and audit of information by processors (KEQ2)(E) For 5 and 6: Targeted semi-structured interview of industry (processors) with qualitative results
(A) Robin Jacob Initial interview Dec 2016. Further surveys to June 2018.(B) and (C) Kimbal Curtis Monthly downloads and random pop-up surveys Dec 2017 to Dec 18(D) Robin Jacob Market audit Oct 2018(E) Mandy CurnowToward end of each interns contract
Target audience
Desired impact Baseline measure as at 2014/2015
Target measure 2018 Collection methodology
Leader and date of collection
Producers
Feedlotters, production groups (7 and 8)
1. Understand the new investment and financing models available to them that may assist in accessing new capital
2. Seek out information and training on improving the quality of their sheep meat
3. Adopt practices that improve the quality of their sheep meat for premium markets
4. To participate in new value chain
5. Sustained increase of total turnoff
6. That producers value access to sheep meat information via an online system
7. To view industry interns as valuable to the industry and their organisations
8. To invest in interns and provide an appropriate training environment
1. Baseline zero as models to be developed
2. Current opportunities to improve understanding not available
3. Quality currently assessed via fatness and carcass weight
4. Currently 0 producers participating in new value chains as chains don’t exist
5. Current 5 year rolling average annual net turnoff (09/10 to 13/14) = $64/head (ref Appendix 1)
6. online system currently not available
7 and 8: No sheep industry interns currently exist to value
1. 80% of training participants demonstrate understanding of models and available finance
2. a) X% producers actively seek information1
b) # producers undertake training on improving meat quality traits1
3. a) X% WA producers measure quality by1 b) 60% producers that undertake training adopt behaviours that improve quality traits1
4. a) X% producers aware of new value chain and1 b) # producers participate in new value chain1
5. 5 year rolling average net turnoff (annual) to 17/18yr = $70/head
6. a) Producers regularly access online system b) Producers rate system higher than 5/10 for value
7 and 8: Four new industry specialists employed1 Values to be determined by May 2016
(F) For 1, 2b and 3b: entry, exit and post-training surveys of producers that have participated in training(G) For 2a, 3a, 4a, 4b: general survey of all WA producers(H) For 5: Production data available from the ABS(B) For 6a: Web access data(C) For 6b: Short survey linked to online system for users of system(E) For 7 and 8: Semi-structured interviews of host producers and relevant groups with qualitative results
(F) Anne Jones workshop entry surveys June 2015 – June 2017; exit surveys June 2016 – 2018(G) Anne Jones March 2018(H) Kimbal CurtisAnnual tracking of ABS data July 2015-18(B) and (C) Kimbal Curtis Monthly downloads and random pop-up surveys Dec 2017 to Dec 18(E) Mandy CurnowToward end of each interns contract
Target Desired impact Baseline measure as Target measure 2018 Collection Leader and date
audience at 2014/2015 methodology of collectionTraining organisations (eg MLA)
1. To conduct training (for processors and producers) recommended by SIBI to transform processors and producers businesses to be able to access new markets (eg feedlotting workshops, market insight workshops)
2. Include content in coursework (for processors and producers) that supports the production of high quality sheep meat
Training currently unavailable
Courses conducted training processors and producers
(I) Communication log, advised by trainers
(I) Anne Jones, Mandy CurnowJune 2015 to July 2018
Industry interns
Understand the cultural framework that new markets operate withinBetter skilled people within the sheep meat industry
0 currently in industry # of placements remaining in industry to be determined
(J) Interviews with interns
(J) Mandy Curnow Toward end of each interns contract
Goal 2: Business and technical skillsKey Evaluation Question:
1. To what extend did producers gain new KASAP relating to genetic selection, higher stocking rates and reproductive rates?2. To what extend did SIBI contribute to the improved business skills of value chain sector?
Target audience
Desired impact Baseline measure as at 2014/2015
Target measure 2018
Collection methodology
Leader and date of collection
RIST 1. Develop a version of the LTEM course that is of particular relevance to WA producers
2. Employ a process which will increase the number of LTEM trainers in WA
1. WA LTEM course being developed
2. Currently 6 trainers in WA
1. WA curriculum available by Feb 2015
2. 10 trainers operating in WA
(K) For 1: signed off by DAFWA(L) For 2: Advised by RIST
(K) Jonathan EnglandConfirm March 2015(L) Jonathan EnglandConfirmed June 2016
Trainers Conduct training for 600 additional producers
Beginning from a baseline of 0
600 producers (I) Comms Log, advised by trainers
(I) Anne Jones, Jonathan EnglandJune 2015 – July 18
Agribusiness To recommend new LTEM course to their clientsTo recommend attendance at RamSelectTo promote the use of ASBVsTo promote resources to assist in the adoption of genetic technologies in WA (eg DAFWA staff and training)
93% and 37% of service providers nationally advise their clients to attend LTEM and RamSelect respectively78% of service providers nationally recommend buying rams using ASBVs
70% / 50% of WA sheep consultants recommend participation in LTEM / RamSelect80% of WA sheep consultants promote the use of ASBVs
(M) Agribusiness survey
(M) Anne Jones, Mandy CurnowJuly 2018
Regional grower groups
1. To establish and administer Lamb Survival Groups from their membership
2. To recommend participation in new
1. Zero groups currently exist
2. Nil as new course not completed
1. 4 groups established
2. 50% of participants influenced by
(N) For 1: Advised by grower group(F) Participant
(N) Katherine Ayres June 15 to July 18(F) Anne Jones
LTEM to their members grower group entry survey June 15 - July 18
Target audience
Desired impact Baseline measure as at 2014/2015
Target measure 2018 Collection methodology
Leader and date of collection
Ram breeders
To provide ASBVs on more rams 49% rams sold with ASBVs (from 2014 WA producer survey)
50% rams sold with ASBVs
(G) General survey of all WA producers
(G) Anne JonesMarch 2018
Producers 1. Participate in new WA LTEM course2. Understand the key principles
associated with improved reproduction and management
3. Set three achievable targets particularly in areas around feed assessment and feed budgeting
4. Improve stocking rate, lamb survival and marking rates by pregnancy scanning and managing ewes appropriately
5. To use genetic information (ASBV and indexes) when selecting rams and ram sources
6. To incorporate behaviours that grow their business
7. To have more confidence in the industry
1. Zero participants2, 3, 4 and 5: 9% of WA producers
condition score or weigh ewes regularly to assess health
40% of producers pregnancy scan, including 14% that scan for litter size
Current marking rate for Merinos is 90%, 95% for meat and maternal
4% of producers have detailed knowledge of ASBVs
6. Relevant business skills to be determined7. Relevant confidence
measures to be determined
1. 600 new participants2. 95% of participants with improved understanding of principles3. 95% of participants set targets. 65% of participants make improvement on targets4. Ave SR, LS and MR of participants improve by 12%. WA producers improve by 5%5. 80% of X RamSelect participants use ASBVs6. Specific anticipated change to be determined7. Value to be determined
(I) For 1: Comms Log, advised by trainers(F) For 2, 3, 4, 5 and 6: entry, exit and post-training surveys of training participants(G) For 4, 5, 6 and 7: general survey of all WA producers(KEQ 1, 2)
(I) Anne Jones, Jonathan EnglandJune 2015 – July 18(F) Anne Jones workshop entry surveys June 2015 – June 2017; exit surveys June 2016 – 2018(G) Anne Jones March 2018
Measurement of projects to support achievement of goals 1 and 2Target audience
Desired impact Baseline measure as at 2014/2015 Target measure 2018
Collection methodology
Date of collection
Producers To implement labour saving or quality control technology that allows them to increase their flock value, profitability, flock size or turnoff
From 2014 producer survey: 8% using automatic drafting equipment, 4% are using electronic ear tags, 2% are using a paddock based ‘Walk over weighing’ system and 1% use Pedigree Matchmaker
New levels to be determined by December 2015
(F) Entry and exit surveys backed up by WA producer survey (G)
(F) June 2015 to July 2018(G) March 2018
Agribusiness To promote the value of implementing electronic flock management technologies
73%, 32%, 66% and 45% of agribusiness is willing or already advising clients to use automatic drafting equipment, electronic ear tags, paddock based ‘Walk over weighing’ system and Pedigree Matchmaker respectively
New levels to be determined by December 2015
(M) Agribusiness survey
(M) Anne Jones, Mandy CurnowJuly 2018
Research and development organisations
To integrate the new information on hard to measure traits (eg Lean Meat Yield and Eating Quality) into their activities
New research unavailable Positive key messages relating to this research published in the MLA annual report
(O) Audit of MLA Annual Reports
(O) Ken Hart, Anne Jones November 2015, 2016, 2017 and 2018
SILC To actively communication the needs of the industry to DAFWATo feel confident that issues affecting the industry are being addressed by R, D and E organisationsTo feedback to industry on progress of R, D and E organisations activities such that confidence in the industry is increased
Communication methods and means of verification to be determined
Target measures to be determined by December 2015
(G) WA producer survey(M) Agribusiness surveySILC survey?
(G) March 2018(M) Anne Jones, Mandy CurnowJuly 2018
Evaluation activitiesAs indicated in the previous section, there are many evaluation activities required to assess the success of this project. Details of the evaluation activities in the ‘Collection Methodology’ and ‘Collection Date’ columns in the previous chapter are collated here for ease of management.Activity code
Activity title
Activity description Date of collection
Leader
A Processor interviews
Informal, semi-structured interviews conducted by activity leader to collect information about how processors are interacting with the new supply chains. This evaluation activity covers processors understanding of and capacity to be involved in the new value chains.The initial interview will be conducted in December 2016 to capture information from the pilot project with V&V Walsh. Further interviews will be conducted with other processors as they come on board the project.
Initial interview: December 2016Further interviews by June 2018
Robin Jacob
B Online information tracking (use)
Collection of data on the usage of the online information system over time including unique number of visitors and unique number of visits
Monthly downloads Dec 2017 to Dec 18
Kimbal Curtis
C Online information system (value)
Short survey of online information system users in the form of a pop-up box linked to the system triggered to come up on users screens after visiting the site
Monthly downloads Dec 2017 to Dec 18
Kimbal Curtis
D New value chain audit
Review information supplied by processors and document new value chain information: specifically what new sheep meat product lines and prices are available for producers to supply to. Contrast this information against previously available product lines and comparative prices offered.
Market audit Oct 2018
Robin Jacob
E Value of interns
To measure the value that industry place on industry interns. This will be done by conducting semi-structured interviews with the host organisations (eg processors, producers) and other relevant industry organisations (eg grower groups) and rating the perceived value of each intern against a list of criteria.
Toward end of each interns contract
Mandy Curnow
F Entry and exit surveys
Short surveys undertaken by producers at the beginning and end of each training event to determine the direct impact of the training on their KASA. Influence of industry (eg grower groups and consultants) on participation in training also tracked through this mechanism.
February 2015 to July 2018
Activity leaders & Anne Jones
G Producer survey
General survey of all WA producers to gauge the impact on producers behaviours caused by the range of extension activities undertaken by the project
Survey conducted March 2018
Anne Jones
H Production data Analysis of five-year rolling average net annual turnoff calculated using publicly available data from the Australian Bureau of Statistics and reported in terms of $ value of combined WA sheep meat products per 100 ewes (WA flock size)
Annual tracking July 15-18
Kimbal Curtis
I Communication Log
Valuable device used to keep track of all the extension and marketing activities undertaken by team members and affiliates
March 2015 – July 2018
Anne Jones
J Intern interviews
Semi-structured interviews conducted by activity leader to provide feedback on interns learning environment and professional development.
Toward end of each interns contract
Mandy Curnow
K WA LTEM course
Just a ‘tick off’ when course signed off by DAFWA and RIST
March 2015 Jonathan England
L WA LTEM trainers
Just a ‘tick off’ when the ideal 10 trainers in WA reached
June 2016 Jonathan England
M Agribusiness survey
Short survey to measure agribusiness’ support of SIBI activities and messages
July 2018 Anne Jones, Mandy Curnow
N Lamb Survival Groups
Just a ‘tick off’ as Grower Groups set up their Lamb Survival Groups (4 required)
June 2015 to July 2018
Katherine Ayres
O Resource flock audit
Collect the MLA Annual Reports of 2015, 2016, 2017 and 2018 and extract relevant comments about the Resource Flock
October 2015, 2016, 2017 and 2018
Ken Hart, Anne Jones
Monitoring scheduleEvaluation Activity 2015 2016 2017 2018
J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N DEvaluation plan reviewWA LTEM course (K)Communication log (I)Training entry & exit surveys (F)Production data (H)Resource flock audit (O)Agribusiness survey (M)Value of interns (E, J)Processor interviews (A)Reported “tick off” by RIST (L)Online info system use (B) &value (C)Producer survey (G)Lamb Survival Groups (N)New value chain audit (D)
Chart legendData collection design phase Benchmarking data collection Data collection Data analysis Reporting
Management and Utilisation StrategyWho will conduct the evaluation & how will credibility be ensured?The overall evaluation plan will be overseen by Team Leader Mandy Curnow, DAFWA and implemented by Anne Jones. The Monitoring Schedule identifies who has responsibility for collecting certain data relevant to evaluation of the project. This information will be collated and reported on by Anne Jones and Mandy Curnow. Some parts of the evaluation field work will be conducted by an external evaluator in order to strengthen the credibility of the evaluation and manage the workload.
How will the evaluation process maintain its relevance?It is recommended that the evaluation plan is reviewed on an annual basis starting from July 2015. All Project Managers and Activity Leaders need to be involved in this review. The evaluation plan will be revised as needed as a result of these reviews.
Who needs to receive the findings of the evaluation?The evaluation of this project is driven by internal project accountability expectations as well as the expectation of the primary funder: the Western Australian Government via the Royalties for Regions Program (RfR). RfR projects are administered by the Department of Regional Development (DRD). RfR expect standard evaluation protocols to be met and DRD will use the evaluation to assess the value of the investment in the project.
Supporting outcome performance indicatorsAs noted in the footnote on page Error: Reference source not found, there are some problems with using GVAP (or change in GVAP) as an outcome performance indicator for a short duration project. These problems include:
Lag in release of GVAP by ABS. Preliminary estimates are expected seven months (January) after the end of a financial year with final estimates usually released in eleven months (May).
The impact of fluctuations in commodity prices. Normal fluctuations in commodities prices (e.g. lamb, mutton, wool, live export) can swamp any underlying change in industry value in individual years
The lack of consideration of the effect of changes in the flock size. The calculation of GVAP does not account for the impact on industry value of either building up the flock (investment) or the selling down of the flock (liquidation).
In response to these problems, an estimated value of the sheep industry calculated from reported production and reported prices for the four products (lamb, mutton, wool, live export) will be reported. This approach allows each of the problems listed above to be addressed. Values for previous years will be provided for comparison.
The estimated industry value correlates well with the ABS produced GVAP for sheep and for wool as shown in the adjacent figure.
Appendix 1 – Estimated value of the sheep industry The table below shows the calculation of the estimated annual value of the sheep industry (5 year average) prior to the commencement of the Sheep Industry Business Innovation project.
For the five years prior to commencement of the project, the industry had an average annual adjusted value3 of $945 million or $64 per head. The target for the project is to increase the estimated value from $64 to $70 per head.
Table A1. Calculation of average estimated value of the Western Australian sheep industry, 2009-10 through 2013-14.
2009-10 2010-11 2011-12 2012-13 2013-14 AverageLamb
Unit value ($/kg) 4.49 4.92 5.23 4.10 4.60 4.67Carcass weight (kg) 20.5 21.1 21.3 20.7 21.0 20.9Lambs slaughtered ('000) 2135.1 2056.6 1733.3 1907.5 2386.3 2043.8Total value (AUD million) 197 214 193 162 231 200
Sheep/muttonUnit value ($/kg) 2.42 3.33 3.00 1.97 2.12 2.57Carcass weight (kg) 21.2 21.7 22.6 22.0 23.0 22.1Sheep slaughtered ('000) 1351.7 1282.0 742.5 1430.7 1562.2 1273.8Total value (AUD million) 69 93 50 62 76 72
Live exportsUnit value ($/hd) 88.1 101.3 113.8 74.8 80.8 91.8Sheep exported (hd) 2.2 2.3 1.8 1.7 1.8 2.0Total value (AUD million) 195 232 207 128 144 180
Meat total Total value (AUD million) 461 539 451 353 451 453
WoolUnit value ($/kg clean) 8.7 10.9 12.1 10.5 11.0 10.7Yield (%) 61.0 61.0 61.1 61.0 61.9 61.2Weight (greasy tonnes) 76,601 79,067 73,223 74,991 79,485 76,673Total value (AUD million) 407 527 541 481 544 500
Grand total Total value (AUD million) 868 1,066 993 833 995 953Value per head (AUD/head) 55 73 71 58 64 64
Flock adjustmentOpening number (million) 15.7 14.7 14.0 14.4 15.5 14.9Closing number (million) 14.7 14.0 14.4 15.5 14.2 14.6Change (million) -1.0 -0.7 +0.4 +1.1 -1.2 -0.3
Flock investment (AUD million) +44 +91Flock liquidation (AUD million) -53 -50 -60
Adj. total value (AUD million) 815 1,016 1,036 924 935 945Adj. value per head (AUD/head) 52 69 74 64 60 64
3 The adjusted value takes account of flock increases (investment) and flock decreases (liquidation)
