3 major lessons learnt

7/28/2019 3 Major Lessons Learnt

1/4

Lesson Learnt / October 2012 Self Assessment Information System 1

SELF ASSESSMENT INFORMATION SYSTEM

FOR SUSTAINBLE SOY PRODUCTIONLESSONS LEARNT IN PARANA

Pieter Sijbrandij, Violaine Laurens, Stefan Lanz and Rui Valena

Solidaridad /Gebana / October 2012

Summary of the most relevant lessons learn on the development, application, report

generation and sharing of the Self-Assessment guide for soy family farmers, as tested and

implemented in the southwest of Paran State, Brazil in 2011/12 by the joint project of

Solidaridad/Gebana/Coopafi.

Development Application Data processing Reporting Sharing Conclusion

On development:

The development of the guide takes more time and effort than planned. It requires a diverse

set of skills which implies in the involvement of several people.

The design of the methodology had to cope with some basic requirements: application in

maximum two hours, self-applicable for secondary school education level, cost effective

through standardizing, facilitating mass processing and covering regional complexity.

So the people involved should know about soy in general and mixed farming systems in the

south of Brazil in specific, be able to communicate in the language of local family farmers,

understand about building questionnaires and formulating the right questions, have insight in

the possibility for data processing, understand the economics of mixed family farming and its

drivers and option, have detailed knowledge of (shifting) environmental legislation, have

knowledge of health and safety regulations, know the set of consolidated labor laws, be

familiar with the RTRS and other certification systems, their principals, criteria indicators and

guidance.

Bringing this set of knowledge and abilities into a room creates a vibrant community, not

always easy to manage but a cross-learning encounter for all involved. Translating subjects into


2/4


questions proved more difficult than imagined. Splitting up as issue in four different

development levels provoked a lot of discussion. Reaching an agreement on how to address

sensitive issues took several rounds. As different aspects had to be re-done a couple of times

and in smaller groups, the first version was seen as a great achievement to be tested in the

field.

On application:

The four Gebana/Coopafi field extensionists trained did a very good job in reaching out to the

local farmers. True, a good part of them was already involved in the project and had visits

planned and issues to be checked an discussed anyway. But it still is a heck of a task to work

with 378 farmers and register carefully the information.

Due to the education level of about half of the participating farmers, the application was a

close interaction process in which the field extensionist worked patiently with the producer

and sometimes his/her family members though the self-assessment guide. Though planned

for taking maximum two hours, the guide proved an good instrument for discussing several

matters. Not registered, but frequently mentioned, a single application led to hours of

discussing and sometimes even visiting parts of the property.

During the application also some methodological errors became apparent. Of these errors

especially two types had direct implications in the processing of data. The first, hard lesson,

was that aiming for natural speech in formulating the question, some questions had been

phrased in the negative. Now, having only the option yes/no as an answer the respondents

became confused whether they should respond: NO I Dont (double negative, like is usual in

Portuguese) , or YES I dont (affirmative that I am not into it). This phrasing a question led to

the exclusion of some questions and to the exclusion of one issue (child labor).

The second methodological error was that in re-doing and revising the set of questions for

each issue and attempting to clean it up, in some cases the questions left would, independent

of the answers chosen, not enable to differentiate between all the levels. That is, in some

cases no identification could be mad whether the producer would score level 1 or level 2. In

two cases this resulted that the respondent was scored in the higher level.

So, of the application we learnt that it is more difficult to ask a question the right way, but that

when done with respect and patience, the self-assessment is a great instrument to engage the

farmer into reviewing his economic, social and environmental situation, the starting point for

looking forward.

Another important learning is that for low level education farmers there are basically two

options: assisted application like the we tested this time, or a more audio-visual way for which

other material should be developed (with nice videos and an application via iPad). As to the

higher educated (that is secondary school complete or higher) the self-assessment can be

completed individually on paper or, in the near future, on-line.


3/4


On data processing:

With 378 questionnaires and around 200 data per questionnaire the amount of data to be

processed was tremendous. No need to say that some computer-power and brains had to be

allocated. Luckily this intelligence was available within project partner Gebana who used some

wiz-kids of its units in Switzerland and the Philippines to

They set up easy data input screens and wrote the formulae for converting the raw data in

individual scores, linking the individual scores to pre-written information and aggregated

scores for the group analysis. A daunting task which was with limited means very well

executed.

Data processing was hampered by some late incoming inputs, the methodological error

mentioned above and the not always clearly described expected output. Although not perfect,

the results were very satisfying and with a couple of months delayed the group report and

individual reports were produced. The process of translating of the information into data

processing led to a description of the basic methodology in English with as precise definitions

as possible.

On reporting:

The consolidation of the group report has taken considerably longer than expected. While the

data processing led to aggregated numbers, the interpretation of these numbers required

quite some discussion.

The discussion was basically spurred by the hard data produced by the system and the lack of

recognition of some of these data by those who know the reality. It was this discussion that led

to identifying some of the weaknesses of the original design. Only once understood why the

data were wrong, the related issue or question was kicked out. Although this created somegaps in information and answers (like on child labour), it made the designer team understand

better the complexity of their proposal.

With this hurtful kicking out of some data, the tough test was to verify whether the results of

the self-assessment guide matched other existing data. This has been verified in three ways:

one by verifying with the smallholders representative whether the group data coincided with

their view of the local situation. The second was using the internal control system of

participating Gebana suppliers to check the validity of individual reports. The third has been

contrasting the output with the external audit as realized by Control Union in August of 2012.


4/4


Although this checking has not been done with statistical analysis, the conclusions of the team

were satisfactory. This implied that the methodology has likely validity and that the assisting

to the application did not lead to significant distortion of the information.

On sharing:

The aggregated results have been presented to over 200 farmers during the Feira de Melado

event. Coopafi farmer representatives, local government, regional university and state rural

extension agency all were present. In talks afterward it was confirmed that the data pointed

out fed well into the local agenda setting. As the sharing event has been during the month

prior to local elections and in a relatively small community, the project partners agreed no

public debate was to be held at that moment and feedback will be sought after the electionshave taken place.

Individual reports are handed out only to the applicant him/herself and explanation is given

when requested. As to date the feed-back on the individual report is limited. The couple of

farmers the team has talked to mentioned they recognized this was they talked about, agreed

with the gaps indicated but did not commit directly to closing this gaps.

End conclusion:

The methodology is promising but not yet at a level for public sharing. Its development and its

application is more labor, time and resources intensive than originally projected. The project

partners have decided to apply for an extension of the project and in this extension would like

to work on an improved version so this can be shared with a broader public.

3 major lessons learnt

Documents