dont know refused
Post on 13-Feb-2017
Embed Size (px)
Dont Know/Refused:A Researchers Perspective on HMIS Data Collection ChallengesBy Susan Walker Rhode Island Coalition for the HomelessBowman Systems Collaborate 2016
Thank you all for joining me. Today Im going to talk about data collection and quality concepts, using our redesigned HUD Entry Questionnaire as an example, and share insights gained from working with case managers. 1
What do I Know About Homelessness and Data?
Occupy Providence 2011-2012, back when I used to call constituents friends.Antiques Dealer since 1998, alongside homeless and marginalized people.Studied Medical Billing & Coding at a for-profit online university. Learned about records and predatory lending.Currently enrolled at Brown School of Public Health, learning and doing data collection and analysis.Recently hired by my dream employer.Im new! If you pay close attention, youll probably catch my mistakes!
This picture was taken the night we decided to break down the Occupy Providence encampment. The city agreed to open a day center for homeless people if we left the park. I didnt camp, but I spent a lot of time at the park. I often cooked and ate with fellow occupiers. Constituents have been guests in my house and sat at my table. Antiques is a funny business. It transcends class. My colleagues in the antiques business range from extremely wealthy people looking to have a losing business for the sake of reducing their tax burden, to people who literally live out of their van and hop from flea market to flea market. Its a fringe culture where people who dont function well in standard jobs and mainstream society have found a place. I got excited about data collection and analysis when I enrolled in an online university for Medical Billing and Coding. The University was a scam, and while I learned a lot, it did not get me a job and out of the antiques business.I wanted my resume to look good, so I went to Brown for Public Health. Ill share my $60,000 education with you for free. I learned all about data collection with a view to publishing papers that other statisticians consider valid. Its a very high data quality standard. 2
Data Collection ConceptsHard to Reach PopulationQuestion OrderingCodingSkip PatternsPrompting with Answer ChoicesProbingLook backs
Data Collection PointsNon-Verbal CommunicationInterviewer BiasClient disposition/SettingSample Size Bias ERROR!
A good interviewer does not necessarily need to be familiar with all these concepts, but needs training and support that incorporates these concepts. Collecting data on homeless people is notoriously hard. Its no surprise that street outreach has the biggest data quality challenges in our COC. Transient people are going to have highly variable data. In public health they train you to ask the most sensitive information towards the end of the questionnaire. This is how its done with the HUD Entry Template, but there is no training to prepare people. Try framing the questionnaire: I need to ask you some questions so we know how to serve you best. Just answer honestly, and not according to what you think the right answer is. Some of these questions may seem very personal. In our COC, the template questions appear in a different order than you enter them in HMIS. This affects data quality. Our data standards cover coding to a certain degree, but not uniformly. I refer to the data standards daily, but I doubt our case managers do. Coding is judgment. I will talk extensively about coding in this presentation. Skip patterns- Is there a reason for missing data, or is it random? Our trainings reveal how our case managers struggle with even simpler concepts. The chronic homeless questions are hard to ask. They are often skipped. Its not a random omission.While the Yes or No questions are supposed to be asked independently, I have no idea how to get the right information without reading the HUD Verification lists of choices. The living situation questions require probing. There is no way to get the data without having a conversation, and a skillful lookback. While data collection points are laid out in the data standards manual, theyre not explicit as you are using HMIS. The face you make when asking a question can influence the answer you get. Interviewer bias: You dont have HIV, Do you?In our COC, some shelters frustrate the clients so much that by the time they see a case manager, theyre probably not in the mood to talk. The saving grace of HMIS data is the size of it. Having a large sample improves validity. Im going to be talking about kinds of bias, for example instruction bias. When the instruction lack clarity, your data skews according the individuals collecting the data.All of these concepts help pinpoint the source of error, however Error itself is Often discussed but rarely quantified. I struggle with the way we report on HMIS data without discussing standard deviation or confidence intervals. I make copious use of cartoons when I send out my error-grams. I try to make people smile. Im not sure if its working.
PersonalityData Quality is influenced by the personalities of
The HMIS end userThe clientThe providers office cultureThe funding managersGovernment officialsThe HMIS administratorsVariable itself
In order to obtain uniform data, it is important to understand that non-uniform personalities are at play.Even the data types have personalities. Theres a rumor in our CoC that a client needs to be receiving SSDI in order to select YES for disability. Thats a personality at work. 4
Form Versus QuestionnaireThe HUD Intake form has issues.You ask a client for their name.You dont ask a client for their name data quality. Has the client been continuously homeless (i.e., on the street, in an emergency shelter, or safe haven) for at least one year? Still on the HUD Template.Our Entry in HMIS is in a very different order. A form gives the impression that the data is somehow uniform, and fits easily into the boxes. A questionnaire, with instructions, communicates that gathering data requires conversation, and entering data requires judgment.
Name and name data quality are two completely different kinds of information. One is a fact about the client. The other is a judgment by the person entering data. Two totally different processes. In my first month on the job, my colleague Emalee told me she was rewriting the Entry form to make it easier to use and fit on less pages (4 v 11)I didnt exactly grab it out of her hands, but I got pretty excited. By this time, I had written and launched 2 full scale surveys, one that went through IRB review, collaborated with my boss, a Brown Faculty Researcher, on 4 surveys, and analyzed the results of 2 surveys. While surveys are different than databases, the principals of questionnaire design are applicable. Understanding how data quality and completeness can interfere with analysis also helps. The data standards show example conversations, but I think specific, in person training on probing, when to list answer choices, and how to frame a look back are important skills to teach. Theoretically, having an interviewer gather the information rather than having a client fill out the form should improve data quality, IF the interviewers are all asking the questions uniformly, and entering the data uniformly. Now that the data standards have changed again, we need to re-write this. I consider it a blessing because I really wanted a period to experiment with the questionnaire and see how our case managers react to it. Trainings on using this questionnaire have revealed a lot of ways that we can continue to improve it. 5
Entry DateThe Entry Date and data entry date are often different: a source of error. There can be overlapping ES entries on the same date. Backdating is great, but can lead to absurdities.I emphasize the importance of the relationship between date and data:By entering a date, they are confirming that all the data in the record is true. If they havent taken care to verify it, there is a problem.
Believe it or not, there are data quality concerns with this field
NameStreet Names Duplicate ClientsData Quality Descriptors?We have found out that we sometimes have 2 clients to one ID because they have the same name.We train Case Managers to enter a name data quality descriptor, but not to ask about it. It is unclear which HMIS Data Elements need to be asked of the client and which do not.
First Strategy: Put all questions that should be asked using words into words. Second Strategy: Write out instructions for filling in data elements that are not asked of clients. You cant assume that case managers are not asking What is your name data quality.The personality of a data element comes out in the kinds of errors it leads to. Our case managers were not sure if you could correct the spelling of the clients name without creating a duplicate client. Any time a data field is programmed to behave differently, thats a personality trait of that variable. 7
Social Security NumberThere is a disincentive for sharing this data. HUDs rationale: unduplicated client counts, and SSN necessary for applications.
Data Quality Descriptor errors: If the Case Manager enters doesnt know or refused, then the blank SSN shouldnt flag.Some shelters were using 4 digits as a matter of procedure. In the event of a security breach, our clients would be vulnerable.
Skip pattern: a concept that everyone is probably familiar with, even if you havent called it that. In the 0213, I noticed nearly every client in one ES had #errorI called them up, and sure enough, the missing data was not random omission, but intentional omission. It was their policy to collect the last 4 digits. I looked up the data standard, emailed the funding manager and the agencys lead HMIS user, and corrected the p