lessons in data and statistical analysispitfalls and problems topics for consideration: source bias...

Lessons in Data and

Statistical Analysis

NCSL Fiscal Analysts Seminar

October 5, 2017

Data types, relevance and reliability

Useful data sources

Tools

Examples

Pitfalls and Problems

Topics for Consideration:

Source Bias

Tips for assessing the credibility of online information

(Center for Public Education, based on Stanford Guidelines for Web Credibility)

1. Does the Web site provide references for research that can be independently verified?

2. Are authors identified? Are their affiliations, credentials, and contact information provided?

3. Who owns or is responsible for the Web site? Is a physical address and complete contact information provided?

4. Does the site describe its mission? Are staff members identified?

5. Does the site carry advertising? If it is run by a not-for-profit organization, are its sources of funding identified?

6. Is the site professional in appearance and quality? How recently has it been updated? Is it free from typographical and grammatical errors?

Evaluating reliability and relevancy

Time Period


Unit of Analysis


individuals

households

Common Adjustments

Inflation (what year = 100?)

Seasonally-Adjusted (SA)

Seasonally-Adjusted at an Annual Rate (SAAR)

Per Capita


Flaws/Errors

Statewide In-Migration vs Utah School Age Enrollment

Statewide In-Migration = 11,000

School Age Enrollment

Growth = 12,161 (School Migration Est. 5,355)

Under estimate = 4,243

Cost = $17 million


Useful data sources

• Bureau of Economic Analysis

• Consumer Expenditure Survey

• Bureau of Labor Statistics

• U.S. Census Bureau

• National Bureau of Economic Analysis

• American Economic Association

• State Vital Statistics Offices

• Global Insight

• Moody’s

• Federal Reserve

Data Sources: BLS Consumer Expenditure Survey (CES)

• Cross-section of spending characteristics of US households

• Useful for analysis of sales tax proposals for narrow taxes/ exemptions:

• Exemption bill on diapers/ feminine hygiene products

• Bill expanding sales tax to services

• Federal Reserve Economic Data (time series)

• BEA, BLS, Census, numerous other data releases

• Always up-to-date on latest data releases

• Example: LA Sales Tax

• Able to manipulate data as needed

Data Sources: FRED

https://fred.stlouisfed.org/series/LASALESTAX

Data Sources: Other Agencies

• Federal Reserve Banks and Board of Governors

• Continuously research policy issues and monitor economic trends

• Timely research on net impacts of sales tax holidays

• Crucial for analysis of LA Second Amendment Sales Tax Holiday bill


• US Energy Information Administration (EIA)

• Continuous research and monitoring

• Timely research on impact of gasoline prices on consumer response

• Useful in analysis of LA Gasoline Tax increase bill


• US Department of Agriculture• Extensive data on agricultural production

• Current data on farm size and production by state• Vastly improved estimate reliability on proposal exempting

irrigation materials from sales tax

Data Sources: Market Research Firms

• Example: Statista• Limited amount of

free data, but enough US-level info to be useful

• Examples:• Expansion of sales

tax to services including digital media

• Diapers/feminine hygiene products sales tax exemption

Other Data Sources

• Professional or Industry Associations

• Often can be a valuable source of very relevant data (but be aware of their mission/motives)

• Example• AOPA: Aircraft painting

service tax exemption

• Think Tanks or Professional Research Groups

• Sometimes can function similarly to an industry association; same caveats apply

• Example• University of New

Orleans Hospitality Research Center: Louisiana Tax Free Shopping Program for foreign tourists

Analytical Tools

Tools

Application Pros Cons

Excel Ubiquitous

Easy to present

results on the fly

“Intuitive”

Hard to track iterative

process

Size limits (better)

Can be slow

Statistical Packages

(R, STATA, SPSS,

SAS, EVIEWS, …)

Good documentation

of process

Sophisticated

modeling techniques

Large datasets/faster

Steeper learning

curves

Expensive

Output presentation

can be tricky

Statistical Packages: STATA Example – “freduse”

Example

• Utah Tax Reform Bill of 2006• Modified Flat Tax

• Went from 6 bracket to one with deductions being converted to credits.

• Tax Commission had access to the filer data but LFA and Governor did not.

• Tax converted the data into an Excel spreadsheet.

• After the bill passed the Tax Commission discovered an error in the spreadsheet understating the revenue loss by $200 million.

• Resulted in a change in statute to allow LFA to access taxpayer data as long as the taxpayer could not be identified.

• Transitioned from Excel to SAS for analysis.

Taxpayer ReceiptTools | budget.utah.gov

Sales Tax Revenue

Total General Fund Revenue

Individual Income Tax Revenue

Total Education Fund Revenue

Enter your income tax paid: $15,000

Enter your sales tax paid (or "x" to estimate from income tax): x

Total: $21,100 $6,100

Public Education48%

Higher Education15%

Social Services14%

Infrastructure10%

Criminal Justice7%

Economic Development

2%

Environment and Natural Resources

3%

Elected Officials1%

General Government

0%

State Sales and Income Tax Uses

http://budget.utah.gov/

Taxpayer Receipt

Statewide

Agency (in M) Your Share

Grand Total 6,266.4$ 21,100.00$

Public Education Total 3,052.6$ 12,315.75$

Higher Education Total 916.8$ 3,240.75$

Social Services Total 884.4$ 2,113.92$

Infrastructure Total 638.4$ 1,545.76$

Buildings -$ -$

Debt Service 643.5$ 1,557.97$

Transportation (5.1)$ (12.21)$

Criminal Justice Total 427.7$ 1,022.49$

Economic Development Total 130.2$ 344.41$

Environment and Natural Resources Total 101.6$ 242.84$

Elected Officials Total 86.6$ 206.97$

General Government Total 28.0$ 66.97$

$11,961

$3,337 $2,072 $1,515 $1,032

$337 $608 $174 $66 $0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

Revenue Simulator

Pitfalls and Problems

La Legislative Fiscal Office

Reliance on External Forecasts Can Be ProblematicHere, La wages & salaries income. History sourced to the BEA, but forecasts provided by Moody’s Analytics. The Moody’s forecast of LA income (La variables in general) has been persistently more optimistic than has actually prevailed. Thus, the Moody’s series has to be adjusted throughout the forecast horizon to something more realistic/likely. A simple extrapolation of underlying trend path or near-term historical performance provides a more likely forecast of this variable for use in sales tax forecasting. {Moody’s forcast implies annual growth rates up to 5% before tapering off; Adjusted rate no more than 2.5%}

0.00

20000.00

40000.00

60000.00

80000.00

100000.00

120000.00

140000.00

1990Q1

1990Q4

1991Q3

1992Q2

1993Q1

1993Q4

1994Q3

1995Q2

1996Q1

1996Q4

1997Q3

1998Q2

1999Q1

1999Q4

2000Q3

2001Q2

2002Q1

2002Q4

2003Q3

2004Q2

2005Q1

2005Q4

2006Q3

2007Q2

2008Q1

2008Q4

2009Q3

2010Q2

2011Q1

2011Q4

2012Q3

2013Q2

2014Q1

2014Q4

2015Q3

2016Q2

2017Q1

2017Q4

2018Q3

2019Q2

2020Q1

2020Q4

LaWages&SalariesIncomeHistoricalandForecasts

qtrlyobs,SAAR

Moody'sForecastAdjusted Moody'sForecast


Reliance on External Forecasts Can Be ProblematicSame issue here, but with regard to oil prices. Moody’s doesn’t have to live on any of its forecasts, but we have to live on ours. External forecasts of oil price seem to be more aggressive than what we are comfortable budgeting. So we adjust them down (an average of Moody’s, EIA, our Natural Resources Dept. – low price scenario, and sometimes NYMEX futures).

0.00

20.00

40.00

60.00

80.00

100.00

120.00

140.00

90:1

91:1

92:1

93:1

94:1

95:1

96:1

97:1

98:1

99:1

00:1

01:1

02:1

03:1

04:1

05:1

06:1

07:1

08:1

09:1

10:1

11:1

12:1

13:1

14:1

15:1

16:1

17:1

18:1

19:1

20:1

21:1

Moody'sAdjusted

Moody'sOilPrice


Even When Good Data Is Available, It Can Be Used BadlyThe 2015/16 Executive Budget proposed to balance the total proposed spending with existing baseline revenue + the recommended additional revenue below from curtailing usage of certain tax credits. The additional revenue figures were taken from the Revenue Dept. annual publication titled “Tax Exemption Budget”, reporting actual credit usage.

from2014teb

AmountOffsettingTax

AmountRefunded

Inventory Tax Credit

Ad Valorem

Credit for Certain

Natural Gas

Ad Valorem Credit

for Offshore

Vessels

Telephone Co

Property Tax Credit

452,676,421 4,534,210 56,406,978 22,643,842

75,961,191 551,179 13,356,040 17,381,873

376,715,230 3,983,031 43,050,938 5,261,969

A bill addressing these 4 credits expected a fiscal note reflecting $429.1 million of additional revenue for FY15/16 (82% of the $525.9 million total recommendation).

This estimate, used in the budget, came directly from Revenue Dept. data concerning the utilization of these credits. To the left, the latest year available of a multi year table of credit use for these 4 credits and all the others mentioned above (from the Revenue Dept.).

∑ = $429.1 million


Even When Good Data Is Available, It Can Be Used BadlyThe budget expected $429.1 million for FY15/16 from these 4 credits. This is the fiscal note they got. A little short of that.

What went wrong? The administration looked at the figures provided by the Dept in the Tax Exemption Budget. That document reports exemption losses (credit losses in this case) by state fiscal year. However, the bill reducing the credits was made effective (appropriately so) on the basis of tax/calendar year, “Applicable to tax years beginning on and after January 1, 2015”. No one in the administration seemed to realize this until the fiscal note was generated. That note explained it, as nicely as possible, as follows:

So what did we do? In addition to many other changes that made the bill almost unrecognizable from the Original version, the effectiveness / applicability language was changed to: “Applicable to all claims for these credits on any return filed onor after July 1, 2015, regardless of the tax year to which the return relates”.


Even When Good Data Is Available, Implementation Can Make Estimates UselessTable below is a sample of near-term history of tax credit usage going into the 2015 legislative session. For many credits history actually went back many years prior. Very good data on actual credit use, annual growth & variation.

Citizens Insurance Credit Inventory Tax Credit

Ad Valorem

Credit for Certain

Natural Gas

Ad Valorem Credit

for Offshore

Vessels

Telephone Co

Property Tax Credit Milk Producers

Technology

Commerciali-

zation

Historic

Residential

Rehab Angel Investor Broadway South

School

Readiness

ChildCare

Provider

Sugarcane

Trailer

Conversion

Research &

Development

from2014teb

AmountOffsettingTax

AmountRefunded

from2013teb

AmountOffsettingTax

AmountRefunded

from2012teb

AmountOffsettingTax

AmountRefunded

from2011teb

AmountOffsettingTax

AmountRefunded

from2010teb

AmountOffsettingTax

AmountRefunded

41,100,487

33,546,054

7,554,433

40,697,004

33,012,766

7,684,238

42,078,539

33,813,798

8,264,741

43,527,560

35,421,468

8,106,092

46,284,427

37,818,083

8,466,344

452,676,421 4,534,210 56,406,978 22,643,842

75,961,191 551,179 13,356,040 17,381,873

376,715,230 3,983,031 43,050,938 5,261,969

419,306,842 4,022,555 41,457,576 24,097,188

77,639,885 1,216,471 5,263,692 15,204,986

341,666,957 2,806,084 36,193,884 8,892,202

381,004,799 3,276,447 43,768,033 24,425,749

65,086,822 866,684 8,394,741 17,817,892

315,917,977 2,409,763 35,373,292 6,607,857

373,990,196 5,407,465 37,454,767 69,304,257

83,704,067 1,108,439 10,254,776 26,628,016

290,286,129 4,299,026 27,199,991 42,676,241

319,710,932 3,825,605 29,795,877 23,306,138

57,553,014 1,129,622 10,430,960 18,386,476

262,157,918 2,695,983 19,364,917 4,919,662

1,555,702 201,377 275,657 1,568,555 8,754,604

112,401 27,030 149,345 690,898 185,497

1,443,301 174,347 126,312 877,657 8,569,107

1,810,000 104,735 303,818 1,819,274 4,950,016

107,934 2,137 180,448 890,545 201,943

1,702,066 102,598 123,370 928,729 4,748,073

1,737,500 104,924 643,476 3,386,710 5,261,983

125,346 20,577 516,380 1,276,181 76,631

1,612,154 84,347 127,096 2,110,529 5,185,352

1,778,750 91,849 250,435 3,493,531 588,663

133,587 8,397 147,909 1,371,238 29,583

1,645,163 83,452 102,526 2,122,293 559,080

1,885,000 317,332 428,553 2,664,283 30,024

148,048 1,851 360,483 1,037,563 21,888

1,736,952 315,481 68,070 1,626,720 8,136

4,662,556

256,260

4,406,296

5,506,820

278,417

5,228,403

5,617,109

261,312

5,355,797

4,685,290

221,578

4,463,712

2,818,400

143,711

2,674,689

2,744,431

23,053

2,721,378

5,733,693

367,868

5,365,825

1,746,142

147,202

1,598,940

880,335

91,732

788,603

197,642

32,656

164,986

24,380,813

794,852

23,585,961

19,703,087

1,444,225

18,258,862

10,795,238

970,834

9,824,404

12,449,259

680,029

11,769,230

2,262,509

62,683

2,199,826

Table 1Summary of Major Revenue Bills, 2015 Session

(in millions)

FY16 Duration Applicable RecoupmentAct 125 / HB 629 Income & Franchise Tax Credits Cut 28% $31.5 3 years All Returns From July 1 3 yearsAct 123 / HB 624 Corporate Income Tax Exclusions and Deductions Cut 28% $122.0 3 years All Returns From July 1 3 yearsAct 133 / HB 805 Five Year Carry-forward of 25% of Inventory Credit $129.0 Permanent All Returns From July 1 None

HCR 8 Suspend Business Utilities Exemption to 1% of Sales Tax $107.2 8/ 27/ 16 Transactions From July 1 NoneAct 94 / HB 119 Increase Cigarette Tax by 50¢/ pack plus vapor products $106.4 Permanent Transactions From July 1 None

Act 109 / HB 402 Equalize Credit for Taxes Paid to Other States $34.0 3 years All Returns From July 1 3 yearsAct 103 / HB 218 Eliminate Net Operating Loss Carry-Backs $29.0 Permanent All Returns From July 1 NoneAct 131 / HB 779 Cap Solar Tax Credit Program $19.0 Permanent All Claims From Jan 1 NoneAct 126 / HB 635 Enterprise Zone Restrictions $5.0 Permanent All Claims From July 1 NoneAct 134 / HB 829 Modify / Cap Film Tax Credit Program $77.0 3 years All Claims From July 1 NoneAct 110 / HB 445 Certificates Of Title Tax Increase $59.5 Permanent Transactions From July 1 NoneAct 147 / SB 271 Reduce Motor Fuels Tax Remitance Discounts $6.0 Permanent Transactions From July 1 NoneAct 109 / SB 93 Prohibits Education Credit If Tuition Deduction Taken $2.3 Permanent From Tax Year 2015 None

Interaction Between Act 123 and Act 103 ($8.0)

Total Additional Revenue Generated $719.9 plus $46.8 million in new fees = $766.6M new money raised

This data and other data on actual usage of credits, deductions, programs was used to pass legislation with estimated effects below.

Ultimately, we couldn’t tell if any of this revenue actually materialized in FY16. Much of it ($404M) was to show up in corporate tax collections.Beginning-of-year forecast = $790M vs end-of-year actual collections = $248M. Look at implementation provisions:(a) 3-year temporary duration (wait us out and receive 100¢ on the dollar), (b) applicable to returns received after 7/1/2015 regardless of tax year of the filing (file quickly and receive 100¢ on the dollar), (c) 3 years to recoup extra tax payments if return filed after 7/1/15 was a final return of an earlier extension (this made the accountants happy).


Lack of data can lead to data collection effort (FY 2016/17)• Sales tax rate increase and base expansion to tax previously exempt transactions estimated based on current collections from the traditional

base, and on data from the Tax Exemption Budget for exempt transactions. • Only a handful of sales tax exempt transactions are reported in the Exemption Budget b/c there are so many of them.• Some 90+ exempt transactions reported in a catch-all line, “All Other Exempt Sales”. A very large number implying a very large amount of

tax receipt should those transactions be taxed.• Fiscal note on the bill assumed only 10% of the implied tax amount from taxation of those transactions. Very controversial assumption.• Total sales tax collections for the year came in $30+M less than forecast, inclusive of the fiscal note.• Revenue Dept implemented a tax form that attempts to capture all the catch-all transactions individually.

lessons in data and statistical analysispitfalls and problems topics for consideration: source bias...

Documents