Working withLarge Lists in SharePoint 2013WHY YOU NEED TO PAY ATTENTION TO THRESHOLDS AND METADATA

Steve StewartAccudata Systems, Inc.7906 North Sam Houston Pkwy W | Suite 300 | Houston, TX 77064

Direct: 281.897.6394 | Fax: 281.897.5001

www.accudatasystems.com

blog.get-spsteve.com @SharePointFly

sstewart@accudatasystems.com

Services we offer:• Advisory Services•Data Center Infrastructure• Security and Mobility•Managed Services• Enterprise Applications•Network•Unified Communications• Assessment & Compliance

Feedback & Other Important Stuff

Use #HoustonSPTF Register to use the Mobile App at SPTF2015.com to give

presentation feedback

Afternoon Reception runs from 2:45-3:30 p.m. in the Exhibit Hall. Refreshments are free and there will be prizes!

Have you ever seen this?

List View Threshold The List View Threshold is set by default to 5,000 items for a web

application in Central Administration Administrators with special permissions have a default list view

threshold of 20,000 items Must be administrator on local server and on farm

Several other thresholds also set (List View Lookup, Permission, etc.)

Administrative Window – Time period for working on large lists without limit

Central Admin - Resource ThrottlingResource throttling settings are accessed in Central Admin fromApplication Management>Manage Web Applications>Web Application>Settings

CA Resource Throttling Settings

Some Important Notes The List View Threshold calculation includes items in

the Recycle Bin. If you delete 1,000 items, they still count against the item limit until they are deleted from the recycle bin.

Search is not affected by the threshold. You can always search for items and get results. Even if the list is well over the threshold

A common misunderstanding is that a view that has been filtered to contain less than the threshold number of items will display items when the list exceeds the threshold. Only views filtered with indexed columns work this way.

What Actions are Blocked When the Threshold is Exceeded?

Operation Description

Add/remove/update a list column

All columns, including lookup and calculated columns, in addition to many kinds of updates, such as a type change or a uniqueness change. Some updates, such as a name change, are not blocked because they do not affect every item in the list.

Add/remove/Update a list content type Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

Create/remove an index Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

Manage files which have no checked-in version A non-indexed recursive query that fails for any list that has more items than the list view threshold.

Non-indexed recursive queriesIncludes filters and some sorts. This operation fails when the list size is greater than the list view threshold. Because there is no index, it does a full scan against the entire list. Also, it returns all items, and it ignores folders.

Cross list queryIncludes queries by the Content Query Web Part and follows the list view threshold setting for auditors and administrators, which by default is 20,000 items. If the operation involves more than 20,000 items, the query fails.

Blocked Operations, cont.Operation Description

Lookup columns that enforce relationship behaviorYou cannot create lookup columns that enforce relationship behavior when the list that is referenced contains more items than the list view threshold.

Deleting a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

Deleting a site If the sum of all items in a site is greater than the list view threshold, deleting the site is prevented because it affects too many items.

Save list as template with data Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

Showing totals in list views Performs a query against every item in the list so it is blocked for any list that has more items than the list view threshold.

Enable/disable attachments in a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

What’s Happening in a Nutshell List View Threshold is a SQL limitation, not a SharePoint limitation

To ensure accurate updates without affecting other users accessing the same table, SQL locks the row being updated

When more than 5,000 rows need to be locked due to the same query, it is more efficient for SQL to lock the entire table … that is all the rows.

It’s there to ensure optimized performance for ALL users So think of it this way. SQL normally eats one tiny piece of pie at a

time except when there are more than 5,000 pieces ordered. In this case, SQL eats the entire pie!

“”

So, just up the number. SharePoint can handle it right?Actually, it’s not really a good idea to increase the

threshold number.

List View Threshold Throughput

5000

10000

50000

100000

0 5 10 15 20 25 30 35 40 45 50

RPS*

Request per Second

List V

iew

Thr

esho

ld

*Values are approximate and for illustration only.

As the size of the list threshold increases, the number of requests per second SharePoint can handle decreases

Does this mean my list can’t be larger than 5,000 items? Of course not.

Feature Max Value TypeNumber of items in a list or library 30 million Supported Limit

Size of an individual file item or item attachment

2 Gigabytes Boundary

Boundaries, Thresholds, and Supported Limits

Boundary - An absolute that cannot be exceeded by design

Threshold - A value that is set to a default and cannot be exceeded unless the default is changed

Supported Limit - Values determined by testing and which represent known limits of the product. Exceeding these limits may result in performance degradation, errors, and other problems.

Thresholds and LimitsThreshold or Limit

Default value

Description

List View Threshold 5,000

Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time. Operations that exceed this limit are blocked. To give you time to make alternative plans, You are warned on the List Settings page when your list has exceeded 3,000 items. The warning contains a help link to this topic.

Unique permissions limit 50,000

Specifies the maximum number of unique permissions allowed for a list or library. Every time you break the inheritance of permissions for an item or folder, it is counted as 1 unique permission toward this limit. If you try to add an item that would lead to exceeding this limit, you are prevented from doing so.

Row size limit 6

Specifies the maximum number of table rows internal to the database used for a list or library item. To accommodate wide lists with many columns, each item is wrapped over several internal table rows, up to 6 rows and up to a total of 8,000 bytes (excluding attachments). For example, if you have a list with many small columns, one that contains hundreds of Yes/No columns, then you could exceed this limit, in which case you would not be able to add more Yes/No columns to the list, but you still may be allowed to add columns of a different type.Note   Administrators can only set this limit by using the object model, not through the user interface.

Thresholds and LimitsThreshold or Limit

Default value

Description

List View Lookup Threshold 8

Specifies the maximum number of join operations, such as those based on lookup, Person/Group, or workflow status columns. If the query uses more than eight columns, the operation is blocked. However, it is possible to programmatically select which columns to use by using maximal view, which can be set through the object model.

List View Threshold size for auditors and administrators

20,000Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time when performed by an auditor or administrator with appropriate permissions. This setting works in conjunction with Allow Object Model Override.

Allow Object Model Override Y

Specifies whether or not developers can perform database operations, such as queries, that request an override of the List View Threshold to the higher limit specified by the List View Threshold for auditors and administrators. An administrator must enable an object model override, and then developers with appropriate permission may then programmatically request that their query use the higher List View Threshold to take advantage of it.

Daily Administrative Time Window Specifies a time period during which resource thresholds and

limits are ignored. An administrator can configure a time window during "off-peak" hours in 15 minute increments and up to 24 hours, such as, 6:00 PM to 10:00 PM or 1:30 AM to 5:15 AM. Note   A database operation or query started within the daily time

window continues until completion (or failure) even if it doesn't finish within the specified time window.

Time can be used to run queries and conduct other administrative tasks such as indexing columns, creating views and deleting large numbers of items.

The administrative window is set in Central Administration for a web application

Is the Threshold Really All or Nothing for the Web Application

Yes and no. You can only change the threshold for the entire web application

using Central Admin But, you can also use PowerShell to disable the threshold for a single

list$web = Get-SPWeb https://domain.com/web/ $list = $web.Lists[“List Name"]$list.EnableThrottling = $false

Additional PowerShell commands are available for setting List thresholds at the web application level. See MSDN for more information:https://msdn.microsoft.com/en-us/library/office/Microsoft.SharePoint.Administration.SPWebApplication_properties.aspx

REST API You can use the REST API to quickly view the list view threshold

settings for a particular site collection without opening Central Admin by accessing the following URL:

<site collection URL>/_api/site/maxItemsPerThrottledOperation   Results will be returned that look like this:

Demonstrations

Views with Indexed ColumnsSupported Column Types Unsupported Column TypesSingle line of text Multiple lines of textChoice (single value) Choice (multi-valued) Number CalculatedCurrency Hyperlink or Picture Date and Time Custom ColumnsLookup (single value)

Indexing the column in the other list or library does not improve performance of the lookup operation.

Lookup (multi-valued)

Person or Group (single value) Person or Group (multi-valued)Yes/No External dataManaged Metadata

Filter IdeasTo filter by: Index the: For example:

Recently changed items Modified column To view only items that have changed in the past week, apply the filter Modified is greater than  [Today]-7.

New items Created column To create a view of only those items that were added in the past week, apply the filter Created is greater than [Today]-7.

My items Created By column To create a view of only those items that you added, apply the filter Created By is equal to [Me].

Items due todayDue Date column (created by you in a list or library)

To create a view of only those items with a due date of today, apply the filter Due Date is equal to [Today].

Discussion board updates Last Updated columnTo create a view of only the discussions that were updated in the past month, apply the filter Last Updated is greater than [Today]-30.

Archive files in a document library Date Modified To create a view of documents not changed since the year 2006 (that you then want to archive), apply the filter Date Modified is less than December 31st 2006.

Find a subset of financial data Region, Year (as two simple indexes)

To create a view of financial data for the NorthEast region in 2008, apply the filter Region equals "NE" AND Year equals 2008 because the Region column is likely to have less values than the Year column.

Index a Column and Create a View Based on it (Demo)

1. Go to list > List Settings2. Scroll to the bottom of the

columns list3. Click Indexed columns4. Click Create a new index5. Select Columns to index

6. Normally, you will pick only a primary column from the drop down. However, you may with some fields select a secondary to create a compound Index. For example, Modified by and Modified (date). Compound indexes can be used with managed metadata.

7. Click Create (max 20 indexes)8. Indexed columns appear in View

builder filter area in the Indexed Columns group.

Create Index Columns

Using InfoPath to Break the Rules

Trick: You cannot use a calculated column as an indexed column, but you can calculate a value in InfoPath and use a rule to transfer the value to a supported column type.

Use Key Filters Key Filters work with managed metadata to help parse a list Prerequisite for using this feature is activation of the Metadata

Navigation and Filtering site feature.

Fields available for Key Filtering are: Content Type Choice Field Managed Metadata Field Person or Group Field Date and Time Field Number Field

Steps to Create a Key Filter

1. Go to List Settings

2. Click Metadata Navigation Settings

3. Add Navigation Hierarchies

4. Add Key Filters5. Click OK

Folders – mainly for libraries You can divide items into folders to get around view thresholds

This is best suited for document libraries Either use initial view with all folders visible or create navigation

(quick launch) to go directly to folder Use Per Location View Settings to customize default and available

views in folders In lists there is no default way to move items into folders after

creation

Document Center

Key Filters Get Us Part Way There, But an Index is Still Needed

Datasheet1. From the Ribbon, click Create View2. Select Datasheet View as the view type3. Under View Settings, expand folders and select Show all items

without folders4. Expand Item Limit and select either Display All Items or Limit

the total number of items returned to option.5. Type 30 in the box below it (this number must be below the list

view threshold)6. Save the view7. Click OK when you receive the warning about flat views.8. Edit the view again, now change

Data Sheet View Settings

Remember unless you are referencing indexed columns, no filters, sorting, totals, etc., can be selected.

Export to Excel From the List/Library Ribbon click Export to Excel You can use the Export to Excel feature to create a read only

version of your list which can be used to perform queries Remove unwanted columns from the Excel worksheet without

affecting your view or the list – columns will not reappear upon refresh

Use the Refresh button to get new additions to the list. You can use Excel tools to generate reports, etc.

Open in Access You need Access 2013 to use this feature in SharePoint 2013 From the Ribbon click Open in Access to create a client that can

be used for: CRUD (Create, Read, Update, Delete) Creating forms Queries Reporting

Even More Powerful – Access Views

1. From the Ribbon select Create View2. Select Access View

Select View Style3. Select an Access View Style4. Click OK5. Make adjustments to the

form and list design, as desired.

6. Use Access to update lists.

Save the Access View App Anywhere

Users can update your list from another site collection You can control permission based on the location where the

document is saved Think about adding a link to the app on the Quick Launch Users can download a copy for more efficiency

Questions

References Designing large lists and maximizing list perfor

mance Techniques for managing large lists Manage lists and libraries with many items SharePoint 2010/2013: List view Lookup thresh

old uncovered

Software Boundaries and Limits for SharePoint 2013

© Copyright 2015 Stephen D. Stewart. All rights reserved.