Balancing Human Creativity with

Algorithms

Marshall Sponder

Lecturer, Zicklin School of Business & Associate Professor, Rutgers Business School

Global Artificial Intelligence Conference

What is covered in this presentation?

1. Introduction2. Darker Side of Algorithms

3. Algorithms and Creativity

4. Textbook & Summary

Introduction

Here today to speak about a way of humanizing Big Data and Algorithms

Marshall Sponder develops and teaches online and hybrid courses at Zicklin School of Business and Rutgers University where he holds a dual appointment.

At Zicklin, he teaches Web Analytics courses while at Rutgers he teaches an online class called Social Media for the Arts. Marshall is the author of Social Media Analytics (McGraw-Hill 2011) and Digital Analytics for Marketing (Routledge, 2017).

Marshall is a Board Member Emeritus at the Web Analytics Association, now called the DAA.

The Digital Marketing Major is part of the success story of Baruch

Digital Marketing CORE courses and some ELECTIVES at Zicklin School of Business

Social Media for the Arts is an Online, Asynchronous learning course I author and teach at Rutgers University –

this semester there is 1436 students and is growing

exponentially every year.

Let’s start with the Darker Side of Algorithms

Uber Surge Pricing generated A $1,100 Bill For an Hour-Long Ride

http://uproxx.com/news/uber-surge-price-new-years-eve/

To this day no one understands what is actually going on inside the Uber Surge Algorithm, though several experts have an approximate idea of how it works.

Instagram uses ‘I will rape you’ threat as a Facebook ad due to a Facebook algorithm

http://metro.co.uk/2017/09/23/instagram-uses-i-will-rape-you-threat-as-a-facebook-ad-due-to-algorithm-6949929/

The trouble is Facebook’s business model is structurally identical whether advertisers are selling shoes, politics or fake diet pills, and whether they’re going after new moms, dog lovers or neo-Nazis. The algorithms don’t know the difference, and Facebook’s customers are not its users.

Algorithms can be as biased as their trainers. Google is more likely to advertise executive-level salaried

positions in the UK to search engine users if it thinks the user is male, according to a Carnegie Mellon study.

Photograph: Yui Mok/PA

https://www.theguardian.com/technology/2016/aug/03/algorithm-racist-human-employers-work#img-2

An algorithm that automatically delivers a ‘don’t interview’ verdict

to candidates with criminal records

disproportionately impacts black job

seekers.

https://www.theguardian.com/technology/2016/aug/03/algorithm-racist-human-employers-work#img-2

One of the challenges in algorithm-based hiring is that currently there is no standard way to measure the

outcome of an algorithm’s choices. https://www.theguardian.com/technology/2016/aug/03/algorithm-racist-human-employers-work#img-2

But it is Important to get algorithms right as they are running

everything now -Many prison

inmates' futures depend on racially biased algorithms

http://theweek.com/articles/627570/many-prison-inmates-futures-depend-racially-biased-algorithms

There is Hope: scientists have devised a way to test whether an algorithm is introducing

gender or racial biases into decision-making.

https://www.theguardian.com/technology/2016/dec/19/discrimination-by-algorithm-scientists-devise-test-to-detect-ai-bias

Police use a Geo-tracking Algorithmic Surveillance Tool, Geofeedia, to Scan Social Media for criminal activity.

https://www.nytimes.com/2016/10/12/technology/aclu-facebook-twitter-instagram-geofeedia.html

Algorithms are best viewed in terms of expected inputs vs. expected outputs, especially when the programming code can not be examined.

Outputs that are several standard deviation above or below the mean or average (such as a 1000.00 cab ride that normally costs 100.00) would be a red flag that there is something wrong or abnormal about what the program (algorithm) is doing. Even without programming knowledge, anyone can begin to examine algorithms in this way.

https://en.wikipedia.org/wiki/Black_box

Watch how you handle big data, FTC warns businesses -'We need systems for auditing the proprietary

algorithms,' an ACLU attorney says

https://www.computerworld.com/article/3019911/big-data/watch-how-you-handle-big-data-ftc-warns-businesses.html

How to balance Algorithms with human creativity and

keep the best of both!

Understanding Algorithms

• Algorithms are a series of instructions that are used to find patterns within a set of data and make decisions upon that data.

• Computer and humans run algorithms.

• Many of the algorithms run on computers are a “black box” and can not be examined closely.

https://piktochart.com/wp-content/uploads/2014/12/Steps-to-choose-an-Infographic.jpeg

In Music – Algorithms are beginning to choose the next hit songs, and perhaps, predetermine them.

A Machine Successfully Predicted the Hit Dance Songs of 2015

https://motherboard.vice.com/en_us/article/bmvxvm/a-machine-successfully-predicted-the-hit-dance-songs-of-2015

Each song is encoded into various measurements that can be operated on by specific algorithms and plotted, as show in the chart on this slide.

The algorithm makes a prediction of the likelihood a song will be a Hit based on it’s position on the cartesian graph, and its calculated score.

A Machine Successfully Predicted probabilities connected with Hit Dance Songs of 2015, and is probably selecting even more hits in 2018

https://motherboard.vice.com/en_us/article/bmvxvm/a-machine-successfully-predicted-the-hit-dance-songs-of-2015

The Washington Post’s robot reporter has

published 850 articles in the

past year –like this one

https://digiday.com/media/washington-posts-robot-reporter-published-500-articles-last-year/?utm_medium=email&utm_campaign=digidaydis&utm_source=publishing&utm_content=170914

Creating Computer Vision and Machine Learning Algorithms

That Can Analyze Works of Art at

Rutgers University.

https://www.mathworks.com/company/newsletters/articles/creating-computer-vision-and-machine-learning-algorithms-that-can-analyze-works-of-art.html?refresh=true&requestedDomain=www.mathworks.com

The creativity algorithms were tested on two data sets containing more

than 62,000 paintings. The algorithm gave high scores to several works

recognized by art historians as both novel and influential, including some

of the works shown in Figure 3.Ranking even higher than Pablo

Picasso’s “Young Ladies of Avignon” (1907) in the same period were

several paintings by Kazimir Malevich. This result initially surprised me, as I knew little about Malevich’s work.

The implications suggest that the art market, including historical art, may be over-rated for some artists, and

under valued, for others.

https://www.mathworks.com/company/newsletters/articles/creating-computer-vision-and-machine-learning-algorithms-that-can-analyze-works-of-art.html?refresh=true&requestedDomain=www.mathworks.com

Exploring world-wide clothing

styles from millions of photos

–@ Cornell University

https://arxiv.org/pdf/1706.01869.pdf

https://arxiv.org/pdf/1706.01869.pdf

Amazing New Algorithms Will Fix Your Photos

Before You Even Take

Them

http://www.sciencealert.com/new-algorithms-can-fix-your-photos-before-you-even-take-them

Google’s Algorithmic Photographers Are Almost as Good as the Real Deal

https://flipboard.com/@flipboard/-googles-algorithmic-photographers-are-a/f-e848f1e2d3%2Fvice.com

New algorithm

gives photos Picasso-style makeovers

http://mashable.com/2015/08/29/computer-photos/#CBRQjLJfrOqc

This Apps' Creations Sure

Look Like Masterworks, But

Is It Art?

https://www.instagram.com/p/BIulqIpD2oL/

http://www.npr.org/sections/alltechconsidered/2016/09/04/492408169/these-apps-creations-sure-look-like-masterworks-but-is-it-art

Algorithms can turn sketches into art with machine learning (Vincent AI)

https://www.cambridgeconsultants.com/vincent

Search Engines such as Bing have Automated Image Detection In Visual Search Queries – searchers can

also use semantic search keywords (supplied) to dig deeper and find images with specific elements in it.

https://www.mediapost.com/publications/article/307695/

Impressive Adobe Algorithm Transfers One Photo’s Style Onto Another

https://petapixel.com/2017/03/29/cornelladobe-show-copy-color-lighting-one-photo-another/

Google releases smart image analysis tools to let robots recognize images, text

https://www.pcworld.com/article/3034967/data-center-cloud/google-releases-intelligent-image-analysis-tools-for-developers.html

Google has a Cloud Vision API (several others have similar offerings) that is easy to build applications around.

I put the building I just moved into, based on a photo I took recently, and the Google Cloud Vision API did a pretty good job of decoding it’s meaning.

https://cloud.google.com/vision/

Text Analytics platforms have been successfully employed, piggybacking off algorithms to detect emotions in text (and images), then isolate it –image taken from a Crimson Hexagon analysis.

Algorithms can detect the most interesting parts of an image and how memorable it is.

http://memorability.csail.mit.edu/demo.html

Geodata can be easily surfaced, algorithmically clustered and charted for instant insights and futher analysis.

Microsoft researchers are teaching AI to write

stories about groups of

photos

https://venturebeat.com/2016/04/14/microsoft-ai-visual-storytelling/

Google created Zero-Shot Translation with a Multilingual Neural Machine Translation System –Computers are teaching themselves (deep learning) how to translate between languages.

https://www.tnooz.com/article/google-translation-update/

An Algorithm can project a 3D face from a 2D image –any image.

http://www.cs.nott.ac.uk/~psxasj/3dme/view.php?name=59bf2f00d5615

http://www.cs.nott.ac.uk/~psxasj/3dme/index.php

The iPhone X will open

new realms for creatives

FaceID still scares many and there is a lot of disagreement about the wisdom of putting it in a phone, still, but ultimately is the natural evolution of where we have been going all along.

Creativity is not limited to the visual arts, this Robot Therapist Talks to Patients Via Facebook - Woebot is available 24/7 via Facebook.

https://www.woebot.io /

With Algorithms doing so much for

us, is there any point in being

creative? I used to paint, but should I

still bother?

https://www.linkedin.com/pulse/warmth-caring-relationships-age-robots-algorithms-ai-bill-murphy

30 years ago, I painted my “Homage to Manet” painting, and struggled, but it’s hard for me to imagine doing the same thing today. I’d rather write and teach. Maybe I’ll go back to painting again someday, when I figure out why I need to.

Last year I used the Prisma app to reimage the Manhattan skyline, this is not a filter, but an

actual algorithm repainting a scene – I used to spend hours, days, trying to paint the same.

For LinkedIn, use third-party algorithms such as snappr.co/photo-analyzer to choose your best photos to use for the profile image. LinkedIn profile pictures are important aspect of a member’s profile, and can be a deciding factor for jobs or other professional opportunities. In this case, we’re using algorithms to help put our best face forward. The same program can be used for any photo.

Everypixel is neural network if your photo is good or not and can be employed with any image media.

Note: using different algorithms on the same photo will produce different results.

https://everypixel.com/aesthetics

Use a Predictive Algorithm to determine the best Tweets to post.

The Retweeted More Algorithm successfully predicted which of two @realdonaldtrump tweets on the same subject would be more viral.

https://chenhaot.com/retweetedmore/

Finding the right things to do and see when traveling

using Google

Trips algorithm

https://get.google.com/trips/

EDM artists can use algorithms such as the

Dance Hit Predictor tool to pick their best

compositions to promote.

http://dorienherremans.com/dance/uploader.php

Algorithms can apply an

artists personal

style to 3D objects in real time

http://stylit.org/

Use algorithms such as LaMem to curate your best Instagram, Facebook, Pinterest and SnapChat photos to share in Social Media, particularly for paid campaigns.

https://www.instagram.com/webmetricsguru/

Use algorithms such as LaMem to curate your best Instagram, Facebook, Pinterest and SnapChat photos to share in Social Media, particularly for paid campaigns.

https://www.instagram.com/webmetricsguru/

score: 0.689 score: 0.281 score: 0.545

score: 0.58 score: 0.81 score: 0.618

score: 0.551 score: 0.645 score: 0.745

score: 0.738 score: 0.545 score: 0.606

Use the Google Cloud API to better annotate the image with the best descriptors and keywords – these are likely to be favored by Google and Social Media Text Analytics engines.

Finish up by coming up with a good tagline for the post with an algorithm such as https://chenhaot.com/retweetedmore/, explored in an earlier slide. Finally, post the image and it’s annotation in Social Media.

Use DeepArt to take make favorite artwork (painting) and turn it into a style that an algorithm uses to repaint any other image it is applied to.

One of my best

paintings from 1988

The building I

live in.

An Algorithm took my original photo and repainted it in the style of my 1988 painting – giving me additional creative ideas.

+ =

What if I wanted to know what Homage to Manet, would have looked like if Manet had painted it, instead of me? ( I suppose, a “Homage” to himself)? – ha!

I used Manet’s “A Bar at the Folies-Bergere” that is pictured in my painting as the filter!

Translating my Homage to Manet into a Manet Style painting did not work out that well for my painting –maybe the “brush” being applied, was wrong.

I guess, not all things we think about, would really work, but at least, now we know.

Assuming Vincent AI becomes a Tablet App (for the iPad, for example) then various styles can be applied to render it.

Ideally, it would be better for artists to program or load in their own styles, and then combine them with what the algorithm (Vincent AI) renders – interacting in a back and forth dance with the algorithm, till the artist gets the rendering they want.

Use Bing Visual Search to find extremely specific imagery to use for inspiration in painting, photography and fashion.

Artists and Illustrators often fail to visualize and convey in drawing and painting the complex geometry and volume of the face when they use photographs.

Using the algorithm from the 3D Face Reconstruction augment to recreate a 3D model of the face, the depth and foreshortening inherit in photographs can compensated for, creating a better model for the portrait.

The angle can be changed, as needed, by the artist, up to the level of what the program allows.

https://en.wikipedia.org/wiki/Abraham_Lincoln#/media/File:Abraham_Lincoln_O-77_matte_collodion_print.jpg

Use the various Vision APIs to generate Metadata around your visual content. Search engines already examine images on webpages and examine if the meanings derived from the images match up with the textual descriptions on the page.

Similarly, creators can put in images that captivate them and use the Vision APIs to come up with descriptors, and tune them for SEO, thereby raising their visibility online and creating a better user experience.

This image was pulled in real time from Picodash.com, a geolocation filtering platform for Instagram, looking at the Metropolitan Museum of Art

Use Algorithms to your advantage – do something creative with the metadata.

Use the metadata to create better creative! This fuels SEO & Social Media.

My Story using the Algorithm:

A young woman wearing glasses walked into the European Painting gallery at the Met and was overjoyed as she stood in front of a Venus and Adonis painting by either Pieve di Cadore or Titian painted in Venice during the Renaissance.

Like the Mona Lisa, there is a mysterious message being communicated to the onlooker. We can only guess and what this mysterious visitor to the MET is trying to say.

Have fun with Algorithms, use them, instead of letting them use you!

What is the green tea mcflurry?

• Back in 2013, McDonald’s Japan released the Matcha Oreo McFlurry, which blended rich vanilla ice cream with powdered green tea and crunchy chocolate biscuit pieces.

• The mix was so popular it returned for an encore season in 2014, and while it was missing from menus last year, the Matcha McFlurry has finally returned with a whole new look, featuring more traditional Japanese flavors.

Use geolocation to explore … anything, find the audience and target.

http://en.rocketnews24.com/2016/03/23/mcdonalds-japan-releases-brand-new-matcha-mcflurry-for-a-limited-time

Found the liked Instagram post and where it was taken then plugged the

latitude/longitude code into Google can resolve the exact location and audience

nearby.

Use the data to iterate, explore, find the story (used Picodash to do this).

Then, use the other approaches outlined here to piggyback off of the algorithms and combine them creatively.

Using Geolocation or a precise location together with emotion tracking can give us a better idea of what people are feeling at the price moment they are touching our content and marketing.

Granted, these technologies are still niche and leading edge (and SCARY), but sooner or later they will become more accessible and easier to work with. Yes, there are scary elements to all of this, but we don’t have to reject these tools outright, we can humanize them, and run them, instead of letting them run us.

http://mashable.com/2017/07/27/disney-facial-recognition-prediction-movies/#hm1qXklrxmqq

I ran a Facebook Ad to increase the enrollment of the Fall 2017 Social Media for the Arts course – I know my audience well.

I ran a Facebook Ad to increase the enrollment of the Fall 2017 Social Media for the Arts course (by as much as 50 students)

With the right approach I made these platforms and algorithms work for me!

Summary -1

1. Most Algorithms are trained by humans and learn from them, inheriting their bias.

2. Algorithms are capable of creativity and often match aspects of our own creativity.

3. Creatives can use Algorithms to increase the effectiveness and scope of their Art.

Summary -2

4. The right workflow solutions for artists have not emerged yet.

5. The tools that exist are too geared to marketers who have short term goals and are probably too expensive to be useful.

6. Most of the existing software is not extensible or user friendly - most remain at the proof of concept level.

New Textbook

• www.routledge.com/9781138190672

• http://bit.ly/DAFM_MS (published 10/4/17)

• amazon.com/author/marshallsponder

• http://www.linkedin.com/in/marshallsponder

• @webmetricsguru

Thank You!Marshall Sponder

Lecturer, Zicklin School of Business

Associate Professor, Rutgers Business School

New Book - http://bit.ly/DAFM_MS