The Cannon DocumentationRelease 0.2.0

Andrew R. Casey, David W. Hogg, Melissa Ness

Jan 28, 2018

Contents

1 User Guide 31.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 Stable Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.2 Development Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.3 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Getting Started Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2.1 Downloading the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2.2 Creating a model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2.3 The training step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.4 The test step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.2.5 Saving the model to disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.3 Tutorials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3.1 Constructing the training set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3.2 Continuum normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3.3 Censoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.3.4 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.4 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.4.1 CannonModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.4.2 Censoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.4.3 Continuum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.4.4 Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151.4.5 Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.4.6 Vectorizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171.4.7 tc command line utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2 License & Attribution 21

Python Module Index 23

i

ii

The Cannon Documentation, Release 0.2.0

The Cannon is a data-driven approach to stellar label determination.

The seminal paper describing The Cannon is Ness et al. (2015). The name, The Cannon, derives from Annie Jump-Cannon, who first arranged stellar spectra in order of temperature purely by the data, without the need for stellarmodels.

This software package is released as part of Casey et al. (2016) and builds on the original implementation of TheCannon by including a number of additional features:

• Easily construct models with complex vectorizers (e.g., cubic polynomial models with 25 labels)

• Analytic derivatives for blazingly-fast optimization at the training step and the test step

• Built-in parallelism to run the training step and the test step in parallel

• Pseudo-continuum normalization using sums of sine and cosine functions

• L1 regularization to discover and enforce sparsity

• Pixel censoring masks for individual labels

• Stratified under-sampling utilities to produce a (more) balanced training set

The Cannon is being actively developed in a public GitHub repository, where you can open an issue if you have anyproblems.

Contents 1

The Cannon Documentation, Release 0.2.0

2 Contents

CHAPTER 1

User Guide

1.1 Installation

You can install the most recent stable version of The Cannon using PyPI or the development version from GitHub.

1.1.1 Stable Version

The easiest way to install the most recent stable version of The Cannon is by using pip. This will install any of theprerequisites (e.g., numpy, scipy), if you don’t already have them:

pip install the-cannon

Note: Make sure you include the - in the-cannon in the command above, otherwise you will install Anna Ho’sversion of The Cannon, which is excellent, but does not include analytic derivatives, regularization, or censoring.

1.1.2 Development Version

To get the source for the latest development version, clone the git repository on GitHub:

git clone https://github.com/andycasey/AnniesLasso.gitcd AnniesLassogit checkout refactor # TODO

Then install the package by running the following command:

python setup.py install

3

The Cannon Documentation, Release 0.2.0

1.1.3 Testing

To run all the unit and integration tests, install nose and then run:

nosetests -v --cover-package=thecannon

1.2 Getting Started Guide

Before we get started, you should know that the following ingredients are required to run The Cannon:

• a training set of stars with known labels (e.g., stellar parameters and chemical abundances),

• pseudo-continuum-normalized spectra of stars in the training set, with all spectra sampled onto the same wave-length points,

• some test set spectra that you want to derive labels from, which has been processed the same way as the trainingset spectra.

In this guide we will provide you with the training set labels and spectra, and the test set spectra. If you want moreinformation about constructing a training set or continuum-normalizing your spectra, see the linked tutorials.

Note: We use Travis continuous integration to test every change to The Cannon in Python versions 2.7, 3.5, and 3.6.The code examples here should work in any of these Python versions.

In this guide we will train a model using APOGEE DR14 spectra and ASPCAP labels to derive effective tempera-ture 𝑇eff , surface gravity log 𝑔, and two chemical abundance ratios ([Fe/H], and [Mg/Fe]. These spectra have beenpseudo-continuum-normalized using a sum of sine and cosine functions (which is a different process ASPCAP usesfor normalization), and individual visits have been stacked.

Here we won’t use any regularization or wavelength censoring, but these can be applied at the end, and theCannonModel object can be retrained to make use of regularization and/or censoring by using the .train()function.

1.2.1 Downloading the data

You can download the required data for this guide using the following command:

wget zenodo-link #TODO

1.2.2 Creating a model

After you have installed The Cannon, you can use the following Python code to construct a CannonModel object:

1 from astropy.table import Table2 from six.moves import cPickle as pickle3 from sys import version_info4

5 import thecannon as tc6

7 # Load the training set labels.8 training_set_labels = Table.read("apogee-dr14-giants.fits")9

4 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

10 # Load the training set spectra.11 pkl_kwds = dict(encoding="latin-1") if version_info[0] >= 3 else {}12 with open("apogee-dr14-giants-flux-ivar.pkl", "rb") as fp:13 training_set_flux, training_set_ivar = pickle.load(fp, **pkl_kwds)14

15 # Specify the labels that we will use to construct this model.16 label_names = ("TEFF", "LOGG", "FE_H", "MG_FE")17

18 # Construct a CannonModel object using a quadratic (O=2) polynomial vectorizer.19 model = tc.CannonModel(20 training_set_labels, training_set_flux, training_set_ivar,21 vectorizer=tc.vectorizer.PolynomialVectorizer(label_names, 2))

Let’s check the model configuration:

>>> print(model)<thecannon.model.CannonModel of 4 labels with a training set of 1624 stars each with→˓8575 pixels>

>>> print(model.vectorizer.human_readable_label_vector)1 + TEFF + LOGG + FE_H + MG_FE + TEFF^2 + LOGG*TEFF + FE_H*TEFF + MG_FE*TEFF + LOGG^2→˓+ FE_H*LOGG + LOGG*MG_FE + FE_H^2 + FE_H*MG_FE + MG_FE^2

# This model has no regularization.>>> print(model.regularization)None

# This model includes no censoring.>>> print(model.censors){}

1.2.3 The training step

The model configuration matches what we expected, so let’s train the model and make it useful:

>>> theta, s2, metadata = model.train(threads=1)2017-03-07 15:35:39,429 [INFO] Training 4-label CannonModel with 1624 stars and 8575→˓pixels/star[====================================================================================================]→˓100% (30s)

The model should take less than a minute to train on a single core. Pixels can be trained independently, so you canparallelize the training step to as many threads as you want using the threads keyword argument.

The .train() function returns the 𝜃 coefficients, the noise residuals 𝑠2, and metadata associated with the trainingof each pixel. The 𝜃 coefficients and scatter terms 𝑠2 are also accessible through the .theta and .s2 attributes,respectively.

1 # Show the noise residuals, but skip the first 300 pixels because2 # they are at the edge of the chip and contain no information.3 # (Note that the last few pixels have s2 = inf because they also4 # contain no information)5 model.s2[300:]6 >>> array([ 2.30660156e-04, 1.38062500e-04, 9.62851563e-05, ...,7 inf, inf, inf])8

1.2. Getting Started Guide 5

The Cannon Documentation, Release 0.2.0

9

10

11

12 fig_scatter = tc.plot.scatter(model)13 fig_scatter.axes[0].set_xlim(0, 3500)14 fig_scatter.savefig("scatter.png", dpi=300)

The code above will produce the following figure, which is zoomed to the first 3500 pixels.

1 model.theta[300:]2 >>> array([[ 0.83280641, 0.07220653, 0.06093662, ..., -0.06230124,3 0.02992542, 0.01622936],4 [ 0.87702919, 0.06771544, 0.02640131, ..., -0.05761867,5 0.02520221, 0.0121453 ],6 [ 0.91777263, 0.05795435, -0.00384247, ..., -0.04377245,7 0.01897458, 0.00580555],8 ...,9 [ 1. , 0. , 0. , ..., 0. ,

10 0. , 0. ],11 [ 1. , 0. , 0. , ..., 0. ,12 0. , 0. ],13 [ 1. , 0. , 0. , ..., 0. ,14 0. , 0. ]])15

16

6 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

17 fig_theta = tc.plot.theta(model,18 # Show the first 5 terms in the label vector.19 indices=range(5), xlim=(0, 3500),20 latex_label_names=[21 r"T_{\rm eff}",22 r"\log{g}",23 r"[{\rm Fe}/{\rm H}]",24 r"[{\rm Mg}/{\rm Fe}]",25 ])26 fig_theta.savefig("theta.png", dpi=300)

This code will produce the figure below, showing the coefficients for the first seven terms in the label vector, includingthe mean flux term, again for the first 3500 pixels. Each coefficient has been normalized by the maximum absolutevalue in order to show relative scales between different coefficients. These first seven coefficients include all linearlabel terms, which is shown by the corresponding label term on the right hand side of each axis.

1.2.4 The test step

The trained model can now be used to run the test step against all APOGEE spectra. First, we will run the test step onthe training set spectra as a sanity check to ensure we can approximately recover the ASPCAP labels.

1 test_labels, cov, metadata = model.test(training_set_flux, training_set_ivar)2

1.2. Getting Started Guide 7

The Cannon Documentation, Release 0.2.0

3 fig_comparison = tc.plot.one_to_one(model, test_labels,4 latex_label_names=[5 r"T_{\rm eff}",6 r"\log{g}",7 r"[{\rm Fe}/{\rm H}]",8 r"[{\rm Mg}/{\rm Fe}]",9 ])

10 fig_comparison.savefig("one-to-one.png", dpi=300)

1.2.5 Saving the model to disk

All CannonModel objects can be written to disk, and read from disk in order to run the test step at a later time. Whena model is saved, it can either be saved with or without the training set fluxes and inverse variances. The training setfluxes and inverse variances aren’t strictly needed anymore once the model is trained, but they can be useful if youwant to re-train the model (e.g., with regularization or censoring), or if you want to run the test step on the spectraused to train the model.

1 model.write("apogee-dr14-giants.model")2 model.write("apogee-dr14-giants-full.model", include_training_set_spectra=True)

8 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

By default the training set spectra are not saved because they can add considerably to the file size. Theapogee-dr14-giants-full.model file size would be smaller given a smaller training set.

>>> ls -lh *.model-rw-rw-r-- 1 arc arc 214M Mar 6 15:58 apogee-dr14-giants-full.model-rw-rw-r-- 1 arc arc 1.1M Mar 6 15:58 apogee-dr14-giants.model

Any saved models can be loaded from disk using the .read() function:

>>> new_model = tc.CannonModel.read("apogee-dr14-giants.model")>>> new_model.is_trainedTrue

1.3 Tutorials

1.3.1 Constructing the training set

You’ll want a balanced training set. #TODO

1.3.2 Continuum normalization

Continuum normalization is the process of removing the black body and instrumental effects of the spectrograph, suchthat the flux values are normalized between 0 and 1.

There are a lot of aspects to continuum normalization, which are beyond the scope of this (or any?) tutorial. Thecrucial aspects are to ensure that the normalization procedure is a linear operation (e.g., not a high-order polynomial),and that the normalization procedure is invariant with respect to both the labels and the spectrum S/N. For example,a normalization procedure that is based on smoothing nearby pixels will give different continuum normalization forlow- and high-S/N spectra of the same star, and will yield a different continuum normalization for metal-rich andmetal-poor stars. If the continuum normalization procedure is dependent on the S/N of the spectra then the (high S/N)training set will have a different continuum normalization to the (lower S/N) test set.

One example of a (pseudo) continuum normalization procedure that meets these constraints is by fitting the continuumwith a sum of sine and cosine functions. This requires a list of continuum pixels to be specified in advance. Howeverin principle, these “continuum pixels” can be chosen at random, as long as the same pixels are used for the continuumnormalization of the training set and the test set.

import thecannon as tc

# The dispersion and flux arrays refer to the wavelength and flux values for# a single spectrum, and the ivar array refers to the inverse variance of those# flux values.

# Here the continuum_pixels is a boolean array the same size as the dispersion# array, indicating whether each entry is a continuum pixel

# The regions parameter can be used to specify -- for example -- separate CCDs# Continuum will be fit separately to each region

continuum, metadata = tc.continuum.sines_and_cosines(dispersion, flux, ivar,continuum_pixels, L=1400, order=3, regions=[

[15090, 15822],

1.3. Tutorials 9

The Cannon Documentation, Release 0.2.0

[15823, 16451],[16452, 16971]

])

1.3.3 Censoring

Censoring allows for labels to be prevented from contributing to the flux at specific pixels.

For example, given a four-label model (𝑇eff , log 𝑔, [Fe/H], [Al/H]), where one label only contributes to a few pixels(e.g., there are few Al lines in this spectrum, so [Al/H] should only contribute to a few pixels), we can use censoringmasks to prevent the [Al/H] label from contributing to the stellar flux except for around the known Al transition.

Censoring masks can be provided for any label. By default, CannonModel objects have no censoring masks. Modelscan be trained with or without censoring, and can be re-trained after a change in the censoring masks.

import thecannon as tc

model = tc.CannonModel.read("apogee-dr12.model")

uncensored_theta = model.theta

print(model.censors)>>> None

print(model.is_trained)>>> True

# Censor [Al/H] everywhere except around the Al line at X Angstroms# todomodel.censors["AL_H"] = tc.censoring.create_mask(

model.dispersion, [[ ],[ ],

])

# Re-train the model.censored_theta, censored_s2, censored_metadata = model.train()

1.3.4 Regularization

Regularization is useful for discovering and enforcing model sparsity. Without any regularization, at a given pixelthere could be contributions from many (e.g., 15) different elemental abundance labels. However we might believethere is only a single, unblended atomic transition that contributes to that pixel, so all 15 elemental abundance labelscannot be contributing. Regularization (specifically we use L1 regularization) encourages the spectral derivatives 𝜃1..𝐾to take on zero values.

Formally, at the training step we fix the 𝐾-lists of labels for the 𝑛 training set stars. At each wavelength pixel 𝑗, wethen find the parameters 𝜃𝑗 and 𝑠2𝑗 by optimizing the penalized likelihood function

𝜃𝑗 , 𝑠2𝑗 ← argmin

𝜃,𝑠

[︃𝑁−1∑︁𝑛=0

[𝑦𝑗𝑛 − 𝑣(ℓ𝑛) · 𝜃]2

𝜎2𝑗𝑛 + 𝑠2

+

𝑁−1∑︁𝑛=0

ln(𝜎2𝑗𝑛 + 𝑠2) + Λ𝑄(𝜃)

]︃

10 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

where

𝑄(𝜃) =

𝐷−1∑︁𝑑=1

|𝜃𝑑| .

Note: The L1 regularization does not act on the first entry in 𝜃, 𝜃0, because this is a mean flux value that we do notexpect to decrease with increasing regularization.

By default, a CannonModel object will have no regularization (Λ = 0). You can set the regularization strength Λusing the .regularization attribute:

import thecannon as tc

model = tc.CannonModel.read("apogee-dr12.model")

model.regularization = 1000.0

When using the .regularization attribute, you can either specify None (equivalent to using model.regularization = 0), a float-like value, or an array of size num_pixels with different regularization strengthsfor each pixel.

Often it is convenient to train a model without any regularization, and then re-train it using a higher regularizationstrength. This is useful because the optimization can take longer at higher regularization strengths (particularly ifstrict convergence tolerances are required), and the previously solved 𝜃 coefficients can be used as an initial startingguess.

# Without any regularization:model.regularization = 0.0nr_theta, nr_s2, nr_metadata = model.train()

# Let's set a strong regularization value:model.regularization = 10e5sr_theta, sr_s2, sr_metadata = model.train()

# Compare the spectral coefficients for, say, [Ti/H] in nr_theta and sr_theta:#TODO

Sometimes you might find that the likelihood landscape is extremely flat with strong regularization in a high dimen-sional label space. The flat likelihood landscape makes it difficult to optimize, and you might have (valid) concernsthat the optimizer has not converged to the best coefficients possible. Be aware that the training step is a convexoptimization problem (when 𝑠2𝑗 is fixed), so the optimizer is working towards the global minimum (and not a localminimum), but the line search in high dimensions may find that the landscape is too flat to continue optimizing.

In these circumstances, you can switch from using the L-BFGS-B optimization algorithm (default; scipy.optimize.fmin_l_bfgs_b) to using Powell’s method (scipy.optimize.fmin_powell). The result fromthe previous trained model will be used as an initial guess, so the coefficients will not have far to optimize. Powell’smethod can have very strict tolerance requirements, and should perform well even if the likelihood landscape is veryflat. However, Powell’s method does not make use of analytic derivatives, so the training time will be considerablylonger.

# Re-train the model using Powell's method, and very strict convergence requirements.# (The $\theta$ values that were trained using BFGS will be used as a starting point)powell_theta, powell_s2, powell_metadata = model.train(

op_method="powell",op_kwds=dict(xtol=1e-8, ftol=1e-8, maxiter=100000, maxfun=100000))

1.3. Tutorials 11

The Cannon Documentation, Release 0.2.0

If you are worried that you could have optimization convergence problems, it is a useful sanity check to performthis dumb, slow optimization: train first with BFGS (default), then re-train using Powell’s method and very strictconvergence criteria. The training time will take longer, but you can plot the spectral derivatives from both trainingsteps and compare the differences!

1.4 API

Common classes and utilities in The Cannon are documented here. For more details, view the source code.

1.4.1 CannonModel

class thecannon.CannonModel(training_set_labels, training_set_flux, training_set_ivar, vectorizer,dispersion=None, regularization=None, censors=None, **kwargs)

A model for The Cannon which includes L1 regularization and pixel censoring.

Parameters

• training_set_labels – A set of objects with labels known to high fidelity. This canbe given as a numpy structured array, or an astropy table.

• training_set_flux – An array of normalised fluxes for stars in the labelled set, givenas shape (num_stars, num_pixels). The num_stars should match the number of rows intraining_set_labels.

• training_set_ivar – An array of inverse variances on the normalized fluxes for starsin the training set. The shape of the training_set_ivar array should match that of train-ing_set_flux.

• vectorizer – A vectorizer to take input labels and produce a design matrix. This shouldbe a sub-class of vectorizer.BaseVectorizer.

• dispersion – [optional] The dispersion values corresponding to the given pixels. Ifprovided, this should have a size of num_pixels.

• regularization – [optional] The strength of the L1 regularization. This should eitherbe None, a float-type value for single regularization strength for all pixels, or a float-likearray of length num_pixels.

• censors – [optional] A dictionary containing label names as keys and boolean censoringmasks as values.

censorsReturn the wavelength censor masks for the labels.

design_matrixReturn the design matrix for this model.

dispersionReturn the dispersion points for all pixels.

in_convex_hull(labels)Return whether the provided labels are inside a complex hull constructed from the labelled set.

Parameters labels – A NxK array of N sets of K labels, where K is the number of labels thatmake up the vectorizer.

Returns A boolean array as to whether the points are in the complex hull of the labelled set.

12 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

is_trainedReturn true or false for whether the model is trained.

classmethod read(path, **kwargs)Read a saved model from disk.

Parameters path – The path where to load the model from.

regularizationReturn the strength of the L1 regularization for this model.

reset()Clear any attributes that have been trained.

s2Return the intrinsic variance (s^2) for all pixels.

test(model, *args, **kwargs)Run the test step on spectra.

Parameters

• flux – The (pseudo-continuum-normalized) spectral flux.

• ivar – The inverse variance values for the spectral fluxes.

• initial_labels – [optional] The initial labels to try for each spectrum. This can be asingle set of initial values, or one set of initial values for each star.

• threads – [optional] The number of parallel threads to use.

thetaReturn the theta coefficients (spectral model derivatives).

train(threads=None, **kwargs)Train the model.

Parameters threads – [optional] The number of parallel threads to use.

Returns A three-length tuple containing the spectral coefficients theta, the squared scatter termat each pixel s2, and metadata related to the training of each pixel.

training_set_fluxReturn the training set fluxes.

training_set_ivarReturn the inverse variances of the training set fluxes.

training_set_labelsReturn the labels in the training set.

vectorizerReturn the vectorizer for this model.

write(path, include_training_set_spectra=False, overwrite=False, protocol=-1)Serialise the trained model and save it to disk. This will save all relevant training attributes, and optionally,the training data.

Parameters

• path – The path to save the model to.

• include_training_set_spectra – [optional] Save the labelled set, normalisedflux and inverse variance used to train the model.

• overwrite – [optional] Overwrite the existing file path, if it already exists.

1.4. API 13

The Cannon Documentation, Release 0.2.0

• protocol – [optional] The Python pickling protocol to employ. Use 2 for compatibilitywith previous Python releases, -1 for performance.

1.4.2 Censoring

class thecannon.censoring.Censors(label_names, num_pixels, items=None, **kwargs)A dictionary sub-class that allows for label censoring masks to be applied on a per-pixel basis to CannonModelobjects.

Parameters

• label_names – A list containing the label names that form the model vectorizer.

• num_pixels – The number of pixels per star.

• items – [optional] A dictionary containing label names as keys and masks as values.

Utilities to deal with wavelength censoring.

class thecannon.censoring.Censors(label_names, num_pixels, items=None, **kwargs)A dictionary sub-class that allows for label censoring masks to be applied on a per-pixel basis to CannonModelobjects.

Parameters

• label_names – A list containing the label names that form the model vectorizer.

• num_pixels – The number of pixels per star.

• items – [optional] A dictionary containing label names as keys and masks as values.

thecannon.censoring.create_mask(dispersion, censored_regions)Return a boolean censoring mask based on a structured list of (start, end) regions.

Parameters

• dispersion – An array of dispersion values.

• censored_regions – A list of two-length tuples containing the (start, end) points of acensored region.

Returns A boolean mask indicating whether the pixels in the dispersion array are masked.

thecannon.censoring.design_matrix_mask(censors, vectorizer)Return a mask of which indices in the design matrix columns should be used for a given pixel.

Parameters

• censors – A censoring dictionary.

• vectorizer – The model vectorizer:

Returns A mask of which indices in the model design matrix should be used for a given pixel.

1.4.3 Continuum

Continuum-normalization.

thecannon.continuum.sines_and_cosines(dispersion, flux, ivar, continuum_pixels, L=1400, or-der=3, regions=None, fill_value=1.0, **kwargs)

Fit the flux values of pre-defined continuum pixels using a sum of sine and cosine functions.

Parameters

14 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

• dispersion – The dispersion values.

• flux – The flux values for all pixels, as they correspond to the dispersion array.

• ivar – The inverse variances for all pixels, as they correspond to the dispersion array.

• continuum_pixels – A mask that selects pixels that should be considered as ‘contin-uum’.

• L – [optional] The length scale for the sines and cosines.

• order – [optional] The number of sine/cosine functions to use in the fit.

• regions – [optional] Specify sections of the spectra that should be fitted separately in eachstar. This may be due to gaps between CCDs, or some other physically- motivated reason.These values should be specified in the same units as the dispersion, and should be givenas a list of [(start, end), . . . ] values. For example, APOGEE spectra have gaps near thefollowing wavelengths which could be used as regions:

>> regions = ([15090, 15822], [15823, 16451], [16452, 16971])

• fill_value – [optional] The continuum value to use for when no continuum was calcu-lated for that particular pixel (e.g., the pixel is outside of the regions).

• full_output – [optional] If set as True, then a metadata dictionary will also be returned.

Returns The continuum values for all pixels, and a dictionary that contains metadata about the fit.

1.4.4 Fitting

Fitting functions for use in The Cannon.

thecannon.fitting.fit_spectrum(flux, ivar, initial_labels, vectorizer, theta, s2, fiducials, scales,dispersion=None, **kwargs)

Fit a single spectrum by least-squared fitting.

Parameters

• flux – The normalized flux values.

• ivar – The inverse variance array for the normalized fluxes.

• initial_labels – The point(s) to initialize optimization from.

• vectorizer – The vectorizer to use when fitting the data.

• theta – The theta coefficients (spectral derivatives) of the trained model.

• s2 – The pixel scatter (s^2) array for each pixel.

• dispersion – [optional] The dispersion (e.g., wavelength) points for the normalizedfluxes.

Returns A three-length tuple containing: the optimized labels, the covariance matrix, and metadataassociated with the optimization.

thecannon.fitting.fit_pixel_fixed_scatter(flux, ivar, initial_thetas, design_matrix, regular-ization, censoring_mask, **kwargs)

Fit theta coefficients and noise residual for a single pixel, using an initially fixed scatter value.

Parameters

• flux – The normalized flux values.

• ivar – The inverse variance array for the normalized fluxes.

1.4. API 15

The Cannon Documentation, Release 0.2.0

• initial_thetas – A list of initial theta values to start from, and their source. Forexample: ‘[(theta_0, “guess”), (theta_1, “old_theta”)]

• design_matrix – The model design matrix.

• regularization – The regularization strength to apply during optimization (Lambda).

• censoring_mask – A per-label censoring mask for each pixel.

• op_method – The optimization method to use. Valid options are: l_bfgs_b, powell.

• op_kwds – A dictionary of arguments that will be provided to the optimizer.

Returns The optimized theta coefficients, the noise residual s2, and metadata related to the opti-mization process.

thecannon.fitting.fit_theta_by_linalg(flux, ivar, s2, design_matrix)Fit theta coefficients to a set of normalized fluxes for a single pixel.

Parameters

• flux – The normalized fluxes for a single pixel (across many stars).

• ivar – The inverse variance of the normalized flux values for a single pixel across manystars.

• s2 – The noise residual (squared scatter term) to adopt in the pixel.

• design_matrix – The model design matrix.

Returns The label vector coefficients for the pixel, and the inverse variance matrix.

thecannon.fitting.chi_sq(theta, design_matrix, flux, ivar, axis=None, gradient=True)Calculate the chi-squared difference between the spectral model and flux.

Parameters

• theta – The theta coefficients.

• design_matrix – The model design matrix.

• flux – The normalized flux values.

• ivar – The inverse variances of the normalized flux values.

• axis – [optional] The axis to sum the chi-squared values across.

• gradient – [optional] Return the chi-squared value and its derivatives (Jacobian).

Returns The chi-squared difference between the spectral model and flux, and optionally, the Jaco-bian.

thecannon.fitting.L1Norm_variation(theta)Return the L1 norm of theta (except the first entry) and its derivative.

Parameters theta – An array of finite values.

Returns A two-length tuple containing: the L1 norm of theta (except the first entry), and the deriva-tive of the L1 norm of theta.

1.4.5 Utilities

General utility functions.

thecannon.utils.short_hash(contents)Return a short hash string of some iterable content.

16 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

Parameters contents – The contents to calculate a hash for.

Returns A concatenated string of 10-character length hashes for all items in the contents provided.

class thecannon.utils.wrapper(f, args, kwds, N, message=None, size=100)A generic wrapper with a progressbar, which can be used either in serial or in parallel.

Parameters

• f – The function to apply.

• args – Additional arguments to supply to the function f.

• kwds – Keyword arguments to supply to the function f.

• N – The number of items that will be iterated over.

• message – [optional] An information message to log before showing the progressbar.

• size – [optional] The width of the progressbar in characters.

Returns A generator.

1.4.6 Vectorizer

BaseVectorizer

A base vectorizer for The Cannon.

class thecannon.vectorizer.base.BaseVectorizer(label_names, terms, **kwargs)A vectorizer class that models spectral fluxes and its derivatives.

get_label_vector(labels, *args, **kwargs)Return the label vector based on the labels provided.

Parameters labels – The values of the labels. These should match the length and order of thelabel_names attribute.

get_label_vector_derivative(labels, *args, **kwargs)Return the derivative of the label vector with respect to the given label.

Parameters labels – The values of the labels to calculate the label vector for.

label_namesReturn the label names that are used in this vectorizer.

termsReturn the terms provided for this vectorizer.

PolynomialVectorizer

A polynomial vectorizer for The Cannon.

class thecannon.vectorizer.polynomial.PolynomialVectorizer(label_names=None,order=None,terms=None,**kwargs)

A vectorizer that models spectral fluxes as combination of polynomial terms. Note that either label_names andorder must be provided, or the terms keyword argument needs to be explicitly specified.

Parameters

1.4. API 17

The Cannon Documentation, Release 0.2.0

• label_names – [optional] A list of label names that are terms in the label vector.

• order – [optional] The maximal order for the vectorizer.

• terms – [optional] A structured list of tuples that defines the full extent of the label vector.Note that terms must be None if label_names or order are provided.

get_human_readable_label_term(term_index, label_names=None, **kwargs)Return a human-readable form of a single term in the label vector.

Parameters

• term_index – The term in the label vector to return.

• label_names – [optional] The label names to use. For example, these could be LaTeXrepresentations of the label names.

Returns A human-readable string representing a single term in the label vector.

get_human_readable_label_vector(mul=u’*’, pow=u’^’, bracket=False)Return a human-readable form of the label vector.

Parameters

• mul – [optional] String to use to represent a multiplication operator. For example, if givingLaTeX label definitions one may want to use ‘cdot’ for the mul term.

• pow – [optional] String to use to represent a power operator.

• bracket – [optional] Show brackets around each term.

Returns A human-readable string representing the label vector.

get_label_vector(labels)Return the values of the label vector, given the scaled labels.

Parameters labels – The scaled and offset labels to use to calculate the label vector(s). Thiscan be a ond-dimensional vector of K labels, or a two-dimensional array of N by K labels.

get_label_vector_derivative(labels)Return the derivatives of the label vector with respect to fluxes.

Parameters labels – The scaled labels to calculate the label vector derivatives. This canbe a one-dimensional vector of K labels (using the same order and length provided byself.label_names), or a two-dimensional array of N by K values. The returning array willbe of shape (N, D), where D is the number of terms in the label vector description.

human_readable_label_vectorReturn a human-readable form of the label vector.

1.4.7 tc command line utility

The Cannon code includes a command line utility called tc.

This command line tool can be used to fit spectra using a pre-trained model saved to disk, and to join many results intoa single table of output labels.

usage: tc [-h] {fit,join} ...

The Cannon

optional arguments:-h, --help show this help message and exit

18 Chapter 1. User Guide

The Cannon Documentation, Release 0.2.0

action:Specify the action to perform.

{fit,join}fit Fit stacked spectra using a trained model.join Join results from individual stars into a single table.

http://TheCannon.io

The fit argument requires the following input:

usage: tc fit [-h] [-v] [-t THREADS] [--parallel-chunks PARALLEL_CHUNKS][--clobber] [--output-suffix OUTPUT_SUFFIX] [--from-filename]model_filename spectrum_filenames [spectrum_filenames ...]

positional arguments:model_filename The path of a trained Cannon model.spectrum_filenames Paths of spectra to fit.

optional arguments:-h, --help show this help message and exit-v, --verbose Verbose logging mode.-t THREADS, --threads THREADS

The number of threads to use.--parallel-chunks PARALLEL_CHUNKS

The number of spectra to fit in a chunk.--clobber Overwrite existing output files.--output-suffix OUTPUT_SUFFIX

A string suffix that will be added to the spectrumfilenames when creating the result filename

--from-filename Read spectrum filenames from file

Once the test step is complete, the results from individual files will be saved to disk. For example, if a spectrum wassaved to disk as spectrum.pkl, then the command tc fit cannon.model spectrum.pkl would producean output file called spectrum.pkl.result. The tc join command can then collate the output from many*.result files into a single table:

usage: tc join [-h] [-v] [-t THREADS] [--from-filename] [--errors] [--cov][--clobber]output_filename result_filenames [result_filenames ...]

positional arguments:output_filename The path to write the output filename.result_filenames Paths of result files to include.

optional arguments:-h, --help show this help message and exit-v, --verbose Verbose logging mode.-t THREADS, --threads THREADS

The number of threads to use.--from-filename Read result filenames from a file.--errors Include formal errors in destination table.--cov Include covariance matrix in destination table.--clobber Ovewrite an existing table file.

1.4. API 19

The Cannon Documentation, Release 0.2.0

20 Chapter 1. User Guide

CHAPTER 2

License & Attribution

The source code is released under the MIT license. If you make use of the code, please cite both the original Ness etal. (2015) paper and Casey et al. (2016):

@ARTICLE{Ness_2015author = {{Ness}, M. and {Hogg}, D.~W. and {Rix}, H.-W. and {Ho}, A.~Y.~Q. and

{Zasowski}, G.},title = "{The Cannon: A data-driven approach to Stellar Label Determination}",journal = {\apj},year = 2015,month = jul,volume = 808,eid = {16},pages = {16},doi = {10.1088/0004-637X/808/1/16},

}

@ARTICLE{Casey_2016author = {{Casey}, A.~R. and {Hogg}, D.~W. and {Ness}, M. and {Rix}, H.-W. and

{Ho}, A.~Q. and {Gilmore}, G.},title = "{The Cannon 2: A data-driven model of stellar spectra for detailed

→˓chemical abundance analyses}",journal = {ArXiv e-prints},archivePrefix = "arXiv",eprint = {1603.03040},year = 2016,month = mar,

}

Here is a list of notable publications that have used or developed upon The Cannon:

• Ho et al.

• others

21

The Cannon Documentation, Release 0.2.0

22 Chapter 2. License & Attribution

Python Module Index

tthecannon.censoring, 14thecannon.continuum, 14thecannon.fitting, 15thecannon.utils, 16thecannon.vectorizer.base, 17thecannon.vectorizer.polynomial, 17

23

The Cannon Documentation, Release 0.2.0

24 Python Module Index

Index

BBaseVectorizer (class in thecannon.vectorizer.base), 17

CCannonModel (class in thecannon), 12Censors (class in thecannon.censoring), 14censors (thecannon.CannonModel attribute), 12chi_sq() (in module thecannon.fitting), 16create_mask() (in module thecannon.censoring), 14

Ddesign_matrix (thecannon.CannonModel attribute), 12design_matrix_mask() (in module thecannon.censoring),

14dispersion (thecannon.CannonModel attribute), 12

Ffit_pixel_fixed_scatter() (in module thecannon.fitting), 15fit_spectrum() (in module thecannon.fitting), 15fit_theta_by_linalg() (in module thecannon.fitting), 16

Gget_human_readable_label_term() (thecan-

non.vectorizer.polynomial.PolynomialVectorizermethod), 18

get_human_readable_label_vector() (thecan-non.vectorizer.polynomial.PolynomialVectorizermethod), 18

get_label_vector() (thecan-non.vectorizer.base.BaseVectorizer method),17

get_label_vector() (thecan-non.vectorizer.polynomial.PolynomialVectorizermethod), 18

get_label_vector_derivative() (thecan-non.vectorizer.base.BaseVectorizer method),17

get_label_vector_derivative() (thecan-non.vectorizer.polynomial.PolynomialVectorizermethod), 18

Hhuman_readable_label_vector (thecan-

non.vectorizer.polynomial.PolynomialVectorizerattribute), 18

Iin_convex_hull() (thecannon.CannonModel method), 12is_trained (thecannon.CannonModel attribute), 12

LL1Norm_variation() (in module thecannon.fitting), 16label_names (thecannon.vectorizer.base.BaseVectorizer

attribute), 17

PPolynomialVectorizer (class in thecan-

non.vectorizer.polynomial), 17

Rread() (thecannon.CannonModel class method), 13regularization (thecannon.CannonModel attribute), 13reset() (thecannon.CannonModel method), 13

Ss2 (thecannon.CannonModel attribute), 13short_hash() (in module thecannon.utils), 16sines_and_cosines() (in module thecannon.continuum),

14

Tterms (thecannon.vectorizer.base.BaseVectorizer at-

tribute), 17test() (thecannon.CannonModel method), 13thecannon.censoring (module), 14thecannon.continuum (module), 14

25

The Cannon Documentation, Release 0.2.0

thecannon.fitting (module), 15thecannon.utils (module), 16thecannon.vectorizer.base (module), 17thecannon.vectorizer.polynomial (module), 17theta (thecannon.CannonModel attribute), 13train() (thecannon.CannonModel method), 13training_set_flux (thecannon.CannonModel attribute), 13training_set_ivar (thecannon.CannonModel attribute), 13training_set_labels (thecannon.CannonModel attribute),

13

Vvectorizer (thecannon.CannonModel attribute), 13

Wwrapper (class in thecannon.utils), 17write() (thecannon.CannonModel method), 13

26 Index