The word canon (not “cannon”) is in origin a religious word, referring to the collection of books accepted by a Christian church to be holy scripture. Large and complex fictional worlds, such as Tolkien’s, have a canon of their own in the collection of writings/other media considered to be valid sources of information about that world. In some cases, such as Star Wars, the copyright holder and licensee (in this example, Lucasfilm) acts like churches in the religious sense by defining what is and is not canon. The canon of Arda is not so easily defined, however, as there is no official policy set by the Tolkien Estate.
Defining canon is a tricky issue and it is perhaps best to view canon as a continuum rather than black-and-white. Generally The Lord of the Rings is accepted as the highest level of canon since it is the most integrated part of the mythology that Tolkien published in his lifetime. This means that if something contradicts The Lord of the Rings, the offending material is not valid. The Hobbit is also generally high in the canon, though it has some incongruous elements (references to matches, for example) that lead some to rank it just below The Lord of the Rings.
The canon question becomes much more complicated when looking at The Silmarillion and related works. Christopher Tolkien, in creating the published Silmarillion, had to sacrifice some things in order to have a cohesive nature. He himself has recognized the sometimes confused nature of the published Silmarillion, so the thoughtful reader must sometimes look to the other books Christopher Tolkien has edited, especially The History of Middle-earth series, to discover the elder Tolkien’s thoughts on certain matters.
In the end there is no one, single standard to apply and each canon question must be considered individually. Unfinished Tales, The History of Middle-earth, The Letters, and various letters and interviews of Tolkien that have still not been published tell us much about Arda, but they must be taken with a grain of salt. The take-home message is that canon is a tricky issue, and perhaps the only book we can accept as canon without reservation is The Lord of the Rings itself.